Skip to main content
PostgreSQL is the only durable store Omniflow has. Valkey holds cache, rate limits, locks, and disposable state, so losing it costs nothing but a warm cache. Everything on this page therefore concerns PostgreSQL.

​
What a backup is

A backup is pg_dump --format=custom --no-owner --no-privileges --compress=6 output, encrypted with AES-256-GCM and written to APP_BACKUP_DIR. Omniflow shells out to pg_dump and pg_restore rather than reimplementing them, so a backup is always restorable with standard PostgreSQL tooling even if Omniflow itself will not start. Encryption is framed rather than a single sealed blob, so a multi-gigabyte dump never has to fit in memory:
Frames are 1 MiB. Each frame’s nonce is the base nonce XOR its index, and each frame authenticates its own index and whether it ends the file. A reordered, duplicated, dropped, or truncated frame fails to open rather than producing a plausible-looking partial dump — which is the only outcome worse than having no backup at all. A file that is not an Omniflow backup fails on the magic bytes instead. Files are named omniflow-<kind>-<UTC timestamp>.dump.enc, for example omniflow-scheduled-20260811T030000Z.dump.enc. kind is scheduled or manual and nothing else — the column has a check constraint, so a third kind needs a migration. The name is unique and the file is created exclusively, so a collision fails rather than overwrites.

​
Who takes one

The operator panel is read-only about backups. There is no panel route and no /v1/admin route that takes or restores one; the only two write paths in the product are the worker’s schedule and the bot’s /ops surface. That is deliberate — a restore is destructive, and it lives behind an explicitly enumerated list of Telegram accounts rather than behind a permission that six roles could be granted.

​
Enabling backups

Store APP_BACKUP_ENCRYPTION_KEY somewhere the database backup itself does not reach — a password manager, a secrets manager, or a sealed envelope. A key that only exists inside the thing it decrypts is not a key.It is a different key from APP_DATA_ENCRYPTION_KEY. Both must be preserved: the backup key to read the dump, the data key to read the encrypted contact values, TOTP secrets, and provider credentials inside it. A restore with the wrong data key produces a database that loads and whose sealed values will not open.
The bot and worker images ship the matching PostgreSQL 18 client tools. In the compose stack both mount a shared backup-data volume at /var/lib/omniflow/backups, so rebuilding a container never discards backups. The database password never reaches the child process’s argv, where any local user could read it from the process list. The DSN is decomposed into PGHOST, PGPORT, PGDATABASE, PGUSER, PGPASSWORD, and PGSSLMODE in the child’s environment instead.

​
Verification

A backup is recorded as completed only after it has been read back: the file is decrypted end to end and its SHA-256 digest compared with the digest computed while writing. A backup that fails either check is deleted and recorded as failed with a classified error code. This runs on every backup, not only on demand. An unverified backup is a guess.

​
Operating from Telegram

Telegram carries operator notifications and backup/restore actions only. Every other administrative control belongs in the web panel.
An account not in that list gets no response at all to /ops — not a refusal, which would confirm the command exists. The command is also ignored outside a private chat with the bot. /ops shows the five most recent backups with file name, status (with the error code when one is recorded), size, and UTC timestamp, plus:
  • 💾 Back up now — takes a manual backup immediately. If one is already running it answers “A backup is already running” rather than starting a second.
  • ♻️ Restore latest… — offered only when a completed backup exists. A two-step confirmation: the first tap only renders the warning, and the second carries the backup identifier the operator saw.
  • 🔄 Refresh — re-renders the list.
The panel shows sizes and timestamps only: no connection details, no file paths, and no encryption material. Both actions write append-only audit events naming the Telegram account, the action, the backup, and the reason: backup.created for a manual backup, then backup.restore_requested and backup.restored around a restore. The restore’s reason is recorded as “restore requested from Telegram by operator <id>”.

​
Restoring

A restore verifies the backup first, then runs pg_restore --clean --if-exists --no-owner --no-privileges --single-transaction with the decrypted stream on stdin. It is deliberately blunt, which is why it insists the caller has confirmed the destructive intent and recorded who asked.
A restore replaces current database contents. Every change made since the backup was taken is lost, including orders placed, payments settled, and support messages received in that window.
It refuses rather than half-running in these cases: Before restoring:
  1. Stop api, bot, and worker. A live worker will keep provisioning against state that is about to be replaced.
  2. Take a fresh backup of the current database, even if you believe it is broken. It is the only way back from a mistaken restore.
  3. Confirm the target backup’s status is completed.
After restoring:
  1. Start worker first and watch /readyz and the fulfillment topic.
  2. Start api and bot.
  3. Reconcile with Remnawave. Provisioning that happened after the backup point is still live in the panel but absent from Omniflow. Scheduled drift detection reports the mismatches within fifteen minutes, and each one is resolvable from the drift view.
  4. Re-check pending payments. A payment that settled after the backup point is picked up by the reconciliation poller or by a provider webhook retry. Both are idempotent, so nothing is charged or credited twice.
The restore runs in the bot process when it is driven from /ops, so its log is where a failure is described even though the on-screen message mentions the worker.

​
Restoring without the bot

The worker binary doubles as a decryption filter, because disaster recovery must not depend on the rest of the stack being able to start. It needs nothing but the binary, the key, and the file:
--decrypt-backup must be the first argument, and the key is read from the environment before any configuration is loaded — no database, no Valkey, and no Remnawave credential is required. It fails with APP_BACKUP_ENCRYPTION_KEY is required to decrypt a backup or APP_BACKUP_ENCRYPTION_KEY must be base64-encoded 32 bytes, and on a wrong key or a damaged file it fails loudly rather than emitting partial output.
Atlas records applied migrations in its own atlas_schema_revisions schema rather than in public. A dump taken by Omniflow carries the whole database, so an ordinary restore brings that history with it. If you rebuild a database by other means, carry that schema across too, or the migrate service will try to re-apply history and the diagnostics bundle will report zero applied migrations.

​
Retention and pruning

Each backup row is stamped with retain_until at creation, APP_BACKUP_RETENTION_DAYS into the future. After every scheduled run the worker prunes up to 100 expired backups: the file is deleted and the row is marked pruned. The row is kept, so the history of which backups existed stays auditable after the files are gone. A file that cannot be removed is logged and skipped, and its row keeps its status, so the next pass tries again rather than losing track of it.

​
Reviewing history

A backup nobody has ever restored is a backup nobody knows works, which is why the restore history sits beside the backup list rather than on a separate screen.

​
Upgrading Omniflow

Migrations are forward-only and applied by the migrate service before any application process starts.
  1. Take a backup and confirm it is completed.
  2. Pull the new images. Pin an explicit tag; floating tags make a rollback ambiguous.
  3. Run docker compose up -d. The migrate service runs atlas migrate apply, and every other service waits for it to complete successfully.
  4. Watch /readyz on each process.
Applying migrations twice is a no-op: Atlas records what has already run. CI proves every supported baseline — the empty database plus each prefix of the migration history — upgrades to head, and that re-applying head changes nothing.
The compose migrate service uses a hard-coded connection string and does not read APP_DATABASE_URL. If you changed the database credentials, change them in compose.yaml as well or the migrator will connect to the wrong database.

​
Rolling back

Omniflow does not ship down-migrations. A schema rollback is a restore:
  1. Stop api, bot, and worker.
  2. Restore the backup taken before the upgrade.
  3. Pin the previous image tags and start the stack.
  4. Reconcile with Remnawave as described above.
This is why step one of an upgrade is a backup, and why the backup is verified before it is trusted.

​
Upgrading PostgreSQL

Omniflow supports PostgreSQL 18. To move between major versions:
  1. Stop api, bot, and worker; leave postgres running.
  2. Take a backup and verify it.
  3. Stop postgres and start the new major version against an empty data directory.
  4. Restore the backup.
  5. Start the application processes.
Do not point a new PostgreSQL major version at an old data directory. pg_upgrade is the alternative if you prefer an in-place upgrade, but the dump-and-restore path is the one this backup format is designed for. Remember that pg_dump and pg_restore must match the server major version; the shipped images pin the PostgreSQL 18 client tools for exactly this reason.

​
Disaster-recovery drill

Run this quarterly. An untested backup is an assumption.
The script is the procedure rather than a description of it. It takes the most recent completed backup, decrypts it through worker --decrypt-backup, restores it into a throwaway PostgreSQL instance on the stack’s own network, compares row counts, opens a sealed value with APP_DATA_ENCRYPTION_KEY, and removes the instance again. It never writes to the live database, and it exits non-zero when the drill fails, so a scheduled run means something. Pass COMPOSE when the stack uses override files, and TABLES to compare a different set:
What it reports, and how to read it: The sealed-value step is the one most often skipped and the one that most often fails: a database restored without its data-encryption key loads perfectly and has unreadable secrets. The script opens a sealed customer-OIDC client secret in a throwaway container, so it needs nothing installed on the host. An installation with no OIDC provider configured has no sealed value for it to check, and it says so rather than passing quietly — check a TOTP secret or a provider credential by hand before trusting that backup. Record the date and the outcome.

​
Beyond the database: full disaster recovery

Everything above answers “PostgreSQL is lost or corrupted.” A host-level disaster is a wider question, and the answer to most of it is: nothing here needs a special procedure, because PostgreSQL is the only durable store — but two things outside it still need a plan.

​
Losing the whole host

A pg_dump restore brings back every row. It does not bring back what only ever lived outside the database:
  • .env, or whatever holds it in your secrets manager.
  • APP_DATA_ENCRYPTION_KEY. This is the one to get right before anything else. Every sealed value — TOTP secrets, OIDC client secrets, provider credentials, customer contact values — is permanently unopenable without it, and there is no re-encryption path. See Security.
  • APP_BACKUP_ENCRYPTION_KEY, stored separately from the backups it decrypts — a backup that carries both is a single artefact that opens itself.
  • APP_OPERATOR_TOKEN, since nothing issues a new one; a lost token means re-bootstrapping the bearer surface.
  • The reverse-proxy configuration and TLS material in front of the stack, which lives in your own estate rather than in Omniflow. See Reverse proxy.
  • DNS, pointed at whatever host replaces the lost one.
None of these are backed up by the backup format on this page, because none of them are rows in PostgreSQL. Keep them wherever you keep secrets that must outlive the thing they protect — never inside the database backup itself.

​
Losing Valkey

This needs no restore at all. Valkey holds cache entries, rate-limit counters, locks, and single-use replay tokens — nothing durable — so a lost Valkey instance is a fresh one started empty. See Deployment. Automatic maintenance mode detection watches Valkey along with PostgreSQL and Remnawave, so an outage pauses new purchases after the same failure streak as any other dependency, rather than customers hitting rate-limit and replay-guard failures inconsistently mid-checkout.

​
RPO and RTO

Stated from what is actually configurable, not a promised figure this page cannot keep:
  • Recovery point objective is governed by APP_BACKUP_INTERVAL_HOURS — the default is 24 hours between scheduled backups, so up to that much recent activity can be lost in the worst case. Shrink the interval for a tighter RPO; a manual backup from /ops before a risky operation shrinks it to zero for that moment.
  • Recovery time objective is dominated by restore time, which scales with database size, plus the reconciliation window after the stack comes back — Remnawave drift resolves within fifteen minutes of the scheduled reconcile pass, and in-flight payments settle on the next reconciler tick. There is no fixed number to promise here; size your own drill’s duration as the baseline for your installation.

Deployment

Compose topology, volumes, and the upgrade path these procedures assume.

Troubleshooting

What each backup error code means and where to read the log.