> ## Documentation Index
> Fetch the complete documentation index at: https://omniflow-vpn.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Deployment

> The compose topology service by service, what has to persist, how migrations are applied, and how to upgrade and roll back

Omniflow ships as one Docker Compose stack for a single Linux server: PostgreSQL, Valkey, a one-shot migration job, the three Go processes (`api`, `bot`, `worker`), and an optional Next.js `web` container behind a compose profile. `compose.yaml` at the repository root is the whole topology — there is no Helm chart, no operator, and no second supported form.

The stateful services stay private to the compose network and Omniflow bundles no reverse proxy, because TLS, body limits and forwarded-header trust are decisions an operator already makes for their own estate. Two maintained ingress examples live under `deploy/proxies/`; see [Reverse proxy](https://omniflow-vpn.mintlify.site/operations/reverse-proxy.md).

## The topology, service by service

| Service | Image or build | Published ports | Waits for | Volumes |
| - | - | - | - | - |
| `postgres` | `postgres:18.4-alpine` | none | — | `postgres-data:/var/lib/postgresql` |
| `valkey` | `valkey/valkey:9.1.1-alpine` | none | — | none, deliberately |
| `migrate` | `arigaio/atlas:1.3.0` | — | `postgres` healthy | `./database/migrations` mounted read-only |
| `api` | `build/go.Dockerfile`, `TARGET=api` | `8080:8080` | `migrate` completed, `valkey` healthy | none |
| `bot` | `build/go.Dockerfile`, `TARGET=bot`, `RUNTIME=postgres` | none | same | `backup-data:/var/lib/omniflow/backups` |
| `worker` | `build/go.Dockerfile`, `TARGET=worker`, `RUNTIME=postgres` | none | same | `backup-data:/var/lib/omniflow/backups` |
| `web` | `build/web.Dockerfile`, profile `web` | `3000:3000` | `api` | none |

Only `api` and `web` publish a host port. PostgreSQL and Valkey are reachable on the internal network alone, which is why nothing in the stack needs a database password you would be comfortable exposing.

`postgres` has a `pg_isready` healthcheck every 5 seconds with 20 retries and `valkey` a `valkey-cli ping` healthcheck, and the application services depend on `migrate` having *completed successfully* rather than merely started. A schema that failed to apply therefore stops the release instead of producing three processes running against a half-migrated database.

Each of `api`, `bot` and `worker` reads `env_file: [.env]`, so all three see the same configuration. The API additionally gets `APP_HTTP_ADDR: :8080` set explicitly in compose. What each variable does is catalogued in [Configuration](https://omniflow-vpn.mintlify.site/getting-started/configuration.md).

### Why `valkey` is configured with no persistence

The command is `valkey-server --save "" --appendonly no`: no snapshots, no append-only file. Valkey holds cache entries, rate-limit counters, locks and single-use replay tokens and nothing else, so losing it costs a window of rate-limit state rather than data. Anything Omniflow cannot afford to lose is in PostgreSQL.

> [!WARNING]
>
> The corollary is that a Valkey restart clears the replay guards that protect payment callbacks and destructive Telegram buttons for the length of their retention window. Restart it during a quiet period rather than in the middle of a campaign.

## What must persist

Two named volumes carry state:

* **`postgres-data`** — everything. Orders, the ledger, entitlements, audit events, sessions, settings, the queue. It is mounted at `/var/lib/postgresql`, the directory the image declares, rather than at the `/var/lib/postgresql/data` that was `PGDATA` up to PostgreSQL 17. Version 18 moved it to `/var/lib/postgresql/18/docker` so a major upgrade can hold two data directories side by side, and mounting the old path leaves this volume empty while the database goes to an anonymous volume that `docker volume prune` eventually removes. If you are upgrading an installation that ran the earlier mount, your data is in that anonymous volume: find it with `docker volume ls -f dangling=true`, or take a backup before changing the mount and restore it afterwards.
* **`backup-data`** — the encrypted `pg_dump` output, mounted at `/var/lib/omniflow/backups` on both `bot` and `worker` because one takes backups on a schedule and the other on demand. Backups live outside the container filesystem so a rebuild never discards them.

If you change where PostgreSQL is mounted — a different compose file, a bind mount, a managed database moved back in-house — check the result rather than assuming it:

```bash theme={null}
sh tools/volume-drill.sh
```

It runs under a compose project of its own, so it never touches a stack already on the host. It asserts that the running `data_directory` is inside the named volume and that the container has no anonymous volumes, then writes a row, recreates the container, and reads it back. CI runs it against the shipped `compose.yaml` on every change.

Support-ticket attachments uploaded through the customer web panel are the exception you have to handle yourself. They are written to `APP_SUPPORT_ATTACHMENT_DIR`, which defaults to `/var/lib/omniflow/support-attachments`, and `compose.yaml` declares **no volume for that path on either `api` or `worker`**. As shipped, a browser-uploaded attachment lives in the API container's writable layer and disappears on the next rebuild, and the worker — which is the process that reclaims orphaned attachment files — cannot see the same directory. If your installation accepts web uploads, add a shared volume mounted at the same path on both services and set `APP_SUPPORT_ATTACHMENT_DIR` to it on both. A worker pointed at a different directory deletes the rows and leaves the bytes behind.

## First deployment

**1. Prepare the environment file**

```bash theme={null}
cp .env.example .env
```

Fill in `APP_DATABASE_URL`, `APP_VALKEY_URL`, `APP_REMNAWAVE_URL`, `APP_REMNAWAVE_TOKEN`, `APP_OPERATOR_TOKEN`, a base64 32-byte `APP_DATA_ENCRYPTION_KEY`, and `APP_TELEGRAM_TOKEN` if you are running the bot. The API refuses to start with a database URL and any of the first five missing, exiting with `commerce requires APP_OPERATOR_TOKEN, APP_DATA_ENCRYPTION_KEY, APP_REMNAWAVE_URL, APP_REMNAWAVE_TOKEN, and APP_VALKEY_URL`.

**2. Bring the core stack up**

```bash theme={null}
docker compose up --build postgres valkey migrate api bot worker
```

`make dev` runs the same set. Add the web application with `docker compose --profile web up --build`, or `make up`.

**3. Read the setup token out of the API log**

```
level=WARN msg="no administrator exists yet; redeem this one-time setup token at /admin/setup" token=… expires_in=30m0s
```

Redeem it at `/admin/setup` within 30 minutes to create the first owner. See [Admin panel](https://omniflow-vpn.mintlify.site/operations/admin-panel.md).

**4. Verify readiness before routing traffic**

```bash theme={null}
curl -s localhost:8080/readyz
```

A `200` with `"status":"ready"` and every check healthy is the gate. What the probes mean, and the one case where a green `/readyz` is misleading, is covered in [Health, metrics, and logs](https://omniflow-vpn.mintlify.site/operations/observability.md).

After that the API is on `http://localhost:8080`, the customer account app on `http://localhost:3000`, and the operator workspace on `http://localhost:3000/admin`.

## Applying migrations

Migrations are forward-only SQL files in `database/migrations/`, applied by Atlas. The `migrate` service runs `atlas migrate apply` against the database and exits; Atlas records what it has already applied in the `atlas_schema_revisions` schema, so re-running it on an up-to-date database is a no-op and the whole stack can be brought up repeatedly without special handling.

There are no down-migrations. That is a deliberate choice rather than an omission: a schema rollback is a restore from a backup, which is the only path that also restores the data the forward migration reshaped. See [Backup and restore](https://omniflow-vpn.mintlify.site/operations/backup-restore.md).

> [!NOTE]
>
> The `migrate` service hard-codes `postgres://omniflow:omniflow@postgres:5432/omniflow?sslmode=disable` in its command and does **not** read `.env`. Changing the database credentials means editing `compose.yaml` as well as `APP_DATABASE_URL`, or the migration job will keep pointing at the old ones.

The same migrations create the durable job queue's tables, so the first deploy needs no separate queue setup — see [Jobs and queues](https://omniflow-vpn.mintlify.site/operations/jobs-and-queues.md#the-queue-schema-ships-with-the-migrations).

For how migrations are authored and reviewed, see [Database](https://omniflow-vpn.mintlify.site/contributing/database.md).

## Images

`build/go.Dockerfile` builds all three Go binaries from one multi-stage file. `CGO_ENABLED=0 go build -trimpath -ldflags="-s -w"` produces a static binary, and a `RUNTIME` build argument selects the final base:

| `RUNTIME` | Base | Used for |
| - | - | - |
| `static` (default) | `gcr.io/distroless/static-debian13:nonroot` | `api` |
| `postgres` | `alpine:3.23.5` with `postgresql18-client`, `ca-certificates`, `tzdata` | `bot`, `worker` |

`pg_dump` and `pg_restore` must match the server's major version, which is why the client is pinned to PostgreSQL 18 alongside the `postgres:18.4-alpine` server. Alpine 3.23 is the first release carrying `postgresql18-client`, so lowering that base breaks the backup path.

Both bases run as `nonroot` (uid 65532 on the Alpine one), and `/var/lib/omniflow` is pre-created and owned by that user so the backup directory is writable without a privileged container.

The web image installs and builds with Bun and serves the Next.js standalone output from `node:26.7-alpine` on port 3000. npm is deleted from the runtime stage: nothing there invokes it, and it arrives from the base image carrying a bundled dependency tree of its own that an image scan reports as findings. Removing it removes the code rather than silencing the scanner.

### Released images

Tagging `v*` builds and pushes `api`, `bot`, `worker` and `web` to `ghcr.io/<repo>/<name>` for `linux/amd64` and `linux/arm64`, tagged `{{version}}` and `{{major}}.{{minor}}`, with full provenance and an SBOM.

Nothing in the repository consumes those images: `compose.yaml` builds from the working tree. Running a pinned release means writing a small override that replaces each `build:` block with an `image:` line. Do not point production at a floating tag — pin the full version, so that what you tested is what restarts.

> [!WARNING]
>
> The release workflow passes only `TARGET` to the build, so the published `bot` and `worker` images use the default `static` base and do **not** contain `pg_dump` or `pg_restore`. Scheduled backups and the `/ops` restore surface need images built with `RUNTIME=postgres`, which is what `compose.yaml` does. If you switch to released images, keep building `bot` and `worker` yourself or backups will fail with a command error.

## Supported topology

| Role | Replicas | Why |
| - | - | - |
| `api` | Any number, behind the proxy | It runs no state-changing background pass. Its only state-changing loop, the maintenance controller, reads the current maintenance state before changing it, so a second replica finds a transition already made |
| `worker` | Any number | River distributes jobs among the replicas, and the periodic passes — order and cart maintenance, payment reconciliation, renewals, sweeps, campaigns, retention, backups — are enqueued by River's elected leader only, so each runs once per interval across the installation |
| `bot` | Exactly one | Telegram delivers updates to one consumer; long polling refuses a second `getUpdates` |

Several workers need the same shared storage one worker already uses: `APP_BACKUP_DIR` on a shared volume, because a backup lands on whichever replica runs the pass, and the same `APP_SUPPORT_ATTACHMENT_DIR` as the API. Every replica opens its own database pool, so each one counts against PostgreSQL's `max_connections` — see below. The full list of passes and in-process loops is in [Jobs and queues](https://omniflow-vpn.mintlify.site/operations/jobs-and-queues.md#periodic-passes-and-in-process-loops).

### Database connections

Each process bounds its PostgreSQL pool with `APP_DATABASE_MAX_CONNS`, a positive whole number; any other value refuses to start. When it is unset:

| Process | Default pool |
| - | - |
| `api` | 20 |
| `bot` | 10 |
| `worker` | 40 |

The defaults sum to 70, inside PostgreSQL's stock `max_connections` of 100, with room left for migrations, backups and an operator's `psql`. The worker refuses to start with fewer than 32 connections: it runs up to 28 jobs at once across its queues, plus 4 for River's own bookkeeping. A `pool_max_conns` parameter in `APP_DATABASE_URL` takes precedence over the variable.

A value in the shared `.env` applies to all three processes. To size them differently, set it per service under `environment:` in a compose override. When you add replicas, add up every replica's pool and keep the total below `max_connections`.

## Exposing the stack

With the base compose file the API answers on host port 8080 and the web app on 3000, which is right for a local run and wrong for the internet. Put a proxy in front:

```bash theme={null}
ACME_EMAIL=ops@example.com \
WEB_DOMAIN=app.example.com \
API_DOMAIN=api.example.com \
BOT_WEBHOOK_DOMAIN=bot.example.com \
docker compose -f compose.yaml -f deploy/proxies/traefik.compose.yaml up -d
```

The overlay uses `ports: !override []` to stop `api` and `web` publishing on the host at all, which needs Docker Compose 2.24.4 or newer; on an older release, delete those `ports` blocks from `compose.yaml` by hand instead. A Caddy example under the same directory does the equivalent work.

Both examples 404 `/metrics` and `/readyz` on the public API host, cap request bodies, and refuse to trust `X-Forwarded-For` from anywhere but private ranges. Set `APP_TRUSTED_PROXIES` to match, or the audit trail and the rate limiter will see the proxy's address instead of the client's. Details, including why a proxy that re-encodes JSON bodies breaks every provider webhook signature, are in [Reverse proxy](https://omniflow-vpn.mintlify.site/operations/reverse-proxy.md).

## Upgrading

**1. Take a backup and confirm it verified**

Every Omniflow backup is decrypted and hashed end to end before it is marked `completed`, so a `completed` row is a backup that has already been proved readable. Do not upgrade on the strength of a `running` one. See [Backup and restore](https://omniflow-vpn.mintlify.site/operations/backup-restore.md).

**2. Read the release notes for migration weight**

Migrations run before the new containers start and the stack waits on them. A migration that rewrites a large table holds the whole release open for its duration.

**3. Pause purchases if the upgrade touches provisioning**

Turning maintenance mode on stops new orders and holds outbound Remnawave calls without touching money already taken. Held fulfillment jobs are snoozed, not failed, and resume when you clear it. See [Maintenance mode](https://omniflow-vpn.mintlify.site/operations/maintenance-mode.md).

**4. Update the source or the pinned image tags, then bring the stack up**

```bash theme={null}
docker compose up -d --build migrate api bot worker
```

`migrate` runs to completion first; the three processes then start against the new schema.

**5. Verify, then clear maintenance**

Check `/readyz` on the API and on the worker's `APP_WORKER_HTTP_ADDR`, confirm the worker logged `worker started`, then clear the maintenance window with a reason.

Configuration is read once at startup and never re-read, so a changed `.env` needs a restart of the affected process. Nothing in Omniflow watches the environment or a config file, and the operator panel's settings sections do not feed the runtime either — see [Installation settings](https://omniflow-vpn.mintlify.site/operations/installation-settings.md).

### Shutdown behaviour during a restart

All three processes install a `SIGINT`/`SIGTERM` handler and drain on a 15-second budget. The API stops accepting connections and lets in-flight requests finish before closing its pools; the worker shuts its operational listener and then calls River's `Stop`, which drains in-flight jobs so a restart never abandons a fulfillment run mid-flight; the bot shuts its webhook and operational servers the same way.

Periodic passes are River jobs and stop with River. The remaining in-process ticker loops exit on the same context and are not waited on. A pass still running when the 15-second budget is spent has its context cancelled, so a backup cut off mid-`pg_dump` is recorded as failed with `cancelled` rather than left `running`.

> [!NOTE]
>
> None of the core compose services declares a `restart:` policy — only the Traefik overlay does. A process that exits stays down until somebody starts it. Add `restart: unless-stopped` in an override if you want the stack to survive a host reboot on its own.

## Rolling back

There is no schema down-path, so a rollback has two shapes depending on what changed.

**No migration ran.** Put the previous image tag or source revision back and bring the stack up. Nothing else is needed.

**A migration ran.** The old binaries do not understand the new schema, so restoring the database is part of the rollback. Restore the pre-upgrade backup, then start the previous version against it. `pg_restore` runs with `--clean --if-exists --single-transaction`, so everything written since the backup point is lost — which is exactly why the backup step precedes the upgrade rather than following it.

> [!WARNING]
>
> A restore into an empty database must also carry the `atlas_schema_revisions` schema, or Atlas will try to re-apply every migration on the next start. A full custom-format dump contains it; a selectively filtered one may not.

The restore path itself — the `/ops` Telegram surface, and the standalone `--decrypt-backup` filter that needs nothing but the worker binary, the key and the file — is documented in [Backup and restore](https://omniflow-vpn.mintlify.site/operations/backup-restore.md).

## Where to look when a deploy misbehaves

* The process exited immediately with `invalid configuration` — a required variable is missing or malformed; the message names the rule it broke.
* `migrate` failed and the application services never started — read its log; the stack is designed to stop here.
* `/readyz` returns 503 — one dependency is failing; the body names which, with a `timeout` or `unavailable` classification. See [Health, metrics, and logs](https://omniflow-vpn.mintlify.site/operations/observability.md).
* Purchases are refused with `maintenance_active` — automatic detection opened a window. See [Maintenance mode](https://omniflow-vpn.mintlify.site/operations/maintenance-mode.md).
* Broader symptom-first guidance is in [Troubleshooting](https://omniflow-vpn.mintlify.site/operations/troubleshooting.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.