> ## Documentation Index
> Fetch the complete documentation index at: https://omniflow-vpn.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Health, metrics, and logs

> The probes each process exposes, the metrics that are actually written, how logging and redaction work, and the real state of tracing

Omniflow's operational surface is deliberately small: three unauthenticated HTTP endpoints, one Prometheus registry per process, and structured JSON logs on stdout. Everything on it is designed for your orchestrator and your monitoring, and none of it belongs on the public API host — both bundled proxy examples return 404 for `/metrics` and `/readyz` on the API domain, because those endpoints answer questions about installation internals to anyone who asks. See [Reverse proxy](https://omniflow-vpn.mintlify.site/operations/reverse-proxy.md).

This page describes what exists. Where a metric is declared but never written, it says so, because an alert built on a series that is never emitted is an alert that never fires.

## The three endpoints

| Endpoint | Answers | Use it for |
| - | - | - |
| `/livez` | `200 {"status":"ok","version":"…"}` as soon as the process is serving | Liveness probe and restart policy |
| `/healthz` | The same handler, byte for byte | Compatibility with existing deployments |
| `/readyz` | `200 {"status":"ready",…}` or `503 {"status":"degraded",…}` | Readiness probe and load-balancer membership |

They are separate because they answer different questions. A process that is running but cannot reach PostgreSQL should be taken out of the pool, not killed and restarted — restarting it does not bring the database back, and a restart loop turns one failing dependency into no capacity at all.

> [!NOTE]
>
> `version` is always `dev`. All three binaries declare `var version = "dev"` and neither the Dockerfile nor the release workflow overrides it with a linker flag, so released images report `dev` on `/livez`, in the operator diagnostics bundle, and in the anonymous telemetry heartbeat. Identify a running build by its image tag, not by this field.

### Where each process listens

| Process | Variable | Default | Notes |
| - | - | - | - |
| `api` | `APP_HTTP_ADDR` | `:8080` | Always serves the probes, even with no database configured |
| `worker` | `APP_WORKER_HTTP_ADDR` | `:8081` | An empty value disables the listener entirely |
| `bot` | `APP_BOT_METRICS_ADDR` | *(unset)* | Unset means the bot serves **no** HTTP at all |

The bot has no probe endpoint out of the box. If your runbook says "check `/readyz` on each process", set `APP_BOT_METRICS_ADDR` first. It is ignored when it equals `APP_BOT_HTTP_ADDR`, the webhook listener's address: publishing metrics on the host Telegram reaches is refused rather than quietly allowed.

## Readiness

Readiness runs one probe per registered dependency and answers 503 while any of them fails.

| Process | Probes registered |
| - | - |
| `api`, with `APP_DATABASE_URL` set | `postgres` (`Ping`), `valkey` (`PING`), `remnawave` (authenticated call) |
| `api`, with no `APP_DATABASE_URL` | none |
| `worker` | `postgres` |
| `bot` | `postgres`, and only when a pool exists |

```json theme={null}
{
  "status": "degraded",
  "checks": [
    {"name": "postgres", "healthy": true, "checkedAt": "2026-08-11T12:00:00Z", "latencyMs": 2},
    {"name": "valkey", "healthy": true, "checkedAt": "2026-08-11T12:00:00Z", "latencyMs": 1},
    {"name": "remnawave", "healthy": false, "error": "timeout", "checkedAt": "2026-08-11T12:00:00Z", "latencyMs": 3001, "consecutiveFailures": 4}
  ]
}
```

`error` is a stable classification and only ever `timeout` (the probe's context expired) or `unavailable`. The driver's own message is dropped on purpose: a pgx or HTTP error routinely contains a DSN, a host name or a credential, and an orchestrator logs this body verbatim.

`consecutiveFailures` counts unhealthy samples in a row and resets on the first success. It is the same counter automatic [maintenance mode](https://omniflow-vpn.mintlify.site/operations/maintenance-mode.md) streaks on, so a readiness body tells you how close a dependency is to pausing purchases.

Results are cached for 2 seconds and each probe is bounded at 3 seconds, both compiled in. The cache exists so a burst of readiness requests cannot become a burst of database round trips.

The Remnawave probe is an **authenticated** call (`ListUsers` for a single record), which means an expired or revoked panel token reports as unavailability rather than passing as a healthy panel. See [Remnawave](https://omniflow-vpn.mintlify.site/integrations/remnawave.md).

> [!WARNING]
>
> An API started with no `APP_DATABASE_URL` registers no probes at all and answers `200 {"status":"ready","checks":[]}`. That is a green light for a process that serves only the four operational endpoints and nothing else. If a deployment mistake blanks the database URL, readiness will not catch it — check that `/v1/catalog` answers as well.

### The same picture in the panel

`GET /v1/panel/overview/health` (permission `system.read`) returns the identical `checks` list, so the panel and the load balancer can never disagree about whether PostgreSQL is up, plus payment-provider and digital-goods-provider status read from the database. The **System → Health** tab renders it and refreshes every 30 seconds. See [Admin panel](https://omniflow-vpn.mintlify.site/operations/admin-panel.md).

## Service-level indicators

Every SLI below is grounded in a metric confirmed live in the table further
down this page. Where a surface has no metric to build one from, that's
stated plainly rather than papered over with an invented number.

| Surface | Indicator | Built from |
| - | - | - |
| API | Availability: share of requests not answering `5xx` | `omniflow_http_requests_total{status="5xx"}` against the same series' total |
| API | Latency: request duration distribution | `omniflow_http_request_duration_seconds` |
| Bot | Update throughput | `rate(omniflow_job_duration_seconds_count{kind="telegram.update"}[5m])` — see the note below on this pair |
| Jobs (fulfillment) | Success rate | `omniflow_jobs_total{kind="fulfillment",outcome="succeeded"}` against the same series' total |
| Jobs (fulfillment) | Duration distribution | `omniflow_job_duration_seconds{kind="fulfillment"}` |
| Payments | Settlement health | `omniflow_payments_total` by `classification` — a rising `underpayment` or `wallet_unavailable` share is the signal, not a single threshold |
| Payments | Webhook trust | `omniflow_provider_webhooks_total{outcome="signature_invalid"}` — any non-zero rate is worth looking at |
| Fulfillment | Drift | Not a Prometheus series today; read from the panel's fulfillment drift view — see [Jobs and queues](https://omniflow-vpn.mintlify.site/operations/jobs-and-queues.md#fulfillment-jobs) |
| Notifications | *(no SLI)* | No delivery metric exists. `omniflow_telegram_sends_total` is declared and never incremented — only inbound updates are counted. Watch delivery through the bot's own failure classification and logs until this is instrumented, not through Prometheus |

> [!NOTE]
>
> The bot's `omniflow_job_duration_seconds{kind="telegram.update"}` histogram
> has no matching `omniflow_jobs_total` counter series, so a rate over the
> histogram's own `_count` is the only way to get Telegram throughput out of
> it — see [What is declared but never written](#what-is-declared-but-never-written).

Machine-readable alert rules and a starting Grafana dashboard built on these
same metrics are in [Reliability and operations](https://omniflow-vpn.mintlify.site/operations/reliability.md#dashboards-and-alerts).

## Metrics

Each process builds its own isolated Prometheus registry, served at `GET /metrics` in the text exposition format alongside the standard Go and process collectors. Every series carries a constant `service` label of `api`, `bot` or `worker`.

Cardinality is bounded by construction. Routes are recorded as chi route **templates** such as `/v1/admin/orders/{orderID}`, never as concrete paths; HTTP status is bucketed to `2xx`, `4xx`, `5xx` or `unknown`; an unmatched request is recorded as `route="unmatched"`. No customer, order or payment identifier is ever a label, because a scrape target retains label values indefinitely and a metrics endpoint is rarely as well protected as an API.

### What is written

| Metric | Type | Labels | Written by |
| - | - | - | - |
| `omniflow_http_requests_total` | counter | `route`, `method`, `status` | Every HTTP request on any process that has metrics |
| `omniflow_http_request_duration_seconds` | histogram | `route`, `method` | The same middleware |
| `omniflow_telegram_updates_total` | counter | `kind`, `outcome` | The bot, once per update. `outcome` is always `handled` |
| `omniflow_job_duration_seconds` | histogram | `kind` | `kind="fulfillment"` from the worker, and `kind="telegram.update"` from the bot's update middleware |
| `omniflow_jobs_total` | counter | `kind`, `outcome` | `kind="fulfillment"` only, with `outcome` `succeeded` or `failed` |
| `omniflow_provider_webhooks_total` | counter | `provider`, `outcome` | Payment webhook intake. `outcome` is `signature_invalid` for a rejected callback, otherwise the settlement classification below |
| `omniflow_payments_total` | counter | `provider`, `operation`, `classification` | Payment settlement. `classification` is one of `paid`, `late`, `underpayment`, `overpayment`, `currency_mismatch`, `wallet_unavailable`, `duplicate` |
| `omniflow_maintenance_mode` | gauge | — | The API's maintenance controller, on every sample |
| `omniflow_dependency_up` | gauge | `dependency` | Every `/readyz` call, and every maintenance sample on the API |

`omniflow_jobs_total` uses the job **kind**, so the fulfillment series is `kind="fulfillment"`. `fulfillment.operation` is the span name, not a label value, and querying for it matches nothing.

### What is declared but never written

Four metrics appear in `/metrics` output with no samples, because nothing in the code ever sets them:

| Metric | State |
| - | - |
| `omniflow_outbox_pending_events` | Never `Set`. The outbox has no dispatcher — see [Jobs and queues](https://omniflow-vpn.mintlify.site/operations/jobs-and-queues.md#the-transactional-outbox) |
| `omniflow_remnawave_requests_total` | Never incremented; the Remnawave client is not instrumented |
| `omniflow_remnawave_errors_total` | Never incremented |
| `omniflow_telegram_sends_total` | Never incremented; only inbound updates are counted |

Two more gaps are worth knowing when you write a dashboard. Digital-goods delivery jobs are uninstrumented — `omniflow_jobs_total{kind="goods_delivery"}` and its duration series never appear, so a stuck delivery is visible in the panel's shop views but not in Prometheus. And `omniflow_job_duration_seconds{kind="telegram.update"}` has no matching counter series, so a rate over the histogram's `_count` is the only way to get Telegram throughput out of that pair.

### Protecting `/metrics` on the API

The API serves `/metrics` on the same listener as the public API. Set
`APP_METRICS_TOKEN` and the route answers only to
`Authorization: Bearer <token>` — configure it as the scrape job's
`authorization` credential in Prometheus. Without a token the route stays open,
as in earlier releases, and the API logs a warning at startup; block the path at
the reverse proxy, as both bundled proxy examples do. The probes `/livez`,
`/readyz`, and `/healthz` are never gated. The worker and the bot serve metrics
on their own listeners, which are not meant to be published, and ignore the
token.

### Turning metrics off

`APP_METRICS_ENABLED` defaults to `true` and is read by the **api** and the **worker**: setting it to `false` leaves the metrics object unbuilt, which removes both the `/metrics` route and the HTTP instrumentation middleware while the probes keep working. The bot ignores it — it constructs metrics unconditionally and serves `/metrics` whenever `APP_BOT_METRICS_ADDR` is set to something other than the webhook address.

The metrics object is nil-safe throughout: every recording helper returns early on a nil receiver, so disabling metrics removes instrumentation rather than introducing a crash path.

### Alerts worth having

Each of these is built on a series something actually writes:

* `omniflow_dependency_up == 0` for longer than `APP_MAINTENANCE_FAILURE_STREAK × APP_MAINTENANCE_PROBE_SECONDS`, which is the point at which detection will pause purchases.
* `omniflow_maintenance_mode == 1` for longer than a planned window. Detection opening a window you did not plan is the signal; it staying open is the incident.
* `rate(omniflow_jobs_total{kind="fulfillment",outcome="failed"}[5m])` above zero — a paid customer is waiting on provisioning that is not landing.
* A rising rate of `omniflow_payments_total{classification="underpayment"}` or `{classification="wallet_unavailable"}` — money arrived and the order did not settle.
* Any `omniflow_provider_webhooks_total{outcome="signature_invalid"}` — a provider callback the installation is refusing, usually a proxy that re-encodes the body or a rotated secret.
* `omniflow_http_requests_total{status="5xx"}` on the `api` service.

Because the outbox gauge is never written, the queue backlog has to be watched through `GET /v1/admin/outbox` or the panel's Outbox tab instead. Read [Jobs and queues](https://omniflow-vpn.mintlify.site/operations/jobs-and-queues.md#the-transactional-outbox) before you decide what a growing count means here.

## Structured logging

All three processes log JSON through `log/slog` with no handler options set, which means the default Info level: the API and bot write to **stdout**, the worker to **stderr**. There is no log-level, log-format or log-destination environment variable anywhere in the repository, so debug logging requires a code change. Collect the streams with your container runtime.

### Redaction

Anything that can reach a log line, an operator notification or a diagnostics bundle passes through a shared redactor first. It removes:

* Telegram bot tokens, and bearer, basic, token, API-key, secret, password and authorization values
* 13-to-19-digit runs, which is card-shaped
* `hwid`, `device_id`, `charge_id` and `invoice_payload` pairs
* **Any** URL in the `http`, `https`, `vless`, `vmess`, `trojan` or `ss` schemes, replaced entirely — even the host identifies the installation a customer connects to

Structured payloads are additionally redacted by key name, matching `token`, `secret`, `password`, `authorization`, `signature`, `api_key`, `hwid`, `subscription_url`, `checkout_url`, `raw_body`, `payload`, `card`, `pan`, `cvv`, `iban`, `email` and `phone`. It is applied to operator notification payloads before the row is written and to the last error on a job before it leaves the process.

The redactor is deliberately aggressive: an over-redacted log line is a minor inconvenience, while a leaked subscription link or bot token is a security incident.

### Classifications instead of messages

Most operational paths log a stable code rather than an error string, which is what makes them safe to log and searchable at the same time:

| Area | Codes |
| - | - |
| Health probes | `timeout`, `unavailable` |
| Remnawave | `remote_user_not_found`, `remnawave_http_<status>`, `remnawave_unavailable` |
| Backups | `command_exit_N`, `verification_failed`, `file_missing`, `cancelled`, `backup_failed` |
| Operator topics | `missing_manage_topics_permission`, `chat_not_found`, `forum_topics_disabled`, `bot_not_in_group`, `telegram_error` |

> [!NOTE]
>
> `/readyz` logs `readiness probe failed` with the number of checks and never which one failed. Read the response body, or scrape `omniflow_dependency_up`, to find out.

### Correlation

`X-Request-ID` is generated by the first middleware in the chain and echoed on every response. Panel mutations store it on the audit event they write, so a support conversation about "what did this operator change" resolves to one request. The bot logs a `correlation_id` on payment paths, and fulfillment operations and their attempt history carry the same identifier, so one customer tap can be followed to the Remnawave call it caused.

## Tracing

Omniflow's code is instrumented with OpenTelemetry at five hops:

| Hop | Span | Attributes |
| - | - | - |
| Inbound HTTP | `http.server` | `otelhttp` defaults |
| Telegram update | `telegram.update` | `messaging.system=telegram`, update kind, update ID |
| Fulfillment job | `fulfillment.operation` | `messaging.system=river`, job kind, attempt |
| PostgreSQL | one span per query | Statement text, trimmed into the span name |
| Outbound HTTP | `http.client` | Method and host only |

Span attributes follow the same rule as metric labels, enforced at a single entry point so no caller has to decide what is safe to record. A Telegram span carries the update ID, which correlates with the logs, but never the Telegram account. Query spans carry statement text and never argument values.

> [!WARNING]
>
> **No trace is ever exported.** Omniflow never installs a tracer provider, imports no OTLP exporter, and reads no `OTEL_*` variable. The global provider is a no-op, so every span above is created and discarded. Setting `OTEL_EXPORTER_OTLP_ENDPOINT` changes nothing, and there is no supported configuration that turns tracing on today.

Two further limits apply if that changes in a fork: the Telegram middleware sets an `Ok` status on the span even when the handler failed, so a failing update is not visible as a span error, and the digital-goods delivery worker opens no span at all.

Until then, correlation across processes is done with the request ID and the fulfillment correlation ID in the logs, and with the operator surfaces described in [Admin operations](https://omniflow-vpn.mintlify.site/operations/admin-operations.md).

## Related

> **[Jobs and queues](https://omniflow-vpn.mintlify.site/operations/jobs-and-queues.md)**
>
> What the worker actually runs, and how to read the dead-letter view.

> **[Maintenance mode](https://omniflow-vpn.mintlify.site/operations/maintenance-mode.md)**
>
> What the dependency streaks in `/readyz` eventually trigger.

> **[Troubleshooting](https://omniflow-vpn.mintlify.site/operations/troubleshooting.md)**
>
> Symptom-first guidance when a probe or a metric points somewhere unhelpful.

> **[Anonymous telemetry](https://omniflow-vpn.mintlify.site/operations/telemetry.md)**
>
> The separate, optional heartbeat Omniflow sends about itself.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.