> For the complete documentation index, see [llms.txt](https://docs.drizz.dev/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.drizz.dev/twins/self-hosting/monitoring.md). # Monitoring Scrape the twins server's metrics, turn on its alerts on each platform, and know what to do when one fires. Scrape the twins server's metrics, turn on its alerts on each platform, and know what to do when one fires. | | | | ------------- | ---------------------------------------------------------------------------------------------------------------------- | | **Metrics** | Prometheus format at `:9464/metrics`, and OTLP where `OTEL_EXPORTER_OTLP_ENDPOINT` is set | | **Runs on** | Kubernetes · AWS · Google Cloud · Azure · Docker Compose | | **Logs** | One JSON line per event | | **Watch out** | Where `networkPolicy.allowFrom` closes the server, name the scraper in `worlds.metrics.scrapers`, or `TwinsDown` fires | ## Metrics The twins server serves its metrics at `:9464/metrics` in Prometheus' format. `TWINS_METRICS_PORT` changes the port; on the Helm chart it is `worlds.metrics.port`, and `0` serves none. It also pushes them over OTLP where `OTEL_EXPORTER_OTLP_ENDPOINT` is set. And it writes a log line when something you must act on happens. An install that watches logs alone is told too. ### The main metrics | Metric | What it tells you | | ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------- | | `twins_ready` | Whether the server is ready. `0` while it is not. | | `http_server_request_duration_seconds` | Request count and latency, by response status. | | `twins_volume_free_bytes`, `twins_volume_size_bytes` | Free and total space on the worlds' volume. | | `twins_tenure_renewals_total` | Renewals of the lease over the worlds' copies, by `result` (`refused`, `failed`). | | `twins_replication_writer_up`, `twins_replication_writer_responding` | Whether Litestream is running and answering. | | `twins_replication_uploads_total` | Copies uploaded. It should keep rising. | | `twins_replication_upload_errors_total`, `twins_replication_sync_errors_total` | Litestream's failures. | | `twins_replication_disk_full` | Litestream paused because the worlds' disk is full. | | `twins_replication_restores_total` | Worlds restored from their copies, by `result`. | The replication and lease metrics apply where the worlds are copied to a bucket (`worlds.replication`, AWS `replication`). ## Turn on alerts {% tabs %} {% tab title="Kubernetes" %} `worlds.metrics.serviceMonitor.enabled` and `worlds.metrics.rules.enabled` give the Prometheus Operator (kube-prometheus-stack, and the managed Prometheus of GKE and AKS) a ServiceMonitor and twins' alerts as a PrometheusRule. ```yaml worlds: metrics: serviceMonitor: {enabled: true, labels: {release: kube-prometheus-stack}} rules: {enabled: true, labels: {release: kube-prometheus-stack}} # where networkPolicy.allowFrom closes the server, the scraper's namespace scrapers: [{namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: monitoring}}}] ``` Each alert in the PrometheusRule carries a `runbook_url`. Where `networkPolicy.allowFrom` closes the server, name the scraper in `worlds.metrics.scrapers`, or the scrape fails and `TwinsDown` fires. {% endtab %} {% tab title="AWS" %} `alarm_topic_arn` turns on CloudWatch alarms. The module's own alarms, from AWS's [recommended alarms](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Best_Practice_Recommended_Alarms_AWS_Services.html): * The server's task not running. * Unhealthy targets: `UnHealthyHostCount` above 0 for five minutes. * Server errors over `alarm_target_5xx_count` (10) a minute, kept up for five minutes. * Postgres short of storage, or over 90% CPU. * EFS at its I/O limit (`PercentIOLimit` at 100 for fifteen minutes), and over `alarm_worlds_metered_io_gib` an hour when you set it. Beside those, one metric filter for each event the server and Litestream log, under the `Twins/` namespace: the lease lost or failing to renew, Litestream not answering or logging errors, copies that do not line up, a restore refused, the worlds' volume low, and the server refusing its records' database. Each alarm's description names its runbook. {% hint style="warning" %} A topic encrypted with KMS takes the alarms only under a customer managed key whose key policy lets CloudWatch use it: `kms:GenerateDataKey*` and `kms:Decrypt` for `cloudwatch.amazonaws.com` ([SNS and AWS services' encrypted topics](https://docs.aws.amazon.com/sns/latest/dg/sns-key-management.html#compatibility-with-aws-services)). Otherwise the alarms change state and no notification arrives. {% endhint %} {% endtab %} {% tab title="Google Cloud" %} `alert_notification_channels` (Cloud Monitoring notification channels) turns on one log-based alert policy per event you must act on: the lease lost or failing to renew, Litestream not answering or logging errors, a restore refused, the worlds' volume low, and the server refusing its records' database. Each policy notifies at most every five minutes and names its runbook. Google Cloud Managed Service for Prometheus can also scrape the chart's metrics (`worlds.metrics`). {% endtab %} {% tab title="Azure" %} `alert_action_group_ids` (Azure Monitor action groups) turns on one log alert rule per event you must act on, queried from Container Insights' `ContainerLogV2` every five minutes: the lease lost or failing to renew, Litestream not answering or logging errors, a restore refused, the worlds' volume low, and the server refusing its records' database. It needs `monitoring` on. Each rule names its runbook. Azure Monitor managed service for Prometheus can also scrape the chart's metrics (`worlds.metrics`). {% endtab %} {% tab title="Docker Compose" %} Scrape port 9464 with your own Prometheus. The Compose file does not publish it, so publish it the way `TWINS_PORT` is published. Or send the server's log to a log tool and alert on the events below. {% endtab %} {% endtabs %} ## Read the server's log The server writes one JSON line per event. | Install | Command | | -------------- | ------------------------------------------------------------------ | | Docker Compose | `docker compose -f deploy/docker-compose.selfhost.yml logs worlds` | | Kubernetes | `kubectl -n twins logs statefulset/twins-worlds` | | AWS (ECS) | `aws logs tail --follow` | To find one kind of event, filter with `jq`: ```bash kubectl -n twins logs statefulset/twins-worlds | jq 'select(.event == "tenure.lost")' ``` ## Alerts and what to do | Alert | Severity | What it means | What to do | | --------------------------- | -------- | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | `TwinsDown` | Critical | Nothing answers the metrics scrape for 2 minutes. Nothing is served. | Check the pod or task is running, and read the previous container's log. On Kubernetes, let the scraper reach port 9464 (`worlds.metrics.scrapers`). | | `TwinsNotReady` | Critical | `/ready` has answered 503 for five minutes. Every request but `/health` is refused. | Read the log, newest first, for the events below. Stop whatever else holds the lease or the lock. The server waits and needs no restart. | | `TwinsLeaseLost` | Critical | Another process took the lease over the worlds' copies. The server stopped writing and exits. | Make sure exactly one install writes each bucket and prefix. The bucket's `/tenure.json` names the holder. | | `TwinsLeaseRenewalsFailing` | Warning | Renewals of the lease are failing. When the term runs out, the server stops. | Restore the server's access to the bucket: credentials, bucket policy, proxy or DNS. Renewals resume by themselves. | | `TwinsReplicationStalled` | Critical | Litestream is not running or not answering. Writes reach no copy. | Restart the server. Then check `twins_replication_uploads_total` rises again. | | `TwinsReplicationErrors` | Warning | Litestream is failing to sync or upload some copies. | Fix access to the bucket or the disk. For `validation error detected`, make sure one install writes the prefix, and tell us the database and the line. | | `TwinsReplicationDiskFull` | Critical | Litestream paused because the worlds' disk is full. | Grow the volume (see `TwinsVolumeLow`). | | `TwinsRestoreRefused` | Critical | A world's copy could not be restored whole. That world will not open; every other world is served. | Restore that world from a point before the damage, or from the volume's backup. Then delete `.db.refused`. | | `TwinsVolumeLow` | Warning | Less than 10% of the worlds' volume is free. | Grow the volume, or remove worlds no longer needed. Old dumps in `backups/` can go. | | `TwinsVolumeCritical` | Critical | Less than 5% of the worlds' volume is free. Writes fail when it fills. | As for `TwinsVolumeLow`, now. | | `TwinsErrors` | Warning | More than 5% of requests failed with a 5xx for ten minutes. | Read the log for `request.failed`. If it is not a readiness or connection problem, send us the lines with their `request.id`. | | `TwinsSlow` | Warning | The slowest 5% of requests took more than two seconds for fifteen minutes. | Give the server more CPU or memory, or move the database or cache closer. Else send us the slow requests' `request.path` and `duration`. |
TwinsDown: what to check * Is the pod or task running? `kubectl -n twins get pod twins-worlds-0`, or the ECS service's events. * Is it restarting? The previous container's log says why: `kubectl -n twins logs twins-worlds-0 --previous`. * On Kubernetes, the NetworkPolicy must let the scraper reach port 9464: `worlds.metrics.scrapers`. Act on the cause the log names. A server refused at start says so (`TwinsNotReady`).
TwinsNotReady: the log events Read the log, newest first: | Event | Meaning | | -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `tenure.waiting` | Another process holds the lease over the copies. It may be an old copy that has not stopped, or a second install pointed at the same bucket and prefix. | | `lock.waiting` | Another process holds `/.writer.lock`. An old pod is still on the volume. | | `lock.unwritable` | The volume is read-only or full, and the server stops. | | `recover.asked` with no `recover.done` | The worlds are still being restored from their copies. A big root takes a while. This is not stuck while `twins_replication_restores_total` still rises. | | `directory.ahead` | A newer version migrated the records' database, and this version refuses it. Run the newer version, or restore the dump from before the upgrade. See [Upgrades](/twins/self-hosting/upgrades.md#rolling-back). | | `dump.refused` | The dump before a migration failed. The line names why: `pg_dump` cannot reach the database, or the disk is full. Fix that, or take a snapshot yourself and set `TWINS_DATABASE_BACKUPS` empty. | Stop whatever else holds the lease or the lock. The server waits for them and needs no restart. On Kubernetes, a node that died with the old pod on it is let go with `kubectl delete node `.
TwinsLeaseLost and TwinsLeaseRenewalsFailing **Lease lost.** Another process took the lease, or this one could not renew it within its term. The server stopped writing at once (`tenure.lost`, `tenure.expired`, then `tenure.stopping`) and exits. Its writes since its last copy are the only ones at risk. Check which process holds it now: the bucket's `/tenure.json` names the holder. Look for two installs on one bucket and prefix, or an old pod that came back after a pause. A server that thawed after a pause writes only into its own, older generation. The new holder's copies are safe, and the old one stops on its own. **Renewals failing** (`tenure.renew.failed`). Read the line's `error`: credentials expired, the bucket's policy, a proxy or DNS on the way to S3 (or the store's `TWINS_REPLICA_ENDPOINT`). Restore the server's access to the bucket.
TwinsReplicationStalled and TwinsReplicationErrors **Stalled.** `twins_replication_writer_up` is 0, or `twins_replication_writer_responding` is 0, or `replication.unresponsive` is in the log. Check Litestream's own lines in the log (`level=ERROR`, `level=WARN`) and the server's CPU and memory: a container at its memory limit can starve Litestream. Restart the server. Litestream starts with it and picks up where its copies end. **Errors.** `level=ERROR` lines, `twins_replication_upload_errors_total` or `twins_replication_sync_errors_total`. The lines name the database and the error: access denied, throttling, a full disk, a network timeout. A `validation error detected` line means copies were written by two processes. The newest generation is what restores read.
TwinsRestoreRefused: putting one world back The restore was refused, or what it put back failed SQLite's integrity check (`recover.refused`). What the restore left is beside the database as `.db.refused`. Check the line's `detail` and `generation`. An older generation of the same database may still restore. * Restore the world from a point before the damage. On AWS: the restore task, with `RESTORE_TIMESTAMP`. Elsewhere: `litestream restore -timestamp` from the generation named. * Or restore it from the volume's backup. See [Backups](/twins/self-hosting/backups.md#restoring). * Then delete `.db.refused`.
TwinsVolumeLow and TwinsVolumeCritical Check `twins_volume_free_bytes`, and which worlds are largest (`du -sh /var/lib/twins/*` in the container). Worlds past their expiry are deleted by the sweep, and removed worlds' copies after `TWINS_WORLDS_RETENTION`. * Grow the volume. Kubernetes: raise `worlds.persistence.size`, where the storage class allows expansion. Compose: the host's disk. EFS grows by itself. * Or remove worlds no longer needed. * Dumps before upgrades are kept in `backups/` on the same volume, the newest five. Old ones can go.
TwinsErrors and TwinsSlow **Errors.** Read the log for `request.failed` and the requests' `status`. 503s come from a server that is not ready (`TwinsNotReady`). Other 5xx mean a database or cache is unreachable (`ConnectionError`), or a fault to report to us. **Slow.** Check the server's CPU and memory against its limits, the database's and the cache's latency, and how many worlds are open: each open world holds its databases.
## Next * [Backups](/twins/self-hosting/backups.md): restore a world or a whole install. * [Upgrades](/twins/self-hosting/upgrades.md): roll back a version the database refuses. * [Troubleshooting](/twins/reference/troubleshooting.md): problems that raise no alert. *** *Last updated: 27 October 2026*