> For the complete documentation index, see [llms.txt](https://docs.drizz.dev/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.drizz.dev/twins/self-hosting/backups.md).

# Backups

Back up the two volumes a self-hosted twins server needs, and restore them on each install.

Back up the two volumes a self-hosted twins server needs, and restore them on each install.

|               |                                                                                          |
| ------------- | ---------------------------------------------------------------------------------------- |
| **Keep**      | `records` (Postgres) and `worlds` (SQLite files)                                         |
| **Runs on**   | Docker Compose · Helm chart · AWS · Google Cloud · Azure                                 |
| **Can lose**  | About a second of worlds with replication. Up to a day without it.                       |
| **Watch out** | A file-by-file copy (`cp`, `rsync`, AWS Backup of EFS) is not safe while the server runs |

## What to keep

Two volumes hold everything:

| Volume    | Holds                                                   |
| --------- | ------------------------------------------------------- |
| `records` | Postgres: people, keys and the record of what each did. |
| `worlds`  | Every world's data, as SQLite files.                    |

Back up both. The cache holds nothing that outlives a restart.

## Backups by install

{% tabs %}
{% tab title="Docker Compose" %}
Snapshot the host's disk. Or stop the server, copy the `worlds` volume, and take a `pg_dump` of the database:

```bash
docker compose -f deploy/docker-compose.selfhost.yml stop worlds
```

Start it again with `docker compose -f deploy/docker-compose.selfhost.yml up -d worlds` once the copy is done.
{% endtab %}

{% tab title="Helm chart" %}
`worlds.replication` streams every world to an S3 bucket as it is written. A server on an empty volume restores them before it serves. See [Replicate worlds to S3](/twins/self-hosting/kubernetes.md#replicate-worlds-to-s3).

Or snapshot the volume with the cluster's CSI driver.

Back up the database too: the bundled Postgres's volume, or your managed database's own backups.
{% endtab %}

{% tab title="AWS" %}
The module backs up both on its own:

* **Worlds' copies in S3** (`replication`, on by default). The server runs [Litestream](https://litestream.io/how-it-works/), which follows each database's write-ahead log and sends what each transaction changed, with a snapshot of each database a day. A restore to a time puts every database back as it stood then. The bucket keeps what Litestream replaces or deletes a week longer (versioning), as Litestream's [S3 guide](https://litestream.io/guides/s3/) advises.
* **AWS Backup of the EFS file system** (`backup`), daily, kept 35 days. It reaches further back than the copies, for a loss noticed late.
* **RDS automated backups**: any point in the last `db_backup_retention_days` (35 by default).
  {% endtab %}

{% tab title="Google Cloud" %}

* **Backup for GKE**, nightly, when Cloud SQL's daily backup starts: twins' namespace with the worlds' disk. Kept `backup_retain_days` (14 by default).
* **Cloud SQL**: `postgres_backup_count` daily backups with a week of point-in-time recovery.
  {% endtab %}

{% tab title="Azure" %}

* **Azure Backup for AKS**, nightly at 22:00 UTC: twins' namespace with the worlds' disk and its volume. Kept `backup_retain_days` (30 by default).
* **Azure Database for PostgreSQL**: `backup_retention_days` (30) of point-in-time backups.

Both default to 30, so every night of one has the other beside it. Keep them equal.
{% endtab %}
{% endtabs %}

## Safe and unsafe copies

| Copy                                                                                                                                                                        | Safe while the server runs? | Why                                                                           |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | ----------------------------------------------------------------------------- |
| A snapshot of a whole block volume: a disk snapshot, Backup for GKE's or Azure Backup's of the disk, a CSI `VolumeSnapshot`, an EBS, Persistent Disk or Azure Disk snapshot | **Yes**                     | It is taken at one instant. SQLite restores it as it would after a power cut. |
| A copy made file by file: `cp`, `rsync`, AWS Backup of EFS                                                                                                                  | **No**                      | Files copied at different moments may not agree.                              |
| Litestream copies (`worlds.replication`, AWS `replication`)                                                                                                                 | **Yes**                     | A copy is always one the database itself could have been at.                  |

AWS says of EFS backups: "inconsistencies ... can occur if you make modifications to your file system while the backup is occurring" ([EFS: backing up](https://docs.aws.amazon.com/efs/latest/ug/awsbackup.html)).

Take a file-by-file copy with the server stopped. Where the newest EFS backup will not open, restore an older one, or replicate the worlds as they are written.

## How much can be lost, and how long a restore takes

What a failure can lose (recovery point) and how long putting things back takes (recovery time), by install. The times are typical. Measure yours with a restore drill.

| Install                                               | Worlds can lose                               | Records can lose                              | Putting worlds back                                                          | Putting records back                 |
| ----------------------------------------------------- | --------------------------------------------- | --------------------------------------------- | ---------------------------------------------------------------------------- | ------------------------------------ |
| Replicating (`worlds.replication`, AWS `replication`) | About a second: Litestream sends every second | The provider's log backup (RDS: five minutes) | Minutes: a server on an empty disk restores each world as it is first opened | The provider's point-in-time restore |
| AWS without replication                               | Up to a day: AWS Backup of EFS, daily         | Five minutes (RDS)                            | Minutes to an hour: restore the recovery point, then the restore task        | RDS point-in-time restore            |
| GKE, AKS without replication                          | Up to a day: the nightly disk backup          | Cloud SQL's or Azure's own                    | Restore the backup of the namespace and its disk                             | The provider's point-in-time restore |
| Compose, or the chart's own Postgres                  | Since your last snapshot of the volume        | Since your last `pg_dump`                     | Copy the volume back                                                         | `pg_restore`                         |

Before every upgrade that migrates, the server also dumps the records. Going back from an upgrade loses nothing written before it. See [Upgrades](/twins/self-hosting/upgrades.md).

## Restoring

Always stop the twins server first, so nothing writes while things are put back. Restore the records and the worlds to the same time.

{% tabs %}
{% tab title="Docker Compose" %}

1. Stop the server: `docker compose -f deploy/docker-compose.selfhost.yml stop worlds`.
2. Copy the `worlds` volume back from your snapshot.
3. Restore the database with `pg_restore`:

   ```bash
   docker compose -f deploy/docker-compose.selfhost.yml exec -T postgres \
     pg_restore --clean --if-exists --no-owner -U twins -d twins < twins.dump
   ```
4. Start the server: `docker compose -f deploy/docker-compose.selfhost.yml up -d worlds`.
   {% endtab %}

{% tab title="Helm chart" %}
With `worlds.replication`, a server on an empty volume restores every world from its copies before it serves. Watch `twins_replication_restores_total` rise while it works.

From a volume snapshot, restore the snapshot with your CSI driver. Restore the database from its own backup.
{% endtab %}

{% tab title="AWS" %}
Put back to a time T (UTC, RFC 3339). The `restore` output gives the commands for the worlds.

1. **Stop the server:**

   ```bash
   aws ecs update-service --cluster <name> --service <name>-worlds --desired-count 0
   aws ecs wait services-stable --cluster <name> --services <name>-worlds
   ```
2. **Restore the database to T**, as a new instance under the old one's name, the old one kept aside ([point-in-time recovery](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PIT.html)):

   ```bash
   aws rds modify-db-instance --db-instance-identifier <name> --new-db-instance-identifier <name>-before-restore \
     --apply-immediately
   aws rds wait db-instance-available --db-instance-identifier <name>-before-restore
   aws rds restore-db-instance-to-point-in-time --source-db-instance-identifier <name>-before-restore \
     --target-db-instance-identifier <name> --restore-time <T> --db-subnet-group-name <name> \
     --vpc-security-group-ids <the stores' security group, <name>-stores> --no-publicly-accessible
   aws rds wait db-instance-available --db-instance-identifier <name>
   ```
3. **Point Terraform at the new instance**, without applying yet (an apply now would start the server):

   ```bash
   terraform state rm 'module.twins.aws_db_instance.twins'
   terraform import 'module.twins.aws_db_instance.twins' <name>
   ```
4. **Restore the worlds to T** with the restore task. It refuses to start while the server holds its lock. It puts every database back into a place of its own, checks each with `PRAGMA integrity_check`, and only then moves them in. It keeps what the root held under `/twins-restore/before-<run>` on EFS.

   * **From the copies:** run it as the `restore` output shows, with `RESTORE_TIMESTAMP=<T>`. A database that did not exist at T, or whose copies do not reach back to T, stops the restore. `RESTORE_ALLOW_MISSING=true` leaves those out. Without `RESTORE_TIMESTAMP`, it restores the copies' latest.
   * **From AWS Backup:** restore the file system's recovery point from before T to the same file system ([restoring EFS](https://docs.aws.amazon.com/aws-backup/latest/devguide/restoring-efs.html)). Then run the restore task with `RESTORE_FROM=backup`.

   `RESTORE_CHECK_ONLY=true` stops after the check, the root untouched.
5. **Start the server:** `terraform apply`. When only the worlds were put back, `aws ecs update-service ... --desired-count 1`. `/ready` answers 200 once the server holds its lock.
6. **Clean up** once the install is checked: delete `<name>-before-restore` (turn its deletion protection off first) and the root's old contents. AWS Backup's `aws-backup-restore_<time>` directory is billed as storage until removed.
   {% endtab %}

{% tab title="Google Cloud" %}
Postgres and the worlds' disk go back together, to one moment: a nightly Backup for GKE backup, and Cloud SQL recovered to the time it was taken. Point-in-time recovery reaches back 7 days. With `P` the project, `R` the region, `N` the module's `name` and `T` the backup's time:

1. Pick the backup and note its `completeTime` as `T`:

   ```bash
   gcloud beta container backup-restore backups list --project P --location R --backup-plan N-twins
   ```
2. Stop the twins server: `kubectl -n twins scale statefulset/twins-worlds --replicas=0`.
3. Recover Cloud SQL to `T`, as a new instance:

   ```bash
   gcloud sql instances clone N-pg N-pg-restored --point-in-time T --project P
   ```
4. Restore the worlds' disk with the restore plan, leaving the cluster's Secrets as they are. Write `filter.yaml`:

   ```yaml
   exclusionFilters:
   - groupKind:
       resourceGroup: ""
       resourceKind: Secret
   ```

   Then:

   ```bash
   gcloud beta container backup-restore restores create restore-1 --project P --location R \
     --restore-plan N-twins-restore \
     --backup projects/P/locations/R/backupPlans/N-twins/backups/<backup> --filter-file filter.yaml
   ```

   Wait for `gcloud beta container backup-restore restores describe restore-1 --project P --location R --restore-plan N-twins-restore` to say `SUCCEEDED`.
5. Hand the clone to the module. Take the old instance out of the state:

   ```bash
   terraform state rm module.gcp.google_sql_user.twins module.gcp.google_sql_database.twins \
     module.gcp.google_sql_database_instance.twins
   ```

   Import the clone in its place:

   ```hcl
   import {
     to = module.gcp.google_sql_database_instance.twins
     id = "projects/P/instances/N-pg-restored"
   }
   import {
     to = module.gcp.google_sql_database.twins
     id = "projects/P/instances/N-pg-restored/databases/twins"
   }
   import {
     to = module.gcp.google_sql_user.twins
     id = "P/N-pg-restored/twins"
   }
   ```

   Set `postgres_instance_name = "N-pg-restored"` on the module, then run `terraform apply`.
6. Start the twins server: `kubectl -n twins scale statefulset/twins-worlds --replicas=1`, then check `kubectl -n twins rollout status statefulset/twins-worlds`.
7. Once it is right, remove the import blocks and delete the old instance: `gcloud sql instances patch N-pg --no-deletion-protection --project P`, then `gcloud sql instances delete N-pg --project P`.
   {% endtab %}

{% tab title="Azure" %}
A restore puts both stores back to the same moment T, the time of the disk's restore point. Microsoft's guides: [Postgres point-in-time restore](https://learn.microsoft.com/azure/postgresql/backup-restore/concepts-backup-restore), [restore AKS from Azure Backup](https://learn.microsoft.com/azure/backup/azure-kubernetes-service-cluster-restore).

1. Pick the restore point: Azure portal, Resiliency, the backup instance `<name>-twins`, its restore points. T is its time, in UTC.
2. Stop the twins server: `kubectl -n twins scale statefulset/twins-worlds --replicas=0`.
3. Restore Postgres to T, as a new server in the same subnet and private DNS zone:

   ```bash
   az postgres flexible-server restore --resource-group <name>-rg --name <new-server> \
     --source-server <old-server> --restore-time <T> \
     --subnet <the database subnet's ID> --private-dns-zone <the Postgres private DNS zone's ID>
   ```
4. Move Terraform onto the new server:

   ```bash
   terraform state rm module.azure.azurerm_management_lock.database[0] \
     module.azure.azurerm_postgresql_flexible_server_configuration.tls \
     module.azure.azurerm_postgresql_flexible_server_database.twins \
     module.azure.azurerm_postgresql_flexible_server.twins
   terraform import module.azure.azurerm_postgresql_flexible_server.twins <new server's ID>
   terraform import module.azure.azurerm_postgresql_flexible_server_database.twins <new server's ID>/databases/twins
   ```

   Set `postgres_server_name = "<new-server>"` on the module, then run `terraform apply -target=module.azure`. The old server stays, locked, until you check the new one.
5. Remove the worlds' claim and volume, keeping the disk until the restore is checked:

   ```bash
   pv=$(kubectl -n twins get pvc worlds-twins-worlds-0 -o jsonpath='{.spec.volumeName}')
   kubectl patch pv "$pv" -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
   kubectl -n twins delete pvc worlds-twins-worlds-0
   kubectl delete pv "$pv"
   ```
6. Restore from Azure Backup for AKS: Resiliency, Recover, the backup instance, the restore point from step 1, the original cluster, namespace `twins`, persistent volumes included, conflicts left on Skip (the default).
7. Run `terraform apply`, then `kubectl -n twins scale statefulset/twins-worlds --replicas=1`. Check it serves, then delete the retained old disk from the node resource group.
   {% endtab %}
   {% endtabs %}

## Common problems

| What you see                            | Fix                                                                                                 |
| --------------------------------------- | --------------------------------------------------------------------------------------------------- |
| One world will not open after a restore | See `TwinsRestoreRefused` in [Monitoring](/twins/self-hosting/monitoring.md#alerts-and-what-to-do). |
| The newest EFS backup will not open     | Restore an older one, or replicate the worlds as they are written.                                  |

## Next

* [Monitoring](/twins/self-hosting/monitoring.md): alerts for the copies, the lease and the volume.
* [Upgrades](/twins/self-hosting/upgrades.md): the dump the server takes before it migrates.

***

*Last updated: 27 October 2026*