> For the complete documentation index, see [llms.txt](https://docs.drizz.dev/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.drizz.dev/twins/self-hosting/upgrades.md).

# Upgrades

Upgrade a self-hosted twins server, and roll back safely when a new version migrated the database.

Change the version and run the command you installed with. Roll back safely when a new version migrated the database.

|                |                                                                                                                |
| -------------- | -------------------------------------------------------------------------------------------------------------- |
| **Runs on**    | Docker Compose · Helm chart · AWS · Google Cloud · Azure                                                       |
| **Safety net** | The server dumps the database with `pg_dump` before it migrates, and keeps the newest five                     |
| **Downtime**   | The server is away while the new version migrates and takes up its worlds. There is no second copy, by design. |
| **Watch out**  | Running the old image over a migrated database is not a rollback. Put the database back first.                 |

## Prerequisites

* On a managed database, the provider's snapshot, taken before you upgrade. See [The automatic backup before migrating](#the-automatic-backup-before-migrating).
* An install from an earlier version: read [Upgrading from earlier install versions](#upgrading-from-earlier-install-versions) first.

## Steps

1. Take the managed database's snapshot, if you have one.
2. Change the version and run the command you installed with:

   | Install             | Change                                       | Then run                                                     |
   | ------------------- | -------------------------------------------- | ------------------------------------------------------------ |
   | Docker Compose      | `TWINS_IMAGE` in `deploy/.env`               | `docker compose -f deploy/docker-compose.selfhost.yml up -d` |
   | Helm chart          | `worlds.image.tag`                           | `helm upgrade`                                               |
   | AWS                 | `twins_image`                                | `terraform apply`                                            |
   | Google Cloud, Azure | `worlds.image.tag` in the release's `values` | `terraform apply`                                            |
3. The server dumps the database, then applies the new version's database migrations as it starts.

## The automatic backup before migrating

Before the server migrates a database that an earlier version migrated, it dumps it with `pg_dump`.

* The dump goes to `backups/` on the worlds volume (`TWINS_DATABASE_BACKUPS`, `/var/lib/twins/backups` by default). On AWS, that is `/twins/backups/` on the worlds' file system.
* A `.json` file beside each dump names the versions and the `pg_restore` command.
* The server keeps the newest five.
* A dump it cannot take stops the upgrade. The log says why.

{% hint style="warning" %}
Setting `TWINS_DATABASE_BACKUPS` empty skips the dump. Do that only after you take a snapshot of your own.
{% endhint %}

On a managed database, take the provider's snapshot before upgrading in any case. It outlives the worlds volume.

{% tabs %}
{% tab title="AWS" %}

```bash
aws rds create-db-snapshot --db-instance-identifier <name> --db-snapshot-identifier <name>-before-<version>
```

{% endtab %}

{% tab title="Google Cloud" %}

```bash
gcloud sql backups create
```

{% endtab %}

{% tab title="Azure" %}

```bash
az postgres flexible-server backup create
```

{% endtab %}
{% endtabs %}

## What happens during the upgrade

{% tabs %}
{% tab title="Helm chart" %}
`helm upgrade` replaces the twins server in place:

1. The StatefulSet stops the old copy, then starts the new one.
2. The new copy applies the database's migrations before its listener opens. It has `worlds.startupSeconds` for this (five minutes by default), the startup probe's budget on `/health`.
3. It takes the worlds' lock and takes up its worlds, then reports ready on `/ready`. This part has no budget: a long restore is waited for, never killed.

The server is away for that time. There is no second copy to take its traffic, by design, and so no PodDisruptionBudget either: a budget for one copy would block every node drain. The console has one and rolls without a gap.

If the new version never becomes ready, or a node dies with the server on it, see [Common problems](#common-problems).
{% endtab %}

{% tab title="AWS" %}
Set `twins_image` to the new version and apply. ECS deploys the server, which applies the new version's migrations as it starts. The new task waits for the old one's lock before it serves. See [Deployments and the one writer](/twins/self-hosting/clouds.md).
{% endtab %}
{% endtabs %}

## Rolling back

A version refuses a database that a newer version migrated, and says so: "the directory was migrated by a newer build". Running the old image over the new tables is not a rollback.

To go back after an upgrade that migrated, put the database back as it was, then start the old version. The worlds keep what was written since. People, keys and audits go back to the dump.

{% hint style="warning" %}
A version from before this check does not refuse a newer database. Restore before starting it all the same.
{% endhint %}

{% tabs %}
{% tab title="Docker Compose" %}
With the server stopped:

```bash
docker compose -f deploy/docker-compose.selfhost.yml cp worlds:/var/lib/twins/backups/<file>.dump before.dump
docker compose -f deploy/docker-compose.selfhost.yml stop worlds
docker compose -f deploy/docker-compose.selfhost.yml exec -T postgres \
  pg_restore --clean --if-exists --no-owner -U twins -d twins < before.dump
TWINS_IMAGE=<the old image> docker compose -f deploy/docker-compose.selfhost.yml up -d worlds
```

{% endtab %}

{% tab title="Helm chart" %}
With the chart's own Postgres:

```bash
kubectl -n twins cp twins-worlds-0:/var/lib/twins/backups/<file>.dump before.dump
kubectl -n twins scale statefulset twins-worlds --replicas=0
kubectl -n twins exec -i twins-postgres-0 -- pg_restore --clean --if-exists --no-owner -U twins -d twins < before.dump
helm rollback twins
```

With an external database, restore its snapshot from before the upgrade, or `pg_restore` the dump into it.
{% endtab %}

{% tab title="AWS" %}
Put the records back from the RDS snapshot you took before the upgrade ([restoring from a DB snapshot](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_RestoreFromSnapshot.html)), or from the dump. Then set `twins_image` back and apply. The worlds keep what was written since.
{% endtab %}

{% tab title="Managed database" %}
Restore the snapshot you took before the upgrade, as the provider does it, then deploy the old version. Or `pg_restore` the dump into it from any machine that reaches it.
{% endtab %}
{% endtabs %}

## Common problems

| What you see                                                 | Fix                                                                                                                                                                                                                                                                                                                                     |
| ------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| "the directory was migrated by a newer build"                | Run the newer version, or put the database back from the dump. See [Rolling back](#rolling-back).                                                                                                                                                                                                                                       |
| The upgrade stops before it migrates: the dump failed        | The log says why. Fix that, or take a snapshot yourself and set `TWINS_DATABASE_BACKUPS` empty.                                                                                                                                                                                                                                         |
| Helm: the new version never becomes ready                    | Roll back, then delete the stuck pod: `helm rollback twins`, then `kubectl -n twins delete pod twins-worlds-0`. A StatefulSet does not replace a pod that never became ready on its own, even after a rollback ([Kubernetes: forced rollback](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#forced-rollback)). |
| Helm: a node died with the server on it                      | Kubernetes waits for the node before it starts the copy elsewhere, so two never write the same volume. Once the node is gone for good, `kubectl delete node <name>` lets it go on.                                                                                                                                                      |
| Helm: the upgrade names a StatefulSet field it cannot change | See "Helm chart: from chart 0.2" below.                                                                                                                                                                                                                                                                                                 |

## Upgrading from earlier install versions

<details>

<summary>Docker Compose: from a version that kept the password in deploy/.env</summary>

Earlier versions kept the database password in `deploy/.env`. Write that same password to the secrets file, because Postgres keeps the one it was first given. Then drop the stopped Redis:

```bash
grep '^TWINS_POSTGRES_PASSWORD=' deploy/.env | cut -d= -f2- | tr -d '\n' > deploy/secrets/postgres_password
docker compose -f deploy/docker-compose.selfhost.yml up -d --remove-orphans
```

Make `deploy/secrets` and the cache password first, as in [Docker Compose](/twins/self-hosting/docker-compose.md).

</details>

<details>

<summary>Helm chart: from chart 0.2</summary>

Chart 0.3 gives each StatefulSet a headless service of its own and the worlds' volume `ReadWriteOncePod`. A StatefulSet cannot change those fields, and the upgrade says so by name.

Delete the two StatefulSets and leave their pods and volumes running, then upgrade, which takes them over. The volumes already made keep their access mode.

```bash
kubectl -n twins delete statefulset twins-worlds twins-postgres --cascade=orphan
helm upgrade twins deploy/charts/twins -n twins --reuse-values
```

The bundled Redis becomes Valkey under the name `cache`, with the same generated password. It holds nothing to carry over.

</details>

<details>

<summary>AWS module: from an earlier version</summary>

Two inputs now ask for a decision. A plan fails until you give them:

* `allowed_cidrs` has no default (it was `["0.0.0.0/0"]`). Name the ranges that reach the load balancer, or `["0.0.0.0/0"]` to keep it open to anyone.
* Without `certificate_arn`, `allow_insecure_http = true` is needed to keep serving plain HTTP.

The first apply changes the running install without touching its data. It deploys the server once.

From the version without the worlds' copies, the first apply also needs Terraform 1.11 and the AWS provider 6.62 (`terraform init -upgrade`). It turns on `replication`, draws a new database password and a new cache token, and keeps Postgres backups 35 days rather than 7. Between the new password and the server's deployment in the same apply, the server cannot open new connections.

Postgres 18 is available on RDS. The module stays on 17 by default. Moving to 18 is a major version upgrade: set `db_engine_version = "18"` with `db_allow_major_version_upgrade = true`, after a snapshot.

`deploy/terraform/aws/README.md` lists every change.

</details>

<details>

<summary>Google Cloud and Azure modules: from an earlier version</summary>

Read the plan before you apply. Both modules now need Terraform 1.11 or later, and move the database password out of the state into Secret Manager or Key Vault. Pass `database_password` and `database_password_version` to the release in the same change, and apply the module and the release together.

Some changes replace the cluster, the node pool or the database server. Read each module's Upgrading section first: `deploy/terraform/gcp/README.md` and `deploy/terraform/azure/README.md`.

</details>

## Next

* [Backups](/twins/self-hosting/backups.md): what to keep, and how to put it back.
* [Monitoring](/twins/self-hosting/monitoring.md): what to do when an alert fires.

***

*Last updated: 27 October 2026*