> For the complete documentation index, see [llms.txt](https://docs.drizz.dev/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.drizz.dev/twins/self-hosting/kubernetes.md). # Kubernetes (Helm) Install twins on any Kubernetes cluster with the Helm chart, from a trial to production. `deploy/charts/twins` installs twins on any cluster, from a trial to production. It runs the same services as the [Docker Compose](/twins/self-hosting/docker-compose.md) install. {% hint style="info" %} **Getting the install files.** The Docker Compose file, Helm chart, Terraform modules and twins images are shared with you by Drizz. Ask your Drizz contact for access before you start. {% endhint %} | | | | ------------- | --------------------------------------------------------------------------------------------------------------- | | **Runs on** | EKS, AKS, GKE Standard and Autopilot, OpenShift, EKS Anywhere, kind, k3s, EKS on Fargate, a cluster of your own | | **You need** | Helm, `kubectl`, and a storage driver that takes `ReadWriteOncePod` | | **Default** | Bundled Postgres and Valkey: enough to try. Managed stores and a gateway or ingress with TLS for production. | | **Watch out** | Set `networkPolicy.allowFrom`. A pod that can reach the server directly can name any address. | ## Prerequisites * A Kubernetes cluster, Helm and `kubectl`. * A CSI driver (the block storage of EKS, GKE and AKS have one) or a provisioner that takes `ReadWriteOncePod`, as the local-path provisioner of kind and k3s does. See [The worlds' volume](#the-worlds-volume). * A network plugin that enforces NetworkPolicies (Calico, Cilium, GKE Dataplane V2, Azure). ## Copy this A trial install with the bundled stores: ```bash helm install twins deploy/charts/twins -n twins --create-namespace kubectl -n twins exec statefulset/twins-worlds -- twins enroll --email you@acme.com kubectl -n twins exec statefulset/twins-worlds -- twins issue --email you@acme.com helm test twins -n twins ``` ## Steps {% stepper %} {% step %} ### Install the chart ```bash helm install twins deploy/charts/twins -n twins --create-namespace ``` This runs the twins server with a bundled Postgres and Valkey cache. That is enough to try twins. For production, see [Production values](#production-values). {% endstep %} {% step %} ### Make the first key Sign-in is off by default, so you mint every key on the server. Enroll an address, then issue a key for it: ```bash kubectl -n twins exec statefulset/twins-worlds -- twins enroll --email you@acme.com kubectl -n twins exec statefulset/twins-worlds -- twins issue --email you@acme.com ``` `issue` prints the key once. Copy it now. The first address enrolled from a company domain owns that organization. See [Keys and limits](/twins/reference/keys-and-limits.md). {% endstep %} {% step %} ### Check it ```bash helm test twins -n twins ``` `helm test` uses the twins image itself. {% endstep %} {% endstepper %} ## What runs | Component | Kind | Copies | Why | | ---------- | ---------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `worlds` | StatefulSet and volume | 1 | The server holds the right to write its worlds alone. A StatefulSet stops the old copy before it starts the new one. Kubernetes restarts it where it fails. | | `postgres` | StatefulSet and volume | 1 | Bundled to try twins. Turn it off (`database.deploy: false`) for a managed Postgres. | | `cache` | Deployment | 1 | Valkey (BSD-3), bundled. Turn it off (`redis.deploy: false`) for a managed cache that speaks the Redis protocol (ElastiCache, Memorystore, Azure Cache). Keeps nothing on disk. | | `console` | Deployment and PodDisruptionBudget | 2 | Optional. A person's session is a sealed cookie, so any copy serves it. | Every pod runs as its image's non-root user with a read-only root filesystem, every capability dropped and the runtime's default seccomp profile. That is the Kubernetes "restricted" pod security standard. No pod mounts a Kubernetes API token. The twins server has a service account of its own (`serviceAccount`) for a cloud identity to be bound to. ## Production values For production, use a managed Postgres and cache, put a gateway or ingress with TLS in front, and turn the console on. ```yaml worlds: extraEnvFrom: [{secretRef: {name: twins-model}}] # OPENAI_API_KEY, for worlds founded from a written brief database: deploy: false external: dsnSecret: {name: twins-db, key: dsn} # postgresql://twins@db.acme.internal:5432/twins?sslmode=require passwordSecret: {name: twins-db-password, key: password} redis: deploy: false external: urlSecret: {name: twins-cache, key: url} # rediss://cache.acme.internal:6379/0 passwordSecret: {name: twins-cache-password, key: password} console: enabled: true gateway: # Gateway API; ingress-nginx is retired enabled: true parentRefs: [{name: public, namespace: gateway-system, sectionName: https}] hosts: {worlds: twins.acme.com, console: console.twins.acme.com} networkPolicy: allowFrom: [{namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: gateway-system}}}] ``` ### Gateway or ingress {% tabs %} {% tab title="Gateway API" %} The routes are HTTPRoutes on a Gateway the cluster runs (Envoy Gateway, Istio, GKE's, AKS's application routing). The Gateway holds the certificate. A Gateway's listener takes routes from its own namespace alone unless it says otherwise: `allowedRoutes.namespaces.from` is `Same` by default ([Gateway API: cross-namespace routing](https://gateway-api.sigs.k8s.io/guides/user-guides/multiple-ns/)). With the Gateway in another namespace, its listener must name twins' namespace (`from: Selector`) or take every namespace (`from: All`). Then check that the routes were taken: ```bash kubectl -n twins get httproute -o jsonpath='{range .items[*]}{.metadata.name}: {.status.parents[*].conditions[?(@.type=="Accepted")].status}{"\n"}{end}' ``` An agent's MCP stream stays open longer than many gateways wait for an answer. `gateway.timeouts` sets the route's own ([Gateway API: HTTP timeouts](https://gateway-api.sigs.k8s.io/guides/user-guides/http-timeouts/)), for example `{request: 1h, backendRequest: 1h}`, or `"0s"` for none. This is an extended feature, so check that the route stays `Accepted`. {% endtab %} {% tab title="Ingress" %} ingress-nginx, the usual Ingress controller, is retired ([Kubernetes, November 2025](https://kubernetes.io/blog/2025/11/11/ingress-nginx-retirement/)). `ingress` still serves an Ingress for a controller that is maintained. ```yaml ingress: enabled: true className: annotations: {} # e.g. cert-manager.io/cluster-issuer: letsencrypt hosts: {worlds: twins.acme.com, console: console.twins.acme.com} tls: enabled: true secretName: "" # one certificate for every host; empty: -tls ``` {% endtab %} {% endtabs %} `worlds.base` is the address every MCP and API URL a world hands out is written against. Left empty, it is `https://` when the ingress is on, else the service's in-cluster address, which only the cluster reaches. ### Secrets The chart never takes a secret written into values. You name a Kubernetes secret instead. Keep each store's password in a secret of its own (`passwordSecret`), apart from its address. The server reads it from a mounted file on every new connection, so a rotated password reaches it without a restart, and the address shows none. A password inside the address still works. The chart generates some secrets: the bundled stores' passwords and the console's cookie secret. It makes each one once, keeps it on every upgrade, and leaves it behind on uninstall with the volumes it unlocks. {% hint style="warning" %} The chart keeps a generated secret by reading the one in the cluster. `helm template` and GitOps tools (Argo CD, Flux) cannot do that, so they make a new one on every render. With those tools, name your own: `database.bundled.existingSecret`, `redis.bundled.existingSecret` and `console.sessionSecret.name`. {% endhint %} ### Certificates of managed stores The chart checks a managed store's certificate where you give the CA: * `database.external.caSecret`: a CA such as Cloud SQL's server CA, with `sslmode=verify-ca`. * `database.external.systemCA`: a certificate from a public CA, such as Azure's, with `sslmode=verify-full`. * `redis.external.caSecret`: a CA such as Memorystore's, named in the address with `ssl_ca_certs`. The [cloud modules](/twins/self-hosting/clouds.md) pass these for you. ### Images The chart pins the images it chooses (the bundled Postgres and Valkey) by digest as well as tag. Set `image.digest` on the twins and console images to pin those too. `values.schema.json` refuses a value of the wrong type, outside its choices or under a key the chart does not have (a misspelt one) before anything is installed. ## The worlds' volume `worlds.persistence.accessModes` is `[ReadWriteOncePod]`. One pod in the whole cluster mounts the volume, where `ReadWriteOnce` allows every pod on one node ([Kubernetes: access modes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes)). It needs a CSI driver (the block storage of EKS, GKE and AKS have one) or a provisioner that takes it, as the local-path provisioner of kind and k3s does. Elsewhere the claim stays `Pending` with an event saying why. Then set `worlds.persistence.accessModes: [ReadWriteOnce]`. The server's own lock on its worlds still keeps a second copy from writing. The claim outlives an uninstall (`persistentVolumeClaimRetentionPolicy: Retain`). `worlds.persistence.size` sets its size (default `10Gi`). ## Network policy NetworkPolicies are on (`networkPolicy.enabled`). They let the bundled Postgres and cache take connections from the twins server alone. `networkPolicy.allowFrom` names who reaches the server and the console: the ingress controller's or gateway's namespace. Empty leaves them open to the whole cluster. {% hint style="warning" %} Set `networkPolicy.allowFrom`. The server takes a caller's address from the forwarded headers of `worlds.trustedProxies` (every private range, by default) for the request allowance. A pod that can reach the server directly can name any address. {% endhint %} The policies need a network plugin that enforces them (Calico, Cilium, GKE Dataplane V2, Azure). Where a policy closes the server, name your metrics scraper in `worlds.metrics.scrapers`. See [Monitoring](/twins/self-hosting/monitoring.md). ## Replicate worlds to S3 `worlds.replication` streams every world to an S3 bucket with Litestream as it is written. A server on an empty volume restores them before it serves. ```yaml worlds: replication: {bucket: acme-twins-worlds, region: eu-west-1, prefix: prod} serviceAccount: annotations: {eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/twins-worlds} # s3 on that bucket ``` The credentials come from the pod's identity (`serviceAccount.annotations`) or `worlds.extraEnvFrom`. A store that speaks S3 with its conditional writes, such as MinIO or Cloudflare R2, is named by `worlds.replication.endpoint`. Its bucket is then addressed by path. {% hint style="info" %} A lease keeps a second server from writing the same copies. The lease is a conditional write, so a store without conditional writes cannot hold it. Google Cloud Storage's S3 interface takes no conditions on a write. {% endhint %} Make sure exactly one install writes each bucket and prefix. See [Backups](/twins/self-hosting/backups.md). ## Special clusters
EKS on Fargate A pod on Fargate can mount only a statically provisioned EFS volume: no EBS, no dynamic provisioning ([EKS: Fargate storage](https://docs.aws.amazon.com/eks/latest/userguide/fargate-pod-configuration.html)). Make the volume and its claim first, then point the twins server at the claim. Use managed stores (RDS, ElastiCache), as the bundled Postgres needs a block volume. ```yaml apiVersion: v1 kind: PersistentVolume metadata: {name: twins-efs} spec: capacity: {storage: 10Gi} accessModes: [ReadWriteMany] persistentVolumeReclaimPolicy: Retain storageClassName: efs-static csi: {driver: efs.csi.aws.com, volumeHandle: "fs-0123456789abcdef0::fsap-0123456789abcdef0"} # an access point owned by uid 999 --- apiVersion: v1 kind: PersistentVolumeClaim metadata: {name: twins-efs, namespace: twins} spec: accessModes: [ReadWriteMany] storageClassName: efs-static volumeName: twins-efs resources: {requests: {storage: 10Gi}} ``` ```bash helm install twins deploy/charts/twins -n twins \ --set worlds.persistence.existingClaim=twins-efs \ --set database.deploy=false,database.external.dsnSecret.name=twins-db \ --set redis.deploy=false,redis.external.urlSecret.name=twins-cache ``` This setup was checked on kind with a statically provisioned volume and claim: the server took the claim and the worlds survived the pod's replacement. Fargate itself was not checked.
OpenShift OpenShift runs every pod as a user ID its project assigns, in the root group. Its `restricted-v2` SCC refuses a pod that names its own. The chart detects OpenShift (a cluster serving `security.openshift.io/v1`) and leaves the user, group and fsGroup out there. `global.compatibility.openshift.adaptSecurityContext` is `auto` (the default), `force` or `disabled`. Every image the chart runs works under an arbitrary user ID in the root group.
## What the chart refuses The chart refuses, by name, an install that cannot serve: * The bundled database or cache off, with no secret naming the managed one. * Sessions on with no identity provider, or sessions off with one named. See [Your own sign-in](/twins/self-hosting/your-own-sign-in.md). * The gateway on with no Gateway named. * The ingress and the gateway both on. * Replication with no region. * An upgrade that would change a StatefulSet's fixed fields. See [Upgrades](/twins/self-hosting/upgrades.md). ## Common problems | What you see | Fix | | ------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- | | The worlds' claim stays `Pending`, with an event saying why | Set `worlds.persistence.accessModes: [ReadWriteOnce]`. The server's own lock still keeps a second copy from writing. | | An HTTPRoute is not `Accepted` | With the Gateway in another namespace, its listener must name twins' namespace (`from: Selector`) or take every namespace (`from: All`). | | An agent's MCP stream stays open longer than the gateway waits | Set `gateway.timeouts`, for example `{request: 1h, backendRequest: 1h}`, or `"0s"` for none. | | `helm template`, Argo CD or Flux make a new secret on every render | Name your own: `database.bundled.existingSecret`, `redis.bundled.existingSecret` and `console.sessionSecret.name`. | | The metrics scrape fails and `TwinsDown` fires | Name your scraper in `worlds.metrics.scrapers`. See [Monitoring](/twins/self-hosting/monitoring.md). | ## Next * [Upgrades](/twins/self-hosting/upgrades.md): `helm upgrade`, and rolling back. * [Backups](/twins/self-hosting/backups.md): the volume and the database. * [Monitoring](/twins/self-hosting/monitoring.md): the ServiceMonitor and twins' alerts. *** *Last updated: 27 October 2026*