Skip to content
Build with XMTP

Deploy on Kubernetes

Create the chart files below in a directory outside the repository. Envoy Gateway terminates TLS and sends h2c to the backend Service. Read the deployment overview for ports, connection budgets, health, shutdown, the ingress contract, and security requirements.

Image requirement: use an image that contains the configuration flag change and rate limits. Every :self-hosted image published after 2026-09-14 has the configuration flag change; an older image exits at startup. The values below set [rate_limits], which an image without rate limits refuses at startup; with an image that has the flag change but not rate limits, remove that table from config. Pin an exact build with ghcr.io/xmtp/backend:sha-<commit>.

Prerequisites

Section titled “Prerequisites”
  • Kubernetes 1.34 or later. Native gRPC probes are stable since 1.27. The preStop.sleep action is stable in 1.34 and enabled by default since 1.30. This guide uses the stable versions of both features.
  • Envoy Gateway v1.7.0, with its Gateway API v1.4.1 CRDs. This is the selected controller version. Its policy schemas are part of the local checks. Use a cluster that can provision a LoadBalancer Service for Envoy.
  • Helm 4.2.4 and kubeconform 0.8.0 for the commands below, plus kubectl.
  • An external database that meets the database requirements. The chart does not install PostgreSQL. Create an existing Secret named xmtp-database in namespace xmtp, with the database URL in key url. Use your secret manager or secure input from a file. Do not put the URL in Helm values, TOML, or command arguments.
  • A DNS name and certificate. Obtain a trusted certificate and private key for xmtp.example.com. Keep the private key outside version control.
  • Access controls that meet the security requirements. The minimal TOML below has no authentication. Restrict the Gateway load balancer to trusted clients before enabling the route, or configure [auth].
  • Optional controllers: the ServiceMonitor needs Prometheus Operator v0.89.0 CRDs and a Prometheus instance that selects its labels. CPU autoscaling needs Metrics Server. Neither is installed by this chart. Install the CRDs before the chart. With serviceMonitor.enabled and no CRD, the install fails with no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1".

Gateway API’s h2c backend protocol is conformance-tested across Envoy Gateway, Cilium, Istio, GKE, NGINX Gateway Fabric, and Traefik. The latter five are UNTESTED alternatives for this guide. Conformance results do not verify this backend deployment. See the Gateway API conformance reports.

Create the chart

Section titled “Create the chart”

Create xmtp-chart/templates. Save each fence at the path in its title. The default value set creates only a Deployment, ConfigMap, and private Service. Replace the image tag before installation. sha-REPLACE_WITH_FULL_COMMIT is a placeholder, not a published image.

xmtp-chart/Chart.yaml
apiVersion: v2
name: xmtp-backend
description: XMTP backend with an external database and optional Envoy Gateway route
type: application
version: 0.1.0
kubeVersion: ">=1.34.0-0"
xmtp-chart/values.yaml
image:
  repository: ghcr.io/xmtp/backend
  tag: sha-REPLACE_WITH_FULL_COMMIT
replicas: 1
databaseSecret:
  name: xmtp-database
  key: url
config: |
  [server]
  # Change this to a reverse-DNS name you own. Never change it again once
  # clients have connected: every client database is bound to it.
  identifier = "org.example.xmtp"


  [database]
  url = "env:XMTP_DATABASE_URL"


  # Envoy connects from the pod network and appends each client's address
  # to X-Forwarded-For. Replace with your cluster's pod CIDR. The limiter
  # takes max_memory_mib (100 MiB by default) at startup.
  [rate_limits]
  trusted_proxies = ["10.244.0.0/16"]
resources:
  requests:
    cpu: 250m
    memory: 256Mi
  limits:
    memory: 1Gi
gateway:
  enabled: false
  hostname: xmtp.example.com
  certificateSecret: xmtp-tls
serviceMonitor:
  enabled: false
  labels: {}
autoscaling:
  enabled: false
  minReplicas: 2
  maxReplicas: 4
  targetCPUUtilizationPercentage: 70
podDisruptionBudget:
  enabled: false
  minAvailable: 1

The ConfigMap holds no secrets. The checksum changes the pod template when the configuration changes. The process mounts a file, so its arguments must use --config-file. --config accepts inline TOML and is not correct here.

xmtp-chart/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-config
data:
  config.toml: |
{{ .Values.config | indent 4 }}
xmtp-chart/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}
spec:
  {{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicas }}
  {{- end }}
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  selector:
    matchLabels:
      app.kubernetes.io/instance: {{ .Release.Name }}
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: {{ .Release.Name }}
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum | quote }}
    spec:
      automountServiceAccountToken: false
      terminationGracePeriodSeconds: 45
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: backend
          image: {{ printf "%s:%s" .Values.image.repository .Values.image.tag | quote }}
          args: ["--config-file", "/etc/xmtp/config.toml"]
          env:
            - name: XMTP_DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: {{ .Values.databaseSecret.name | quote }}
                  key: {{ .Values.databaseSecret.key | quote }}
          ports:
            - name: grpc
              containerPort: 5050
            - name: metrics
              containerPort: 9464
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          resources:
{{ toYaml .Values.resources | indent 12 }}
          startupProbe:
            grpc:
              port: 5050
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 120
          readinessProbe:
            grpc:
              port: 5050
            periodSeconds: 5
            timeoutSeconds: 2
          livenessProbe:
            grpc:
              port: 5050
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3
          lifecycle:
            preStop:
              sleep:
                seconds: 15
          volumeMounts:
            - name: config
              mountPath: /etc/xmtp
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: {{ .Release.Name }}-config

startupProbe allows 120 failures at 5 s intervals: about 600 s for startup. Increase this budget if migrations need more time. In apps/backend/src/main.rs, run awaits server::initialize(config), which takes the validated configuration, before TcpListener::bind. Initialization finishes primary database migrations before it returns. The RPC port cannot pass a probe until that work finishes. Kubernetes delays readiness and liveness probes until the startup probe succeeds.

The image has no shell. Use native grpc: probes and preStop.sleep, with no exec command. The 15 s sleep gives endpoint changes time to reach the Gateway before SIGTERM. This delay needs a real rollout test; it is not a guarantee.

The sleep runs inside the termination grace period. The source defaults are DEFAULT_DRAIN_DURATION_MS = 10_000 in crates/xmtp_backend_config/src/lib.rs and OTLP_FLUSH_TIMEOUT = 5 s in crates/xmtp_logging/src/telemetry.rs. The chart therefore uses 45 s = 15 s sleep + 10 s drain + 5 s flush + 15 s margin. A 30 s grace period would leave no margin. Increase it if you increase the sleep or the configured drain. See shutdown for client reconnect behavior.

xmtp-chart/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: {{ .Release.Name }}
  labels:
    app.kubernetes.io/instance: {{ .Release.Name }}
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/instance: {{ .Release.Name }}
  ports:
    - name: grpc
      port: 5050
      targetPort: grpc
      appProtocol: kubernetes.io/h2c
    - name: metrics
      port: 9464
      targetPort: metrics

Gateway and TLS

Section titled “Gateway and TLS”

Use an HTTPRoute for the whole hostname. GRPCRoute does not cover HTTP/1.1 gRPC-Web or its CORS preflight. An HTTPRoute and a GRPCRoute with the same hostname conflict under the Gateway API rules. Do not add a GRPCRoute beside this route. appProtocol: kubernetes.io/h2c selects HTTP/2 to the backend for both client transports.

The GatewayClass is cluster-scoped. Its name includes the release namespace and name so separate releases do not claim the same class.

xmtp-chart/templates/gateway.yaml
{{- if .Values.gateway.enabled }}
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: {{ .Release.Namespace }}-{{ .Release.Name }}
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: {{ .Release.Name }}
spec:
  gatewayClassName: {{ .Release.Namespace }}-{{ .Release.Name }}
  listeners:
    - name: https
      hostname: {{ .Values.gateway.hostname | quote }}
      port: 443
      protocol: HTTPS
      tls:
        mode: Terminate
        certificateRefs:
          - group: ""
            kind: Secret
            name: {{ .Values.gateway.certificateSecret | quote }}
      allowedRoutes:
        namespaces:
          from: Same
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: {{ .Release.Name }}
spec:
  parentRefs:
    - name: {{ .Release.Name }}
      sectionName: https
  hostnames:
    - {{ .Values.gateway.hostname | quote }}
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      timeouts:
        request: 0s
        backendRequest: 0s
      backendRefs:
        - name: {{ .Release.Name }}
          port: 5050
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: {{ .Release.Name }}
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: {{ .Release.Name }}
  timeout:
    http:
      requestTimeout: 0s
      maxStreamDuration: 0s
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
  name: {{ .Release.Name }}
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: {{ .Release.Name }}
  tls:
    alpnProtocols: ["h2", "http/1.1"]
  timeout:
    http:
      requestReceivedTimeout: 0s
      streamIdleTimeout: 0s
{{- end }}

These are Envoy Gateway v1.7.0 timeout fields. requestReceivedTimeout removes the client request read deadline. streamIdleTimeout removes the stream inactivity deadline in either direction, including response sends. The route timeouts and backend policy remove the upstream response deadline and maximum stream duration. Here 0s disables each limit. This permits long subscriptions, including quiet periods. Apply the shared security controls because idle streams can retain resources. No body buffering, gRPC conversion, CORS rewrite, or retry policy is added. All ingress checks still need a client test.

Client addresses

Section titled “Client addresses”

Backend images with rate limits limit each caller, and charge a caller without a credential to its address. The backend’s peer is an Envoy pod, so it finds the client in X-Forwarded-For, which Envoy Gateway appends the downstream address to: the v1.7 client traffic task shows a request sent with X-Forwarded-For: 1.1.1.1,2.2.2.2 arriving with 1.1.1.1,2.2.2.2,10.244.0.9. The backend reads the list from the right and takes the first address outside trusted_proxies, so a value a client wrote itself is never chosen.

Set trusted_proxies to your cluster’s pod CIDR, for example from the --cluster-cidr of your control plane or your CNI’s configuration. 10.244.0.0/16 is only the common default. If the list misses the Envoy pods, the backend ignores the header, and every caller without a credential shares one allowance. Any pod in a listed network can choose the address it is charged to, so if the cluster runs workloads you do not trust, list a narrower network that holds only the Envoy pods.

Envoy sees the client’s own address only if nothing in front of it replaces the source address. Envoy Gateway’s Envoy Service uses externalTrafficPolicy: Local by default (API reference), so the cluster does not replace the client’s source address. Kubernetes notes that this keeps the client’s address only behind a cloud load balancer that forwards packets; one that terminates the connection replaces it with its own, and every caller without a credential then shares one allowance. If that load balancer’s address is inside a network in trusted_proxies, such as the pod CIDR, the backend skips it as well and takes an address the client wrote into X-Forwarded-For, so the client chooses the address it is charged to. Check your provider’s load balancer, and keep its addresses out of trusted_proxies. This guide has not verified the client address on a live cluster.

The limiter allocates rate_limits.max_memory_mib, 100 MiB by default, at startup: the image measured about 115 MiB resident at startup with the defaults. The chart’s 256Mi memory request and 1Gi limit leave room; keep the request above max_memory_mib plus your working set.

Create the certificate Secret separately from Helm. The required manifest has this shape. Replace both placeholders with PEM data from your certificate provider before applying it. tls.crt must contain the certificate chain. Do not commit the populated file or send it to Helm as a value.

certificate.yaml
apiVersion: v1
kind: Secret
metadata:
  name: xmtp-tls
  namespace: xmtp
type: kubernetes.io/tls
stringData:
  tls.crt: |
    REPLACE_WITH_PEM_CERTIFICATE_CHAIN
  tls.key: |
    REPLACE_WITH_PEM_PRIVATE_KEY

Optional monitoring and scaling

Section titled “Optional monitoring and scaling”
xmtp-chart/templates/optional.yaml
{{- if .Values.serviceMonitor.enabled }}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: {{ .Release.Name }}
  labels:
{{ toYaml .Values.serviceMonitor.labels | indent 4 }}
spec:
  selector:
    matchLabels:
      app.kubernetes.io/instance: {{ .Release.Name }}
  endpoints:
    - port: metrics
      path: /metrics
      interval: 30s
{{- end }}
{{- if .Values.autoscaling.enabled }}
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: {{ .Release.Name }}
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ .Release.Name }}
  minReplicas: {{ .Values.autoscaling.minReplicas }}
  maxReplicas: {{ .Values.autoscaling.maxReplicas }}
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
{{- end }}
{{- if .Values.podDisruptionBudget.enabled }}
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: {{ .Release.Name }}
spec:
  minAvailable: {{ .Values.podDisruptionBudget.minAvailable }}
  selector:
    matchLabels:
      app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}

Save this second value set beside the chart. Match serviceMonitor.labels to your Prometheus selector. The monitor selects Services in its own namespace. The Gateway exposes only the application port. Apply network policy or cluster network controls to keep the metrics port limited to monitoring clients.

enabled.yaml
gateway:
  enabled: true
serviceMonitor:
  enabled: true
  labels:
    release: prometheus
autoscaling:
  enabled: true
podDisruptionBudget:
  enabled: true

The CPU request enables utilization-based scaling. CPU is only a starting signal; validate capacity with subscription load. Size the database for maxReplicas plus the rolling update surge, using the shared connection budget. Terminating pods can keep connections during the grace period, so reserve capacity for them too. The PDB limits voluntary evictions. It does not control Deployment rollouts or prevent node failures. With one replica, minAvailable: 1 blocks an eviction. Use at least two replicas when you enable it.

Install

Section titled “Install”

Set an immutable image tag in values.yaml, and prepare the database Secret and access controls, before you run these commands.

Install the pinned controller, which includes its Gateway API and Envoy policy CRDs:

Terminal window
helm install eg oci://docker.io/envoyproxy/gateway-helm \
  --version v1.7.0 --namespace envoy-gateway-system --create-namespace \
  --wait=watcher --timeout 10m
kubectl create namespace xmtp
kubectl apply -f certificate.yaml

Use a certificate renewal process that updates xmtp-tls before expiry. For an existing namespace, omit kubectl create namespace. Create the database Secret there before the next command. Install Prometheus Operator and Metrics Server before using enabled.yaml; otherwise enable only the Gateway with --set gateway.enabled=true in place of -f enabled.yaml.

Terminal window
helm upgrade --install xmtp ./xmtp-chart --namespace xmtp \
  -f enabled.yaml --rollback-on-failure --wait=watcher --timeout 15m
kubectl -n xmtp rollout status deployment/xmtp --timeout=15m
kubectl -n xmtp get gateway xmtp
kubectl -n xmtp get httproute xmtp -o yaml
kubectl -n xmtp get backendtrafficpolicy,clienttrafficpolicy

These commands use Helm 4 spellings: --rollback-on-failure replaces --atomic, and --wait=watcher names the wait strategy. The 15-minute Helm budget exceeds the startup probe budget. Check Gateway Accepted and Programmed, route Accepted and ResolvedRefs, and policy Accepted conditions. Helm readiness alone does not prove that the route works.

Point the DNS name at the Gateway address. Use https://xmtp.example.com as the SDK backend URL. For a configuration edit, change values.yaml and repeat the upgrade command. The checksum causes a rollout. An external Secret change does not change the checksum; restart the Deployment to refresh its environment. Check migration limits before any image upgrade.

Local manifest checks

Section titled “Local manifest checks”

Run these checks from the directory that contains xmtp-chart and enabled.yaml. They need no cluster. Use Python 3 with PyYAML to create local JSON schemas from the pinned upstream CRDs:

prepare-schemas.py
import json
from pathlib import Path
from urllib.request import urlopen


import yaml


sources = {
    "gateway.networking.k8s.io": (
        "https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/"
        "v1.4.1/config/crd/standard/",
        ["gatewayclasses", "gateways", "httproutes"],
    ),
    "gateway.envoyproxy.io": (
        "https://raw.githubusercontent.com/envoyproxy/gateway/"
        "v1.7.0/charts/gateway-helm/crds/generated/",
        ["backendtrafficpolicies", "clienttrafficpolicies"],
    ),
    "monitoring.coreos.com": (
        "https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/"
        "v0.89.0/example/prometheus-operator-crd/",
        ["servicemonitors"],
    ),
}




def strict(schema):
    if isinstance(schema, dict):
        if "properties" in schema and "additionalProperties" not in schema:
            schema["additionalProperties"] = False
        for value in schema.values():
            strict(value)
    elif isinstance(schema, list):
        for value in schema:
            strict(value)




for group, (base, resources) in sources.items():
    for resource in resources:
        with urlopen(f"{base}{group}_{resource}.yaml") as response:
            crd = yaml.safe_load(response)
        for version in crd["spec"]["versions"]:
            if not version["served"]:
                continue
            schema = version["schema"]["openAPIV3Schema"]
            strict(schema)
            # Kubernetes supplies metadata outside the custom resource schema.
            schema["properties"]["metadata"] = {"type": "object"}
            kind = crd["spec"]["names"]["kind"].lower()
            path = Path("schemas") / group / f"{kind}_{version['name']}.json"
            path.parent.mkdir(parents=True, exist_ok=True)
            path.write_text(json.dumps(schema))

An unreachable source fails this script. Do not skip missing schemas. Run it, then check both named value sets with Kubernetes 1.34 schemas:

Terminal window
python3 prepare-schemas.py
helm lint --strict ./xmtp-chart
helm lint --strict ./xmtp-chart -f enabled.yaml

Use Bash with pipefail so a failed render also fails the pipeline:

Terminal window
set -o pipefail
helm template xmtp ./xmtp-chart --namespace xmtp --kube-version 1.34.0 | \
  kubeconform -strict -summary -kubernetes-version 1.34.0 \
  -schema-location default \
  -schema-location 'schemas/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json'
helm template xmtp ./xmtp-chart --namespace xmtp --kube-version 1.34.0 \
  -f enabled.yaml | \
  kubeconform -strict -summary -kubernetes-version 1.34.0 \
  -schema-location default \
  -schema-location 'schemas/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json'

Clean up

Section titled “Clean up”

Each platform terminates TLS and programmes DNS differently, so check TLS at the Gateway, external DNS, gRPC-Web and CORS through your controller, and the route timeouts against the shared ingress checks after you install.

Remove a test install with:

Terminal window
helm uninstall xmtp --namespace xmtp
kubectl -n xmtp delete secret xmtp-tls
helm uninstall eg --namespace envoy-gateway-system

Uninstall the controller only if this test owns it. Confirm that its cloud load balancer is gone. Helm can leave CRDs behind. The external database, database Secret, DNS record, and certificate renewal process are not chart resources. Remove disposable test resources separately; retain production data.

As a legacy footnote, ingress-nginx is retired. Its best-effort maintenance ended in March 2026. Do not use it for a new installation. See the retirement notice.

XMTP.org · XMTP status · XMTP.chat

Privacy policy · Terms of service

CC BY 4.0 · Copyright © 2024-present XMTP.