Self-Hosted Orbs on Kubernetes

Amp’s Helm chart runs self-hosted orbs on GKE, EKS, and AKS. It installs the orb provider and a dispatch program in one pod. Orbs run as gVisor sandboxes on Substrate, an open source sandbox runtime for Kubernetes that the chart installs with the provider.

How the Chart Works

The pod runs two containers. The spawner container runs amp orb provider and receives jobs from Amp. The provider container asks Substrate for a sandbox, which starts as a pod on one of your worker nodes, and saves a paused orb’s disk to your storage bucket, so it can resume on any worker.

Supported Platforms

The install bundle has the full prerequisites and commands for each platform:

The chart defaults to one orb type of 16 vCPU and 32 GiB with one spare worker, so size the node pool for the number of orbs you expect to run at once. To let the node pool scale to zero when no orb runs, give the type min: 0 and warm: 0 and set workers.autoFill: true; the provider then keeps spare workers only where they fill nodes already running an orb, shared among the types by demand. See “Scale to zero” in self-hosted.md in the bundle.

Before You Start

Use a dedicated cluster that meets the platform runbook’s prerequisites, and install from a machine with cloud credentials, kubectl, and Helm 3.14 or newer with cluster-admin access. Terraform examples for EKS and AKS are in the bundle.

Set up a provider in Amp first. The chart needs its token.

Network Requirements

In addition to the outbound access every self-hosted orb needs, allow:

Install with Amp

Run Amp on a machine that can reach your cluster. Download the install bundle:

export AMP_URL=https://ampcode.com
curl -fsSLo bundle.zip "$AMP_URL/docs/orbs/self-hosted/bundle.zip"
unzip bundle.zip
cd amp-self-hosted-orbs
amp

The bundle includes AGENTS.md, this guide, the platform runbooks, a smoke test, and Terraform examples. Then use this prompt:

Install

Follow gke.md, eks.md, or aks.md in the bundle to create the cluster and storage, then run its helm install command. The provider token is not a chart value: put it in the Kubernetes Secret amp-orb-spawner (key AMP_ORB_PROVIDER_TOKEN) in the release namespace before installing, so Helm, your values files, and your GitOps repository never hold it. For example, once the GKE prerequisites are met:

kubectl create namespace amp-orbs
kubectl -n amp-orbs create secret generic amp-orb-spawner \
  --from-file=AMP_ORB_PROVIDER_TOKEN=/secure/path/provider-token
helm install orb-provider oci://us-central1-docker.pkg.dev/amp-prod-0/orb-provider/chart \
  -n amp-orbs \
  --set cloud=gke --set amp.url=https://ampcode.com \
  --set snapshots.location=gs://<bucket>/amp-orbs/ \
  --wait --timeout 15m

Without --version, Helm installs the latest chart; run helm show chart on the chart URL to see which version that is. The token file must contain only the token. See self-hosted.md in the bundle for the storage identity values EKS and AKS need. To rotate the token, update the Secret and run kubectl -n amp-orbs rollout restart deployment/amp-orb-provider.

To install the CLI from a private npm registry, create a Secret in the release namespace with a .npmrc key that holds the registry configuration and its authentication, and set:

spawner:
  npm:
    registry: https://npm.example.com/repository/npm/
    existingConfigSecret: ampspawner-npm

See “Internal npm registries” in self-hosted.md for what the registry must serve.

Orb Types

Each orb type on the provider in Amp gets its own worker pool in the chart. Set workers.types to the same names, with the size of each, and workers.default to the one an orb gets when its thread names none:

workers:
  types:
    - { name: medium, cpu: 4, memoryGi: 8, min: 0, warm: 1 }
    - { name: large, cpu: 8, memoryGi: 16, min: 0, warm: 1 }
    - { name: xlarge, cpu: 16, memoryGi: 32, min: 0, warm: 1 }
  default: large

min is the number of workers the pool always keeps, and warm the idle workers it keeps ready beyond the busy ones, so the next orb of that type starts without waiting for a pod. Workers of different types share nodes; a worker needs memoryGi plus 1 GiB of a node’s memory. An orb that names a type the chart does not have fails to start, and its thread’s setup log names the types the chart has.

A thread keeps its orb type for life, so keep a type in workers.types until no thread uses it. To rename a type, add the new name here and in Amp, then delete the old one from both once its threads are done. Changing a type’s cpu or memoryGi starts a new worker generation the same way a worker image upgrade does: new orbs land on the new workers, and orbs still running on the old ones are paused onto them after workers.retireAfterSeconds.

Test the Install

1. Run NAMESPACE=amp-orbs bash ./smoke-test.sh from the bundle. It spawns an orb, waits for it to boot, and deletes it.
2. Create a thread on the provider with your final firewall and proxy rules in place. Run a command and open Files and Changes. Open Terminal and Desktop, start a dev server, and open its portal.
3. After Amp shows the orb paused, send another message and check that a file you created is still there.

Troubleshoot

If an orb does not start, read both container logs and the thread’s setup log:

kubectl -n amp-orbs logs deploy/amp-orb-provider -c provider
kubectl -n amp-orbs logs deploy/amp-orb-provider -c spawner

If the cause is not on your side, send a bug report from the thread and include the log lines.

Upgrade

The provider keeps its CLI up to date on its own. Upgrade to the latest chart with:

helm upgrade orb-provider oci://us-central1-docker.pkg.dev/amp-prod-0/orb-provider/chart \
  -n amp-orbs --reset-then-reuse-values --wait --timeout 15m

This keeps your configured values. Running orbs stay on the worker generation they started on and move to the new one when they next pause; a pause and resume in a thread afterwards proves the new generation works. If you installed a chart from before worker generations, do the one-time step in “Upgrading from a chart before worker generations” in self-hosted.md first. After changing the npm config Secret, restart the provider deployment.

The chart carries the guides and smoke test for its own version under docs/ (helm pull oci://us-central1-docker.pkg.dev/amp-prod-0/orb-provider/chart --untar), so you can read the upgrade notes of the chart you are about to install, not of this page.

Uninstall

Follow the uninstall steps in the platform runbook. Removing the provider from workspace settings does not delete the cluster or its storage. It revokes the provider token right away: the controller receives no more jobs, and the orbs of threads that use the provider stop working.