Deploying to production¶
The server is the web UI and REST API for research, writing and long-document jobs, shared by a team. This page covers running it reliably: architecture, three ways to deploy, settings, API keys, monitoring, backups and upgrades.
Architecture¶
HTTPS SQLite job database (WAL), on one volume
users ──► reverse proxy ──► swarm serve ──enqueue──► jobs · events · api_keys
(Caddy/nginx) (API, UI) ▲ │
│ claim │ progress
swarm worker ─┴────────┘ (×N, heartbeats)
- The API never runs jobs itself (unless you use its embedded worker). It checks the key, rate limit and budget, then puts the job in the queue.
- Workers claim jobs in a transaction, so no two workers ever get the same job. They run it and stream progress into the database, where the browser follows it live (Server-Sent Events, which resume after a reconnect).
- Jobs survive crashes. A worker heartbeats every 10 seconds. If one dies, its jobs go back to the queue (up to 2 attempts), and after that they're marked failed with a reason.
- Restarts don't lose work. On a normal shutdown (SIGTERM) a worker hands its running jobs straight back to the queue, so upgrades lose nothing.
- What runs on the server: research,
writeanddocjobs.build,analyzeandreviewexecute code or read local files, so they're deliberately CLI-only.
Option 1: Docker Compose with HTTPS (one server)¶
Needs Docker, a domain pointing at the server, and ports 80 and 443 open.
git clone https://github.com/Chetan0246/multi-agent-swarm && cd multi-agent-swarm
cp deploy/.env.example deploy/.env # set SWARM_DOMAIN, model and search keys
docker compose -f deploy/compose.yaml up -d --build
docker compose -f deploy/compose.yaml exec api swarm keys create --name alice --daily-usd 5 --rpm 30
docker compose -f deploy/compose.yaml up -d --scale worker=3 # more parallel jobs
Caddy gets and renews a Let's Encrypt certificate automatically and streams progress events without buffering. Every container:
- runs as a non-root user with a read-only filesystem;
- has no Linux capabilities and runs with
no-new-privileges; - has 45 seconds to stop gracefully.
Only Caddy publishes ports.
Option 2: Kubernetes¶
kubectl create namespace swarm
kubectl -n swarm create secret generic swarm-secrets \
--from-literal=ANTHROPIC_API_KEY=... --from-literal=TINYFISH_API_KEY=... \
--from-literal=SWARM_METRICS_TOKEN=$(openssl rand -hex 24)
# edit the host names in deploy/k8s/swarm.yaml, then
kubectl -n swarm apply -f deploy/k8s/swarm.yaml
kubectl -n swarm exec deploy/swarm -c api -- swarm keys create --name alice --daily-usd 5
- One pod, two containers: the API and a worker share a persistent volume, and the pod has readiness and liveness probes, a restricted security context and resource limits.
- Ingress: an nginx Ingress with cert-manager, with buffering off so progress events stream.
- Single replica: the database is SQLite, so the Deployment is one replica with a
Recreatestrategy. Scale jobs with the worker's--concurrency, or by adding worker containers to the pod.
Option 3: systemd on a VM¶
deploy/systemd/ has a hardened swarm-api.service (with ProtectSystem=strict,
NoNewPrivileges and a write-only state directory) and a template swarm-worker@.service.
The install steps are at the top of swarm-api.service. Put TLS in your own reverse proxy
(Caddy, nginx), and list the host name in SWARM_ALLOWED_HOSTS.
Settings¶
Everything is set through environment variables (or [server] in swarm.toml):
| Variable | Default | What it does |
|---|---|---|
SWARM_STATE_DIR |
~/.local/share/swarm (/data/state in the image) |
where jobs.sqlite lives |
SWARM_ALLOWED_HOSTS |
localhost names | host names the server answers to (blocks DNS rebinding) |
SWARM_API_TOKEN |
one admin key; prefer swarm keys per user |
|
SWARM_METRICS_TOKEN |
lets a Prometheus scraper read /metrics |
|
SWARM_MAX_USD_PER_JOB |
2 | hard cap per job, whatever the request asks for |
SWARM_DEFAULT_RPM |
60 | requests per minute for keys without their own limit |
SWARM_JOB_TIMEOUT_S |
1800 | a job running longer is stopped |
SWARM_RETENTION_DAYS |
30 | finished jobs older than this are deleted |
SWARM_WORKER_CONCURRENCY |
2 | jobs at once in the embedded worker |
SWARM_EMBEDDED_WORKER |
1 | 0 = API only (same as --no-worker) |
SWARM_ALLOW_MOCK |
true | set to false in production |
SWARM_CORS_ORIGINS |
browser origins allowed to call the API from other sites | |
SWARM_MAX_BODY_BYTES |
2000000 | larger requests get 413 |
SWARM_TRUSTED_PROXIES |
127.0.0.1 | proxies whose X-Forwarded-* headers are trusted |
SWARM_LOG_FORMAT, SWARM_LOG_LEVEL |
text (json in the image), INFO | log format and level |
The model and search settings are the same as for the CLI: ANTHROPIC_API_KEY, or
SWARM_PROVIDER=openai with OPENAI_BASE_URL, OPENAI_API_KEY, SWARM_MODEL and
SWARM_PRICE; plus TINYFISH_API_KEY.
API keys and budgets¶
swarm keys create --name alice --daily-usd 5 --rpm 30 # prints the key once
swarm keys create --name ops --admin # sees every job and /metrics
swarm keys list
swarm keys revoke <id>
- Keys are stored only as hashes, so a leaked database doesn't leak usable keys.
- Each key sees only its own jobs. Admin keys see all of them.
- Daily budgets can't be overshot. A key's budget counts what its jobs spent today plus
the caps of its jobs still queued or running, so a burst of submissions can't go over it.
Going over returns
402; going over the rate limit returns429withRetry-After. - No public server without a key.
swarm serverefuses to listen on a public interface until at least one key orSWARM_API_TOKENexists.
Monitoring¶
| Endpoint | For |
|---|---|
/healthz |
liveness: the process is up |
/readyz |
readiness: the database answers (503 if not) |
/metrics |
Prometheus, with an admin key or SWARM_METRICS_TOKEN |
The metrics are:
- swarm_http_requests_total and swarm_http_request_duration_seconds: requests by route and
status, with latency histograms;
- swarm_jobs{kind,status}: job counts;
- swarm_job_spend_usd{kind}: spend by job type;
- swarm_queue_oldest_age_seconds: how long the oldest waiting job has waited;
- swarm_process_uptime_seconds.
Job, spend and queue figures come from the database, so they're correct however many workers run.
Suggested alerts:
swarm_queue_oldest_age_seconds > 600: jobs are waiting; add workers or check they run.- A rising
swarm_jobs{status="error"}: read the worker logs. swarm_job_spend_usdgrowing faster than expected: check the keys' budgets.
Logs are JSON lines with a request_id. The same id is returned in the X-Request-ID
response header, so a user's report can be traced to the exact log lines. Logs record key
ids, never keys or prompts.
Backups and upgrades¶
- Backup while running:
sqlite3 /data/state/jobs.sqlite ".backup /backup/jobs-$(date +%F).sqlite"
(WAL mode makes this safe during writes.)
- Upgrade: pull or build the new image, then docker compose up -d. Workers hand running
jobs back to the queue on SIGTERM, and the new workers pick them up. Released images are at
ghcr.io/chetan0246/multi-agent-swarm:<version>. They're built for amd64 and arm64, with
provenance and an SBOM attached.
Security checklist¶
- [ ] HTTPS in front (Caddy in the Compose setup, or your own proxy or Ingress).
- [ ]
SWARM_ALLOWED_HOSTSlists exactly your host names. - [ ] One key per user, each with
--daily-usd; no shared admin token. - [ ]
SWARM_ALLOW_MOCK=false, andSWARM_MAX_USD_PER_JOBset to what you'll accept. - [ ]
/metricsscraped withSWARM_METRICS_TOKEN, not left open. - [ ]
deploy/.env(or the Kubernetes Secret) readable only by the operator. It's gitignored and left out of image builds. - [ ] Backups of
jobs.sqliteon a schedule.
Limits, plainly¶
- One host for the job database. It's SQLite, so the API and workers share one disk. That carries many workers on one machine. Spreading them over several machines needs a shared database server.
- Rate limits are per API process. With several API replicas, each enforces its own window. Daily dollar budgets are shared, because they're kept in the database.