Production deployment

Topology#

docker-compose.prod.yml is standalone: it is not combined with the development docker-compose.yml. It serves a single domain. Caddy sends /api/* and /ws to the backend (internal port 8080) and everything else to the frontend (internal port 3000).

Only Caddy publishes ports: 80, 443 TCP and 443 UDP. Every other service lives on the internal network, declared with internal: true and therefore without internet access. The runtime also joins the egress network to call model provider APIs, and Caddy joins the edge network.

The frontend uses relative paths (/api/v1) and derives the WebSocket URL from window.location (wss:// under HTTPS), so a single-domain deployment does not need any baked NEXT_PUBLIC_* variables.

Requirements#

• A Linux server with Docker Engine and the Docker Compose v2 plugin, with ports 80 and 443 open in the firewall.
• A domain with a DNS A record (and AAAA if you have IPv6) pointing to the server. Let's Encrypt fails if DNS does not resolve yet.
• Access to the images on GHCR. If the packages are private, run docker login ghcr.io with a token that has read:packages. Alternative: build the images on the server itself with --build.
• A DeepSeek key: the production compose file requires DEEPSEEK_API_KEY. The Anthropic key is optional.

Step-by-step installation#

The examples use example.com as the domain and OWNER and TAG as placeholders. All state lives in named volumes, and .env.prod stays out of git: the repository .gitignore already ignores .env.*.

1. Clone the repository on the server and copy the .env.prod.example template to .env.prod.
2. Restrict its permissions with chmod 600 .env.prod.
3. Fill in the variables. The block below shows a complete minimal .env.prod.

git clone <repo-url> && cd <repo>
cp .env.prod.example .env.prod
chmod 600 .env.prod
$EDITOR .env.prod

# Domain and TLS
DOMAIN=example.com
ACME_EMAIL=ops@example.com
ALLOWED_ORIGINS=https://example.com

# Images published by CI
OWNER=acme
TAG=sha-1a2b3c4

# Secrets: a different openssl rand -hex 32 value for each
POSTGRES_PASSWORD=...
REDIS_PASSWORD=...
JWT_SECRET=...
RUNTIME_TOKEN=...

# Spending and demo
BUDGET_USD=25
ENABLE_DEMO_RESET=false

# Model providers
DEEPSEEK_API_KEY=sk-...
ANTHROPIC_API_KEY=
ALLOWED_PROVIDERS=

Generating secrets and validating the configuration#

Generate a different value for POSTGRES_PASSWORD, REDIS_PASSWORD, JWT_SECRET and RUNTIME_TOKEN. The backend does not start if JWT_SECRET is shorter than 32 bytes, and the production compose file sets AUTH_ENABLED=true. RUNTIME_TOKEN is not in the template, but you should add it: the compose file passes it to both backend and runtime, and without it calls between them are not authenticated.

The compose file's required variables make validation fail when missing: DOMAIN, ACME_EMAIL, POSTGRES_PASSWORD, REDIS_PASSWORD, OWNER, TAG, JWT_SECRET, ALLOWED_ORIGINS and DEEPSEEK_API_KEY.

openssl rand -hex 32   # repeat for POSTGRES_PASSWORD, REDIS_PASSWORD, JWT_SECRET and RUNTIME_TOKEN

# Validate before starting (fails if a required variable is missing)
docker compose -f docker-compose.prod.yml --env-file .env.prod config -q

Starting and checking health#

Every service defines a healthcheck and dependencies use condition: service_healthy. Postgres, Redis and the runtime start first; the backend waits for all three, the frontend for the backend and Caddy for both. The backend and the runtime expose /healthz. Caddy obtains and renews the certificate automatically.

C="docker compose -f docker-compose.prod.yml --env-file .env.prod"

$C pull          # option A: images published by CI
# $C build       # option B: build on the server
$C up -d
$C ps            # every service should become healthy
curl -fsS https://example.com/ -o /dev/null && echo OK
$C logs -f backend

Updating, migrations and rollback#

To update, change TAG to a sha-<short> value published by CI and bring the stack up again. Always take a pg_dump before deploying a version that carries migrations.

At startup the backend applies the missing embedded SQL migrations, in order and inside a single transaction, and records each one in the schema_migrations table. It never rolls migrations back on its own: the backend/migrations/down/ directory holds rollback scripts you apply by hand.

If the new version fails and the schema has already changed, you have two paths: restore the previous dump and go back to the old TAG (you lose what was written afterwards), or ship a new corrective migration, which is preferable once new data exists. If the migration only added columns or tables, the previous TAG usually keeps working without touching the database.

# 1) backup (see the next section)
# 2) point to the new tag
sed -i 's/^TAG=.*/TAG=sha-<NEW>/' .env.prod
$C pull
$C up -d

# Rollback without schema changes: go back to the previous TAG
sed -i 's/^TAG=.*/TAG=sha-<PREVIOUS>/' .env.prod
$C pull && $C up -d

# Applied migrations
$C exec postgres sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "select * from schema_migrations order by applied_at"'

Postgres backups and restore#

Postgres is the system of record. Redis keeps an AOF in the redisdata volume, but treat it as a cache and queue. Schedule pg_dump with cron, copy the files off the server and test a restore at least once.

mkdir -p backups
$C exec -T postgres sh -c 'pg_dump -U "$POSTGRES_USER" -d "$POSTGRES_DB" -Fc' \
  > backups/workforce-$(date +%F-%H%M).dump

# Restore
$C stop backend
$C exec -T postgres sh -c 'dropdb -U "$POSTGRES_USER" --if-exists "$POSTGRES_DB" && createdb -U "$POSTGRES_USER" "$POSTGRES_DB"'
$C exec -T postgres sh -c 'pg_restore -U "$POSTGRES_USER" -d "$POSTGRES_DB" --no-owner' < backups/FILE.dump
$C start backend

Go-live checklist#

• DNS points to the server, 80 and 443 are open and docker compose ps only shows published ports on caddy.
• .env.prod has 600 permissions and is outside git.
• JWT_SECRET, POSTGRES_PASSWORD, REDIS_PASSWORD and RUNTIME_TOKEN are unique and random (32 hex bytes or more).
• ALLOWED_ORIGINS is exactly https://example.com and ENABLE_DEMO_RESET is false.
• The provider key is valid, ALLOWED_PROVIDERS reflects the company data policy and BUDGET_USD holds the cap you want.
• TAG points to a specific sha-<short> that passed CI, not to latest.
• Every service is healthy, sign-in works and the WebSocket connects at wss://example.com/ws.
• The Postgres backup is scheduled and a restore has been tested.
• Logs rotate (json-file driver, 10 MB times 5 files, already configured) and the memory and CPU limits fit the host.

Image publishing in CI#

The .github/workflows/ci.yml workflow builds and tests backend, runtime and frontend, runs govulncheck and npm audit (non-blocking), hadolint, a smoke test and Playwright E2E. On every push to main that passes, the publish job pushes three images to GHCR (aiworkforce-backend, aiworkforce-agent-runtime and aiworkforce-frontend) tagged sha-<short> and latest, using GITHUB_TOKEN with packages: write.

In the repository settings, allow GITHUB_TOKEN to write packages and, after the first publish, adjust the packages' visibility. For a single-domain deployment, leave the repository variables NEXT_PUBLIC_API_URL and NEXT_PUBLIC_WS_URL undefined.

Reduced layout on a shared server#

deploy/vps holds the public demo layout, built for a server that already runs its own Caddy owning ports 80 and 443. That compose file publishes no ports: backend, frontend and the static landing join the existing Caddy's network (declared as external) and everything else stays on a private network.

It has real accounts: Postgres, AUTH_ENABLED=true, trial accounts (TRIAL_ENABLED=true), ENABLE_DEMO_RESET=false and TRUSTED_PROXIES pointing at the Caddy network. It has no Redis, so events and rate limits live in process memory. The runtime runs with SIMULATION=true, with no keys and no cost.

The frontend is built with absolute NEXT_PUBLIC_API_URL and NEXT_PUBLIC_WS_URL, and the app site adds X-Robots-Tag: noindex, nofollow. deploy.sh runs git pull, builds at low priority and runs up -d --wait; backup.sh, meant for a daily cron job, saves a compressed pg_dump and deletes those older than 14 days.

example.com        -> landing (static)
app.example.com    -> /api/* and /ws -> wf-backend:8080 ; rest -> wf-frontend:3000
(the existing Caddy terminates TLS; this project publishes no ports)