Upgrade

For the compose stacks: the quick start (docker-compose.yml at the repository root) and the production compose. Kubernetes: docs/deploy/on-premise.md.

Three things have versions, and each rolls back on its own:

Before 1.0, a minor version can change the schema and the configuration: read the CHANGELOG for every version you skip.

Backups#

scripts/backup.sh saves what a restore needs, while the stack runs:

scripts/backup.sh                                                     # quick start
scripts/backup.sh -f deploy/compose/production/docker-compose.yml    # production
# backup: done: backups/guardspec-20261106T060839Z

Not included, on purpose: models (pulled again and verified against the manifest, or pinned with L2_MODEL), images (pulled by tag), and secrets kept outside .env (a secret manager: back those up there).

The directory is created with mode 700 and holds every secret: encrypt it (age, gpg, your backup tool's encryption) and keep it off the host. API keys are stored as SHA-256 hashes, so the backup cannot reveal them; password hashes are bcrypt.

Schedule it with cron or a systemd timer, for example daily:

15 3 * * * cd /opt/guard-spec && scripts/backup.sh -f deploy/compose/production/docker-compose.yml -o /var/backups/guardspec/$(date +\%F) && find /var/backups/guardspec -mindepth 1 -maxdepth 1 -mtime +14 -exec rm -rf {} +

Restore#

Into an empty database only (a new host, a new compose project, or after docker compose down -v); a database with tables is refused:

scripts/restore.sh -f deploy/compose/production/docker-compose.yml backups/guardspec-20261106T060839Z

It checks the dump's SHA-256, writes the backup's .env when there is none (and refuses a different MFA_ENC_KEY in an existing one), starts Postgres, creates the cluster-wide role guardspec_app (a database dump does not carry roles), runs pg_restore, puts Caddy's data back, then starts the stack: migrate applies any migrations newer than the dump and db-roles sets the guardspec_app password from .env. Logins, 2FA, API keys and the internal CA keep working.

Checked on 2026-11-06 (arm64, Docker Desktop): a backup of a production compose stack (1 organization, 1 user, 1 API key, 1 policy, 26,004 audit rows) restored into a separate compose project. Checksums of the organizations, users, API keys, policies and audit rows matched; the old admin password and API key worked; an injection was blocked and a benign prompt passed over HTTPS verified with the original Caddy root.

Managed databases: docs/managed-postgres.md.

Upgrading#

scripts/backup.sh [-f ...]                          # 1. a backup to roll back to
git fetch --tags && git checkout v<new-version>     # 2. new migrations/ and compose files
# 3. .env: GUARDSPEC_VERSION=<new-version>; compare .env.example for new settings
docker compose [-f ...] pull                        # 4. images first, so the switch is short
docker compose [-f ...] up -d --wait                # 5. migrate and db-roles run before the services
curl -s localhost:8080/readyz | jq .detector        # 6. or https://<GATEWAY_HOST>/readyz

• Step 2 matters: the migrate job applies the migrations/ directory of your checkout, so checkout and GUARDSPEC_VERSION must be the same release. A release source archive works the same as a checkout.
• During step 5 the gateway restarts; requests in flight fail and clients retry. One host cannot avoid that; for no downtime, run several gateway replicas behind a load balancer (Kubernetes, Helm chart).
• Models: if the new image recommends a new model version, models-init pulls it before the gateway starts (with DETECTOR_REQUIRE_ML=true, the production default, a failed pull stops up before the new gateway starts). With L2_MODEL / L3_MODEL pinned, the gateway keeps the pinned version.
• New classifier version: to try it before switching, keep L2_MODEL pinned and set SHADOW_L2_MODEL=guardspec-classifier@<new>. It scores the same traffic without deciding; compare in the admin UI (Models). Then switch without a restart with POST /admin/models/reload and {"l2_model":"guardspec-classifier@<new>","shadow_l2_model":"off"} (GATEWAY_ADMIN_TOKEN), and set L2_MODEL in .env for the next start (or set it and up -d gateway, which restarts). A reload checks the files' SHA-256 and keeps the running model if anything fails; see README, Shadow mode and model reload.
• Check after the upgrade: /readyz lists the layers you expect, the admin UI loads, a known injection is blocked (README quick start).
• With images released after 0.1.1, also run doctor inside the gateway: docker compose [-f ...] exec gateway /app/doctor. It checks the database, that the migration version matches the image's migrations/, the model files against the manifest's SHA-256, which layers will load, the upstream and the security settings; exit code 1 means a check failed (-strict also fails on warnings, -json for scripts). See README, Kubernetes.

Rolling back#

Images, no schema change#

When the release you leave added no migration (compare migrations/ of the two tags, or MANIFEST's migration version with the new one):

git checkout v<old-version>
# .env: GUARDSPEC_VERSION=<old-version>
docker compose [-f ...] up -d --wait

Images and schema#

A newer schema with older code is not supported, and the *.down.sql migrations drop what the newer version added, data included; they are not the tested path. Restore the backup taken before the upgrade instead:

docker compose [-f ...] down -v      # removes the upgraded database: keep the backup safe first
git checkout v<old-version>
# .env: GUARDSPEC_VERSION=<old-version> (restore.sh keeps an existing .env)
scripts/restore.sh [-f ...] backups/guardspec-<before-upgrade>

Audit rows written between the backup and the rollback are lost. To keep them, export them first (GET /v1/organizations/{id}/requests, or a pg_dump -t requests).

down -v also removes the models volume of the production compose (pulled again) and Caddy's data (restored from the backup).

Models only#

No image or schema change: pull the old version and pin it.

docker compose [-f ...] --profile models run --rm models-pull guardspec-classifier@<old>
# .env: L2_MODEL=guardspec-classifier@<old>
docker compose [-f ...] up -d gateway
curl -s localhost:8080/readyz | jq .detector.l2_model

Without a restart: keep the pin in .env for the next start, and reload the running gateway with POST /admin/models/reload and {"l2_model":"guardspec-classifier@<old>"} (needs GATEWAY_ADMIN_TOKEN). SIGHUP reloads the targets the gateway already has, not a changed .env.

Details, the L3 embedder and air-gapped installs: docs/models.md.

A failed upgrade#

• migrate exits non-zero: the services do not start (they wait for it). docker compose logs migrate names the migration. A failed migration leaves schema_migrations.dirty = true; fix the cause, then restore the backup rather than forcing the version.
• models-init fails: offline, or a mirror is missing the new version. Set MODELS_MIRROR, or pin the previous L2_MODEL / L3_MODEL, and up again.
• The gateway is unhealthy: docker compose logs gateway; with DETECTOR_REQUIRE_ML=true a missing model stops it at start, by design.

Source: docs/upgrade.md @ ca69dad29f02