Operations

Audit privacy#

Audit rows (requests) store scores, the decision, and a SHA-256 and the length of the request body and of a buffered upstream response. They store no prompt or response content unless content storage is enabled, either for every organization with AUDIT_STORE_CONTENT=true on the gateway, or for one organization with PATCH /v1/organizations/{id} and {"audit_store_content": true} (admin role). Streamed responses are not buffered, so their response columns stay empty.

Each request gets one audit row. action is allow, block, error (the upstream failed or the client left before it answered) or output_leak (the canary showed up in the answer).

Shadow mode and model reload#

Shadow mode. With SHADOW_L2_MODEL=guardspec-classifier@v2 (or SHADOW_L2_MODEL_DIR), the gateway loads a second classifier next to the active one. For each sampled request (SHADOW_SAMPLE), every text the detector scores is scored again by the candidate, and its L2 verdict is fused with the same request's L1 and L3 scores and the candidate's own calibration and fusion weights. The request's comparison is the largest fused score of each side over its texts, and each side's decision at the direct threshold (0.85).

• It runs asynchronously, never in the response path: the active result is returned at once and the candidate runs on its own goroutine and its own ONNX limiter (SHADOW_MAX_INFLIGHT), so it never takes a slot an active request waits for. When every slot is busy the text is not compared (skipped) rather than queued. The audit writer, already off the response path, waits up to SHADOW_TIMEOUT for it; a slower candidate is recorded as timeout. The cost is CPU: a candidate roughly doubles L2 work for sampled traffic, so lower SHADOW_SAMPLE on a busy gateway.
• It never changes a decision, a response or a policy input.
• Results go to shadow_results (migration 017), one row per compared request, written in the audit row's transaction and row-level secured per org like requests. A row holds the two model names, scores and decisions and a status (ok, timeout, error, skipped), never content, whatever the audit privacy settings. Rows are deleted with their request (audit retention).
• Policies, sanitizer blocks and the indirect threshold are not replayed: the comparison is of the model layers.

Compare in the admin UI (Models), or with the API: GET /v1/organizations/:orgId/shadow?window=24h|7d|30d (viewer: agreement rate, shadow_would_block / shadow_would_allow, statuses, score means, p50/p95, the shadow − active delta and a 10-bin histogram) and GET /v1/organizations/:orgId/shadow/disagreements?direction=would_block|would_allow (admin: the disagreeing requests, newest first, paged; no content).

Reloading models without a restart. Two triggers, both on the gateway:

• kill -HUP <gateway pid> (docker compose kill -s HUP gateway) reloads the current targets from disk: the pinned version, the manifest's recommended one when none is pinned, and the shadow candidate.
• POST /admin/models/reload with Authorization: Bearer $GATEWAY_ADMIN_TOKEN reloads and can switch targets: {"l2_model":"guardspec-classifier@v2"}, {"l2_model_dir":"/models/x"}, {"shadow_l2_model":"guardspec-classifier@v3b"} or {"shadow_l2_model":"off"}. It answers 200 with the new generation, or 409 with the error. GET /admin/models returns the status. The routes do not exist unless the token is set; they are on the API port, so keep the token secret and the port off the Internet as usual.

A reload resolves the target, checks every file against the manifest's SHA-256 (the built-in manifest, or MODELS_MANIFEST; a directory that is not an install of a listed version is refused), loads the classifier fully and scores a warm-up text, and only then swaps it in atomically, with its own calibration and fusion weights. Requests in flight finish on the model they started with, which is closed once the last of them is done; none is dropped or fails. On any error the running models stay, the error is logged and /readyz reports it (detector.models.reload.error). A change made by the endpoint lasts until the next restart; pin it in .env (L2_MODEL) to keep it. With several gateway replicas, reload each one.

/readyz reports under detector.models: the running classifier as name@version and whether its files matched the manifest (l2_verified), the manifest's recommended version and whether it is installed, the versions installed in the cache, update_available, the shadow candidate and the last reload. The Models page shows it through the control plane (GATEWAY_URL, read server side); nothing calls the Internet. A new version shows up when a new manifest recommends it (a new image, or MODELS_MANIFEST); pull it (models pull), then reload.

Upstream behaviour#

UPSTREAM_LLM_TIMEOUT (default 5m) is the longest wait for the first byte and for any later silence; a long stream is not cut off while data flows. If the client hangs up before the upstream answers, the upstream request is cancelled at once. A client that only half-closes its side (shutdown(SHUT_WR)) sends the same FIN as one that left, and the gateway cannot tell them apart without writing to the connection, so by default it cancels too. Set CLIENT_HALF_CLOSE=wait to serve such clients: the gateway then cancels only on a connection reset and otherwise relies on the timeout.

Output guard#

The gateway checks the model's answer before the client sees it. It rewrites choices[].message.content (and choices[].text for legacy completions) in whole responses. In streams it keeps a 256-character window per choice, so a pattern split across chunks is still caught, and rebuilds each chunk with its id, model and other fields unchanged. A stream therefore runs about 256 characters behind the upstream. Held text is flushed at finish_reason and at [DONE]. An upstream body that is not JSON cannot be checked and gives 502.

• Images and links. Markdown images, HTML <img> tags and reference definitions whose host is not on the allowlist are removed ([REDACTED:image]). Links to other hosts keep their text and lose the URL. Bare image URLs that carry a query string are replaced with [REDACTED:url]. With the default empty allowlist every image and link is removed. An <img> tag is kept only if every src and srcset URL is on the allowlist.
• Personal data. Each match is replaced with [REDACTED:<type>]: email, phone (Vietnamese numbers in the +84, 84 and 0 forms, mobile and landline), cccd (12 digits starting with 0), card (13 to 19 digits that pass the Luhn check) and, when enabled, ipv4.
• Canary. When enabled, the gateway puts a random token for each request in the system prompt (appended to a leading system message, or as a new one; legacy completions get none). If the model repeats it, the response is refused with 400 content_filter (in a stream, an error event that ends it), or the token is redacted. Either way the row's action is output_leak.
• Reasoning. The reasoning and reasoning_content fields that Ollama and vLLM send are guarded like the answer, or dropped.
• Refusals and tool calls. refusal is guarded like the answer. Tool-call arguments (tool_calls[].function.arguments, and the older function_call) are parsed as JSON and each string is guarded on its own, so they stay valid JSON. In a stream, each call's arguments are held whole and sent when its choice finishes. A canary in tool-call arguments refuses the response even in redact mode.
• Content parts. Text parts are guarded. A part that is not text cannot be checked: a whole response gets 502, and a stream ends with an error event.

The audit row counts what was removed per type in pii_entities, as output:email=2. It never stores the content. These settings apply to every organization; there are no per-org output settings yet.

Indirect injection and tool poisoning#

Text that reaches the model from a third party (tool results, retrieved documents, and the tool descriptions an MCP server supplies) is untrusted. The gateway scores it apart from the user's own text.

User and tool text is sanitized whether or not it is classified. A key that differs from tools or functions only in case is refused with 400. While tool_descriptions is on, so is a tools field that is not a list, a name or description that is not a string, and a schema nested deeper than 32 levels.

Scoring. Each tool result and each tool definition gets its own score. An instruction found in a tool result is indirect_injection; anything flagged in a tool description is tool_poisoning. A tool description's score is the larger of the detector's and the MCP signals': a secret path (~/.ssh, id_rsa, /etc/shadow), <IMPORTANT>, "do not mention this to the user", "send all emails to" or invisible characters reach 0.9 on their own; weaker signals ("Before using this tool", mcp.json, a description over 1500 bytes, ...) count 0.5 each and only block together. The request is blocked when the largest untrusted score reaches INDIRECT_BLOCK_THRESHOLD. The audit row's injection_score and injection_label hold the strongest finding, direct or not. Untrusted text is limited to 128 KiB per request (413 above).

Spotlighting applies to tool results only:

The sanitizer strips spoofed markers from client text and raises the anomaly score when it finds one, so a tool result cannot close the block it sits in. An org can override scan_roles and spotlight through PUT /v1/organizations/:orgId/guard-settings; the gateway reads them on every request and fails closed with 503 if it cannot.

POST /guardrail/scan takes text, documents (at most 64, 128 KiB in total; source is web, email, file, tool or other), or both. The response has injection_score, injection_label, anomaly_score, had_tag_chars, had_bidi, bytes_removed, sanitized_text, blocked, verdict and a documents[] entry per document with its own score, category, detector_category, rules, blocked and verdict. A document is blocked at INDIRECT_BLOCK_THRESHOLD or when its anomaly score reaches 0.6. (toxicity_score is still in the response for compatibility and is always 0.)

POST /guardrail/scan/tools checks each tool's description and every description nested in inputSchema (at most 256 tools). Each tool gets score, category, signals, blocked, verdict and fields (the descriptions that scored at least 0.3 or carried a signal, by path).

go run ./cmd/eval -path tool scores the eval set through this untrusted-content path.

Per-organization exemplars#

/v1/organizations/:orgId/exemplars stores {"label": "block"|"allow", "text": "...", "source": "..."} (at most 1,000 per org, 4,000 characters each). A block text counts as an attack for that org's requests (injection.category tenant_exemplar), and its calibrated L3 score is a floor for the fused score; an allow text lowers L3 only. The gateway embeds each org's exemplars once and reloads them when they change (checked at most every 5 s). They need the L3 model. Tenant exemplars live in the database only, never in this repository.

Metrics#

Both services serve Prometheus metrics at /metrics on a listener of their own (METRICS_ADDR), so the endpoint is not reachable through the API port. Keep that port on the internal network.

The control plane exports guardspec_control_plane_requests_total, guardspec_control_plane_request_duration_seconds, guardspec_audit_retention_runs_total{result} and guardspec_audit_retention_deleted_rows_total. Labels take values from fixed sets only; API key IDs are never a label.

Tracing#

OpenTelemetry tracing is off unless OTEL_EXPORTER_OTLP_ENDPOINT (or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT) is set. Spans go over OTLP/HTTP; gRPC is not supported. The gateway continues a W3C traceparent from the client and creates a server span per request with children sanitize, detect, policy, upstream, output_guard and output_guard.stream. OTEL_PROPAGATE_UPSTREAM=true sends traceparent to the upstream; it is off by default because it tells the upstream the trace ID.

Rate limits#

Each API key gets two token buckets: requests per minute and, optionally, tokens per minute (estimated as body bytes / 4). A key's own rpm_limit and tpm_limit win; keys without one get RATE_LIMIT_RPM / RATE_LIMIT_TPM. Over the limit: 429 with Retry-After and {"error":{"message":"...","type":"requests","code":"rate_limit_exceeded"}} (type is tokens for the token limit); no audit row. Buckets are kept in memory per gateway replica, and a key's limits are cached for 30 seconds.

POST  /v1/organizations/{id}/api-keys           {"alias":"bot","rpm_limit":600,"tpm_limit":200000}
PATCH /v1/organizations/{id}/api-keys/{prefix}  {"rpm_limit":null}   # back to RATE_LIMIT_RPM

Users, invites and projects#

Admins invite people with POST /v1/orgs/{id}/invites {"email":"a@example.com","role":"member","ttl_hours":72}. No e-mail is sent: the response carries a one-time gsinv_... token that the admin passes on, and POST /v1/invites/accept {"token":"...","password":"...","display_name":"A"} creates the user (or adds the membership to an existing account whose password matches). Only the token's SHA-256 is stored. Nobody can invite with a role above their own.

A project groups API keys inside an organization. A key or a policy created with a project_id belongs to that project; the gateway evaluates the project's policies first, then the org-wide ones.

Identity, SSO and two-factor authentication#

Sessions. A login creates a row in sessions; the token (12 hours) carries its id as sid. Logout, DELETE /v1/auth/sessions/:id, a password change (other sessions), disabling a user and an admin's revoke-sessions end sessions at once on the replica that made the change and within 15 seconds on the others. Tokens issued before migration 013 have no sid and are refused: everyone logs in again after the upgrade.

Two-factor authentication (TOTP, RFC 6238: SHA-1, 6 digits, 30 s, one step of skew). POST /v1/auth/mfa/totp/setup returns the secret and an otpauth:// URL (the UI draws the QR code itself); .../confirm {code} turns it on and returns 10 recovery codes, shown once and stored as bcrypt hashes. Enrollment is trust on first use: whoever completes it at the first login after an org requires 2FA (the account's password holder) chooses the authenticator. A login of a user with 2FA returns {"mfa_required": true, "mfa_token", "methods": ["totp","recovery"]}; POST /v1/auth/mfa/verify {mfa_token, code} finishes it. The mfa_token is a 5-minute JWT allowing 5 attempts and one success; a TOTP code is refused for the step it was used in (and earlier ones), a recovery code works once. Ten wrong codes in a row lock the user's second factor for 15 minutes (doubling every 5 more, at most 24 hours; 429 mfa_locked, logged as mfa.locked), whatever the token or address. The secret is encrypted with MFA_ENC_KEY (AES-256-GCM, bound to the user id). Back the key up with the database: without it no user with 2FA can log in.

A member's 2FA reset and session revocation act on the user everywhere, so they need an instance administrator or an owner of every organization the user belongs to (403 not_sole_authority otherwise).

Org security (PUT /v1/organizations/:orgId/security, owner): require_mfa makes members without 2FA enroll at their next login ({"mfa_enroll_required": true, "mfa_token"}; that token can only call the 2FA setup endpoints) and refuses existing sessions without 2FA (403 mfa_required); sso_satisfies_mfa lets SSO sessions pass; allowed_login_methods (password, google, oidc:<provider id>; unset allows all) refuses sessions of other methods (403 login_method_not_allowed); session_lifetime_hours (1-12) shortens sessions for this org; domain_join adds SSO users of a verified email domain on first login, with a role up to admin. A domain can be joined only if the instance administrator listed it in the provider's allowed_domains.

SSO (OpenID Connect, Google as a preset). An instance administrator (users.is_instance_admin; the bootstrap admin, which is the earliest owner of the default org at migration 013) adds providers at /v1/admin/identity-providers {"type":"google","client_id":"...","client_secret_env":"OIDC_SECRET_GOOGLE","allowed_domains":["example.com"],"auto_provision":true}. Register <PUBLIC_BASE_URL>/v1/auth/oidc/<id>/callback as the redirect URI. The flow uses PKCE (S256), state and nonce (stored server-side for 10 minutes); the state is also bound to the browser by an HttpOnly, SameSite=Lax cookie set at start (login CSRF), so the UI navigates to .../start rather than fetching it. The callback verifies the ID token through the provider's JWKS and requires email_verified; Google logins with allowed_domains must also carry a matching hd claim. The callback redirects to /ui/#/auth/callback?code=<60 s one-time code> (or ?error=<code>), which POST /v1/auth/oidc/exchange turns into the same answer as a password login, so 2FA still applies. Identities match on (provider, subject). A new identity whose email matches an existing account links to it only when the provider has trust_email_for_linking (off by default) and the account has no password and no identity; otherwise the login is refused with account_exists, and the owner of the account links the identity from their profile: POST /v1/auth/identities/:providerId/link/start returns the provider's URL, the callback sends the browser to /ui/#/account?link_code=<60 s code>, and POST /v1/auth/identities/link/complete {code} with the same user's token binds it (a code is refused for anyone else, so a link URL sent to a victim links nothing). A manual link never runs domain_join. New users are created only with auto_provision and an allowed domain. Emails are stored lower-cased.

Activity log. Member, invite, API key, policy, upstream, security and config changes and 2FA resets are recorded per org (GET /v1/organizations/:orgId/activity); SSO provider and instance user changes, logins (success and failure) and the user's own 2FA events at GET /v1/admin/activity. Details never hold secrets; a failed login keeps the email masked (a***@example.com).

Bootstrap. ADMIN_EMAIL/ADMIN_PASSWORD create the first owner and instance administrator once. A later start does nothing if an instance administrator exists or the bootstrap ran before, so a deleted bootstrap admin stays deleted.

Audit retention#

The control plane deletes audit rows older than AUDIT_RETENTION_DAYS in batches, at start and then every AUDIT_RETENTION_INTERVAL. Every replica runs the job; a Postgres advisory lock lets one of them work at a time.

Kubernetes#

The chart is in deploy/helm/guard-spec. It deploys the gateway and the control plane against an external Postgres; Postgres is not bundled. Its optional parts are off by default (the defaults render only the two Deployments, their Services and the models PVC); values.yaml documents each.

kubectl create secret generic guard-spec-secrets \
  --from-literal=POSTGRES_PASSWORD=... --from-literal=GUARDSPEC_APP_PASSWORD=... \
  --from-literal=JWT_SECRET=... --from-literal=ADMIN_PASSWORD=... \
  --from-literal=MFA_ENC_KEY="$(openssl rand -base64 32)"   # back it up with the database
helm install gs deploy/helm/guard-spec \
  --set externalDatabase.url='postgres://guardspec_app@db.example:5432/guardspec?sslmode=require' \
  --set externalDatabase.host=db.example \
  --set migrations.enabled=true --set dbRoles.enabled=true \
  --set controlPlane.adminEmail=admin@example.com
kubectl exec deploy/gs-guard-spec-gateway -c gateway -- ./doctor   # checks the install (images after 0.1.1)

• Migrations: migrations.enabled=true applies migrations/ (copied from the gateway image of the release) with golang-migrate in a pre-install / pre-upgrade hook, before dbRoles sets the application role password. It is off by default, as before: then migrate yourself first, as the owner role (migrate -path migrations -database "postgres://ps:...@host/guardspec" up). The services never migrate at startup.
• Cloud and OpenShift examples: deploy/helm/guard-spec/examples/ has values for EKS, AKS, GKE and OpenShift (random UIDs, Routes). They are rendered and schema-checked in CI but not verified on those platforms.
• Exposure and scaling: ingress (or route on OpenShift), gateway.autoscaling / controlPlane.autoscaling (HPA), *.podDisruptionBudget, *.topologySpreadConstraints, networkPolicy (API and metrics ports per source; optional egress restriction to DNS, the database and listed destinations) and serviceAccount (annotations for EKS IRSA, GKE or AKS Workload Identity; the API token is never mounted).
• Shadow mode and model reload: gateway.shadow (off while model and modelDir are empty), gateway.modelsManifest (MODELS_MANIFEST) and gateway.adminToken.enabled (GATEWAY_ADMIN_TOKEN from existingSecret, off by default). The control plane gets GATEWAY_URL pointing at the gateway Service (controlPlane.gatewayURL; - leaves it unset), and with networkPolicy on, the chart lets it reach the gateway's port 8080. See Shadow mode and model reload. doctor reads the manifest of MODELS_MANIFEST too, unless -manifest is given.
• Air-gapped or mirrored registries: docs/airgapped.md mirrors the images with their cosign signatures and attestations and the models, verifies them offline, points the chart at the mirrors and shows an External Secrets example.
• doctor (/app/doctor in the gateway and control-plane images released after 0.1.1, and in the release archives) checks the database, the migration version, the model files against the manifest's SHA-256, which detection layers will load, the upstream and the security settings. Exit code 0: nothing failed (warnings allowed); 1: a check failed (-strict: or warned); 2: usage error. -json for scripts.
• Secrets come from existingSecret. GUARDSPEC_APP_PASSWORD reaches both services as PGPASSWORD, so URLs carry no password. POSTGRES_PASSWORD is used only by the optional migrations and dbRoles hook jobs. MFA_ENC_KEY (see Configuration) is optional in that Secret; it can come from controlPlane.extraEnvFrom instead.
• Metrics are served on separate ClusterIP Services (*-metrics), so exposing an API Service never exposes /metrics. serviceMonitor.enabled=true adds a Prometheus Operator ServiceMonitor.
• Models come from a PVC mounted read-only at /models in the gateway (models.existingClaim, or the chart's own with models.persistence.enabled=true, the default). The release gateway image is built with -tags onnx and runs L1 + L2 + L3 when the models are there; with an empty PVC it runs L1 only. With models.persistence.enabled=false and no existingClaim nothing is mounted at /models: use that with gateway.image.repository=ghcr.io/guard-spec/guard-spec-gateway-full, which has the models inside. models.pull.enabled=true adds an init container that runs ./models pull (Hugging Face, or models.pull.mirror) into the claim, or into a per-pod emptyDir without one, checking every file's SHA-256 against the manifest; a ReadWriteOnce claim serves one node only, so several replicas on several nodes need ReadWriteMany or the emptyDir. The readiness probe uses /readyz, which lists the active layers and answers 503 only when a layer the configuration requires (DETECTOR_REQUIRE_ML=true) is missing.
• Resources: the gateway defaults (request 500m CPU / 1Gi, limit 2Gi) fit L1 + L2 + L3, which use about 900 MiB right after start.
• Security: pods run as non-root with a read-only root filesystem, no capabilities and the RuntimeDefault seccomp profile. The dbRoles job runs as UID 70 (the postgres image's user) unless dbRoles.podSecurityContext sets one; on OpenShift it lets the SCC pick the UID.
• Rate limits are per replica, so gateway.replicaCount (or the autoscaler) multiplies the effective limit.
• Tested by the kind-e2e CI job (tests/kind/e2e.sh): images built from the change, an in-cluster Postgres and fake upstream, the migrations and dbRoles hooks, network policies, and EN / VI injections blocked with L1 only.

Source: README.md#operations @ ca69dad29f02