Routing

An organization defines named upstreams (OpenAI-compatible APIs). For a request the guard allowed, the gateway picks one in this order:

the route_target of the route rule that decided the request;
the default upstream of the API key's project;
the default upstream of the organization;
UPSTREAM_LLM_URL;
the stub answer.

A target or default that is missing, disabled, or whose key variable is not set or not allowed for the org fails the request closed with 502 (route_unavailable); the gateway never falls back to another upstream. The audit row records the upstream's name in route_target (null for UPSTREAM_LLM_URL and the stub), the action route when a route rule decided, and in model the model actually sent upstream (the upstream's model when it rewrote the request; policies still see the requested one). Upstreams are read per request in the same transaction as the policies, so a change applies to the next request.

An upstream is {"name", "base_url", "model", "api_key_env", "timeout_ms", "enabled"}. name is lowercase letters, digits and dashes, unique per org. When model is set, the gateway replaces the request's model with it. timeout_ms defaults to 300000. No secret is stored: api_key_env names a gateway environment variable (UPSTREAM_KEY_[A-Z0-9_]{1,64}) that holds the key, so set it in the gateway's environment (compose: add it under gateway.environment).

Which org may use which key is the operator's decision, in UPSTREAM_KEY_ORGS on both services:

UPSTREAM_KEY_ORGS=UPSTREAM_KEY_MAIN=*;UPSTREAM_KEY_ACME=<org-uuid>,<org-uuid>

* means every organization and has to be written out; the default is empty, so no org can name any key variable. Without this, any user could create an org, point an upstream at a host they control with api_key_env: UPSTREAM_KEY_MAIN, and receive your key. The control plane refuses an upstream or config file that names a variable not allowed for the org (400, or 422 for the file), and the gateway checks again on every request, so rows saved before you narrowed the list fail closed (502).

Addresses. base_url must be http(s) without credentials, a query or a fragment. Link-local, multicast and unspecified addresses and the cloud metadata addresses (169.254.0.0/16, fd00:ec2::254, 100.100.100.200, 192.0.0.192, metadata.google.internal) are always refused, also after DNS when the gateway connects. Loopback, RFC 1918, unique local (fc00::/7) and CGNAT (100.64.0.0/10) addresses are refused at connect time unless UPSTREAM_ALLOW_PRIVATE=true. The trade-off: with true, org admins can reach anything on the gateway's network (an Ollama next to it, but also internal services); with false, org upstreams must be public. Your own UPSTREAM_LLM_URL is never restricted. Compose sets true so that Ollama on the host works.

# An upstream, the org default, and a rule that sends suspicious requests to a local model
curl -s $CP/organizations/$ORG/upstreams -H "$AUTH" -H 'Content-Type: application/json' \
  -d '{"name":"safe-model","base_url":"http://ollama:11434/v1","model":"qwen3:4b"}'
curl -s -X PUT $CP/organizations/$ORG/default-upstream -H "$AUTH" -H 'Content-Type: application/json' \
  -d '{"upstream_id":"<id of another upstream>"}'
TREE=$(curl -s $CP/policy/parse -H "$AUTH" -H 'Content-Type: application/json' \
  -d '{"text":"injection.score > 0.5"}' | jq .condition)
curl -s $CP/organizations/$ORG/policies -H "$AUTH" -H 'Content-Type: application/json' \
  -d "{\"name\":\"routing\",\"rules\":[{\"name\":\"suspicious\",\"action\":\"route\",\"route_target\":\"safe-model\",\"condition\":$TREE}]}"

# What would happen: every rule's outcome, the decision and the upstream
curl -s $CP/organizations/$ORG/policies/simulate -H "$AUTH" -H 'Content-Type: application/json' \
  -d '{"vars":{"injection.score":0.7,"request.model":"gpt-4o"}}' | jq .

Deleting an upstream that a rule or a default still uses returns 409 with the users in references; so does renaming one that a rule names.

Simulator. POST /organizations/:orgId/policies/simulate takes {"policy_id", "rules", "project_id", "vars"}: rules (an unsaved rule list) or policy_id (one policy, even disabled) or neither (the enabled policies the gateway would run for a key of project_id, in its order). vars sets any schema field; fields not given are zero. It returns trace (every rule, with matched and error), decision (action is error when the gateway would answer 503, for example a rule that does not compile) and upstream (source is rule, project, org, env or stub; error is set when the gateway would answer 502). Nothing is stored or sent upstream.

Config as code. GET /organizations/:orgId/config returns the org's upstreams, defaults and policies as YAML; PUT applies such a file in one transaction, and ?dry_run=true only reports the changes:

version: 1
default_upstream: main
project_defaults:          # project slug: upstream
  web: safe-model
upstreams:
  - name: main
    base_url: https://api.openai.com/v1
    api_key_env: UPSTREAM_KEY_MAIN
  - name: safe-model
    base_url: http://ollama:11434/v1
    model: qwen3:4b
policies:
  - name: block-injection
    project: null            # or a project slug
    enabled: true
    rules:
      - name: high l2
        when: scores.l2 > 0.9 && injection.category != "benign"
        action: block
      - name: suspicious to safe model
        when: injection.score > 0.5
        action: route
        route_target: safe-model

The file is the whole truth: upstreams and policies it does not list are deleted, and defaults it does not set are cleared (projects themselves are not created or deleted). Objects are matched by name; a rule keeps its ID when its name is unchanged inside the same policy, so audit rows keep naming it. The order of policies in the file is their evaluation order. Every when is parsed like /policy/parse and validated like a saved rule; a when that starts with ! must be quoted, since YAML reads ! as a tag. Errors come back as 422 {"errors":[{"path":"policies[0].rules[1].when","message":"..."}]} and nothing is written; success is {"applied":true,"changes":[{"kind","name","op"}]}.

GET refuses with 409 when the stored config would not apply again: two policies with the same name (the file names policies by name), or rules that no longer validate, such as rules naming a removed field. The body lists each problem with the policy's id and the policy API call that fixes it ({"error", "problems": [{"policy_id", "policy", "message", "fix"}]}). Policy names are unique per org: creating a policy, or renaming one, onto a name another policy has returns 409.

curl -s $CP/organizations/$ORG/config -H "$AUTH" > guard.yaml
curl -s -X PUT "$CP/organizations/$ORG/config?dry_run=true" -H "$AUTH" -H 'Content-Type: text/yaml' --data-binary @guard.yaml

Source: README.md#routing @ ca69dad29f02