Token & Cost Control¶

Two of the nine detection axes answer a commercial question rather than a safety one. out_of_scope refuses off-topic traffic before the upstream call, so those tokens are never billed. prompt_complexity routes easy prompts to a cheap model and hard ones to a capable one. Both come from the pass you are already paying for — no extra latency, no extra model call. A third lever, the token_saving switch, cuts upstream spend without changing what the model reads.

out_of_scope — refusing before you pay¶

An LLM charges you for every token it reads and every token it writes. A question your assistant was never meant to answer costs you the full price of a wrong answer — and then costs you again in support, in trust, and (for a regulated deployment) in an audit record you have to explain.

out_of_scope moves that decision in front of the upstream call. When the axis is part of the blocking set, the gateway returns a refusal directly and the request never leaves the building:

prompt ──▶ detector (one pass) ──▶ out_of_scope 0.998 ──▶ refusal
                                                          │
                                        upstream LLM ─────┘ never called
                                        input tokens: 0
                                        output tokens: 0

Declaring the scope¶

The axis is conditional by construction: "off topic" has no meaning until somebody says what the topic is. The scope is read from the system message(s) of the incoming request, and in G-1 Studio the Application's policy.scope field supplies the default:

curl -s -X PUT http://localhost:8080/v1/glad/apps/travel_support/policy \
  -H "Content-Type: application/json" \
  -d '{
    "policy": {
      "scope": "You are a customer-support assistant for a travel booking website. You help travellers search flights, hotels and car rentals, change or cancel bookings, and explain fare rules, baggage allowances and refund policies. Stay strictly on travel topics."
    }
  }'

A client that sends its own system message overrides it for that conversation. Geodesia's own injected constitutional prompt is added after this read and can never be mistaken for the scope.

Turning it into an actual block¶

out_of_scope is an additional axis: it ships annotate-only, and the gateway-wide GW_PROMPT_BLOCK_AXES will not promote it — that switch is global and silent, and an annotate-grade signal should not gain the power to refuse every customer's traffic from one line in a deployment file.

Enable it where the decision belongs: on the Application that wants it, through its enforcement policy.

curl -X PATCH "$G1/v1/glad/apps/$APP_ID" \
  -H "Authorization: Bearer $ADMIN_KEY" -H "Content-Type: application/json" \
  -d '{"policy": {"enforcement": {"out_of_scope": "block"}}}'

(The same control is a dropdown in g1-studio → Applications → Policy → Enforcement.) Scoping it to one Application is the point: an assistant with a tight declared scope wants the refusal, a general-purpose one does not, and the two can share a gateway.

The refusal comes back in the shape the caller expects — an OpenAI finish_reason: "content_filter" response, an Ollama done message, or a stream frame for streaming clients — with the full verdict attached:

{
  "id": "geodesia-block-1754131200000",
  "choices": [{"index": 0,
               "message": {"role": "assistant",
                           "content": "[Geodesia blocked — out of scope (input)]"},
               "finish_reason": "content_filter"}],
  "geodesia": {
    "schema_version": "1.0",
    "event": "final",
    "decision": "blocked",
    "mode": "blocking",
    "reason": {"stage": "input", "axis": "out_of_scope", "detail": null},
    "additional_axes": {
      "out_of_scope": {"score": 0.998, "threshold": 0.90, "flagged": true, "available": true, "role": "advisory"}
    }
  }
}

(Other axes omitted.) reason.axis names the axis that refused the request; the full object is described in the Response Format reference.

The blocked call is still written to the audit ledger with its axis scores — you keep the compliance record without paying for the generation.

What it is worth¶

The saving is exact and you can read it off your own traffic: every request the axis refuses is a request whose input and output tokens you did not buy. Because the gateway logs blocked calls with prompt_blocked = 1, the Cost & FinOps view can show refused volume next to spend, and the Application metrics endpoint reports it directly:

curl -s http://localhost:8080/v1/glad/apps/travel_support/metrics | jq

{ "total": 12840, "prompt_blocked": 1173, "answer_blocked": 12, "hallucinated": 41, "grounded": 9902 }

Complexity routing (Model A → Model B)¶

Most production traffic is easy. "What time is it in Rome?", "Where is my order?", "Summarise this paragraph" — these do not need the frontier model you are paying frontier prices for. A minority of prompts genuinely do.

prompt_complexity is a binary complex / simple classifier over the user's prompt. When routing is enabled, the gateway compares it against a threshold and picks the upstream binding accordingly:

The common case is same provider, same key, different model — so setting complex_binding.model alone is enough. Model B can also be a completely different provider (a local Ollama for the cheap tier, a hosted API for the hard tier), in which case you set its upstream_type, base_url and key too.

Where it runs in the pipeline¶

Routing happens after the input-block decision and before generation:

1. one detector pass → safety axes + out_of_scope + prompt_complexity;
2. if an input guardrail fired → refuse (a blocked prompt is never routed — you do not pay to route something you refused);
3. if p(prompt_complexity) > threshold → swap in Model B's binding;
4. forward, generate, validate the answer as usual.

The model that actually answered is echoed in the response payload and written to the cost ledger — not the configured default — so your FinOps numbers stay honest when routing is on.

Configuring it¶

In G-1 Studio → Applications, the "Route complex prompts to a different model" toggle reveals the threshold slider and the Model B fields, with a ↻ discover button that lists the models available on that upstream. If the served checkpoint has no prompt_complexity axis, the Studio shows an amber warning and the controls stay inert.

Over the API:

curl -s -X PUT http://localhost:8080/v1/glad/apps/support_bot/routing \
  -H "Content-Type: application/json" \
  -d '{
    "complex_routing": {
      "enabled": true,
      "threshold": 0.5,
      "complex_binding": { "model": "gpt-5" }
    }
  }'

{
  "complex_routing": {
    "enabled": true,
    "threshold": 0.5,
    "complex_binding": {"upstream_type": null, "base_url": null, "model": "gpt-5",
                        "region": null, "api_key": "***"}
  },
  "config_version": 7
}

Set the Model B token rates in the Application's cost block so the ledger prices each call at the rate of the model that answered:

curl -s -X PUT http://localhost:8080/v1/glad/apps/support_bot/cost \
  -H "Content-Type: application/json" \
  -d '{"cost": {"input_per_mtok": 0.15, "output_per_mtok": 0.60,
                "complex_input_per_mtok": 1.25, "complex_output_per_mtok": 10.00}}'

Reading the routing decision¶

Whenever routing is enabled, the response's geodesia payload carries a routing block — which axis decided, what it scored, the threshold it was compared against, and which model actually answered:

"geodesia": {
  "routing": {
    "enabled": true,
    "used_complex_model": true,
    "axis": "prompt_complexity",
    "score": 0.8134,
    "threshold": 0.5,
    "model": "gpt-5"
  }
}

used_complex_model: false with axis_missing: true means the served detector has no prompt_complexity axis and the request fell back to Model A. Because the block is present on every routed call, the trade-off stays auditable after the fact — you can always show why a given answer came from the cheap model.

Failure behaviour¶

Routing never fails a request. If the axis is unavailable — an older checkpoint, GB_EXTRA_AXES not enabled — the gateway logs a distinct [complex-routing] axis 'prompt_complexity' unavailable marker and falls back to Model A. That is a silent cost regression rather than an outage (all "complex" traffic collapses onto the cheap model), which is exactly why it is logged loudly. Check supports_axis.prompt_complexity on GET /v1/glad/apps/meta before relying on it.

Token saving — spend less without changing what the model reads¶

token_saving is a platform-wide switch, off by default, available from 0.4.2. Turning it on never changes a byte of the messages the upstream model receives, and never changes the text your client gets back. It does not compress, prune, summarise or reorder anything: every one of those techniques changes the answer.

Turning it on¶

curl -s -X POST http://localhost:8800/v1/glad/gateway/config \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $GW_API_TOKEN" \
  -d '{"token_saving": true}'

The change applies to the next request and is saved to GW_CONFIG_FILE, so it survives a restart. Send {"token_saving": false} to turn it off again. The Authorization header is needed only when the gateway runs with GW_API_TOKEN set. Two other ways to set it:

• GW_TOKEN_SAVING=1 at boot;
• the Token saving checkbox in G-1 Studio under Settings → Gateway.

Check the current value:

curl -s http://localhost:8800/v1/glad/gateway/config | jq .token_saving

What it does¶

Two things the switch deliberately leaves out:

• Azure OpenAI. It does not document prompt_cache_key, and its strict schema rejects unknown fields with a 400.
• Bedrock cachePoint markers. Bedrock bills a cache write at 1.25× the input price, so a prefix that is never read again would cost more, not less.

Reading the result¶

"geodesia": {
  "decision": "allowed",
  "token_saving": {
    "enabled": true,
    "prompt_tokens": 1500,
    "cached_prompt_tokens": 1280,
    "prompt_cache_key": "gateway"
  }
}

With the switch off the section is absent and the response is byte-identical to earlier releases.

What it is worth, measured¶

We measured it with two real gateways, one with the switch on and one with it off, in front of one OpenAI-compatible upstream that logs exactly when its connection drops:

• Identical traffic. The upstream received the same messages from both gateways. The two payloads differed only by prompt_cache_key, and the clients received the same text.
• Bedrock is where the saving is real. Without the switch, a halted stream left the SDK worker reading until its 256-event queue was full, then hanging with the connection open. With a simulated stream at 10 ms per event, the worker read 260 events and never closed without the switch, and read 4 events then closed with it.
• Over HTTP the early close did not save tokens. Python already drops the connection when the handler returns. After a halt the upstream generated the same number of tokens with and without the switch: 12 on a system-prompt leak, 73 on an energy barrier. The switch makes that close deterministic and visible in the log.
• The OpenAI hit-rate gain from prompt_cache_key depends on your traffic and was not measured against the live API. Read cached_prompt_tokens / prompt_tokens before and after turning the switch on.

Choosing the threshold¶

The two axes want opposite kinds of tuning, because their errors cost different things.

Both are per-Application. A public support bot, an internal research assistant and a compliance reviewer can share one deployment and one detector while drawing these lines in completely different places.

See also¶

• Detection Axes — what all nine axes score and how enforcement is grouped
• Policy Lens — set these thresholds against your own traffic, with the counterfactual computed before you apply it
• Cost & FinOps — spend, forecast and budget enforcement per Application
• Managing Applications — where scope, complex_routing and the cost rates live in the config