Token & Cost Control¶

Two of the nine detection axes answer a commercial question rather than a safety one. out_of_scope refuses off-topic traffic before the upstream call, so those tokens are never billed. prompt_complexity routes easy prompts to a cheap model and hard ones to a capable one. Both come from the pass you are already paying for — no extra latency, no extra model call.

out_of_scope — refusing before you pay¶

An LLM charges you for every token it reads and every token it writes. A question your assistant was never meant to answer costs you the full price of a wrong answer — and then costs you again in support, in trust, and (for a regulated deployment) in an audit record you have to explain.

out_of_scope moves that decision in front of the upstream call. When the axis is part of the blocking set, the gateway returns a refusal directly and the request never leaves the building:

prompt ──▶ detector (one pass) ──▶ out_of_scope 0.998 ──▶ refusal
                                                          │
                                        upstream LLM ─────┘ never called
                                        input tokens: 0
                                        output tokens: 0

Declaring the scope¶

The axis is conditional by construction: "off topic" has no meaning until somebody says what the topic is. The scope is read from the system message(s) of the incoming request, and in G-1 Studio the Application's policy.scope field supplies the default:

curl -s -X PUT http://localhost:8080/v1/glad/apps/travel_support/policy \
  -H "Content-Type: application/json" \
  -d '{
    "policy": {
      "scope": "You are a customer-support assistant for a travel booking website. You help travellers search flights, hotels and car rentals, change or cancel bookings, and explain fare rules, baggage allowances and refund policies. Stay strictly on travel topics."
    }
  }'

A client that sends its own system message overrides it for that conversation. Geodesia's own injected constitutional prompt is added after this read and can never be mistaken for the scope.

Turning it into an actual block¶

out_of_scope is an additional axis: it ships annotate-only, and the gateway-wide GW_PROMPT_BLOCK_AXES will not promote it — that switch is global and silent, and an annotate-grade signal should not gain the power to refuse every customer's traffic from one line in a deployment file.

Enable it where the decision belongs: on the Application that wants it, through its enforcement policy.

curl -X PATCH "$G1/v1/glad/apps/$APP_ID" \
  -H "Authorization: Bearer $ADMIN_KEY" -H "Content-Type: application/json" \
  -d '{"policy": {"enforcement": {"out_of_scope": "block"}}}'

(The same control is a dropdown in g1-studio → Applications → Policy → Enforcement.) Scoping it to one Application is the point: an assistant with a tight declared scope wants the refusal, a general-purpose one does not, and the two can share a gateway.

The refusal comes back in the shape the caller expects — an OpenAI finish_reason: "content_filter" response, an Ollama done message, or a stream frame for streaming clients — with the full verdict attached:

{
  "id": "geodesia-block-1754131200000",
  "choices": [{"index": 0,
               "message": {"role": "assistant",
                           "content": "[Geodesia blocked — out of scope (input)]"},
               "finish_reason": "content_filter"}],
  "glad_decision": "blocked",
  "geodesia": {
    "glad_decision": "blocked",
    "flagged_axis": "out_of_scope",
    "axis_energy": {"out_of_scope": {"p_detector": 0.998, "threshold": 0.90, "flag": true}}
  }
}

The blocked call is still written to the audit ledger with its axis scores — you keep the compliance record without paying for the generation.

What it is worth¶

The saving is exact and you can read it off your own traffic: every request the axis refuses is a request whose input and output tokens you did not buy. Because the gateway logs blocked calls with prompt_blocked = 1, the Cost & FinOps view can show refused volume next to spend, and the Application metrics endpoint reports it directly:

curl -s http://localhost:8080/v1/glad/apps/travel_support/metrics | jq

{ "total": 12840, "prompt_blocked": 1173, "answer_blocked": 12, "hallucinated": 41, "grounded": 9902 }

Complexity routing (Model A → Model B)¶

Most production traffic is easy. "What time is it in Rome?", "Where is my order?", "Summarise this paragraph" — these do not need the frontier model you are paying frontier prices for. A minority of prompts genuinely do.

prompt_complexity is a binary complex / simple classifier over the user's prompt. When routing is enabled, the gateway compares it against a threshold and picks the upstream binding accordingly:

The common case is same provider, same key, different model — so setting complex_binding.model alone is enough. Model B can also be a completely different provider (a local Ollama for the cheap tier, a hosted API for the hard tier), in which case you set its upstream_type, base_url and key too.

Where it runs in the pipeline¶

Routing happens after the input-block decision and before generation:

1. one detector pass → safety axes + out_of_scope + prompt_complexity;
2. if an input guardrail fired → refuse (a blocked prompt is never routed — you do not pay to route something you refused);
3. if p(prompt_complexity) > threshold → swap in Model B's binding;
4. forward, generate, validate the answer as usual.

The model that actually answered is echoed in the response payload and written to the cost ledger — not the configured default — so your FinOps numbers stay honest when routing is on.

Configuring it¶

In G-1 Studio → Applications, the "Route complex prompts to a different model" toggle reveals the threshold slider and the Model B fields, with a ↻ discover button that lists the models available on that upstream. If the served checkpoint has no prompt_complexity axis, the Studio shows an amber warning and the controls stay inert.

Over the API:

curl -s -X PUT http://localhost:8080/v1/glad/apps/support_bot/routing \
  -H "Content-Type: application/json" \
  -d '{
    "complex_routing": {
      "enabled": true,
      "threshold": 0.5,
      "complex_binding": { "model": "gpt-5" }
    }
  }'

{
  "complex_routing": {
    "enabled": true,
    "threshold": 0.5,
    "complex_binding": {"upstream_type": null, "base_url": null, "model": "gpt-5",
                        "region": null, "api_key": "***"}
  },
  "config_version": 7
}

Set the Model B token rates in the Application's cost block so the ledger prices each call at the rate of the model that answered:

curl -s -X PUT http://localhost:8080/v1/glad/apps/support_bot/cost \
  -H "Content-Type: application/json" \
  -d '{"cost": {"input_per_mtok": 0.15, "output_per_mtok": 0.60,
                "complex_input_per_mtok": 1.25, "complex_output_per_mtok": 10.00}}'

Reading the routing decision¶

Whenever routing is enabled, the response's geodesia payload carries a routing block — which axis decided, what it scored, the threshold it was compared against, and which model actually answered:

"geodesia": {
  "routing": {
    "enabled": true,
    "used_complex_model": true,
    "axis": "prompt_complexity",
    "score": 0.8134,
    "threshold": 0.5,
    "model": "gpt-5"
  }
}

used_complex_model: false with axis_missing: true means the served detector has no prompt_complexity axis and the request fell back to Model A. Because the block is present on every routed call, the trade-off stays auditable after the fact — you can always show why a given answer came from the cheap model.

Failure behaviour¶

Routing never fails a request. If the axis is unavailable — an older checkpoint, GB_EXTRA_AXES not enabled — the gateway logs a distinct [complex-routing] axis 'prompt_complexity' unavailable marker and falls back to Model A. That is a silent cost regression rather than an outage (all "complex" traffic collapses onto the cheap model), which is exactly why it is logged loudly. Check supports_axis.prompt_complexity on GET /v1/glad/apps/meta before relying on it.

Choosing the threshold¶

The two axes want opposite kinds of tuning, because their errors cost different things.

Both are per-Application. A public support bot, an internal research assistant and a compliance reviewer can share one deployment and one detector while drawing these lines in completely different places.

See also¶

• Detection Axes — what all nine axes score and how enforcement is grouped
• Policy Lens — set these thresholds against your own traffic, with the counterfactual computed before you apply it
• Cost & FinOps — spend, forecast and budget enforcement per Application
• Managing Applications — where scope, complex_routing and the cost rates live in the config