Framework Compatibility & DevOps Guide¶

G1-Proxy sits in front of your inference server as a thin, stateless HTTP hop — it never loads your weights and never replaces your serving stack. Any framework that speaks the OpenAI or Ollama wire format works. The one thing that varies between them is whether they expose per-token log-probabilities, which decides whether the closed-book axis can run.

Configure It — Three Equivalent Ways¶

Every framework below is configured identically; only the values change.

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -H "Content-Type: application/json" \
  -d '{
    "upstream_type": "vllm",
    "upstream_base_url": "http://generator:8000",
    "upstream_model": "meta-llama/Llama-3-8B-Instruct",
    "upstream_api_key": ""
  }'

GW_UPSTREAM_TYPE=vllm
GW_UPSTREAM_URL=http://generator:8000
GW_UPSTREAM_MODEL=meta-llama/Llama-3-8B-Instruct
GW_API_KEY=                 # only for hosted services
GW_BLOCK_INPUT=1            # enforce on input
GW_BLOCK_OUTPUT=1           # enforce on output

Health, Readiness & Probes¶

The gateway's /health is designed to be both a liveness and a readiness signal.

curl http://localhost:8800/health

{
  "ok": true,
  "upstream_type": "vllm",
  "upstream": "http://localhost:8000",
  "internal_vllm": "stopped",
  "logprobs": true,
  "axes": 5
}

The product backend exposes its own lightweight check at GET /v1/glad/health for the compliance plane.

At a Glance¶

The One Rule That Decides Everything: Log-Probabilities¶

There is exactly one upstream capability that changes how many detection axes you get:

Does the upstream return per-token log-probabilities?

Log-probabilities express how confident the model was when it chose each word. Geodesia uses them for the closed-book fabrication axis (catching confidently-invented facts when there is no grounding context).

No setting toggles this — the gateway probes it on the first request and /health reports the result. You can recover the 5^th axis later with a log-prob sidecar.

Compatibility Matrix¶

* Hosted providers expose log-probabilities on most—but not all—models and tiers; the gateway falls back to 8 axes automatically when they are absent.

Where Geodesia Sits¶

The gateway is the only thing your application talks to. It is stateless; all durable state lives in the shared audit database.

Choosing the Right Type¶

Anything OpenAI-compatible that is not vLLM / SGLang / TensorRT-LLM uses type: openai.

Per-Framework Setup¶

vLLM¶

The recommended production backend on NVIDIA GPUs. Returns log-probabilities natively → all 9 axes.

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3-8B-Instruct --port 8000

docker run --gpus all -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3-8B-Instruct

Configure:

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{"upstream_type":"vllm","upstream_base_url":"http://localhost:8000",
       "upstream_model":"meta-llama/Llama-3-8B-Instruct"}'

SGLang¶

OpenAI-compatible, log-probabilities supported → 5 axes. Default port 30000.

python -m sglang.launch_server \
  --model-path meta-llama/Llama-3-8B-Instruct --port 30000

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{"upstream_type":"sglang","upstream_base_url":"http://localhost:30000",
       "upstream_model":"meta-llama/Llama-3-8B-Instruct"}'

TensorRT-LLM¶

NVIDIA's maximum-performance engine, fronted by an OpenAI-compatible server (TensorRT-LLM serving frontend or Triton). Log-probabilities supported → 5 axes.

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{"upstream_type":"trtllm","upstream_base_url":"http://localhost:8000",
       "upstream_model":"llama-3-8b"}'

llama.cpp¶

llama-server is OpenAI-compatible and returns log-probabilities → 5 axes, even on CPU or Apple Silicon. Use type: openai.

./llama-server -m ./models/llama-3-8b-instruct.Q4_K_M.gguf \
  --host 0.0.0.0 --port 8080

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{"upstream_type":"openai","upstream_base_url":"http://localhost:8080",
       "upstream_model":"llama-3-8b-instruct","upstream_api_key":""}'

Ollama¶

The easiest local runtime. Ollama ≥ 0.12 exposes per-token log-probabilities on its OpenAI-compatible /v1/chat/completions endpoint, so the closed-book fabrication axis is enabled automatically — the proxy probes that endpoint on the first request and turns it on with no extra configuration.

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{"upstream_type":"ollama","upstream_base_url":"http://localhost:11434",
       "upstream_model":"llama3.2"}'

Older Ollama — recover the closed-book axis with a log-prob sidecar¶

On Ollama < 0.12 (no native log-probs) the closed-book axis is gated and every other axis runs normally. You can recover it by running a second server for the same model that does expose log-probabilities — typically llama.cpp on the same GGUF. The gateway re-derives the answer through the sidecar to recover the signal (only used when native log-probs are absent).

./llama-server -m ./models/llama3.2.gguf --port 8080   # the sidecar

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{"upstream_type":"ollama","upstream_base_url":"http://localhost:11434",
       "upstream_model":"llama3.2",
       "ollama_logprob_sidecar_url":"http://localhost:8080",
       "ollama_logprob_sidecar_model":"llama3.2"}'

OpenAI API and Hosted Services¶

Use type: openai for the OpenAI API and any hosted OpenAI-compatible provider.

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{"upstream_type":"openai","upstream_base_url":"https://api.openai.com",
       "upstream_api_key":"sk-...","upstream_model":"gpt-4o"}'

Text Generation Inference (TGI)¶

Hugging Face TGI exposes an OpenAI route and returns log-probabilities → 5 axes. type: openai, default port 8080.

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{"upstream_type":"openai","upstream_base_url":"http://localhost:8080",
       "upstream_model":"tgi"}'

LM Studio¶

Local OpenAI-compatible server (default port 1234) with log-probabilities → 5 axes.

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{"upstream_type":"openai","upstream_base_url":"http://localhost:1234",
       "upstream_model":"local-model"}'

LocalAI¶

Self-hosted OpenAI drop-in (default port 8080). type: openai.

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{"upstream_type":"openai","upstream_base_url":"http://localhost:8080",
       "upstream_model":"gpt-3.5-turbo"}'

Internal (self-managed vLLM)¶

The gateway launches and manages its own vLLM subprocess, starting it on selection and freeing the GPU when you switch away — ideal for single-GPU hosts.

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{"upstream_type":"internal",
       "internal_vllm_cmd":"python -m vllm.entrypoints.openai.api_server --model my-model --port 8000",
       "internal_vllm_url":"http://localhost:8000",
       "upstream_model":"my-model"}'

Production Topologies¶

1 · Single node — Docker Compose¶

The simplest production unit: the model server and the gateway side by side.

The repository ships deploy/docker-compose.gateway.yml with two services — generator (the model) and glad-gateway (the validator) — and two profiles:

# Fixed checkpoint
docker compose -f deploy/docker-compose.gateway.yml --profile fixed up -d

# New model / new customer
docker compose -f deploy/docker-compose.gateway.yml --profile customer up -d

A minimal .env to drive it:

# .env
GW_UPSTREAM_TYPE=vllm
GW_UPSTREAM_URL=http://generator:8000
GW_UPSTREAM_MODEL=meta-llama/Llama-3-8B-Instruct
GW_BLOCK_INPUT=1
GW_BLOCK_OUTPUT=1
GW_MAXLEN=2048

2 · Kubernetes¶

The gateway is stateless → run it as a Deployment with N replicas behind a Service, mount the upstream as another Service, keep secrets in a Secret, and wire /health to probes.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: geodesia-gateway
spec:
  replicas: 3
  selector: { matchLabels: { app: geodesia-gateway } }
  template:
    metadata: { labels: { app: geodesia-gateway } }
    spec:
      containers:
        - name: gateway
          image: registry.geodesia.ai/gateway:1.0   # your licensed image
          ports: [{ containerPort: 8800 }]
          env:
            - { name: GW_UPSTREAM_TYPE,  value: "vllm" }
            - { name: GW_UPSTREAM_URL,   value: "http://model:8000" }
            - { name: GW_UPSTREAM_MODEL, value: "meta-llama/Llama-3-8B-Instruct" }
            - { name: GW_BLOCK_INPUT,    value: "1" }
            - { name: GW_BLOCK_OUTPUT,   value: "1" }
            - name: GW_API_KEY
              valueFrom: { secretKeyRef: { name: geodesia-secrets, key: upstream-api-key } }
          readinessProbe:
            httpGet: { path: /health, port: 8800 }
            initialDelaySeconds: 10
            periodSeconds: 10
          livenessProbe:
            httpGet: { path: /health, port: 8800 }
            initialDelaySeconds: 30
            periodSeconds: 20

---

apiVersion: v1
kind: Service
metadata: { name: geodesia-gw }
spec:
  selector: { app: geodesia-gateway }
  ports: [{ port: 8800, targetPort: 8800 }]

---

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: geodesia-gateway }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: geodesia-gateway }
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 70 } }

3 · systemd (bare metal / VM)¶

# /etc/systemd/system/geodesia-gateway.service
[Unit]
Description=Geodesia G-1 Gateway
After=network-online.target

[Service]
Environment=GW_UPSTREAM_TYPE=vllm
Environment=GW_UPSTREAM_URL=http://127.0.0.1:8000
Environment=GW_UPSTREAM_MODEL=my-model
Environment=GW_BLOCK_INPUT=1
EnvironmentFile=-/etc/geodesia/gateway.env       # secrets here
ExecStart=/opt/geodesia/bin/start-gateway --host 0.0.0.0 --port 8800
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

systemctl enable --now geodesia-gateway
systemctl status geodesia-gateway

Observability¶

Every response carries a geodesia{} block — your richest telemetry source. Ship it to your logging pipeline and build dashboards on it.

Alerts worth wiring:

• /health ok=false for > 1 probe → gateway down
• axes drops 5 → 4 unexpectedly → upstream stopped returning log-probs
• block rate spikes on prompt_safety or jailbreak → possible attack / misconfig
• block rate spikes on halluc_context → upstream model or RAG regression
• audit-chain verification failing (GET /v1/glad/chain/verify) → integrity incident

Security Hardening¶

• Terminate TLS at the edge (nginx / ingress); keep the gateway and the model on a private network. Never expose the model port publicly — the gateway is the only intended entry point.
• Inject secrets, never bake them. GW_API_KEY and any license token come from a Kubernetes Secret, Docker secret, or vault — not from the image or git.
• Turn on enforcement in production: GW_BLOCK_INPUT=1 and GW_BLOCK_OUTPUT=1. In passthrough mode the gateway annotates but does not withhold — appropriate for monitoring, not for protection.
• Lock down the compliance plane (:8199) to operators only; it exposes the kill switch, FRIA, and audit exports.
• Verify the audit chain on a schedule (GET /v1/glad/chain/verify) and alert on any failure.

Scaling & Performance¶

Zero-Downtime Model / Framework Switch¶

The gateway hot-reloads configuration — you can repoint it without dropping traffic.

1. Test the new backend first (/upstream/test) — confirm reachability and log-prob support.
2. Switch (POST /v1/glad/gateway/config or roll new env vars). The closed-book axis is cross-model and needs no per-model step — it works on the new upstream immediately.

In Kubernetes, prefer a rolling update of the Deployment with the new env vars; the readiness probe keeps traffic on healthy pods throughout.

Troubleshooting Runbook¶

See Also¶

• Upstream Backends — the connection test and upstream-adapter mechanics
• Gateway Configuration — every GatewayConfig field
• Environment Variables — the full GW_* reference
• Enforcement Modes — blocking vs passthrough
• Detection Thresholds — per-axis tuning
• API Response Format — the geodesia{} payload