Framework Compatibility & DevOps Guide¶

Geodesia G-1 is model-agnostic and framework-agnostic. It sits in front of your inference server as a thin, stateless validation layer and inspects every request and response — it does not care how the tokens are produced, only that the server speaks a protocol it understands.

The golden rule: if your serving stack exposes an OpenAI-compatible /v1/chat/completions endpoint (or the Ollama API), Geodesia G-1 drops in front of it with zero changes to your application — you only change the base_url your client points at.

This page is the operator's reference: the compatibility matrix, per-framework setup, and the production deployment, scaling, observability, security, and troubleshooting playbook.

At a Glance¶

The One Rule That Decides Everything: Log-Probabilities¶

There is exactly one upstream capability that changes how many detection axes you get:

Does the upstream return per-token log-probabilities?

Log-probabilities express how confident the model was when it chose each word. Geodesia uses them for the closed-book fabrication axis (catching confidently-invented facts when there is no grounding context).

No setting toggles this — the gateway probes it on the first request and /health reports the result. You can recover the 5^th axis later with a log-prob sidecar.

Compatibility Matrix¶

* Hosted providers expose log-probabilities on most—but not all—models and tiers; the gateway falls back to 4 axes automatically when they are absent.

Where Geodesia Sits¶

flowchart LR
    APP([Your application<br/><small>OpenAI client</small>]):::app
    GW[Geodesia Gateway<br/><small>:8800 · stateless</small>]:::gw
    LLM[Upstream LLM<br/><small>vLLM · SGLang · TRT-LLM · llama.cpp · Ollama · hosted</small>]:::llm
    DB[(Audit Database<br/><small>shared</small>)]:::db
    PROD[Product Backend<br/><small>:8199 · compliance</small>]:::svc

    APP -->|base_url → :8800/v1| GW
    GW <-->|forward + validate| LLM
    GW -.->|one audited row<br/>per call| DB
    PROD -.->|reads| DB

    classDef app fill:#3f51b5,color:#fff,stroke:#283593;
    classDef gw fill:#1565c0,color:#fff,stroke:#0d47a1;
    classDef llm fill:#5e35b1,color:#fff,stroke:#311b92;
    classDef db fill:#37474f,color:#fff,stroke:#263238;
    classDef svc fill:#00838f,color:#fff,stroke:#005662;

The gateway is the only thing your application talks to. It is stateless; all durable state lives in the shared audit database.

Choosing the Right Type¶

flowchart TD
    Q{How is your<br/>model served?} --> A[Local OpenAI-compatible<br/>server already running]
    Q --> B[Only the Ollama app]
    Q --> C[A hosted API<br/>OpenAI / Azure / Groq / …]
    Q --> D[Nothing yet —<br/>let the gateway run it]

    A --> A1{Which engine?}
    A1 -->|vLLM| TV[type: vllm]:::t
    A1 -->|SGLang| TS[type: sglang]:::t
    A1 -->|TensorRT-LLM| TT[type: trtllm]:::t
    A1 -->|llama.cpp · TGI ·<br/>LM Studio · LocalAI| TO[type: openai]:::t

    B --> TB[type: ollama<br/>+ optional logprob sidecar]:::t
    C --> TC[type: openai<br/>+ api_key]:::t
    D --> TD[type: internal]:::t

    classDef t fill:#1565c0,color:#fff,stroke:#0d47a1;

Anything OpenAI-compatible that is not vLLM / SGLang / TensorRT-LLM uses type: openai.

Configure It — Three Equivalent Ways¶

Every framework below is configured identically; only the values change.

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -H "Content-Type: application/json" \
  -d '{
    "upstream_type": "vllm",
    "upstream_base_url": "http://generator:8000",
    "upstream_model": "meta-llama/Llama-3-8B-Instruct",
    "upstream_api_key": ""
  }'

GW_UPSTREAM_TYPE=vllm
GW_UPSTREAM_URL=http://generator:8000
GW_UPSTREAM_MODEL=meta-llama/Llama-3-8B-Instruct
GW_API_KEY=                 # only for hosted services
GW_BLOCK_INPUT=1            # enforce on input
GW_BLOCK_OUTPUT=1           # enforce on output

Per-Framework Setup¶

vLLM¶

The recommended production backend on NVIDIA GPUs. Returns log-probabilities natively → 5 axes.

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3-8B-Instruct --port 8000

docker run --gpus all -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3-8B-Instruct

Configure:

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{"upstream_type":"vllm","upstream_base_url":"http://localhost:8000",
       "upstream_model":"meta-llama/Llama-3-8B-Instruct"}'

SGLang¶

OpenAI-compatible, log-probabilities supported → 5 axes. Default port 30000.

python -m sglang.launch_server \
  --model-path meta-llama/Llama-3-8B-Instruct --port 30000

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{"upstream_type":"sglang","upstream_base_url":"http://localhost:30000",
       "upstream_model":"meta-llama/Llama-3-8B-Instruct"}'

TensorRT-LLM¶

NVIDIA's maximum-performance engine, fronted by an OpenAI-compatible server (TensorRT-LLM serving frontend or Triton). Log-probabilities supported → 5 axes.

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{"upstream_type":"trtllm","upstream_base_url":"http://localhost:8000",
       "upstream_model":"llama-3-8b"}'

llama.cpp¶

llama-server is OpenAI-compatible and returns log-probabilities → 5 axes, even on CPU or Apple Silicon. Use type: openai.

./llama-server -m ./models/llama-3-8b-instruct.Q4_K_M.gguf \
  --host 0.0.0.0 --port 8080

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{"upstream_type":"openai","upstream_base_url":"http://localhost:8080",
       "upstream_model":"llama-3-8b-instruct","upstream_api_key":""}'

Ollama¶

The easiest local runtime. Its chat API does not expose log-probabilities → 4-axis mode by default (everything except closed-book fabrication).

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{"upstream_type":"ollama","upstream_base_url":"http://localhost:11434",
       "upstream_model":"llama3.2"}'

Recover the 5^th axis with a log-prob sidecar¶

Run a second server for the same model that does expose log-probabilities — typically llama.cpp on the same GGUF. The gateway re-derives the answer through the sidecar to recover the signal.

flowchart LR
    GW[Geodesia Gateway]:::gw -->|generate| OLL[Ollama<br/><small>:11434 · no logprobs</small>]:::o
    GW -.->|re-derive for<br/>closed-book signal| SC[llama.cpp sidecar<br/><small>:8080 · same GGUF · logprobs</small>]:::s
    classDef gw fill:#1565c0,color:#fff,stroke:#0d47a1;
    classDef o fill:#5e35b1,color:#fff,stroke:#311b92;
    classDef s fill:#00838f,color:#fff,stroke:#005662;

./llama-server -m ./models/llama3.2.gguf --port 8080   # the sidecar

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{"upstream_type":"ollama","upstream_base_url":"http://localhost:11434",
       "upstream_model":"llama3.2",
       "ollama_logprob_sidecar_url":"http://localhost:8080",
       "ollama_logprob_sidecar_model":"llama3.2"}'

OpenAI API and Hosted Services¶

Use type: openai for the OpenAI API and any hosted OpenAI-compatible provider.

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{"upstream_type":"openai","upstream_base_url":"https://api.openai.com",
       "upstream_api_key":"sk-...","upstream_model":"gpt-4o"}'

Text Generation Inference (TGI)¶

Hugging Face TGI exposes an OpenAI route and returns log-probabilities → 5 axes. type: openai, default port 8080.

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{"upstream_type":"openai","upstream_base_url":"http://localhost:8080",
       "upstream_model":"tgi"}'

LM Studio¶

Local OpenAI-compatible server (default port 1234) with log-probabilities → 5 axes.

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{"upstream_type":"openai","upstream_base_url":"http://localhost:1234",
       "upstream_model":"local-model"}'

LocalAI¶

Self-hosted OpenAI drop-in (default port 8080). type: openai.

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{"upstream_type":"openai","upstream_base_url":"http://localhost:8080",
       "upstream_model":"gpt-3.5-turbo"}'

Internal (self-managed vLLM)¶

The gateway launches and manages its own vLLM subprocess, starting it on selection and freeing the GPU when you switch away — ideal for single-GPU hosts.

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{"upstream_type":"internal",
       "internal_vllm_cmd":"python -m vllm.entrypoints.openai.api_server --model my-model --port 8000",
       "internal_vllm_url":"http://localhost:8000",
       "upstream_model":"my-model"}'

Production Topologies¶

1 · Single node — Docker Compose¶

The simplest production unit: the model server and the gateway side by side.

flowchart TB
    subgraph HOST [" Single host (GPU) "]
        direction LR
        GEN[generator<br/><small>vllm/vllm-openai · :8000</small>]:::llm
        GWC[glad-gateway<br/><small>:8800</small>]:::gw
        VOL[(volume:<br/>audit + config + RAG)]:::db
        GWC --> GEN
        GWC --- VOL
    end
    EDGE[Reverse proxy / TLS<br/><small>nginx · :443</small>]:::edge --> GWC
    CLIENT([Clients]):::app --> EDGE
    classDef app fill:#3f51b5,color:#fff,stroke:#283593;
    classDef edge fill:#455a64,color:#fff,stroke:#263238;
    classDef gw fill:#1565c0,color:#fff,stroke:#0d47a1;
    classDef llm fill:#5e35b1,color:#fff,stroke:#311b92;
    classDef db fill:#37474f,color:#fff,stroke:#263238;
    style HOST fill:#1565c00d,stroke:#1565c0,stroke-width:2px;

The repository ships deploy/docker-compose.gateway.yml with two services — generator (the model) and glad-gateway (the validator) — and two profiles:

# Pre-calibrated checkpoint
docker compose -f deploy/docker-compose.gateway.yml --profile fixed up -d

# First-boot auto-calibration (new model / new customer)
docker compose -f deploy/docker-compose.gateway.yml --profile customer up -d

A minimal .env to drive it:

# .env
GW_UPSTREAM_TYPE=vllm
GW_UPSTREAM_URL=http://generator:8000
GW_UPSTREAM_MODEL=meta-llama/Llama-3-8B-Instruct
GW_BLOCK_INPUT=1
GW_BLOCK_OUTPUT=1
GW_MAXLEN=2048

2 · Kubernetes¶

The gateway is stateless → run it as a Deployment with N replicas behind a Service, mount the upstream as another Service, keep secrets in a Secret, and wire /health to probes.

flowchart TB
    ING[Ingress / TLS]:::edge --> SVC[Service: geodesia-gw<br/><small>ClusterIP :8800</small>]:::svc
    SVC --> P1[Pod gw-1]:::gw
    SVC --> P2[Pod gw-2]:::gw
    SVC --> P3[Pod gw-N]:::gw
    P1 & P2 & P3 --> GSVC[Service: model<br/><small>vLLM Deployment on GPU nodes</small>]:::llm
    P1 & P2 & P3 -.-> PVC[(PVC / managed DB<br/><small>shared audit store</small>)]:::db
    HPA{{HPA<br/><small>scales on CPU / RPS</small>}}:::hpa -.->|replicas| SVC
    classDef edge fill:#455a64,color:#fff,stroke:#263238;
    classDef svc fill:#00838f,color:#fff,stroke:#005662;
    classDef gw fill:#1565c0,color:#fff,stroke:#0d47a1;
    classDef llm fill:#5e35b1,color:#fff,stroke:#311b92;
    classDef db fill:#37474f,color:#fff,stroke:#263238;
    classDef hpa fill:#ef6c00,color:#fff,stroke:#e65100;

apiVersion: apps/v1
kind: Deployment
metadata:
  name: geodesia-gateway
spec:
  replicas: 3
  selector: { matchLabels: { app: geodesia-gateway } }
  template:
    metadata: { labels: { app: geodesia-gateway } }
    spec:
      containers:
        - name: gateway
          image: registry.geodesia.ai/gateway:1.0   # your licensed image
          ports: [{ containerPort: 8800 }]
          env:
            - { name: GW_UPSTREAM_TYPE,  value: "vllm" }
            - { name: GW_UPSTREAM_URL,   value: "http://model:8000" }
            - { name: GW_UPSTREAM_MODEL, value: "meta-llama/Llama-3-8B-Instruct" }
            - { name: GW_BLOCK_INPUT,    value: "1" }
            - { name: GW_BLOCK_OUTPUT,   value: "1" }
            - name: GW_API_KEY
              valueFrom: { secretKeyRef: { name: geodesia-secrets, key: upstream-api-key } }
          readinessProbe:
            httpGet: { path: /health, port: 8800 }
            initialDelaySeconds: 10
            periodSeconds: 10
          livenessProbe:
            httpGet: { path: /health, port: 8800 }
            initialDelaySeconds: 30
            periodSeconds: 20
---
apiVersion: v1
kind: Service
metadata: { name: geodesia-gw }
spec:
  selector: { app: geodesia-gateway }
  ports: [{ port: 8800, targetPort: 8800 }]
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: geodesia-gateway }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: geodesia-gateway }
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 70 } }

3 · systemd (bare metal / VM)¶

# /etc/systemd/system/geodesia-gateway.service
[Unit]
Description=Geodesia G-1 Gateway
After=network-online.target

[Service]
Environment=GW_UPSTREAM_TYPE=vllm
Environment=GW_UPSTREAM_URL=http://127.0.0.1:8000
Environment=GW_UPSTREAM_MODEL=my-model
Environment=GW_BLOCK_INPUT=1
EnvironmentFile=-/etc/geodesia/gateway.env       # secrets here
ExecStart=/opt/geodesia/bin/start-gateway --host 0.0.0.0 --port 8800
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

systemctl enable --now geodesia-gateway
systemctl status geodesia-gateway

Health, Readiness & Probes¶

The gateway's /health is designed to be both a liveness and a readiness signal.

curl http://localhost:8800/health

{
  "ok": true,
  "upstream_type": "vllm",
  "upstream": "http://localhost:8000",
  "internal_vllm": "stopped",
  "logprobs": true,
  "axes": 5
}

The product backend exposes its own lightweight check at GET /v1/glad/health for the compliance plane.

Observability¶

Every response carries a geodesia{} block — your richest telemetry source. Ship it to your logging pipeline and build dashboards on it.

Alerts worth wiring:

• /health ok=false for > 1 probe → gateway down
• axes drops 5 → 4 unexpectedly → upstream stopped returning log-probs
• block rate spikes on prompt_safety or jailbreak → possible attack / misconfig
• block rate spikes on halluc_context → upstream model or RAG regression
• audit-chain verification failing (GET /v1/glad/chain/verify) → integrity incident

Security Hardening¶

flowchart LR
    NET([Public]):::pub --> TLS[TLS termination<br/><small>nginx / ingress</small>]:::edge
    TLS --> GW[Gateway :8800<br/><small>private network only</small>]:::gw
    GW --> LLM[Model :8000<br/><small>never exposed</small>]:::llm
    SEC[(Secret store<br/><small>GW_API_KEY</small>)]:::sec -.-> GW
    classDef pub fill:#c62828,color:#fff,stroke:#8e0000;
    classDef edge fill:#455a64,color:#fff,stroke:#263238;
    classDef gw fill:#1565c0,color:#fff,stroke:#0d47a1;
    classDef llm fill:#5e35b1,color:#fff,stroke:#311b92;
    classDef sec fill:#00838f,color:#fff,stroke:#005662;

• Terminate TLS at the edge (nginx / ingress); keep the gateway and the model on a private network. Never expose the model port publicly — the gateway is the only intended entry point.
• Inject secrets, never bake them. GW_API_KEY and any license token come from a Kubernetes Secret, Docker secret, or vault — not from the image or git.
• Turn on enforcement in production: GW_BLOCK_INPUT=1 and GW_BLOCK_OUTPUT=1. In passthrough mode the gateway annotates but does not withhold — appropriate for monitoring, not for protection.
• Lock down the compliance plane (:8199) to operators only; it exposes the kill switch, FRIA, and audit exports.
• Verify the audit chain on a schedule (GET /v1/glad/chain/verify) and alert on any failure.

Scaling & Performance¶

Zero-Downtime Model / Framework Switch¶

The gateway hot-reloads configuration — you can repoint it without dropping traffic.

sequenceDiagram
    autonumber
    participant Op as Operator
    participant GW as Gateway
    participant New as New upstream
    Op->>GW: POST /upstream/test (new backend)
    GW->>New: probe reachability + logprobs
    New-->>GW: ok · models · logprobs
    Op->>GW: POST /v1/glad/gateway/config (switch)
    Op->>GW: POST /calibrate  (if base model changed)
    GW-->>Op: streams calibration progress
    Note over GW: next request uses the new config + calibration

1. Test the new backend first (/upstream/test) — confirm reachability and log-prob support.
2. Switch (POST /v1/glad/gateway/config or roll new env vars).
3. Recalibrate the closed-book axis if you changed the base model (Llama → Qwen, etc.). Different quantizations of the same model can share a calibration.

curl -X POST http://localhost:8800/calibrate   # streams until done

In Kubernetes, prefer a rolling update of the Deployment with the new env vars; the readiness probe keeps traffic on healthy pods throughout.

Troubleshooting Runbook¶

See Also¶

• Upstream Backends — the connection test and closed-book calibration mechanics
• Gateway Configuration — every GatewayConfig field
• Environment Variables — the full GW_* reference
• Enforcement Modes — blocking vs passthrough
• Detection Thresholds — per-axis tuning
• API Response Format — the geodesia{} payload