Upstream Backends¶

G1-Proxy is model-agnostic: it sits in front of anything that speaks the OpenAI or Ollama API — vLLM, SGLang, TensorRT-LLM, Ollama, OpenAI, Bedrock, Vertex, Azure. You configure the upstream once and every Application inherits it, or binds its own.

Configuring the Backend¶

Via the UI¶

Navigate to Settings → Service Connection. You will find:

1. Service type — a dropdown to select vLLM, SGLang, TRT-LLM, OpenAI, Ollama, or Internal.
2. URL — the base URL of the upstream (http://host:port).
3. API key — required for OpenAI and hosted services; leave empty for local deployments.
4. Test connection — sends a probe request to the upstream and reports reachability, latency, available models, logprob support, and a sample reply.
5. Model — populated automatically from the upstream's model list after a successful test.
6. Save — persists the configuration to GW_CONFIG_FILE so it survives restarts.

The "Exposed API (OpenAI-compatible)" section shows the gateway's own base URL (http://host:port/v1). Point your downstream application here.

Via the API¶

# Update gateway configuration
curl -s -X POST http://localhost:8800/v1/glad/gateway/config \
  -H "Content-Type: application/json" \
  -d '{
    "upstream_type": "vllm",
    "upstream_base_url": "http://localhost:8000",
    "upstream_model": "my-model",
    "upstream_api_key": ""
  }'

Any field in GatewayConfig can be updated. Changes take effect immediately on the next request.

Testing a Connection¶

The /upstream/test endpoint performs a full connection probe and returns a diagnostic summary. It is called automatically when you click Test connection in the UI.

curl -s -X POST http://localhost:8800/upstream/test \
  -H "Content-Type: application/json" \
  -d '{
    "url": "http://localhost:8000",
    "type": "vllm",
    "api_key": "",
    "model": "my-model"
  }'

Response:

{
  "reachable": true,
  "models": ["my-model"],
  "has_logprobs": true,
  "sample_reply": "OK",
  "latency_ms": 117,
  "error": null,
  "type": "vllm",
  "model": "my-model",
  "closed_book_available": true,
  "axes": 5
}

Supported Backend Types¶

logprobs covers the generated tokens and gates the closed-book axis. prompt_logprobs covers the tokens of the question — a different feature, and the one that decides which detector generation you can calibrate.

Backend-Specific Notes¶

vLLM¶

vLLM is the recommended backend for production deployments on NVIDIA GPUs. It returns per-token log-probabilities natively, enabling all 5 detection axes.

# Start vLLM (example)
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3-8B-Instruct \
  --port 8000

# Configure gateway
curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{"upstream_type":"vllm","upstream_base_url":"http://localhost:8000","upstream_model":"meta-llama/Llama-3-8B-Instruct"}'

SGLang¶

SGLang is fully OpenAI-compatible and supports log-probabilities. Use type sglang:

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{"upstream_type":"sglang","upstream_base_url":"http://localhost:30000","upstream_model":"meta-llama/Llama-3-8B-Instruct"}'

TensorRT-LLM¶

TensorRT-LLM deployments typically front-end with an OpenAI-compatible server. Use type trtllm:

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{"upstream_type":"trtllm","upstream_base_url":"http://localhost:8000","upstream_model":"llama-3-8b"}'

OpenAI API¶

To use the actual OpenAI API (or any hosted OpenAI-compatible service such as Azure OpenAI, Together AI, Groq, Mistral AI):

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{
    "upstream_type": "openai",
    "upstream_base_url": "https://api.openai.com",
    "upstream_api_key": "sk-...",
    "upstream_model": "gpt-4o"
  }'

Cloud upstreams (Bedrock / Vertex / Azure)¶

Beyond the local OpenAI-compatible servers above, an Application can bind directly to a managed cloud LLM:

The gateway ships native adapters for each, so no OpenAI-compatible shim is required. One caveat: Bedrock and Vertex do not return per-token log-probabilities, so the halluc_closedbook axis is automatically disabled for Applications bound to those upstreams (the remaining axes run normally).

See Cloud Upstreams for the full per-provider configuration — adapter setup, IAM/ADC authentication, region binding, and the logprobs caveat in detail.

Ollama¶

Ollama is a popular tool for running open-source models locally. Ollama ≥ 0.12 exposes per-token log-probabilities on its OpenAI-compatible /v1 endpoint, which the proxy probes automatically — so the closed-book axis works out of the box. On older Ollama (< 0.12) that axis reports available: false, and you can recover it with a log-prob sidecar (see Framework Compatibility → Ollama).

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{
    "upstream_type": "ollama",
    "upstream_base_url": "http://localhost:11434",
    "upstream_model": "llama3.2"
  }'

Legacy only (Ollama < 0.12) — recover the 5^th axis with a logprob sidecar¶

On older Ollama (< 0.12) that predates native log-probabilities, you can still recover closed-book detection with a logprob sidecar: a second instance of llama.cpp or another OpenAI-compatible server serving the same model with log-probability support. The gateway deterministically re-derives the answer through the sidecar to recover the needed signals. On Ollama ≥ 0.12 this is unnecessary.

# Configure sidecar (same model, different server, WITH logprobs)
curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{
    "upstream_type": "ollama",
    "upstream_base_url": "http://localhost:11434",
    "upstream_model": "llama3.2",
    "ollama_logprob_sidecar_url": "http://localhost:8080",
    "ollama_logprob_sidecar_model": "llama3.2"
  }'

Internal (self-managed vLLM)¶

When upstream_type is set to internal, the gateway launches and manages its own vLLM subprocess. It starts vLLM when selected and frees GPU memory when you switch to an external backend. This is useful for single-GPU deployments where you want the gateway to own the entire lifecycle.

curl -X POST http://localhost:8800/v1/glad/gateway/config \
  -d '{
    "upstream_type": "internal",
    "internal_vllm_cmd": "python -m vllm.entrypoints.openai.api_server --model my-model --port 8000",
    "internal_vllm_url": "http://localhost:8000",
    "upstream_model": "my-model"
  }'

Closed-Book Fabrication — no per-model setup¶

The closed-book fabrication detector is cross-model: it ships ready to use and runs on any upstream out of the box, with no per-model calibration or training step. When you switch to a different base model, nothing to do — point the gateway at the new upstream and the closed-book axis works immediately (provided the upstream returns per-token log-probabilities; otherwise that one axis is gated and the rest run unchanged).