LiteLLM proxy¶
LiteLLM is the model gateway of the lab: a self-hosted, OpenAI-compatible proxy that normalizes about 50 model aliases across direct APIs, subscription plans, and aggregators behind a single endpoint, a single master key, and per-consumer virtual keys.
What runs where¶
- Proxy: a stateless deployment behind an internal service; the UI and the
/v1API share the same proxy. The admin UI sits behind Caddy auth on the private mesh — not exposed to the public internet. - Database: usage, keys, and spend records live in managed PostgreSQL.
- Redis: rate limiting and routing state.
- Config: the alias catalog is a versioned ConfigMap — providers, models, fallbacks, and cost metadata are declarative, reviewed, and rolled out through the normal pipeline.
Virtual keys per consumer¶
Every consumer (each agent gateway, each local harness) gets its own virtual key with a name, budget, and model allowlist. Benefits:
- Revocation is instant and scoped: kill one key, not the master.
- Spend is attributable: the proxy records usage per key, which feeds cost telemetry per tenant and per workload.
- Leaks are bounded: a compromised virtual key exposes only the aliases and budget granted to it — never the master key or upstream credentials.
The alias catalog¶
The catalog is organized into sections, each with a purpose:
| Section | Examples | Purpose |
|---|---|---|
| Prefixed single models | sub/anthropic/…, sub/openai/…, sub/google/…, payg/nous/…, payg/tokenrouter/…, direct/deepseek/… |
One model on one access path: sub/ = subscription shim, payg/ = pay-as-you-go provider, direct/ = the vendor's own API |
| Tier groups | chat-main, chat-fast, chat-balanced, chat-smart, research-deep, code-fast, code-smart |
Fixed tiers per workload, each with an explicit fallback chain |
| Workload aliases | chat-cron, compression, chat-browser, chat-free |
Background and special workloads with their own cost policy |
| Vision / embeddings / video | vision, memory-embedding, video |
Special-purpose models with their own fallbacks |
Why subscription shims are interesting¶
Part of the catalog is backed by personal subscription plans exposed
through small OpenAI-compatible shims (a ChatGPT plan proxy and CLIProxyAPI
for Claude and Gemini plans): the shim holds the plan's OAuth/token
credentials, refreshes them in memory, and exposes a standard
/v1/chat/completions interface to LiteLLM. This turns flat-rate plans into
first-class routing targets — the same pattern a company would use to pool
enterprise plan entitlements behind one gateway. ToS and breakage risks are
documented per shim, and every shim-backed tier has a pay-as-you-go fallback
leg. Plans also end: when a plan expires or drops access, its aliases leave
the catalog and the tier moves to another provider in one config PR.
Operational patterns¶
max_tokensparity between a tier and its fallback so a failover does not re-truncate a reply.- Cost metadata on aliases (
input_cost_per_token,output_cost_per_token) so spend records are accurate even for plan-backed aliases (plan cost is policy, not per-token). - Rollout discipline: config changes are PRs; the proxy restarts quickly and the catalog diff is reviewable.
- Readiness: the proxy answers health checks so the ingress and the fleet's router can fail fast.
Wire compatibility
Because the endpoint is OpenAI-compatible, any tool that speaks the OpenAI
API — including the local coding harnesses (see Harness integration) —
can be pointed at the lab gateway with two environment variables:
OPENAI_BASE_URL and OPENAI_API_KEY (the virtual key).