Skip to content

LLM Gateway

Every model in the lab is reachable through one OpenAI-compatible endpoint: a central LiteLLM proxy that owns all provider credentials and exposes about 50 model aliases to every harness and agent.

LLM gateway — harnesses through LiteLLM to the providers
Every harness, every model, one endpoint: LiteLLM normalizes about 50 aliases into fixed workload tiers.

What the gateway buys you

  • One key, one endpoint, one wire format. A harness that speaks the OpenAI API can use any model in the lab — DeepSeek, Qwen, MiMo, GLM, MiniMax, GPT, Claude, Gemini — without changing code.
  • Provider credentials never reach consumers. Harnesses and gateways use virtual keys; the master key and upstream keys live only in the proxy.
  • Aliases are policy. A consumer asks for chat-smart; the proxy decides which provider and which account serve it, and can change that decision without touching the consumer.
  • Cost control. The cheapest tier that answers is the default; paid providers are explicit opt-ins or last-resort fallbacks; budgets are enforced per tenant.
  • Failover is declarative. Every tier has an explicit fallback chain with a different paid provider per leg, so one provider outage degrades a tier instead of killing it.

The alias tiers

Each workload asks for a fixed tier alias. There is no automatic router: the caller picks the tier, and the proxy picks the backing (see Routing policy).

Alias Use when Backing (current policy)
chat-main Highest-volume agent chat DeepSeek V4.1 Flash (inference platform, PAYG)
chat-fast Cheap, fast — mechanical edits DeepSeek V4.1 Flash (Alibaba Model Studio token plan)
chat-balanced Middle ground — function calling Qwen 3.8 Flash (Alibaba Model Studio token plan)
chat-smart Hard turns — best quality Claude Opus 5.5 (subscription shim)
research-deep Deep reasoning, long turns MiMo v2.6 Pro (Nous, PAYG)

Coding tiers (code-fast, code-smart), cron, compression, vision and embedding aliases complete the catalog.

Reliability engineering

  • Provider diversity per chain: a paid provider appears at most once in each fallback chain (free legs are exempt), and a script audits this rule. A quota exhausted on one provider cannot take down two legs at once.
  • Continuity routes: subscription- and plan-backed tiers fall back to a pay-as-you-go provider, so a plan outage degrades to a slightly more expensive provider instead of a dead endpoint.

Note

Names of current backings are descriptive of the routing policy; the routing plane is designed so backings can change without consumer changes.