Skip to content

Routing policy

The lab routes by fixed tier aliases per workload. Each caller asks for the tier that fits its job (chat-main, chat-fast, chat-smart, research-deep, …), and the proxy decides which provider serves that tier. The tier table is in LLM Gateway.

History: the Auto Router

Until September 2026 a smart-router alias classified each request (keyword rules, a cheap LLM classifier, a fail-open scorer) and picked a tier. It was retired: fixed tiers per workload are cheaper to reason about, easier to audit, and fail more predictably. The agent fleet moved to fixed tiers in the same change.

Why fixed tiers

  • Predictable cost. A workload's spend follows from its tier, not from a classifier decision per request.
  • Simple audits. The catalog is one reviewed ConfigMap; a reader can see which provider serves which workload.
  • Backings change without consumer changes. Moving chat-fast to a new provider is one config PR; no harness or agent changes.

Fallback chains

Every tier declares an explicit fallback chain in the proxy config:

fallbacks:
  - chat-fast: ["<same model, other paid provider>", "<free model>"]

Rules for every chain:

  1. A paid provider appears at most once in a chain. Free legs are exempt.
  2. A tier never lists another tier as a fallback (that makes a loop).
  3. max_tokens stays the same between a tier and its fallbacks, so a failover does not re-truncate a reply.
  4. A script audits rule 1 on every catalog change.

Why this matters

Model selection is a reliability and cost problem, not only a quality problem. Fixed tiers with audited fallback chains keep it reviewable: one config file, explicit provider order, and no hidden decision per request.