Who this is for
Engineering teams running LLM features at production volume
Route each request to the cheapest model that can do the job. Mix Anthropic, OpenAI, and local inference behind a single API, with fallbacks, caching, and observability.
Outcomes
- 30–60% cost reduction vs using a single top-tier model for everything
- Automatic fallback when a provider rate-limits or degrades
- One client-side API, many models behind it
Why routing matters
A well-tuned router sends extractions to Haiku, summaries to Sonnet, hard reasoning to Opus, and anything OpenAI-specific to GPT-5.5. Running everything on the flagship model is wasteful; running everything on the cheap model degrades quality. Routing by intent, not by vibes, is the difference.
What the runtime includes
Prompt caching so repeated system prompts do not get re-billed on every request. Streaming passthrough so the UX is never blocked by the router. Observability hooks so you can see which model answered each request and at what cost. Fallback chains so a single provider outage does not take you down.
Local inference
For sensitive data or offline use, the router can include a local model (Mistral or Llama-family) as a first-pass. The router decides per request whether the local model's answer is good enough or whether to escalate to a hosted model — cutting both cost and latency on the hot path.
Related products
Models typically involved
Frequently asked questions
What is LLM model routing?
LLM model routing is the practice of sending each request to the model best suited to the task — by quality, latency, and cost. A router classifies the incoming request and picks among Claude Opus, Sonnet, Haiku, GPT-5.5, or local models.
How much can model routing save?
Typical savings are 30–60% versus running everything on the flagship model, depending on request mix. The biggest wins come from moving short extraction and classification tasks off of Opus onto Haiku.