How LLM Routing Actually Works in Production
A popular interview question.
The operating system for AI research labs!
Transformer Lab is an open-source ML platform that orchestrates GPUs across any cloud and runs any training or eval workflow you define:
supports LoRA, QLoRA, DPO, ORPO, SIMPO
use it from a GUI, CLI, or agent skill.
works with MLX, vLLM, Ollama, and HF Transformers
one-click convert between HF, GGUF, and MLX
diffusion training and inference built in
LLM-as-a-judge evals plus the EleutherAI harness
submits jobs to Slurm and SkyPilot from the same UI
runs on Apple Silicon, NVIDIA, and AMD
The same interface a solo researcher uses on a MacBook also drives a 64-GPU cluster. Nothing else does that today.
Find the GitHub repo here → (don’t forget to star it ⭐️)
How LLM routing actually works in production
A tricky LLM interview question:
Your agent runs everything on a frontier LLM, so you add a routing layer that sends several easy calls to a 15x cheaper LLM.
The router is working as expected, but the bill is still the same as before.
Where did the savings go?
On a high-level, model routing works as follows:
Add a classifier in front of the agent
Send easy prompts to the cheap model
Send hard ones to an expensive model
While this looks simple, under the hood, a lot more engineering must go into this layer before you can actually realize any savings.
And the reason is in how agents consume tokens.
A typical agent task is never one LLM call. Instead, one prompt fires several LLM calls in sequence, like planning, tool use, and analyzing results.
Each call sends the accumulated context back through the model, so the input grows every step.
To avoid redundantly processing the same tokens on every call, providers built prompt caching.
Essentially, the input a model has already seen costs about 90% less than fresh input.
The problem is that this cache is specific to a model.
So every time the router switches models mid-task, the accumulated warm cache gets thrown away, and the full context is re-billed at cold rates.
This means a cache-hot expensive model can still cost less than a cache-cold cheap model.
Production routers solve this by making the routing decision once per task instead of once per call.
The first call routes normally, and the chosen model gets pinned to a session ID. Every call after that goes to the same model, so the cache stays warm and the context stays coherent.
When the task changes, the pin resets, and routing starts fresh.
The savings don’t disappear, but rather they move one level up.
Each new task still gets routed by difficulty, so easy tasks land on a cheap model and hard ones on an expensive model.
The pin only stops switching inside a task, which is exactly where switching costs more than it saves.
This is called model affinity, and it’s the last layer of a 4-stage production routing layer that runs on every request.
A) Guardrail filter
The prompt goes through a safety check. Jailbreaks and unsafe inputs get caught before they reach the router, and every call is logged and traced from this point.
B) Router model
A small routing model reads the prompt, infers the domain and action of the request, and matches it against candidate models.
The model has to be tiny, because a routing decision that costs as much as the call it’s routing saves nothing.
C) Selection policy
A cost policy picks the winner from the candidates, preferring the cheapest or the fastest, depending on the config.
Prices are picked from pricing catalogs, so the choice adapts when providers reprice, and the runner-up stays on standby as the fallback if the winner fails.
D) Model affinity
After selection, the winning model gets pinned to the session ID. Every later call in the task goes to it, so the cache never resets and the context never splits.
To see this in practice, the exact pipeline is already implemented in Plano, an open-source proxy that runs locally between your agent and your model providers.
The config for all four layers is defined in one YAML config file, and swapping any of it never touches agent code.
The routing model in stage 2 uses Arch-Router, a 1.5B model available on HF. It’s trained on human preference data, i.e., what developers actually pick for each task type.
So routing reflects real-world preference instead of benchmark rank, and you define those preferences in plain English in the config.
It also implements an observability console with a per-request cost column, so you can see exactly which model answered each request and what it cost.
Here’s the repo: https://github.com/katanemo/plano
(don’t forget to star it ⭐ )
We put this exact pipeline in front of my Hermes agent, and the usage dropped 2x, without changing a line of agent code.
Here’s the full article we wrote →
Good day!
P.S. For those wanting to develop “Industry ML” expertise:
At the end of the day, all businesses care about impact. That’s it!
Can you reduce costs?
Drive revenue?
Can you scale ML models?
Predict trends before they happen?
We have discussed several other topics (with implementations) that align with such topics.
Here are some of them:
Learn everything about MCPs in this crash course with 9 parts →
Learn how to build Agentic systems in a crash course with 14 parts.
Learn how to build real-world RAG apps and evaluate and scale them in this crash course.
Learn sophisticated graph architectures and how to train them on graph data.
So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.
Learn how to run large models on small devices using Quantization techniques.
Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using Conformal Predictions.
Learn how to identify causal relationships and answer business questions using causal inference in this crash course.
Learn how to scale and implement ML model training in this practical guide.
Learn techniques to reliably test new models in production.
Learn how to build privacy-first ML systems using Federated Learning.
Learn 6 techniques with implementation to compress ML models.
All these resources will help you cultivate key skills that businesses and companies care about the most.
Awesome and very detailed brother
Thanks for sharing