Your best model shouldn’t be answering every request
Most production LLM traffic isn’t equally demanding.
A greeting, a one-line lookup, and a codebase refactor can all hit the same endpoint. If your best model handles all three, you’re paying more than you need to for the first two.
TrueFoundry’s Auto Routing moves that decision into the AI gateway.
Your app sends requests to a virtual model. The gateway classifies each request as simple, medium, or complex, then routes it to the model assigned to that tier.
The classifier uses heuristics. It runs in-process and looks at signals like code and reasoning phrases, so it adds no latency or cost.
Conversations stay pinned, so follow-ups don’t move a complex thread to a cheaper model. If a tier’s target model fails, the request moves up to the next tier.
Against an all-Opus baseline, it cut costs by 69% while retaining 98% of quality across 11 graded benchmark datasets.
Get started with Auto Routing here →
Thanks to TrueFoundry for partnering today!
Contrastive Language Model, clearly explained
NVIDIA and Stanford just challenged Jev.
Their new System 1 architecture runs up to 9x faster. It is called a Contrastive Language Model, or CLM.
Like Jev, CLM is not designed to generate text. It handles the small, repeated decisions inside AI systems, such as choosing a tool, ranking a patch, routing a request, or selecting the next action.
But CLM reaches those decisions differently.
Instead of generating an answer token by token like an LLM, it treats decision-making as a retrieval problem.
Here is how it works.
Encode the state
CLM takes the current situation, such as an agent’s context or the state of a game, and converts it into a vector. The state also carries the question being asked, so the context and the question are encoded together.
It uses a frozen Qwen3-8B model with a small trainable state projection head.
Encode every possible action
A separate action head converts each candidate into the same vector space.
In the dino game example, the candidates are jump, duck, and run. CLM does not invent a fourth option. It only evaluates the actions supplied by the application.
Learn which states and actions belong together
During training, the correct state-action pair is pulled closer while incorrect pairs are pushed apart.
A batch of N examples produces an N × N similarity matrix. The matching pairs sit on the diagonal. Every other pairing becomes a negative example.
This contrastive training uses InfoNCE, the same general mechanism behind systems such as CLIP and dense retrieval.
The team trains in three stages that get progressively harder, starting with question-answer pairs, then adding plausible wrong answers as hard negatives, and finishing on steps from real agent runs.
Turn similarity into a decision
At inference, CLM measures the cosine similarity between the state and every candidate action.
A softmax converts those scores into a probability distribution. The application can choose the winner, apply a confidence threshold, or escalate an uncertain result.
The real speed advantage comes from separating states and actions.
Actions can be embedded once and cached. If an agent repeatedly chooses between the same tools, CLM only needs to encode the changing state and compare it with stored action vectors.
That replaces repeated generation with one embedding pass and a set of cheap dot products. With the three dino moves, that’s one encoder call per step instead of four.
The researchers report that CLM-8B performs on par with Jev across computer-use, gaming, and tool-calling evaluations while reaching up to 9x lower latency. It ties Jev on the dino and Mario games and trails slightly on tool calling and Wikipedia link navigation. The improvement is largest when actions repeat or the candidate set grows.
CLM still has limits. It cannot generate new actions, its probabilities are relative to the supplied candidates, and its strongest verifier results require task-specific fine-tuning.
But its central idea is powerful.
The entire research is open-source, including the code.
That’s all for today!
Thanks for reading.