Your KV cache library is stealing your throughput
Adding a KV cache layer to your inference stack can cost you throughput, due to a reason that’s unrelated to caching per se.
Cache management runs inside the inference engine’s process today, and the two jobs have opposite bottlenecks:
Moving cached tensors between GPU, CPU, and disk waits on I/O.
Running attention is heavy math on the GPU.
In one process, the engine runs them one at a time, so the GPU sits idle while blocks move between memory tiers.
Google’s TurboQuant shrinks the KV cache to 3 bits per value with no accuracy loss. But if you run it inside the engine, you still lose more than 20% of inference throughput.
The compression works, but it still costs based on where it runs.
LMCache (open-source) pulls cache management into its own process next to the engine. The two share GPU memory and pass a short list of block IDs between them, nothing heavier.
Cache work stops stalling inference.
Two GPUs read the same memory instead of copying tensors back and forth. LMCache searches GPU memory, CPU RAM, local SSD and cloud storage at the same time, then streams from whichever one answers first.
On H200s running Qwen3-235B with 50 concurrent users, that is 14x faster time-to-first-token, 4x faster decoding, and a startup that drops from minutes to seconds. The full benchmarks are in the repo.
It works with vLLM, SGLang, and TensorRT-LLM, on NVIDIA and AMD, and it is open-source.
GitHub repo: https://github.com/LMCache/LMCache
(don’t forget to star it ⭐️)
How semantic code navigation cuts agent token costs by up to 36%
Microsoft cancelled Claude Code for 5,000 engineers this year, after token costs climbed to $500 to $2,000 a month per engineer.
Microsoft wasn’t alone. Uber’s CTO said the company went through its entire 2026 AI coding budget in four months.
Both stories were framed around the bill. Neither asked what the agent was actually doing with the tokens.
Here’s the problem. Most teams assume the answer is writing code, and that more output means a higher bill.
That assumption doesn’t hold up once you look at what an agent actually does in a session.
A coding agent starts a task with no map of your codebase. You describe what should change, and the agent has to find where that change belongs before it writes a single line.
The finding is where the tokens go, not the writing.
In this article, we’ll look at exactly how that finding breaks down, and what a different approach to it looks like in practice.
When the map is missing
An agent has one tool to build a map of your codebase, text search. For a lot of tasks, that’s enough.
The trouble starts when the name you search for doesn’t line up with the place you actually need to change, and it breaks in three distinct ways.
First, a name can appear at every location that needs to change and also at hundreds of locations that don’t. The agent has no way to tell the difference except to open and read each match.
That costs tokens and time before it writes anything.
Second, two things can share a name and mean something different. One method might take different arguments than another with the same name.
A local variable can shadow a field. The text lines up, the identity doesn’t, and telling them apart requires knowing what each one actually refers to.
Third, sometimes the code you need to find shares no words with what you searched for. It’s connected structurally instead, a class implementing an interface without ever naming it near the code that changes.
A method can be invoked without its name ever appearing in the code that calls it, through a callback or a dynamic reference. Text search has no way to find something it was never given the words for.
First and second cost you tokens. The agent reads more, reasons longer, spends more to reach the same answer it would have gotten instantly from a real map.
The third doesn’t cost you tokens the same way. A missing structural connection shows up in two different situations, and only one of them is risky.
A missed location in a straightforward rename fails loudly, the build breaks, and the failure is visible right away. That case is safe, since you catch it immediately.
A missed location in a behavior change reached through indirection is the risky one. Nobody wrote a test for a connection nobody knew existed.
So the code compiles, the existing tests pass, and the change ships.
The bug surfaces somewhere else, later, with no obvious link back to what caused it.
This is the part of the token conversation that budget dashboards don’t capture.
Semantic code navigation
If text search is the wrong tool for this, the direct fix is to stop treating the codebase as a body of text at all.
Instead, treat it as a graph. Classes, methods, fields, and interfaces become nodes.
The real relationships between them, calls, implements, extends, references, become edges. Each node and edge carries the exact file and line it lives at.
An agent working against that graph doesn’t search for a name and hope the match is relevant. It asks a direct structural question instead, “find every place that implements this interface.”
It gets back exact locations, not a list of string matches it still has to read through and reason about.
This is conceptually close to what an IDE already does when it offers “find all references” or “go to implementation.”
The difference is the agent can ask that question directly, mid-task, instead of relying on a human to look it up first.
A different way to navigate code
Semantic code navigation solves where the agent looks. It doesn’t solve whether what it writes there is correct.
That gap is worth keeping in mind as you read what this actually looks like built.
A graph like this can be rebuilt without a compiler or language server. That means it stays usable on code that doesn’t currently compile, which is the normal state of code mid-edit.
That matters because the agent’s map stays current while it’s actively changing things, instead of going stale until the next clean build.
A controlled test measured this against real, previously merged commits from open source projects. Each task ran ten times per side, once with a plain agent and once with the same agent given access to this kind of navigation.
Every run was required to pass the actual build and tests before it counted.
Across six tasks in four languages, cost fell in every one, from 5% on a straightforward change up to 36% on a Java interface change.
The pattern behind the wins was consistent. Every task where cost fell significantly involved a change that had to land identically across every related location, a shared interface, a renamed package, a consistent argument order.
That’s exactly the situation where text search can’t cleanly enumerate every site on its own.
The same system that finds the location also needs to verify what actually gets written there, as the code is produced rather than after the PR exists.
The reasoning for keeping that check separate from the agent that wrote the code is straightforward. A model grading its own work has no independent way to catch what it already believed was correct.
A different methodology, run by something other than the model itself, is what actually catches it.
This is what Sonar built into their product Sonar Vortex, a Semantic navigation and real-time verification running inside the agent’s own loop.
The takeaway
Microsoft and Uber both watched a token bill climb past what anyone had modeled for it.
Neither story asked the question that actually explains the number.
Whether the agents doing that work were finding everything they needed to find, or just running longer because they never had a map to begin with.
This is the real cost of agentic coding at scale, and it doesn’t show up as a line item.
It shows up as every session an agent spends searching for something a real map would have handed it directly.
The next time a token bill looks larger than it should, ask what the agent was actually looking for before you switch tools or cut usage.
You can read the full study here →
Thanks to Sonar for partnering today!
Label smoothing for regularization
The entire probability mass belongs to just one class in typical classification problems, and the rest are zero:
This can sometimes impact its generalization capabilities since it can excessively motivate the model to learn the true class for every sample.
Regularising with Label smoothing addresses this issue by reducing the probability mass of the true class and distributing it to other classes:
In the experiment below, I trained two neural networks on the Fashion MNIST dataset with the same weight initialization.
One without label smoothing.
Another with label smoothing.
The model with label smoothing (right) resulted in better test accuracy, i.e., better generalization.
When not to use label smoothing?
Label smoothing is recommended only if you only care about getting the final prediction correct.
But don’t use it if you also care about the model’s confidence since label smoothing directs the model to become “less overconfident” in its predictions, resulting in a drop in the confidence values for every prediction:
That said, L2 regularization is another common way to regularize models. Here’s a guide that explains its probabilistic origin: The Probabilistic Origin of Regularization.
Good day!