How a GPU Actually Works

The intuition an LLM engineer needs. Understand techniques like quantization, speculative decoding, and continuous batching in one place.

Aug 17, 2026

Mock infrastructure for AI apps (open-source)

Every CI run of an AI app today sends real requests to providers like OpenAI or Anthropic.

Like any other LLM call, this too gets billed at actual API rates. So for teams with high commit volumes, this accumulates into a meaningful chunk of API spend.

One common hack devs use is that instead of invoking the LLM API, the test calls a fake local server that speaks the same API and returns a dummy response.

The catch is that the dummy response is a copy of what the provider returned on the day it was saved, and providers keep adding fields and changing types.

So the tests keep passing against a schema that’s no longer valid, while the real integration breaks in production.

A smart approach is now actually implemented in CopilotKit’s recently open-sourced aimock project.

Every day, the repo’s own CI sends a handful of requests to the real API and the same requests to the fake server, then compares both against the official client library’s type definitions.

Those are the only real API calls in the whole setup, and they run on the repo’s own keys, not in anyone else’s CI.

A single team can push hundreds of commits a day, and thousands of teams are already doing that with coding agents.

All of those runs stay offline, because one repo checks against the real API on everyone’s behalf.

When a check fails, a coding agent updates aimock’s built-in response schema, the full test suite has to pass, and a patch version ships to npm.

By simply upgrading the package, the corrected schema gets reflected in every project using it.

The capability is not just limited to a single provider.

The same server works for Claude, OpenAI, Gemini, Bedrock, Azure, Ollama, plus MCP tools, A2A agents, AG-UI event streams, vector DBs like Pinecone and Qdrant, and search, speech, image, and video endpoints.

Here’s the repo: https://github.com/CopilotKit/aimock

(don’t forget to star it ⭐)

How a GPU actually works

A GPU rated for close to a thousand trillion operations per second returns a few dozen tokens per second when it serves a 70B model, with the utilization monitor reading high the whole time.

Nothing is broken in that setup, and a bigger chip barely improves the token generation speed.

Our latest issue explains why, starting from the asymmetry the entire design rests on. Doing arithmetic is cheap, and fetching the numbers to do arithmetic on is expensive.

That asymmetry explains why a GPU carries thousands of simple units sharing one controller, and why it hides the wait for data rather than shortening it.

The same mechanism is why the utilization number means so little, since a chip starved for data looks identical to one running at full throughput.

We walk the memory ladder next, from the register file through shared memory and L2 down to HBM, and show where each level sits relative to an SM.

The number that comes out of all this is arithmetic performed per byte fetched.

Current hardware breaks even near 300 operations per byte at 16-bit precision, and anything below that line is limited by memory rather than compute.

Generating a token performs two operations on every weight it reads, so the ratio lands near one. A single request runs three hundred times below break-even, which is exactly why the arithmetic units go idle.

The token rate follows from the same division. 140 GB of weights over 3.3 TB/s of bandwidth is 42 milliseconds per token, or roughly 24 tokens per second.

Every optimization in common use turns out to be one of two moves, either increasing the work done per fetch or decreasing the bytes fetched. Batching, quantization, fusion, and FlashAttention stop looking like separate tricks once it is clear which move each one makes.

Knowing that a neural network does matrix multiplications, and that weights are numbers sitting in memory, covers the prerequisites.

The full issue is completely free.

Read it here →

Prefix caching vs CacheBlend

If your system prompt and tool definitions are stable, prompt caching is the single highest-leverage optimization available today.

Cached input tokens get up to 90% cheaper, and hit rates of 60 to 85% are realistic.

But it comes with one rigid rule. The cached portion must be an exact, byte-for-byte prefix of the new request. Change a single character in that region, and you get a full cache miss.

That rule breaks in three situations you hit constantly:

→ RAG with multiple documents. You cached document A alone and document B alone. A query now needs both. Document B’s cached state was computed without any awareness of A, so it’s invalid and gets recomputed from scratch.

→ Document order changes. The same three documents appear in a different order across requests. Every permutation is a cache miss, even though the content is identical.

→ Growing conversation history. Each new turn changes everything after the stable prefix, so earlier cached states beyond it become useless.

Alibaba Cloud’s production data puts numbers on this. 10% of KV cache blocks serve 77% of all hits. The rest sits in storage and never gets reused, because prefix matching won’t allow it.

CacheBlend, a research paper from the LMCache (EuroSys 2025 Best Paper Award), attacks exactly this.

In modern transformers, tokens overwhelmingly attend to their own local context, and only a small fraction of tokens carry real connections across document boundaries.

So instead of recomputing everything after the first cached document, CacheBlend reuses every document’s cache as-is and selectively recomputes just those few boundary tokens.

This results in 2 to 4x faster processing on multi-document queries with no quality loss.

The order problem disappears with it. Shuffle the same documents however you like, and every permutation stays cached, where prefix caching recomputes all of them every time. The bottom of the diagram shows that side by side.

The shift is from caching prefixes to caching knowledge. Every document in your knowledge base becomes a reusable cached asset, regardless of what order it appears in or what sits next to it.

CacheBlend ships inside LMCache, the open-source cache management layer that runs outside the inference engine and integrates with vLLM, SGLang, and TensorRT-LLM, on both NVIDIA and AMD GPUs.