Researchers built a new AI inference engine that:

• reduces self-hosting costs by ~4x

• runs a full agentic pipeline on one GPU

• is a drop-in for the OpenAI API

And it serves 20+ model architectures, not just LLMs.

Here’s the problem with engines like vLLM that it solves.

Most agent pipelines today run 4-5 small models under the hood:

• an embedder for retrieval

• a reranker for precision

• an extractor for entities

• and an LLM for generation

The standard way to serve them is one server per model.

vLLM serves the LLM, TEI serves the embedder, and everything else gets a custom FastAPI wrapper.

Each server reserves its own slice of GPU memory and holds it whether traffic arrives or not. GPUs are billed by the hour, so idle time costs the same as busy time.

This is why switching to small models rarely reduces the bill. The cost is never in the calls but rather in maintaining the servers.

The structural fix is serving every model from one process that loads and evicts models based on traffic.

Superlinked open-sourced a new inference engine that does exactly that.

​SIE (Superlinked Inference Engine)​ is an Apache 2.0 server that runs 85+ models behind one API.

Four calls cover the whole pipeline:

• encode() returns vectors

• score() returns relevance scores

• extract() returns entity spans

• and generate() runs small open LLMs.

Models load on first request and are evicted least-recently-used, so one GPU serves a rotating set of models instead of sitting siloed behind one.

It runs anywhere from a laptop to a Kubernetes cluster, and it plugs into Qdrant, Weaviate, Chroma, LanceDB, LangChain, and LlamaIndex.

You can find the repo here: ​https://github.com/superlinked/sie​

(don’t forget to star 🌟)

WebMCP by Google, clearly explained!

When an agent buys something on a website, they usually navigate it via screenshots.

It captures the page, finds something that looks like a button, clicks, waits, and captures again. It reads the screen the way a person would, only slower, and spends tokens in each turn.

That works often in a demo, but it is fragile if the website is redesigned.

However, the website already knows exactly what it can do. It has a search, a cart, a checkout, a booking flow, and none of that is written down anywhere a program can read. All of it belongs to a layout that was built for people.

So the failure is not that agents read pages badly. Pages were never written for anything except people.

Here is an example.

WebMCP is a browser API from the Chrome and Edge teams that lets a site write those actions down.

The site names each action, search or add to cart, or book a slot describes it in words a model can read, and lists the inputs it accepts.

WebMCP is one of six ways an agent can reach an app. Going through all six in order shows why this one is the better bet.

Six ways an agent can reach an app

The six run from furthest away from the interface to closest to it.

1. The raw API. Your script hits the company’s backend directly with an API key. It is precise and fast, but you had to find the endpoints yourself, you manage the key, and the website is never involved.

2. A backend MCP server. The company builds a server that describes its actions as named tools, and your agent connects to it. Someone who understands the product defined those tools, which helps, though the user interface is still skipped.

3. Computer use. Your agent sees the live page as an image and clicks around. There is nothing to set up. It is slow, every look costs money, and a layout change confuses it.

4. Browser automation. Your agent reads the page’s underlying code instead of a picture of it, which is more reliable than pixels. The tools are generic, so the agent still has to work out meaning from anonymous divs and buttons.

5. WebMCP. The page declares its own actions with names, descriptions, and typed inputs, and your agent calls them.

6. The site’s built-in assistant. The company ships its own chat box, picks the model, and pays for the tokens. Your agent stays outside, so you cannot bring your own agent to the site.

Problems with the six methods

Three things vary across them. They differ in whose agent does the work, in what the user has to configure before anything happens, and in what the agent receives once it arrives.

Every option except one gives up at least one of the three.

The raw API and the backend MCP server provide clean typed actions, but you do the configuring and the website itself never comes into it.

Computer use needs no setup and hands the agent pixels to work out on its own.

Browser automation gives structure, though it is the same generic structure for every site on the internet.

The built-in assistant is free and precise, and it is not your agent, so nothing it learns about you carries anywhere else.

WebMCP keeps all three, since you bring your own agent, you configure nothing, and you get named actions instead of guesswork.

Why declaring is better

Instead of the agent working out what a button does, the site says what it does. A few things follow from that.

• There is no guessing step. The agent gets a list of actions with typed inputs, so there is no interpretation stage where a wrong click quietly does the wrong thing.

• The action runs in the session you are already in. It executes inside your browser tab, so there is no API key for the agent to hold, no separate login, and no token to pass around. You are already signed in, so the agent already has access.

• The available actions change with the page. A logged-out visitor’s agent sees a handful of read-only actions like search and product lookup. After signing in, the site adds the rest, including order history, cart, and checkout. Nothing special happens on the agent’s side, it just reads the list again.

• Your interface stays in front of the user. The action runs on the visible page, so the user watches it happen, and your product does not get reduced to an API that somebody else’s chat window is calling.

• Any model can invoke it. Inputs are described with JSON Schema, the same format Claude, GPT, and Gemini already use for tool calling, so you describe your actions once.

Setup in code

A tool is a plain JavaScript object handed to the browser. The code goes in your page’s own front-end script, the same JavaScript that already runs when someone loads the site. You register each tool once on page load, and from then on any agent visiting that page can see it and call it.

document.modelContext.registerTool({
  name: "add_to_cart",
  description: "Add a product to the shopping cart",
  inputSchema: {
    type: "object",
    properties: {
      productId: { type: "string" },
      quantity: { type: "number" }
    },
    required: ["productId"]
  },
  async execute({ productId, quantity }) {
    await addToCart(productId, quantity);
    return `Added ${quantity} to the cart`;
  }
});

There are four parts, and only one of them is new work.

The name is what the agent calls. The description is written in plain English, because a language model reads it to decide whether this is the right action. The schema says which inputs are valid, so bad arguments never reach your code.

The last part is the function that runs. It calls addToCart, the same function already sitting behind your own button, so you are not building a second version of your product for agents. You are pointing at the one you have.

If the thing you want to expose is already a form, you write no JavaScript at all. You add two attributes to the form markup already sitting in your HTML.

<form toolname="search_flights"
      tooldescription="Search available flights between two cities">
  <input name="from">
  <input name="to">
  <button type="submit">Search</button>
</form>

The browser reads the form, works out that it takes a from and a to, and builds the schema itself.

How to try this?

A site that tells an agent what it can do gets cleaner, more reliable results than a site that makes the agent guess from pixels.

It is still early. One browser family has shipped it, the standard is not final, and only the browser’s own agent calls these tools today.

The cost of trying is close to nothing. If you own a site, the cheapest place to start is a form you already have. You add the two attributes, open the page in a browser that supports the trial, and watch an agent use it.

Read the official docs here →

The illustration below is a summary of how agents access web apps today.

Good day!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

Datalab Marker v2 is an open-source parsing pipeline that runs up to 23.7 pages per second on a single B200 GPU.

It outputs Markdown, JSON, or HTML, and under the hood it runs Surya 2, a single 650M parameter model that handles OCR, layout, reading order, and table recognition together, across more than 90 languages.

The usual way to scale parsing is to add more workers. Each worker loads its own copy of the model into GPU memory and feeds it one page at a time.

A GPU gets its speed from running many pages together in one pass, and a private copy fed one page at a time never gets that. It also sits idle while its worker reads the PDF and assembles the output on the CPU.

So you pay for multiple copies of the model and get close to the throughput of one.

Marker v2 runs many lightweight CPU workers that all talk to one shared Surya inference server, so pages from every worker batch together on one model instance that stays busy.

The parent process reads how much that server can handle and splits the capacity across the workers, so adding workers fills the pipeline instead of flooding it.

Throughput now scales with the server rather than with how much VRAM you can afford per process. That single B200 sustains 2.9 pages per second in balanced mode, 7.4 in fast, and 23.7 with OCR off.

It also skips work it does not need. Marker reads the PDF’s own text layer first and calls the model only where that layer falls short, like scans, equations, and low-confidence tables.

Three modes let you set the trade-off:

• balanced runs the model for layout and re-OCRs bad pages, highest quality, best on GPU

• fast leans on the text layer with minimal model calls, far cheaper per page

• disable_ocr skips the model entirely and runs on CPU for clean digital files

The default is balanced on GPU and fast on CPU or MPS.

One thing worth knowing before you switch. Fast mode reads equations out of the text layer instead of looking at the page, so math accuracy falls off sharply, and a math-heavy corpus should stay on balanced.

On olmOCR-bench, Allen AI’s harness of 1,403 PDFs, balanced scores 76.0% against MinerU at 72.7% while running over 5x more pages per second, and against Docling at 50.3%.

Find the repo here →

Why KV cache stores K and V vectors but never Q?

LLMs are autoregressive, so each token is predicted from every token before it, one at a time.

This autoregressive nature has a direct consequence inside the model.

A forward pass over <n> tokens produces <n> hidden states, but only the last one is projected to logits and is required to generate the next token.

So to understand why the KV cache just stores K and V vectors, we must backtrack to see how exactly the last hidden state is produced.

Let’s walk through this with a 10-token prompt.

1) Prefill:

All 10 tokens go through the model in one forward pass, in parallel (with causal masking), since the whole prompt is already known.

At every layer, each of the 10 positions produces a query, a key and a value vector, and attention at each position runs against all positions up to it.

This pass is compute-heavy, and it’s why the first token takes noticeably longer than the ones after it. TTFT is mostly prefill.

2) The first output token:

To generate the 11th token, only the 10th token’s hidden state is needed. So this is projected from the hidden-dim to vocab-dim to generate logits over vocab.

These logits then go through softmax and sampling to generate token 11.

3) Back-track the hidden state:

The last hidden state is the last row of the feedforward block’s output. The feedforward block is position-wise (it’s applied to each row independently), so that row comes from the last row of the attention output before it.

So now we need to see how the last row of attention is computed.

4) Attention matrix:

QKᵀ for a 10-token prompt will give a 10 × 10 matrix.

Row <i> will have the dot product of query <i> with every key.

Row 10 is therefore Q₁₀·K₁, Q₁₀·K₂, all the way to Q₁₀·K₁₀.

Notice that only Q₁₀ appears in it. Q₁ through Q₉ only belong to their corresponding rows 1-9, and those rows’ hidden states we already discarded because they were never needed.

The last row of attention goes through softmax and multiplies the full stack of value vectors, V₁ through V₁₀, to give the last row of the attention output.

So the last hidden state depends on exactly three things: Q₁₀, every key, and every value.

5) Generating token 12:

Token 11 is appended, and this time, we need row 11’s hidden state to generate token 12.

Mathematically, the attention operation turns out to be Q₁₁ against K₁ through K₁₁, then multiplied by V₁ through V₁₁.

K₁ through K₁₁ and V₁ through V₁₁ are bit-for-bit what prefill + first token produced since under causal masking, a token’s key and value depend on that token and the ones before it, never on anything after, so appending token 11 cannot change anything at position 3.

6) The cache state:

Overall, this implies that you just need to retain the keys and values at each decoding step, and compute only the new position’s Q, K, and V.

Each decode step requires one query vector, which is never used again, so they are never cached across the decoding process.

The visual below explains the entire process:

That said, KV cache is only one of four separate caching layers in an LLM stack.

The other three are prefix caching on the server, prompt caching billed by a provider, and a semantic cache that skips the model entirely.

We wrote a full breakdown of all four caches in LLM serving that you should know as an AI engineer, with code for each.

You can read it here →

Good day!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

A long analysis run can go on for tens of minutes. Somewhere in the middle, a new file shows up, or the review criteria change, and most systems give you one option, which is to start over.

Everything the run already finished gets discarded with it.

The reason is that the plan only exists inside the prompt. A deep research pipeline usually decides its steps up front and then executes them to completion, so there’s no object to edit while the run is live. Regenerating the whole thing is the only safe move.

The solution is to keep task state outside the conversation as an explicit structure, covering the plan, the steps that finished, the artifacts each one produced, and which steps depend on which.

Once dependencies are explicit, a new requirement doesn’t invalidate the run. It invalidates a subgraph.

The system checks the new input against completed steps, keeps the artifacts that are still valid, marks only the downstream dependents as stale, and replans that part. This is closer to an incremental rebuild than a fresh start.

The visual below explains this:

The same state object also keeps parallel work interruptible.

When subagents run branches concurrently and write results back into shared state as they finish, a change committed while branches are still running reaches the ones that haven’t started yet.

This implies that producing a correct answer and carrying a task through to a checkable result are different capabilities, and the second one is mostly a state management problem.

To see this in practice, Apodex is an AI lab working on exactly this, and Apodex 1.1 is their model for long analysis tasks.

A task there starts from actual files like spreadsheets, PDFs, and datasets. The model inspects and cleans the data, picks a method, writes and runs the code, repairs steps that fail, and returns figures and files.

The plan, the finished steps, the outputs, and the exceptions stay on a task board while it runs, which is where a mid-task change gets absorbed.

In their Deep Discover mode, the model decides whether to split the task at all, how many subagents to run, and when to merge what they return.

Before a key claim is shipped, a separate check runs it against the evidence and the code output. If a citation doesn’t match or a computed number conflicts with the reported one, it gets flagged, sent back, and the correction stays on record.

FrontierAgent is their open-source runtime for this, with a command-line TUI. One mode runs a single agent through the task in order, and the other puts a coordinator in front of parallel subagents. It starts with one command on macOS and Linux, and it doesn’t need Docker.

GitHub repo: github.com/ApodexAI/FrontierAgent (don’t forget to star it ⭐)

Apodex 1.1 mini is the open-weight 35B model, and it runs locally with FrontierAgent: huggingface.co/collections/apodex/apodex-11

The web workspace runs the full model, and new accounts get registration credits. Upload one of your own datasets and run a task end to end: apodex.ai

Both apodex-1.1 and apodex-1.1-mini are available on the API, and core model access is free for two weeks as a limited-time offer here: platform.apodex.ai

On a long trajectory, the only supervision is the final outcome, so a run that fails late gets no credit for the early work that was sound, and a run that succeeds gets no penalty for the weakly grounded steps along the way.

Their method, PIVOT-RL, locates the decision where the run went wrong, keeps the prefix that was working, restores the environment state, and builds a shorter continuation task from that point.

So you get to keep what’s still valid, redo only the rest, in training and at runtime.

The technical report on arXiv covers the method in full →

Thanks to Apodex for partnering today!

​Implementing Siamese Network with Contrastive Learning​

A few days back, we covered contrastive learning.

Today, let’s implement it.

On a side note, CLIP (Contrastive Language–Image Pretraining) is a model developed by OpenAI that creates a shared representation space for text and images.

Unlike traditional models that handle text or images in isolation, CLIP allows us to compare and reason about text and images together, which makes it a key component of multimodal systems like Retrieval-Augmented Generation (RAG).

A key element of this model is Contrastive learning (which is also reflected in its name).

​We covered this in Part 5 of the RAG crash course here (open access) →​

To recap, it is a popular self-supervised learning technique that teaches models to learn useful representations by comparing samples.

Here’s the overview:

Create a dataset of face pairs:

• If a pair belongs to the same person, the true label will be 0.

• If a pair belongs to different people, the true label will be 1.

After creating this data, define a network like this:

Pass both inputs through the same network to generate two embeddings.

• If the true label is 0 (same person) → minimize the distance between the two embeddings.

• If the true label is 1 (different person) → maximize the distance between the two embeddings.

Contrastive loss (defined below) helps us train such a model:

where:

• y is the true label.

• D is the distance between two embeddings.

• margin is a hyperparameter, typically greater than 1.

Next, let’s look at the implementation.

We start with some standard imports:

Next, we download/load the MNIST dataset:

Now, recall that to build a Siamese network, we have to create image pairs:

• In some pairs, the two images will have the same true label.

• In other pairs, the two images will have a different true label.

To do this, we define a SiameseDataset class that inherits from the Dataset class of PyTorch, and here, we implement this data creation logic:

• Line 5: We obtain the current instance.

• Line 7: We randomly decide whether this instance should be paired with the same class or not.

• Lines 9-12: If flag=1, continue to find an instance until we get an instance of the same class.

• Lines 14-17: If flag=0, continue to find an instance until we get an instance of a different class.

• Line 23:

  • If the two labels are different, the true label for the pair will be 1.

  • If the two labels are the same, the true label for the pair will be 0.

After defining the class, we create the dataset objects below:

Next, we define the neural network:

• The two input images are fed through the same network to generate an embedding (outputA and outputB).

Moving on, we define the contrastive loss:

Almost done!

Next, we define the dataloader, the model, the optimizer, and the loss function:

Finally, we train it:

And with that, we have successfully implemented a Siamese Network using PyTorch.

Results

Let’s look at some results using images in the test dataset:

We can generate a similarity score as follows:

• Image pair #1: Similarity is high since both images depict the same digit:

• Image pair #2: Similarity is low since both images depict different digits:

• Image pair #3: Similarity is high since both images depict the same digit:

• Image pair #4: Similarity is low since both images depict different digits:

It works as expected!

As mentioned above, CLIP (Contrastive Language–Image Pretraining) is a model developed by OpenAI that creates a shared representation space for text and images.

Unlike traditional models that handle text or images in isolation, CLIP allows us to compare and reason about text and images together, which makes it a key component of multimodal systems like Retrieval-Augmented Generation (RAG).

A key element of this model is Contrastive learning (which is also reflected in its name).

​We covered this in Part 5 of the RAG crash course here (open access) →​

Good day!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

Agent memory stored as a knowledge graph produces one graph per user, per team, and per project. A single enterprise customer can run millions of them.

Most of those graphs are cold at any given moment, and a single query reads exactly one of them.

Neo4j and similar systems are built for the opposite workload. One large graph stays resident in memory, and every request passes through a Cypher parser and a query optimizer before it reaches the data.

On millions of small idle graphs, the cost follows provisioned memory rather than the reads actually served. Isolation also becomes a filter on one shared graph rather than a separate graph per tenant.

Adding hardware stopped helping Zep, so the team built Konig around this access pattern.

Konig keeps hot graphs in RAM, recently used graphs on local NVMe, and idle graphs in object storage, loading them back when a query arrives. Encryption keys and retention rules attach to each graph.

Because each query reads a single graph, PageRank runs inline in milliseconds rather than as a batch job. Retrieval holds a p95 under 100ms from a thousand graphs up to tens of millions, and end-to-end latency stays under 200ms.

Konig also keeps two in-memory layouts of every graph and converts between them on demand.

We read this interesting blog by the Zep team that covers the full design, including the AVX-512 search kernels and bi-temporal facts, here →

​KV vs Prefix vs Prompt vs Semantic Caching​

Four things in an LLM stack store four different objects, and all of them get called caching.

• The KV cache stores attention tensors for one request.

• Prefix caching stores those same tensors on the server, keyed by a hash chain over token IDs.

• Prompt caching is the provider’s billed version of that same lookup, at 0.1x the base input rate on a read against a 1.25x premium on the write.

• A semantic cache stores finished response strings, keyed by cosine similarity over an embedding.

The first three are exact-match and correctness-neutral, so a miss costs you money and latency. The fourth is fuzzy-match, and it will hand you a wrong answer with a 200.

So today let’s go through all four, what each one stores, and what quietly breaks it.

1) The KV cache

During prefill, the model computes a key and value vector for every prompt token at every layer and stores them.

Decoding then attends over those stored vectors and appends one new pair per generated token, instead of recomputing the whole sequence each step.

Queries don’t get cached, and the reason is causal masking. A token’s query vector is used once, at the step that token is processed, and never read again. Its key and value are read by every token that comes after it, so those are the two most important vectors to save.

Without storing them, each decode step is a matrix-matrix multiply over the full sequence. With it, the step becomes a matrix-vector multiply over one new token, which is far fewer FLOPs.

The video below depicts LLM inference with and without KV caching:

While this reduces the computation on each token, you have to load the entire cache from HBM on every single step, so decode stops being compute-bound and becomes memory bandwidth-bound.

Attention kernels finish faster than the cache can be streamed in, and the GPU spends most of a decode step waiting on memory.

The cache is also the thing that decides how many requests fit on the box. Its size is fixed by the model shape and grows linearly with token count, since every layer holds a key and value tensor for every KV head.

For a 70B model at BF16, a single 128K context holds around 40 GB of cache, comparable to the entire model at 4-bit weights.

These are some ways to reduce this. For instance, Grouped-query attention shares one key and value head across a group of query heads, which shrinks the cache and raises the FLOPs done per byte loaded.

Multi-head latent attention in the DeepSeek line compresses the whole thing into a latent vector. FP8 cache quantization roughly doubles capacity and trades a little numerical accuracy for it.

The engine frees those blocks when the request finishes. A 20-turn chat therefore prefills turns 1 through 19 again on turn 20, at full cost.

We built all of this from scratch (with implementation) in ​Part 12 of the RAG systems course​, including the prefill and decode split and the cache memory formula, with the tensors printed out at each step. ​Read it here →​

2) Prefix caching

Prefix caching is the engine that does not need to free the blocks above.

vLLM stores the cache as fixed-size blocks of 16 tokens and identifies each block by a hash over the parent block’s hash plus the token IDs inside it.

Chaining the parent hash into the child turns a block lookup into a prefix lookup, since a block only matches if everything before it matched too.

The scheduler walks the incoming blocks in order and stops at the first miss. A hit increments that block’s reference count, which also pins it against eviction while a request is using it.

Everything from the miss onward gets fresh allocation and a fresh prefill.

Only complete blocks get indexed, so a trailing partial block is recomputed every time. That makes block size a real tuning decision. Larger blocks mean fewer table lookups and better memory locality, smaller blocks mean finer-grained sharing and less waste at the tail.

Eviction reduces hit rates, as expected.

The cache and the running batch draw from the same GPU memory pool, so a larger cache means fewer concurrent sequences, and under pressure vLLM drops unreferenced blocks by least recent use.

Mixed traffic makes this worse, because long shared prefixes occupy the most blocks and are the ones whose loss actually hurts.

Before you turn this on, you should know two things

• It saves prefill only, so decode time is unchanged and crediting a whole speedup to the cache will overstate it.

• And the hashing itself costs something, so on traffic with genuinely unique prompts, benchmarks have measured a throughput regression rather than a gain.

There’s a third problem, which is workload dependent, and it impacts RAG the most. A RAG prompt includes a system instruction, then retrieved chunks, then the query, and the chunks change per request and change order between requests. Two requests that retrieve the same documents in a different order share nothing at all under the chain hash.

The same ​Part 12 of the RAG course​ covers the solutions to it. It also explains the problem with the obvious repair of prefilling each chunk on its own and stitching the caches together, and the problems with it. ​Read it here →​

3) Prompt caching

On a hosted model, you don’t get any block table or the eviction policy. Instead, you get a price sheet over the provider’s own prefix reuse, plus two knob.

The cached object is still KV tensors, not your prompt text, and it still requires an exact prefix match on the fully rendered context.

Anthropic charges 1.25x the base input rate to write an entry and 0.1x to read it, with a higher write multiplier if you want it for a longer time. OpenAI applies the same two multipliers on its current models.

The premium cost is recovered in subsequent requests since anything reused inside the TTL will avoid any recomputation.

A read can only find an entry that some earlier request wrote, and writes happen only at a breakpoint you placed.

Each call checks your breakpoint, and on a miss it walks backward through a limited number of blocks looking for an older write. Anthropic caps that at 20 blocks, so adding more than 20 blocks of conversation between two calls pushes the last write out of range and the hits stop.

All of this is about reusing whatever the provider happened to keep around.

The other thing you can do is to prefill your corpus once, deliberately, before any query arrives, and pay to store the resulting cache instead of paying to recompute it.

​Part 13 of the RAG course​ extensively works through that, including the storage arithmetic and how many queries a preloaded cache has to serve before the offline prefill pays for itself. ​Read it here →​

4) Semantic caching

The three techniques above save prefill work and still run the model.

A semantic cache embeds the incoming prompt, runs a nearest-neighbor search over stored prompts, and returns a stored response outright when the similarity exceeds a threshold.

That’s why it saves output tokens as well as input. It’s also why every request must bear an embedding round trip, including every miss.

On the similarity threshold:

• If you increase it, the hit rate collapses while you keep paying for embeddings on every call.

• If you decrease it, the hit rate climbs alongside the rate of confidently wrong answers.

• Published defaults range from 0.75 to 0.97 depending on who you ask, which tells you it’s a property of your traffic rather than a value to copy.

This is not a fully reliable technique per se since some failures can bypass any threshold value, because they come from what embeddings represent.

For instance:

• Negated sentences sit close together in vector space.

• Two prompts sharing a sentence frame but differing in one operational value score are near-identical, because the frame contributes most of the vector.

Putting the four side by side

Three of the four techniques discussed above are correctness-neutral, so their misses show up in cost and latency and nowhere else.

The semantic cache works in a different way, so hit rate is not the right metric to report here.

Takeaways for production

All of these have failure points:

• If you have any variable in the front of the prompt, like A timestamp, request id or user name in the system prompt, this invalidates every block after it. Always put stable content first, variable content last, and a marker on the boundary.

• Tool schemas are usually placed ahead of the system prompt, so a reorder can invalidate the whole cache.

• Check the settings that get rendered into the prompt. On Anthropic, toggling web search, citations, thinking config or tool_choice rewrites the prompt text and invalidates downstream blocks. A/B testing two reasoning efforts splits your cache in two.

• Summarizing history rewrites the prefix, so the next call pays full price on cold tokens. Truncating tool outputs in place keeps the prefix byte-identical and the cache alive.

• Cache entries are keyed to a model, so routing to a cheaper one still prefills the whole accumulated history at cold rates.

As further reading:

• ​RAG course Part 12 covers the prefill and decode split and why prefix caching underperforms on RAG, with implementations →​

• ​RAG course Part 13 covers preloading a corpus into a cache before any query arrives →​

• ​RAG course Parts 14 and 15 cover cache compression that has to run before a query exists, plus the training-based approaches and what production involves →​

👉 Over to you: which of these four layers has cost you the most debugging time?

Good day!

They split one coding task across four agents by role, as a planner, implementer, tester, and reviewer.

The goal was to test whether splitting agents by job title is a good way to divide the work.

And they found agents spent more tokens on coordination than on the work itself.

They call it the telephone game, where each handoff degrades what the next agent receives.

OpenAI and Google also built machinery for this rather than leaving it to the prompt.

• OpenAI added a “handoffs” primitive to the Agents SDK

• Google’s ADK controls how much parent context reaches a sub-agent.

Both are design-time wiring. You declare which agents are reachable from which, and both ends run inside the same framework.

This setup assumes you know which agent feeds which before the run starts, but plenty of agent work isn’t known in advance.

Switch, from Flint AI, is built for that case. It puts agents and people in the same chat channel, so any two of them can work together without being wired to each other in advance.

Say the error rate on a checkout service rises. An on-call agent queries the logs and finds the deploy behind it. Reading that, someone decides the next step is a chart against last week’s baseline.

That decision did not exist a minute ago, so no handoff would have been declared for it. The charting agent is a separate session with an empty context window, often a different framework on a different machine.

So the person makes the routing call and then moves the payload too, by reading the first agent’s answer and typing a version of it into the second.

This creates three problems:

1) The charting agent has no record of the earlier session, so on the next incident it recomputes work the first agent already did.

2) The charting agent only sees the conclusion without the queries behind it, so it either trusts the summary or reruns the retrieval itself.

3) When the first agent finds no regression, nobody types that anywhere, and the rest of the team never learns it was checked.

With Switch, since everyone is working in a shared channel, the person still decides what runs next, but they stop carrying the payload.

The charting agent joins a channel that already holds the first agent’s report, so it points at that report directly instead of rerunning the queries, and the next incident starts from a channel that shows the earlier one.

The team reads the same thread the agents do, so a no-regression result is visible the moment the first agent posts it.

Switch connects agents built with Claude Code, OpenAI Codex, OpenCode, or any HTTP/MCP-compatible framework directly to the tools your team already uses, like Slack, Teams, Discord, Telegram, and Mattermost.

Here’s the GitHub repo →

Thanks to Flint AI for partnering today!

[Hands-on] Turn scientific figures into structured data with Mistral OCR

Scientific papers often hide their most important findings inside figures rather than paragraphs.

Reading those figures manually takes researchers around 36 minutes per paper. An AI-assisted workflow can reduce that to 26.6 seconds.

For a systematic review covering 200 papers, that’s roughly two weeks versus one afternoon.

The bottleneck isn’t that charts are difficult to read. Most document AI never actually looks at them. It reads the surrounding text instead.

So we built an agentic workflow using Mistral OCR 4 that reads every chart in a scientific paper and returns structured data for each figure.

We tested it on biotech, where scientific papers are especially figure-heavy, but the same architecture works across other scientific domains and even fields like finance.

Here’s the complete system running end-to-end 👇

Before building it, though, it’s worth understanding why existing PDF pipelines miss so much information in the first place.

Let’s begin!

Why PDF parsers miss scientific figures

Imagine running a standard PDF parser on a 12-page medical research paper.

It extracts around 8,000 words of clean text and saves 11 figures as image references. The body text says, “As shown in Figure 3, treatment group B demonstrated significantly improved outcomes.”

If you open Figure 3, you’ll find a chart comparing how three treatment groups perform over time. It contains the actual measurements, trends, and statistical results that support the paper’s conclusion.

The parser saves that chart as a PNG, but the numbers are gone.

And this isn’t unique to medical research. The same thing happens anywhere the important information is contained within a figure rather than in the surrounding text, such as scientific bar charts, line charts, scatter plots, graphs, financial reports, etc.

Standard PDF parsers aren’t fully broken. They’re doing exactly what they were built to do, which is extract the text layer.

The problem is that many important findings never exist as text in the first place. It also combines tables, charts, diagrams, and images into one.

What good document intelligence requires

Reading a research paper involves several different tasks, all happening at once.

Some information is written as text. Some lives are inside charts and diagrams. And turning all of that into structured data is another task entirely.

Trying to solve everything in one pass usually mixes those jobs together. A more reliable approach is to keep them separate.

Here, two separations matter more than anything else.

OCR reads text, and Vision reads charts

OCR is excellent at recovering written text from a page. But a chart isn’t just text.

• Axis values, legends, data points, colors, and relationships between them all carry meaning that OCR was never designed to understand.

• Dense tick labels get misread, small legends disappear, and the structure of the chart is lost.

• The analysis step needs to look at the image itself, not just the text around it.

Keeping extraction and analysis separate

Once the chart has been isolated, interpreting it is a different problem from extracting it.

If OCR errors flow directly into the reasoning step, those mistakes compound throughout the pipeline.

• Separating extraction from analysis means each stage receives a clean, well-defined input instead of inheriting every mistake made before it.

• Everything else follows naturally: (a) a chart-aware extraction, (b) a consistent schema for every figure, and (c) a structured output.

• All can be indexed, searched, or stored in a vector database without additional cleanup.

Using Mistral OCRv4 for extraction

Earlier, we saw that standard PDF parsers miss much of what makes scientific figures valuable.

Part of the reason is how they read a page in the first place:

• Most parsers break a page into small blocks and process each one independently before stitching the results back together. That works well for ordinary documents, but scientific papers depend heavily on layout.

• A legend in the corner gets separated from the chart it belongs to. A caption loses its connection to the figure it describes. The extraction still succeeds, but the document’s meaning does not.

Mistral OCR takes a different approach:

• Instead of processing isolated blocks, it performs a Document AI pass over the entire page, understanding both the layout and the content together.

• A single request returns the OCR text, page layout, embedded figure images, and structured extractions.

• When you provide a JSON schema, the model fills that schema directly instead of returning free-form text.

For documents filled with dense axis labels, tiny legends, and complex layouts, that whole-page understanding keeps figures connected to the information around them.

Later, that becomes the foundation for everything else in this pipeline.

Everything from one request

Everything in the extraction stage comes from a single OCR request. So it’s actually one API call that returns:

• page text

• page layout

• every embedded figure

• and structured figure metadata

import os
from mistralai.client import Mistral

api_key = os.environ["MISTRAL_API_KEY"]

ocr_response = client.ocr.process(
    model="mistral-ocr-4-0",
    document={
        "type": "file",
        "document_url": uploaded.id
    },
    include_image_base64=True,
    include_blocks=True,
    confidence_scores_granularity="page",
)

The request asks Mistral OCRv4 to return both the extracted document and the embedded figures in the same response, while preserving enough layout information for later matching.

Note: The code above pins mistral-ocr-4-0, the version this workflow was built on. Mistral has since shipped OCR 4.1, a model update that keeps every capability used here and reads busy, marked-up pages more precisely. Bounding boxes align to each element instead of drifting, callouts on dense technical diagrams stay as separate regions instead of merging into one, and multi-column pages return each column on its own rather than collapsing them together. The mistral-ocr-latest alias now points to 4.1, so moving this pipeline over is a change to the model string and nothing else.

The practical benefit here is that there is no second extraction pass, no intermediate conversion step, and no additional state to manage between stages.

Rather than letting the model invent its own response structure, the pipeline defines exactly what every extracted figure should contain.

The document_annotation_format argument is the schema argument.

_FIGURE_SCHEMA = {
    "type": "object",
    "properties": {
        "figures": {
            "type": "array",
            "description": "...",
            "items": {
                "type": "object",
                "properties": {
                    "figure_id": {"type": "string", "description": "..."},
                    "page_number": {"type": "integer", "description": "..."},
                    "chart_type": {"type": "string", "description": "..."},
                    "title": {"type": "string", "description": "..."},
                    "x_axis": {"type": "string", "description": "..."},
                    "y_axis": {"type": "string", "description": "..."},
                    "data_summary": {"type": "string", "description": "..."},
                    "conditions": {"type": "array", "items": {"type": "string"}, "description": "..."},
                    "caption": {"type": "string", "description": "..."},
                },
            },
        }
    },
}

_DOCUMENT_ANNOTATION_FORMAT = {
    "type": "json_schema",
    "json_schema": {
        "name": "figures",
        "schema_definition": _FIGURE_SCHEMA,
        "strict": False,
    },
}

Each figure is returned with structured fields such as its title, caption, chart type, axis labels, experimental conditions, and a summary of the visible data.

Here’s what that looks like for a single figure:

{
  "figures": [
    {
      "figure_id": "Figure 3",
      "page_number": 4,
      "chart_type": "line",
      "title": "Change in body weight over 68 weeks",
      "x_axis": "Time (weeks)",
      "y_axis": "Mean percent change in body weight (%)",
      "data_summary": "Semaglutide group shows a steady decline reaching approximately -14.9% at week 68, while the placebo group plateaus near -2.4%.",
      "conditions": ["Semaglutide 2.4 mg", "Placebo"],
      "caption": "Figure 3. Mean percent change in body weight from baseline to week 68 in the semaglutide and placebo groups."
    }
  ]
}

Explore more in the Mistral OCRv4 docs →

Matching figures to their images

The OCR response already contains both structured figure information and the embedded images. What it doesn’t include is a direct link between them.

The pipeline creates that link itself.

It breaks the extracted caption into words and compares those against the OCR text, looking for the strongest overlap.

Once a figure is matched to its page, the next unused image on that page is assigned to it.

A small cursor ensures that when multiple figures appear on the same page, each one receives its own image instead of a repetitive selection.

If a caption is too short or ambiguous to match confidently, the pipeline falls back to the page number reported by Mistral OCRv4.

By the end of this step, every extracted figure carries both structured metadata and the actual image it came from, ready for visual analysis.

Measuring extraction confidence

Mistral OCRv4 can also return a confidence score for every processed page.

The pipeline averages those page-level scores into a single document confidence, as an average confidence percentage.

In a production pipeline, this becomes a useful signal for automatically flagging low-confidence pages before their extractions flow into a downstream knowledge base.

Notice what we’ve built so far. At this stage, the pipeline has only extracted the document into a structured representation and recovered every figure.

It still hasn’t tried to understand what those charts actually mean.

That’s the next layer.

Coordinating agentic figure analysis

Once Mistral OCRv4 has extracted the figures, a CrewAI flow orchestrates the rest of the pipeline.

Rather than sending every extracted image straight to an LLM, the flow first prepares the workload before any model is called.

It:

• validates that the document actually contains figures to analyze

• removes duplicate figures that point to the same source image

• batches the figures into groups of five for efficient inference, with automatic retries if a request fails

• passes only clean, validated output from one step to the next

With the figures prepared, the flow passes each image to a multimodal agent for analysis.

The agent looks at the figure itself and not just the surrounding caption to understand what the chart is showing.

The agent runs on Mistral Small 4, wired into CrewAI as a multimodal LLM:

from crewai import Agent

figure_analyst = Agent(
    config=agents_config["figure_analyst"],
    llm=LLM(
        model="mistralai/mistral-small-2603",
        reasoning_effort="none",
    ),
    multimodal=True,
)

So the pipeline uses two Mistral models for two different jobs:

1. Mistral OCRv4 extracts the figures and their surrounding document context.

2. Mistral Small 4 reads the extracted figure images and interprets what they show.

For each figure, the agent reads the axis labels, legends, plotted data, and visual trends alongside the caption.

It then returns structured fields including the chart type, main finding, variables being compared, quantitative observations, interpretation, and knowledge-base tags.

Before a figure reaches the final output, the agent performs two validation checks.

1. It verifies that the image is actually a quantitative chart rather than a micrograph, western blot, or photograph, and that the figure is legible and consistent with its caption.

2. Figures that fail either check are still recorded for traceability, but they’re excluded from the results shown in the interface.

The whole pipeline

By this point, we've looked at each component individually.

Here's how they work together from the moment a paper is uploaded to the moment structured figure intelligence appears in the application.

1. User uploads a scientific PDF.

2. Mistral OCRv4 runs a pass to identify and structure all figures

3. The agent flow validates extraction quality and routes accordingly

4. A multimodal analyst agent produces structured intelligence

5. Results are displayed in the UI with a figure thumbnail.

That said, this pipeline isn’t only about processing figures accurately and faster. It changes what you can do with an entire paper collection.

• Coverage: Every figure in every paper becomes a structured record. Nothing disappears because a chart type was unfamiliar or a caption was vague.

• Consistency: Every figure goes through the same extraction pipeline, the same multimodal agent, and the same schema. The output no longer depends on who happened to review the paper.

• Queryability: Variables, entities, chart types, quantitative findings, and knowledge-base tags are structured from the start, making them immediately searchable and ready for downstream RAG or analytics.

A systematic review of 200 papers can easily contain 1,600–3,000 figures. Reviewing and cataloging those manually takes weeks, produces inconsistent annotations, and leaves results that are difficult to search afterward.

The result isn't just another OCR output. Every paper becomes a structured collection of searchable figure records, complete with extracted metadata, chart summaries, confidence scores, and the original figure images.

Here’s what one processed paper looks like in the application:

With a structured pipeline, every figure becomes searchable data instead of another image buried inside a PDF.

Running the pipeline in production

The demo in this article uses a public paper. Production deployments often don’t.

Many organizations process unpublished experiments, proprietary datasets, clinical studies, or internal research.

Every document sent to a cloud API reveals what you’re working on, even if the model never stores the data.

For those environments, keeping document processing inside your own infrastructure stops being a preference and becomes a requirement.

Mistral OCRv4 is available as a self-hosted deployment, so the same extraction pipeline can run entirely inside your own environment.

One practical advantage of this architecture is that the application doesn’t change. The extraction request, schema, and downstream agent pipeline stay the same whether OCR runs through the hosted API or a self-hosted deployment.

Moving from one to the other is primarily a deployment decision rather than an application rewrite.

Explore Mistral OCRv4 through Mistral Studio →

​Demo GitHub repo →

Good day!​

Brendan Short, who has spent years building GTM systems at companies like Zoom, made a pointed observation about re-engagement pipelines.

Most teams wait nine months after a lost deal before reaching out again, but the right moment isn’t calendar-based.

Instead, it’s when something just changed at that company, say a new VP joined, a funding round closed, a leadership hire landed in the right function.

The problem is that the signal and the timing don’t live in the same place. Who just joined a company and what that company announced recently reside in different records.

Seltz is a web index built for exactly this. Its people and news scopes return full structured records in a single call, so the join happens in one pass instead of being assembled from fragments.

Build with Seltz's web index

Here’s how this helps:

• One call per scope instead of a chain of search, fetch, and parse steps.

• Full structured records back, not fragments you reassemble into an answer.

This article is a hands-on walkthrough of building a multi-agent GTM research pipeline with Seltz. But before the build, let’s understand why this class of question is harder than a standard search.

The join problem

Most search queries have one answer that can be found in one place.

“Who is the CEO of Nvidia” returns one record, “What did OpenAI announce last week” returns one article, and the agent moves on.

GTM research doesn’t work that way. An outreach agent needs to find which companies on a target list hired someone into a data or AI leadership role last quarter and what just happened there that makes now the right moment to reach out.

That answer doesn’t exist as a single page anywhere on the web. It only exists once you read a person’s full role history, cross-reference the company’s recent news, and merge both.

A standard search API does not return that record. It returns a ranked list of links, each with a snippet, meaning the two or three lines of text pulled from around the keywords that matched on the page.

A snippet is enough to tell you a page is worth opening and never enough to answer from. So the agent fetches the page, parses the HTML, pulls out the fields it needs, and repeats that for every person and every company on the list.

When each call returns a full structured record instead, the join becomes a merge across two complete records rather than an assembly job across fragments.

That’s the problem this pipeline is built to solve.

How the pipeline works

The pipeline runs three agents in sequence, each with a focused job.

• Agent 1, the Signal Hunter, queries the Seltz news scope for recent trigger events at a list of target companies. It looks for leadership hires, funding rounds, product launches, or expansion announcements within your specified time window.

• Agent 2, the People Enricher, takes the names surfaced by Agent 1 and queries the Seltz people scope for each one. It retrieves the full career record, every role with dates, prior companies, and education.

• Agent 3, the Outreach Strategist, receives both outputs and merges them. It produces a ranked list of contacts, each with the trigger event, the full background, and a first outreach line written around the specific signal.

• Neither Agent 1 nor Agent 2 fetches a second page. Each Seltz call returns a complete document, so the join in Agent 3 is a merge across full records, not a reconstruction from fragments.

Building it

We use CrewAI for orchestration, OpenRouter as the LLM provider, and the Seltz MCP server as the retrieval tool for all three agents.

The MCP integration means any agent in the crew can call Seltz through a single connected tool, no custom wrappers needed.

Setup

pip install crewai

CrewAI’s MCPServerHTTP connects to any HTTP-based MCP server and exposes its tools directly to agents via the mcps parameter.

from crewai import Agent, Task, Crew, Process
from crewai.mcp import MCPServerHTTP

MODEL = "openrouter/anthropic/claude-sonnet-4-6"
SELTZ_API_KEY = "your_seltz_api_key"
TARGET_COMPANIES = ["Datadog", "Cohere", "Weights & Biases"]

The Seltz MCP server runs at https://mcp.seltz.ai/mcp. One connection object, passed directly to whichever agents need retrieval access.

seltz_mcp = MCPServerHTTP(
    url="https://mcp.seltz.ai/mcp",
    headers={"x-api-key": SELTZ_API_KEY},
    cache_tools_list=True,
)

Defining the agents

signal_hunter = Agent(
    role="GTM Signal Hunter",
    goal="Find recent trigger events at target companies",
    backstory="""You search Seltz news scope to find leadership hires, 
    funding rounds, and expansion announcements at target companies. 
    You extract the person involved, their role, the event, and the date.
    Always use scope='news' when searching.""",
    mcps=[seltz_mcp],
    llm=MODEL,
    verbose=False,
)

people_enricher = Agent(
    role="People Research Specialist",
    goal="Retrieve full career records for identified contacts",
    backstory="""Given a person's name and company, you query Seltz people scope 
    to retrieve their complete role history, prior companies, and education.
    Always use scope='people' when searching for people.""",
    mcps=[seltz_mcp],
    llm=MODEL,
    verbose=False,
)

outreach_strategist = Agent(
    role="Outreach Strategist",
    goal="Merge trigger events and profiles into a ranked outreach list",
    backstory="""You synthesise research from the signal hunter and people enricher 
    into a ranked list of outreach opportunities, each with a specific first line 
    written around the trigger event.""",
    llm=MODEL,
    verbose=False,
)

Note that the Outreach Strategist has no mcps. It reasons over what the first two agents return, not over additional retrieval.

Defining the tasks

signal_task = Task(
    description=f"""Search the Seltz news scope for recent trigger events 
    at these companies: {TARGET_COMPANIES}.
    
    For each company, look for leadership hires in data or AI functions, 
    funding rounds, product launches, or expansion announcements 
    published after 2025-01-01.
    
    Use scope='news' and from_date='2025-01-01'.
    
    Return: company name, person name, their role, the event, and why 
    this is a good moment to reach out.""",
    agent=signal_hunter,
    expected_output="Structured list of trigger events per company with person, role, event, and outreach rationale"
)

enrich_task = Task(
    description="""For each person identified in the trigger events, 
    query the Seltz people scope to retrieve their full career record.
    
    Search by person name and company. Use scope='people'.
    
    Return: full role history with dates, prior companies, education, 
    and LinkedIn URL for each person.""",
    agent=people_enricher,
    expected_output="Full structured profiles for each identified contact",
    context=[signal_task]
)

outreach_task = Task(
    description="""Using the trigger events and enriched profiles, 
    build a ranked outreach list.
    
    For each contact produce:
    - Full name and current role
    - Company
    - The trigger event that makes now the right moment
    - A specific first outreach line that references the trigger event 
      and their background
    - A signal strength score from 1 to 10
    
    Rank by signal strength, strongest first.""",
    agent=outreach_strategist,
    expected_output="Ranked outreach list with personalised first lines",
    context=[signal_task, enrich_task]
)

Running the crew

crew = Crew(
    agents=[signal_hunter, people_enricher, outreach_strategist],
    tasks=[signal_task, enrich_task, outreach_task],
    process=Process.sequential,
    verbose=True,
)

result = crew.kickoff()
print(result)

We wrapped this pipeline in a Streamlit app to make it easier to run and watch the agents work in sequence.

The sidebar shows the three agents and which Seltz scope each one calls.

The main panel updates as each agent completes, showing trigger events found, profiles enriched, and outreach list ready. The ranked cards below show the final output.

Each outreach message is written around the specific trigger event and the person's background, context that only exists because both the people record and the news record were complete enough to read and join.

When to chain with open web search

Seltz handles two jobs in this pipeline well. It reliably returns full records at the director and regional-leader layer, and surfaces recent news with complete article text.

For the single most senior executive at a company, open web search is the stronger first call.

The reliable pattern is to pin the top name on the open web, then use Seltz to enrich the full layer below.

Similarly, for discovery queries where you don’t yet know which companies to target, open web search handles that job. Seltz earns its place once you have a list and need depth on each one.

A pipeline that chains both gets the best of each, using open web for discovery and Seltz for the depth that makes the join possible.

Brendan Short’s observation was about timing being arbitrary. When the retrieval layer returns complete records, the timing problem solves itself, and the signal finds you.

Get started with Seltz here →

Build with Seltz's web index

Thanks to Seltz for partnering today!

​5 context compaction strategies for LLM agents​

Compacting your agent’s context can cut its tokens and still raise costs.

This sounds counterintuitive, but token count and billed amount are two different quantities in how prefix caching works.

A long agent session resends its entire history on every call, appending the model’s reply and the tool output to the transcript each turn.

This stays affordable because the leading span of each request is byte-identical to the previous one.

Providers bill that span as a cache read, which on Anthropic is 10% of base input, so a context that grows only at the tail stays cheap no matter how large it gets.

Compaction edits the front of the transcript rather than the tail. The harness replaces the original turns with a summary, so everything from the edit point onward stops matching what was cached.

Consider a session having 100K tokens of history.

• Normally, the next call reads that history from cache at a tenth of base rate, which works out to 10K tokens at full price.

• But compact it down to a 10K summary, and there is nothing left to match, so those 10K bill as a cache write at 1.25x base input, which comes to 12.5K. The context is ten times smaller, and the call costs more.

To be fair, that cost is recovered over the turns that follow. But harnesses trigger compaction on a token threshold, so a long session compacts repeatedly and each event resets it.

None of this makes compaction wrong. Context windows are finite, and there are five strategies used in practice.

> Truncation drops the oldest tokens once the limit is close. It is the cheapest to implement and the only one that permanently loses early decisions.

> Rolling summarization merges each new summary into a persistent state instead of regenerating from scratch, and still moves the cache boundary every time.

> Prompt compression scores each token with a small model and drops the low-relevance ones. LLMLingua reports up to 20x compression at small accuracy loss, and LLMLingua-2 does the same scoring with a BERT-sized encoder.

> RAG-based retrieval moves the history into a vector DB and injects back only what matches the current query, so retrieval precision becomes the failure mode instead.

The first three delete text outright, and RAG moves it into a store the agent only sees again if retrieval fetches it.

> KV cache eviction runs at the serving layer and drops the entries least likely to be needed, either by attention score, as in H2O and SnapKV, or by position, as in StreamingLLM.

The full history still goes to the model, and what gets dropped is the KV tensors the GPU computed for those tokens. Since those tensors are derived from the tokens, eviction costs prefill work rather than information.

They can always be recomputed. KV blocks that no longer fit in GPU memory can move to CPU DRAM, local NVMe, or a remote store, then load back on the next request instead of being recomputed during prefill.

LMCache implements this as an open-source layer for vLLM, SGLang, and Dynamo. Through CacheBlend, it reuses cached blocks at any position in the prompt rather than only the leading span.

Repo: ​https://github.com/LMCache/LMCache​

(don’t forget to star it ⭐)

Good day!

We have released three new parts in the RAG Systems course, all covering one idea: what if the model reads the knowledge base before any query arrives, and every request skips retrieval, chunking, and embedding entirely?

• Part A on naive preloading, effective length, and provider economics here →

• Part B on compressed preloading and query-agnostic requirement here →

• Part C on modular preloading, trained preloading, and production deployment here →

The three parts cover:

• Naive preloading and why the advertised context window is not the real context limit

• Cost economics and the break-even math for both self-hosted and provider-hosted caching

• Why most published KV cache compression methods fail for preloading

• Modular preloading where each passage is cached independently

• Trained preloading where the corpus is distilled into a compact cache through a training run

• How to treat the cache as production infrastructure with storage tiers

• When preloading is the wrong choice entirely

• Hands-on experiments across all three parts.

Start reading Part A here →

Why care?

Every time a RAG system answers a question, it re-reads the same documents the model has likely already seen on earlier requests.

For instance, processing 16,000 input tokens with a 14B model on an NVIDIA L20 takes over 5.5 seconds before a single output token appears.

And prefill scales quadratically with input length, so doubling the context roughly quadruples the attention cost.

For production systems serving thousands of requests per hour against the same knowledge base, prefill is the dominant line item on the inference bill.

Preloading eliminates that line item.

The model reads the corpus once, the resulting KV cache is stored, and every subsequent query skips straight to decoding.

• Provider-side caching (Gemini, Claude) offers up to 90 percent reduction on input token costs.

• And in self-hosted setups, the savings come from the GPU compute not done.

But preloading is not as simple as caching everything and forgetting about it.

The advertised context window is not the usable context window. Models that accept 128K tokens can start degrading at 32K on tasks requiring inference rather than keyword matching.

And most published KV cache compression methods cannot be used for preloading at all, because they need the query at compression time, which defeats the entire point.

And modular caching, where each passage is stored independently, runs into the same cross-attention failures we covered in the KV cache reuse deep dive.

These are engineering constraints that determine whether preloading works for a given system, and they are the kind of thing that only surfaces after you have already built it.

These three parts cover the full spectrum so you can make that decision upfront.

• Part A: Naive preloading, effective length, and provider economics →

• Part B: Compressed preloading and the query-agnostic requirement →

• Part C: Modular preloading, trained preloading, and production deployment →

👉 Over to you: Is your RAG system re-reading the same documents on every request?

How to cut agent tokens by 2.7x (using an open harness)

An agent run comes back correct, and costs more than the task looked like it should. The model did what you asked, so there is no obvious place to look for the difference.

The difference lies in the code wrapped around the model. That layer decides how much context the model sees, how often it gets called, which tools it can reach, and what carries from one step to the next. The usual name for it is the harness.

An agent that costs more than it should is rarely a model problem but rather a runtime problem.

LangChain showed how much room there is in that layer. They moved deepagents-cli from 52.8% to 66.5% on Terminal Bench 2.0 with the model pinned to gpt-5.2-codex, taking it from outside the top 30 to rank 5.

The model stayed the same, and the harness did the rest.

Capability improves that way, and so does cost, because the same decisions that control what the model reads also control how many times it reads it.

Today we’ll look at what a production harness owns, why two harnesses produce workloads this far apart, and how those decisions come together.

The objective of a harness beyond the execution loop

Anthropic’s documentation calls the SDK behind Claude Code an agent harness, and OpenAI’s Codex team uses the same term. Beyond the loop, a harness handles tool execution, context, memory, state persistence, errors, and permission.

Claude Code, Codex, and OpenCode assume a developer, a machine, and a terminal, which works when the person running the agent built it. Putting that agent in front of real users changes the problem.

• A server restarts while a task is running

• A sensitive action needs approval an hour later from a different device

• Several users run the same agent at once

• Conversations pick up where they left off without seeing another user’s state

Production runtimes move concerns to the server, managing sessions, execution state, and concurrent users independently of the process running the agent.

The harness starts affecting cost here, deciding what survives between runs and what gets sent to the model again. The model does the work. The harness decides how that work runs.

Why the same task can cost three times more

A tool returns a 50,000-token JSON payload at step four. Nothing removes it from the conversation, so by step nineteen the model has read those fifteen more times.

That is 800,000 input tokens for one tool response, all billed at input rates.

Tool definitions do the same. Each carries a name, description, and input-output schema, so MCP servers fill a large part of every prompt before the actual work.

A larger context window does not help. You pay on every model call, not once when you assemble the context.

Context is one lever. The other is how often the harness calls the model. Every tool round trip creates another call, planning, verification, and reflection on top.

TrueFoundry’s open harness, TrueForge, works both levers. It splits context into what the model needs at the start and what accumulates while it executes.

What TrueForge loads before the run starts

TrueForge keeps skills and tool definitions lightweight at startup, then loads details when the agent needs them.

Skills are git-backed SKILL.md packs. The agent starts with the name & description and reads the body from the sandbox only when the skill is relevant.

Tool definitions default to deferred. With preload set to false, the agent starts with the MCP server’s name and description instead of every schema, then discovers what it needs through four calls:

1. list_tools returns the tool names available on a given MCP server.

2. get_tool_info returns one tool’s description with its input-output schema.

3. get_tool_output_schema returns the output shape alone, read before writing a Code Mode script so the agent isn’t guessing keys from raw JSON.

4. call_tool invokes a tool on a server by name once the agent knows its need.

If an internal platform server exposes 100 tools and the agent uses two, the other 98 never enter the prompt.

The bottom line is load what the run needs, not everything the system supports.

What happens during the run

Take a support question that needs information from several systems.

Which accounts have the most open tickets right now, and what are they about?

That means querying an issue tracker, matching tickets to accounts in a CRM, and reading documents. The tool calls are manageable, but the data return is not.

Large tool responses go to disk

Suppose the tracker returns 400 open tickets, each with a title, description, comments, labels, assignee, and timestamps. The agent needs the account ID and subject line, and the rest rides into every following model call.

TrueForge writes large responses to a sandbox file, keeping a short preview and the path in context. The agent reads/parses that file when it needs the full result.

Parallel calls create another case. If several responses return together and cross the context budget, TrueForge offloads the largest first until the batch fits.

Subagents keep intermediate data out of the root context

Now suppose twelve accounts come back, each needing its own lookups.

The root agent can work through them one by one, carrying every lookup and raw record in its own context. Or it creates a subagent per account.

Each subagent works in its own context and returns a short summary, so the root sees twelve summaries instead of twelve accounts’ worth of raw records.

That is context isolation on top of parallel execution.

Code Mode collapses tool chains into one script

Counting tickets per account is a data join.

Without Code Mode, the agent pulls both sets, brings them into context, matches the IDs, and counts across several model runs.

With Code Mode, it writes a Python script that calls both tools, joins them in code, and prints the table. The IDs stay inside the script instead of passing through a model run, and only the output enters context.

Compaction summarizes history past 50,000 tokens

The first three control what enters context during the run. Compaction handles the case where the conversation itself becomes the payload.

Past the default 50,000-token threshold, TrueForge writes a structured summary of the intent, decisions, files and artifacts, errors and fixes, and next steps, and that replaces the older messages. The full event history stays on the server.

The four strategies in a single run

A single run can use all four, and they reduce model workload in different ways:

• Offloading and compaction cut the context carried forward

• Subagents move intermediate work into separate contexts

• Code Mode removes multi-tool data processing from the model loop

The runtime controls both things we started with: what the model reads and how many times it works through it.

Why the sandbox never holds your credentials

Reducing context helps only if the runtime can safely let the agent touch real systems.

Running generated code in a sandbox is the usual isolation. But a sandbox holding the model’s API keys and MCP credentials gives that code the same secrets as the harness.

TrueForge separates them. The harness and its credentials stay on the server, and the sandbox handles code, files, and shell.

When a Code Mode script calls call_tool, the request routes back through the harness, which applies the stored credentials, calls the MCP server, and returns the result to the sandbox.

import asyncio
from collections import Counter
from mcp_client import call_tool

async def main():
    tickets = await call_tool(
        “tracker”,
        “search_issues”,
        body={”status”: “open”, “limit”: 500},
    )
    accounts = await call_tool(
        “crm”,
        “query_records”,
        body={”object”: “Account”, “fields”: [”id”, “name”]},
    )

    names = {a[”id”]: a[”name”] for a in accounts[”records”]}
    counts = Counter(t[”account_id”] for t in tickets[”issues”])

    for account_id, count in counts.most_common(10):
        print(f”{names.get(account_id, account_id):<28} {count:>4}”)

asyncio.run(main())

The 400 tickets and the account records stay in the sandbox, and the script never receives the credential that authenticates it.

Approvals apply here too. A script invoking a gated tool pauses for approval, so Code Mode is not a backdoor around the permission model.

What it takes to build this yourself

None of these strategies is complicated in isolation. The production behavior is what takes longer:

• Offloading has to decide which of several parallel responses to remove first.

• Code Mode has to let generated code call MCP tools without ever receiving their credentials.

• Deferred loading has to expose a tool’s output shape without loading the entire schema.

The same pattern continues underneath the agent. Production runtime needs sessions that survive restarts, streams clients that can rejoin, approvals that stay pending, credentials you can refresh without exposing them to generated code, and accounting per model message to show where the workload came from.

Describing a single optimization is easy. Making them work together, safely and predictably, is the runtime engineering.

Adopting an open harness

So you adopt a runtime instead of building one. The next question is how much control you have over it.

With a closed harness, the layer deciding what context reaches the model, which tools it calls, and how often it runs is code you cannot inspect or change. When a run behaves strangely, you see the result and not the machinery producing it.

TrueForge is MIT licensed and self-hostable. You can read how it manages context, loads tools, and controls execution, and change that behavior when the defaults do not fit. Self-hosting also keeps the data in your environment.

The benchmark that follows puts the difference into numbers.

Benchmarks

TrueForge is tested against Claude Managed Agents and deepagents on 14 tasks from DevRev’s Enterprise-Bench, pulling from a Jira-style tracker, a Salesforce-style CRM, and a Drive-style document store, all exposed through MCP servers.

Each runtime got the same model, tools, and a fresh session per task. An LLM judge scores answers against the criteria without knowing which harness produced it, and a task counts as solved only if it meets every one.

Starting with task success, where it is a three-way tie. Across 14 tasks, a one-task difference does not show that one harness solves tasks better.

The difference worth reading is how much work each harness did to get there.

• TrueForge reached the same answers on about 40% of Claude Managed Agents’ tokens and under a quarter of deepagents’.

• It also finished faster, at 40 minutes per run against 63 and 64.

The gap comes down to how much each harness carries. TrueForge drives from a compact instruction, plans fewer tool calls, and trims history. The deepagents library carries planning, a virtual filesystem, and a subagent on every turn.

Changing the model is the other lever. TrueForge with GLM-5.2 solved roughly the same number of tasks at 75% lower cost than Claude Agents on Opus 4.8.

The benchmark harness, system prompt, and grader are all open-source, so you can run the same setup against your own models and datasets.

TrueFoundry’s full agent-harness benchmark →

What it already runs

TrueForge runs AskTFY, the copilot inside TrueFoundry’s AI Gateway, where it works across production traces to investigate failures and surface patterns.

NetApp and Automatiq are building agents on it too, with Daytona supplying the sandbox and search, inference, and guardrail providers plugging in around it. The harness is the execution layer, and the systems around it change underneath.

Running it on your machine

TrueForge runs locally as a single process on SQLite. The only requirement is Node 22.13 or newer.

npx @truefoundry/trueforge

Open http://localhost:8790 and configure the runtime once in Settings, after which every agent reuses it.

Local mode has no login, so keep it on your own machine. To serve multiple users, run the same runtime in hosted mode, backed by Postgres and Redis and deployed through Docker Compose or Helm.

Add your model provider under Settings → Models, connect MCP servers under Connectors, and add a Daytona key under Sandbox providers for sandboxed code execution, file offloading, and skills.

Then create an agent. Choose the model, attach connectors and skills, add instructions, and save it to the library.

The chat UI is the fastest way in, but the deeper controls live in the agent spec, and everything runs through the API and SDK. You can build agents into your own product or theme the UI into a branded interface backed by your server.

Conclusion and final takeaway

Two harnesses ran the same model on the same 14 tasks and landed within a task of each other. One spent 2.7× the tokens.

That difference came from what each runtime put in front of the model on every run. You either make those decisions deliberately or inherit them by default.

Harness complexity shrinks as models improve. Manus was rebuilt five times, each pass deleting machinery the model no longer needed, and Anthropic has removed planning steps from Claude Code as newer models internalize them.

Some parts are harder to replace. A better reasoning model cannot stop the harness from sending a 50,000-token payload sixteen times.

That makes the harness worth engineering on purpose. The models will keep changing, and the runtime will still decide what each one sees, how often it runs, and what carries forward.

Check out TrueForge (MIT licensed) on GitHub →

Explore the documentation and quickstart →

👉 Over to you: when an agent run costs more than you expected, do you check the model first or the trace first?

Thanks to TrueFoundry for working with us on this issue!

Adding a KV cache layer to your inference stack can cost you throughput, due to a reason that’s unrelated to caching per se.

Cache management runs inside the inference engine’s process today, and the two jobs have opposite bottlenecks:

• Moving cached tensors between GPU, CPU, and disk waits on I/O.

• Running attention is heavy math on the GPU.

In one process, the engine runs them one at a time, so the GPU sits idle while blocks move between memory tiers.

Google’s TurboQuant shrinks the KV cache to 3 bits per value with no accuracy loss. But if you run it inside the engine, you still lose more than 20% of inference throughput.

The compression works, but it still costs based on where it runs.

LMCache (open-source) pulls cache management into its own process next to the engine. The two share GPU memory and pass a short list of block IDs between them, nothing heavier.

Cache work stops stalling inference.

Two GPUs read the same memory instead of copying tensors back and forth. LMCache searches GPU memory, CPU RAM, local SSD and cloud storage at the same time, then streams from whichever one answers first.

On H200s running Qwen3-235B with 50 concurrent users, that is 14x faster time-to-first-token, 4x faster decoding, and a startup that drops from minutes to seconds. The full benchmarks are in the repo.

It works with vLLM, SGLang, and TensorRT-LLM, on NVIDIA and AMD, and it is open-source.

GitHub repo: https://github.com/LMCache/LMCache

(don’t forget to star it ⭐️)

How semantic code navigation cuts agent token costs by up to 36%

Microsoft cancelled Claude Code for 5,000 engineers this year, after token costs climbed to $500 to $2,000 a month per engineer.

Microsoft wasn’t alone. Uber’s CTO said the company went through its entire 2026 AI coding budget in four months.

Both stories were framed around the bill. Neither asked what the agent was actually doing with the tokens.

Here’s the problem. Most teams assume the answer is writing code, and that more output means a higher bill.

That assumption doesn’t hold up once you look at what an agent actually does in a session.

A coding agent starts a task with no map of your codebase. You describe what should change, and the agent has to find where that change belongs before it writes a single line.

The finding is where the tokens go, not the writing.

In this article, we’ll look at exactly how that finding breaks down, and what a different approach to it looks like in practice.

When the map is missing

An agent has one tool to build a map of your codebase, text search. For a lot of tasks, that’s enough.

The trouble starts when the name you search for doesn’t line up with the place you actually need to change, and it breaks in three distinct ways.

First, a name can appear at every location that needs to change and also at hundreds of locations that don’t. The agent has no way to tell the difference except to open and read each match.

That costs tokens and time before it writes anything.

Second, two things can share a name and mean something different. One method might take different arguments than another with the same name.

A local variable can shadow a field. The text lines up, the identity doesn’t, and telling them apart requires knowing what each one actually refers to.

Third, sometimes the code you need to find shares no words with what you searched for. It’s connected structurally instead, a class implementing an interface without ever naming it near the code that changes.

A method can be invoked without its name ever appearing in the code that calls it, through a callback or a dynamic reference. Text search has no way to find something it was never given the words for.

First and second cost you tokens. The agent reads more, reasons longer, spends more to reach the same answer it would have gotten instantly from a real map.

The third doesn’t cost you tokens the same way. A missing structural connection shows up in two different situations, and only one of them is risky.

A missed location in a straightforward rename fails loudly, the build breaks, and the failure is visible right away. That case is safe, since you catch it immediately.

A missed location in a behavior change reached through indirection is the risky one. Nobody wrote a test for a connection nobody knew existed.

So the code compiles, the existing tests pass, and the change ships.

The bug surfaces somewhere else, later, with no obvious link back to what caused it.

This is the part of the token conversation that budget dashboards don’t capture.

Semantic code navigation

If text search is the wrong tool for this, the direct fix is to stop treating the codebase as a body of text at all.

Instead, treat it as a graph. Classes, methods, fields, and interfaces become nodes.

The real relationships between them, calls, implements, extends, references, become edges. Each node and edge carries the exact file and line it lives at.

An agent working against that graph doesn’t search for a name and hope the match is relevant. It asks a direct structural question instead, “find every place that implements this interface.”

It gets back exact locations, not a list of string matches it still has to read through and reason about.

This is conceptually close to what an IDE already does when it offers “find all references” or “go to implementation.”

The difference is the agent can ask that question directly, mid-task, instead of relying on a human to look it up first.

A different way to navigate code

Semantic code navigation solves where the agent looks. It doesn’t solve whether what it writes there is correct.

That gap is worth keeping in mind as you read what this actually looks like built.

A graph like this can be rebuilt without a compiler or language server. That means it stays usable on code that doesn’t currently compile, which is the normal state of code mid-edit.

That matters because the agent’s map stays current while it’s actively changing things, instead of going stale until the next clean build.

A controlled test measured this against real, previously merged commits from open source projects. Each task ran ten times per side, once with a plain agent and once with the same agent given access to this kind of navigation.

Every run was required to pass the actual build and tests before it counted.

Across six tasks in four languages, cost fell in every one, from 5% on a straightforward change up to 36% on a Java interface change.

The pattern behind the wins was consistent. Every task where cost fell significantly involved a change that had to land identically across every related location, a shared interface, a renamed package, a consistent argument order.

That’s exactly the situation where text search can’t cleanly enumerate every site on its own.

The same system that finds the location also needs to verify what actually gets written there, as the code is produced rather than after the PR exists.

The reasoning for keeping that check separate from the agent that wrote the code is straightforward. A model grading its own work has no independent way to catch what it already believed was correct.

A different methodology, run by something other than the model itself, is what actually catches it.

This is what Sonar built into their product Sonar Vortex, a Semantic navigation and real-time verification running inside the agent’s own loop.

The takeaway

Microsoft and Uber both watched a token bill climb past what anyone had modeled for it.

Neither story asked the question that actually explains the number.

Whether the agents doing that work were finding everything they needed to find, or just running longer because they never had a map to begin with.

This is the real cost of agentic coding at scale, and it doesn’t show up as a line item.

It shows up as every session an agent spends searching for something a real map would have handed it directly.

The next time a token bill looks larger than it should, ask what the agent was actually looking for before you switch tools or cut usage.

You can read the full study here →

Thanks to Sonar for partnering today!

Label smoothing for regularization

The entire probability mass belongs to just one class in typical classification problems, and the rest are zero:

This can sometimes impact its generalization capabilities since it can excessively motivate the model to learn the true class for every sample.

​Regularising​ with Label smoothing addresses this issue by reducing the probability mass of the true class and distributing it to other classes:

In the experiment below, I trained two neural networks on the Fashion MNIST dataset with the same weight initialization.

• One without label smoothing.

• Another with label smoothing.

The model with label smoothing (right) resulted in better test accuracy, i.e., better generalization.

When not to use label smoothing?

Label smoothing is recommended only if you only care about getting the final prediction correct.

But don’t use it if you also care about the model’s confidence since label smoothing directs the model to become “less overconfident” in its predictions, resulting in a drop in the confidence values for every prediction:

That said, L2 regularization is another common way to regularize models. Here’s a guide that explains its probabilistic origin: ​The Probabilistic Origin of Regularization​.

Good day!

• ​RAG vs. CAG, explained visually!​

• What is (was?) GIL in Python?

• ​​What is Contrastive Learning?​

​RAG vs. CAG, explained visually!​

RAG is great, but it has a major problem:

Every query hits the vector DB, even for static information that hasn’t changed in months.

This is expensive, slow, and unnecessary.

Cache-Augmented Generation (CAG) fixes this by letting the model keep static information in its key-value (KV) memory, which is what the model builds internally for every token it reads.

And you can take this one step ahead by fusing RAG and CAG as depicted below:

Here’s how it works in simple terms:

• In a regular RAG setup, your query goes to the vector database, retrieves relevant chunks, and feeds them to the LLM.

• But in RAG + CAG, you divide your knowledge into two layers.

  • The static, rarely changing data, like company policies or reference guides, gets cached once inside the model’s KV memory.

  • The dynamic, frequently updated data, like recent customer interactions or live documents, continues to be fetched via retrieval.

This way, you get faster inference, lower costs, and less repeated work.

The trick is being selective about what you cache.

Only cache static, high-value knowledge that rarely changes.

If you cache everything and you’ll hit context limits. separating “cold” (cacheable) and “hot” (retrievable) data keeps this system reliable.

To use this, you can actually start right away since OpenAI and Anthropic already support prompt caching in their APIs.

One thing to know before you scale it.

Prompt caching matches on an exact prefix, byte for byte. So a cached layer only gets reused when it sits at the very front of the context in the same order every time.

So if you:

↳ so reorder two cached policy documents and both turn into a miss

↳ cache document A alone and document B alone, then query both, and the second one misses because the model computed its cached state without ever seeing the first

In production, this looks like a small fraction of your cached blocks serving almost all the hits. The rest just sits there.

The way out comes from how attention behaves. Tokens attend mostly to their own local neighborhood, and only a few reach across document boundaries.

CacheBlend is a technique that recomputes those few and reuses everything else from the separately cached documents.

Multi-document queries run two to four times faster, quality holds, and order stops mattering.

The method is implemented in LMCache, which is fully open source.

GitHub repo: ​https://github.com/LMCache/LMCache​

(don’t forget to star 🌟)

What is (was?) GIL in Python?

For years, the GIL (global interpreter lock) has been the single biggest bottleneck for multi-threaded Python code.

With Python 3.14, you can run Python without the GIL for the first time. Here’s everything you need to know about GIL in Python.

Let’s dive in to learn more today!

Some fundamentals

• A process is isolated from other processes and operates in its own memory space. This isolation means that if one process crashes, it typically does not affect other processes.

• Multi-threading occurs when a single process has multiple threads. These threads share the same resources, like memory.

What is GIL?

Simply put, GIL (global interpreter lock) restricts a process from running more than ONE thread at a time.

So essentially, a process can have multiple threads, but ONLY ONE can run at a given time.

This means the process cannot use multiple CPU cores for performance optimization, which means multi-threading leads to similar performance as single-threading.

Let’s understand with a code demo!

• First, we start with some imports and define a long function:

• Single threading, wherein we invoke the same function twice, takes 0.432 seconds.

• With multi-threading, we create two threads, one for each function, and this takes 0.428 seconds:

The reason for similar run-time, despite multi-threading, is…

GIL.

On a side note, we do experience a run-time boost with multi-processing:

The above three scenarios (single-threading, multi-threading, and multi-processing) can be explained visually as follows:

• Single-threading: A single thread executes the same function twice in order:

• Multi-threading: Each thread is assigned the job to execute the function once. But due to GIL, only one thread can run at a time:

• Multi-processing: Each function is executed under a different process:

If this is clear, let’s answer two questions now:

1) Why has Python been using GIL even when it is suboptimal?

Thread safety.

When multiple threads run in a process and share the same resources (such as memory), problems can arise when they try to access and modify the same data.

For instance, say we want to run two operations with two threads on a Python list:

• If t1 runs before t2, we get the following output:

• If t2 runs before t1, we get the following output:

We get different outputs!

This can lead to race conditions, where the outcome depends on the timing of the threads’ execution.

This, and a few more reasons, made it convenient to execute just one thread at a time.

On a side note, GIL usually affects CPU-bound tasks and not I/O-bound tasks, where multi-threading can still be useful.

2) If multi-processing works, why not use that as a workaround?

This is easier said than done.

Unlike threads, which share the same memory space, processes are isolated.

As a result, they cannot directly share data as threads do.

While there are inter-process communication (IPC) mechanisms like pipes, queues, or shared memory to exchange information between processes, they add a ton of complexity.

Thankfully, Python 3.14 allows us to disable GIL, which means a process can fully utilize all CPU cores.

This video depicts the run-time difference:

We have been testing Python 3.14 lately, and we’ll share these updates in a detailed newsletter issue soon.

That said, if you want to get hands-on with actual GPU programming using CUDA, learn about how CUDA operates GPU’s threads, blocks, grids (with visuals), etc., we covered it here: ​Implementing (Massively) Parallelized CUDA Programs From Scratch Using CUDA Programming.

👉 Over to you: What are some other reasons for enforcing GIL in Python?

​What is Contrastive Learning?​

Contrastive Learning is a popular self-supervised learning technique that teaches models to learn useful representations by comparing samples.

Let’s understand more by considering a real-world task.

As an ML engineer, imagine you are responsible for building a face unlock system.

Let’s look through some possible options:

1) Building a Binary classifier

Output 1 if the true user is opening the mobile; 0 otherwise.

Initially, you can ask the user to input facial data to train the model.

But that’s where you identify the problem.

Their inputs will belong to “Class 1.”

Now, you can’t ask the user to find someone to volunteer for “Class 0” samples since it’s too much hassle for them.

Also, you need diverse “Class 0” samples. Samples from just one or two faces might not be sufficient.

The next possible solution you think of is…

Maybe ship some negative samples (Class 0) to the device to train the model.

Might work.

But then you realize another problem:

What if another person wants to use the same device?

Since all new samples will belong to the “new face” during adaptation, what if the model forgets the first face?

2) How about transfer learning?

• Train a neural network model (base model) on some related task → This will happen before shipping the model to the user’s device.

• Next, replace the last few layers of the base model with untrained layers and ship it to the device.

The first few layers would have learned to identify the key facial features, and from there on, training on the user’s face won’t require much data.

But yet again, you realize that you shall run into the same problems you observed with the binary classification model, since the new layers will still be designed to predict 1 or 0.

Solution: Contrastive learning using Siamese Networks

At its core, a Siamese network determines whether two inputs are similar.

It does this by learning to effectively map both inputs to a shared embedding space (the blue layer above):

• If the distance between the embeddings is LOW, they are similar.

• If the distance between the embeddings is HIGH, they are dissimilar.

They are beneficial for tasks where the goal is to compare two data points rather than to classify them into predefined categories/classes.

This is how it will work in our case:

Create a dataset of face pairs:

• If a pair belongs to the same person, the true label will be 0.

• If a pair belongs to different people, the true label will be 1.

After creating this data, define a network like this:

Pass both inputs through the same network to generate two embeddings.

• If the true label is 0 (same person) → minimize the distance between the two embeddings.

• If the true label is 1 (different person) → maximize the distance between the two embeddings.

Contrastive loss (defined below) helps us train such a model:

where:

• y is the true label.

• D is the distance between two embeddings.

• margin is a hyperparameter, typically greater than 1.

Here’s how this particular loss function helps:

• When y=1 (different people), the loss will be the following, which will be minimum when D is close to the margin value, leading to more distance between the embeddings.

• When y=0 (same person), the loss will be the following, which will be minimum when D is close to 0, leading to a low distance between the embeddings.

This way, we can ensure that:

• when the inputs are similar, they lie closer in the embedding space.

• when the inputs are dissimilar, they lie far in the embedding space.

Siamese Networks in face unlock

Here’s how it will help in the face unlock application.

First, train the model on several image pairs using contrastive loss.

This model (likely after model compression​) will be shipped to the user’s device.

During the setup phase, the user will provide facial data, which will create a user embedding:

This embedding will be stored in the device’s memory.

Next, when the user wants to unlock the mobile, a new embedding can be generated and compared against the available embedding:

• Action: Unlock the mobile if the distance is small.

Done!

Note that no further training was required here, like in the earlier case of binary classification.

Also, what if multiple people want to add their face IDs?

No problem.

Create another embedding for the new user.

During unlock, compare the incoming user against all stored embeddings.

Here’s some further hands-on reading to learn how to build on-device ML applications:

• Learn how to build privacy-first ML systems (with implementations): ​Federated Learning: A Critical Step Towards Privacy-Preserving Machine Learning.

• Learn how to compress ML models and reduce costs: ​Model Compression: A Critical Step Towards Efficient Machine Learning.

👉 Over to you: Siamese Networks are not the only way to solve this problem. What other architectures can work?

Good day!

• The harness decides your token bill, not the model.

• ​Kimi K3’s sandbox problem finally has an open-source fix.

The harness decides your token bill, not the model

A large share of an agent’s token bill is the model rereading things it already read.

That isn’t the model’s doing. The runtime around it decides what goes into every prompt and how often the model gets called.

For example, an agent queries a CRM at step four and gets back 400 rows. Those rows pile up in the conversation history.

By step nineteen, the model has read those rows fifteen more times, and you pay input rates for every one of them.

It happened because your harness assembled that prompt on every turn and kept the rows in it.

That gives you two levers: how much context the harness carries forward, and how often it calls the model.

There are four practical ways to keep the prompt from growing:

• Load tool schemas on demand. A server with 100 tools doesn’t need to put all 100 into every prompt when the agent only calls two.

• Offload large results to disk. Turn a large response into a short preview and a file path instead of replaying the entire result on every turn.

• Delegate to subagents. Let a subagent spend thirty tool calls in its own context and return one summary to the root agent.

• Run toolchains in code. One script calls three tools, joins the results, and returns a table instead of three turns each dragging a full response.

But reducing context is only half the job. You also need to control how often the model gets called.

A good harness should avoid unnecessary planning, verification, and reflection when the work can be completed in fewer steps.

TrueFoundry’s open-source agent harness, TrueForge, is built around both of those controls.

It sits between the model and the tools, deciding what goes into every prompt and when another model call is needed. It also breaks token usage down across the harness, skills, instructions, tools, and messages.

DevRev’s Enterprise-Bench is where this gets tested, on multi-step tasks of the kind where an agent pulls records from one system and reconciles them against another.

TrueFoundry ran TrueForge there against Claude Managed Agents, both on the same model, and both finished the same number of tasks.

The tie is the part that matters, because it means the gap underneath is not a quality tradeoff.

TrueForge reached that score on close to a third of the tokens, with roughly 40% fewer trips back to the model. For the same result, that comes out around 2.7x cheaper than Claude Managed Agents.

Swapping in an open model went further. TrueForge with GLM-5.2 scored a little higher than either setup above, and the entire benchmark run cost about $3 at list prices.

Since it’s open-source, you can swap the model underneath without rewriting the agent, and run the whole thing inside your own environment when the data cannot leave it.

All of this comes down to the runtime around the model, the context it carries, the tools it exposes, and how many times it goes back to the model.

That is what a production harness owns.

The full task list, the per-run numbers, and the MIT-licensed code are on GitHub:

Here’s the GitHub repo →

(Don’t forget to star 🌟)

Kimi K3’s sandbox problem finally has an open-source fix

Training Kimi K3 required over 51 million sandboxes across 1.5 million images, and no existing tool could handle all three things they needed at once.

The three requirements were strong workload isolation, native GPU access inside the sandbox, and fast environment forking for parallel agent trajectories.

The Kimi team built AgentENV from scratch to get all three together.

AgentENV runs isolated microVMs with Firecracker, supports forking a running sandbox from its exact state without side effects, and at peak was spinning up tens of thousands of sandboxes within seconds.

Building that took a dedicated infrastructure team. Most agents running today execute inside containers, which share the host kernel, the core layer of the operating system every process on that machine runs on.

Others run inside stitched runtimes that each solve one of the three properties but not all of them together.

smolvm (GitHub Repo) is an open-source VM runtime built to cover all three in a single binary.

The way it works is that each workload runs in its own isolated VM with GPU access and the ability to fork the environment mid-run, with no separate runtimes to manage.

This article walks through setting up smolvm, building an isolated agent environment, and packing it into a single portable artifact.

But before the build, let's understand why getting all three sandbox properties right is harder than most teams expect.

The sandbox problem in agentic systems

Running one agent in a sandbox is manageable. Fifty parallel trajectories from the same checkpoint, each isolated and with GPU access, is where no single tool keeps up.

Consider a coding agent inside a container, executing code it just wrote. That code runs on the same kernel as every other process on the host.

The container draws a process-level boundary around it, but the kernel underneath is shared. If that kernel gets exploited, every other workload on the machine is reachable too.

GPU access makes the picture worse. Most VM runtimes cannot expose a GPU inside the sandbox at all, and the container runtimes that can are back to sharing the host kernel.

So with today’s tools, gaining the GPU means giving the isolation back.

Forking is the property almost nothing supports.

Say you want to try five different next steps from the same point in an agent’s run. You either replay the whole run five times to get there, or you fork the live environment and throw away the copies you don’t need.

Most runtimes only give you the first option.

It works the same way as forking a repo on GitHub, you get an exact copy of the current state, and it runs independently from there. Nothing you do inside the fork touches the original.

This is the gap the Kimi team hit at 51 million sandboxes, and most teams building agentic systems hit it quietly, long before that scale.

smolvm covers all three from one runtime, without stitching anything together.

How smolvm works

smolvm gives every workload its own complete virtual machine, not a slice of the host’s.

If something goes wrong inside one, it stays there and never reaches the others.

Think of it like a hotel instead of a shared apartment. Every guest gets their own room, their own walls, and their own utilities, instead of splitting one kitchen and one hallway with the rest of the building.

That kind of separation usually comes at a cost, a slow start while a full virtual machine loads its hardware before it is even ready.

smolvm skips that step and starts the machine directly, which is why it still boots in under 200ms, against the fifteen to thirty seconds a typical VM takes.

You also do not have to guess how much RAM or CPU a workload will need upfront.

Each machine defaults to 4 vCPUs and 8 GiB of RAM, but smolvm only holds onto what the workload is actually using at any moment and gives the rest back automatically as usage drops.

On top of the isolation, smolvm layers in a few more things:

• OCI images, so anything on Docker Hub runs directly with no Docker installation needed.

• GPU access, a real GPU inside the VM, with the host driver doing the actual compute.

• Forking, save a running environment’s exact state and start an independent copy from it.

This walkthrough demos isolation and GPU access on the CLI. Forking is available today through the runtime and SDK.

Now let’s set it up, build an isolated agent environment with GPU access, and pack it into a single portable artifact that runs anywhere.

Setup

smolvm installs with one command on macOS and Linux:

curl -sSL https://smolmachines.com/install.sh | bash

Once installed, we spin up an ephemeral VM to verify the environment is running:

smolvm machine run --net --image alpine -- sh -c "echo 'Hello from a microVM' && uname -a"

The VM runs the command and cleans itself up on exit. Nothing persists between runs unless we explicitly create a named machine.

For this walkthrough, we need a persistent environment. We use python:3.12-alpine as the base image so Python is available inside the VM from the start:

smolvm machine create --net --image python:3.12-alpine --name agentvm
smolvm machine start --name agentvm

From here we can run commands inside it and install packages, and the state survives restarts.

Verifying isolation and GPU access

On macOS, GPU support works out of the box with no extra installs.

On Linux, you’ll need to install a host Vulkan driver first from your system package manager:

# Alpine
apk add virglrenderer mesa-vulkan-intel

# Debian/Ubuntu
apt install virglrenderer0 mesa-vulkan-drivers

Once the host dependencies are in place (or on macOS, right away), we can launch a GPU-enabled VM:

smolvm machine run --gpu --net --image python:3.12-alpine -- python3 -c "import sys; print('GPU VM running Python', sys.version)"

The output confirms the guest has a working GPU device, with the host still doing the actual compute behind it.

For CUDA workloads on NVIDIA hardware, smolvm supports a separate --cuda flag. Unlike --gpu, which gives the guest a general graphics device, --cuda routes CUDA calls through the host NVIDIA driver without installing a driver inside the VM.

Network access is off by default. An agent running inside smolvm cannot reach the host network unless we explicitly pass --net, and even then we can restrict it to specific hosts only:

# no network by default, this fails
smolvm machine run --image alpine -- ping -c 1 1.1.1.1

# allowed host only, this succeeds
smolvm machine run --net --image alpine --allow-host registry.npmjs.org -- wget -q -O /dev/null https://registry.npmjs.org

# host not in allow list, this fails
smolvm machine run --net --image alpine --allow-host registry.npmjs.org -- wget -q -O /dev/null https://google.com

Each workload runs in its own VM with its own kernel. Even if agent code crashes or tries to read host files, it cannot reach anything outside its own VM.

The host filesystem, credentials, and network are invisible from inside unless explicitly shared.

With the environment built and the isolation verified, the next step is packing it up.

Packing it into a portable artifact

Once the environment is configured, we pack its entire state into a portable artifact:

smolvm pack create --image python:3.12-alpine -o ./agentvm

This produces two files. agentvm is a self-contained stub binary, and agentvm.smolmachine holds everything needed to rebuild the environment exactly as it is now.

To share this environment, anyone with the .smolmachine file can create a named machine from it and start the exact same setup, with no image pull and no extra steps:

smolvm machine create --name agentvm-copy --from agentvm.smolmachine
smolvm machine start --name agentvm-copy
smolvm machine exec --name agentvm-copy -- python3 --version

The Kimi team had to build custom infrastructure to get this same result.

Here, the same isolated environment with GPU access runs identically whether it's on a laptop, a teammate's machine, or a production server.

When containers are still the right call

smolvm is not a container replacement for every workload. Containers are faster to build and the ecosystem around them is larger.

For stateless services that do not execute untrusted code, the shared kernel is not a meaningful risk.

smolvm earns its place when the workload executes untrusted code, needs GPU access inside the sandbox, or needs to branch a live environment mid-run.

For agent systems that need all three, it is currently the only open-source option that covers them without stitching runtimes together.

What Kimi had to build, you can run today

The Kimi K3 paper describes 51 million sandboxes built on custom infrastructure that took a dedicated team to engineer.

The three properties they needed, isolation, GPU access, and fast forking, had no single open-source answer at the time.

In this walkthrough, we went from a single install command to a portable agent environment with GPU access and network isolation that boots anywhere in under 200ms.

None of that required a cloud dependency, a separate runtime for each property, or a dedicated infrastructure team.

smolvm is Apache-2.0 licensed and runs entirely on your own machine.

smolvm GitHub repo → (5.5k stars)

(Don’t forget to star it ⭐)

Good day!

• ​A technical LLM interview question.

• Grok Bot Masterclass.

• ​6 types of contexts for Agents.

A technical LLM interview question:

Your LLM chatbot replies to a query in about 3 seconds, and users are complaining.

So you move the model onto a GPU with 3x the memory bandwidth.

The end-to-end latency barely improves.

Why did this happen?

Continue reading to learn more:

Latency in an LLM app is a placement problem disguised as a model problem.

If you profile the 3 seconds, the model itself may usually account for around 400ms of it.

So doubling the generation saves just 200ms out of 3000, which is under 7%.

The rest is spread across stages that never touch the GPU.

The request first travels to whatever region the app runs in, and a cross-continent round trip costs a few hundred milliseconds before any code executes.

Then the request handler starts. On a container-based serverless platform, this adds another 100 to 500 ms of cold start, paid before auth, rate limiting, or prompt assembly even begins.

Retrieval adds its own hop, and the response streams back across the same distance.

Optimizing a stage that was already fast cannot alter the latency that’s majorly affected by other stages.

Those other stages are slow for a structural reason.

An LLM app runs two workloads that want opposite machines.

• The request path is short, spiky, and needs to sit close to users

• Inference is long-running, GPU-bound, and billed hourly, whether requests arrive or not.

So the actual decision is not which model to run, but where each of these two workloads runs.

There are three options, each with its own tradeoffs:

> A dedicated GPU box removes inference cold starts, but it bills around the clock and lives in one location, so distant users wait out the round trip on every request

> Container-based serverless scales to zero, but the request path pays a cold start, and most of these platforms have no GPU behind them.

> Edge runtimes start in under a millisecond, because a WebAssembly module carries no OS or container image to boot. They handle the request path well and cannot hold a model.

So the answer is not to pick one, but to split the app across two of them.

The request path runs close to users, and inference runs on a dedicated GPU it calls into.

That also explains the failed upgrade. More memory bandwidth made a stage that was already fast faster, and left the 2.6 seconds around it untouched.

To actually learn how it’s done in practice, Akamai’s GitHub has a reference implementation for each half:

• vllm-on-lke serves Qwen2.5-7B-Instruct behind an OpenAI-compatible endpoint on one RTX 4000 Ada GPU in Linode Kubernetes Engine, with Terraform creating the cluster, both firewalls, and the GPU operator in one apply.

• akamai-functions-llm-chatbot covers the front, where a WebAssembly API checks a KV cache and only calls the GPU-backed instance on a miss.

Both are available on Akamai’s new Developer Hub, alongside several production-grade tutorials and code samples.

It also links to Edge Case, their Discord, where four developer advocates architect and deploy a production app live every other Wednesday.

If you create a new Akamai Cloud account, you can also get $300 in credits for joining.

Join here: developers.akamai.com

Thanks to Akamai for partnering today!

Grok Bot Masterclass

There is a habit that spread quietly among people running AI agents this year. They leave the laptop lid half-open so the agent can keep working. When they close it, the run dies.

That habit belongs to agents that run on your own machine. The hosted ones solved the lid problem long ago, and over the past year they have been solving the memory problem too, in pieces.

For instance:

• Devin restores a saved machine snapshot at the start of every session, so anything you did not commit or capture is gone.

• Manus puts its sandbox to sleep between tasks and recycles it after enough inactivity, with an always-on cloud computer available.

• ChatGPT Work keeps browser cookies, so once you sign in to a site through the takeover flow, later runs pick it up.

Each of those persists one thing, for one agent, usually scoped to a session or a project. You still get a machine per task, and the roster of agents does not share anything.

SpaceXAI launched Grok Bot last week. Instead of a machine per task, you get a machine per person. It stays on, it is tied to your account, and the whole roster of Bots shares it.

This guide covers what a Bot actually is, how the system is put together underneath, how it compares to Hermes Agent and the other options, how to set it up on macOS, and which roles are worth building first.

By the end, you will have a working roster on your own machine, a saved skill, and a routine that runs while your laptop is closed.

Let’s begin!

​Due to a size limit, the email has been cut off. You can read the full Grok Bot masterclass issue on our website here →

What Grok Bot is

The one-line version is that a Bot is a named teammate you message, and the actual computing happens one layer below it.

The word “bot” suggests a separate program running somewhere. It is closer to a saved role with its own memory, sitting on a machine it shares with every other role you create.

A Bot is defined by three things you write:

• A short name

• One primary job

• And a description of how it should work.

In exchange, you get a persistent conversation thread, and the Bot keeps working context across turns.

As per the official docs, a Bot retains stable working preferences, important facts, and summaries from its work. It does not replay every prior message.

The computer is the other half. Every Bot on your account uses one persistent cloud computer with a browser, a filesystem, and a terminal. It is assigned to your user account, not to any individual Bot.

Each Bot gets its own screen on that shared machine. Several Bots can therefore drive browser and desktop tools at the same time, though one Bot runs one computer-use task on its screen at a time

In the docs and in the app, a Bot means one persistent named agent. It is not a separate machine, a separate container, or a separate model instance.

As a useful analogy, you can relate it to an office setting.

The cloud computer is the office, and each Bot is a coworker with their own desk. There are different people, jobs, and notebooks, but one filing cabinet and one set of keys.

That analogy also explains the trade-off, which we will return to in detail. Shared keys make handoffs free. They also mean the roster shares one set of consequences.

In summary, a Bot is a role plus a memory plus a thread. The machine underneath is singular, persistent, and shared.

How it is built

Before the setup, let’s look at how all the pieces fit together.

Specifically, there are five parts you should know about Grok Bot:

The persistent cloud computer

The computer runs in SpaceXAI’s cloud, not on your Mac. So closing the app or the laptop does not stop a background turn or a scheduled routine.

There’s a durability contract, which means that there is a shared workspace at /workspace, and files, browser state, and supported sign-ins are designed to survive normal computer updates and recovery.

Everything else is explicitly replaceable. Temporary directories, manually installed packages, and uncommitted application state carry no guarantee. Durable project files belong in the shared workspace.

SpaceXAI has not published the operating system, the image contents, or the machine specification. One widely shared report from a user who inspected the environment describes Debian with roughly eight virtual CPUs, around sixteen gigabytes of memory, and a disk in the hundred-gigabyte range, with no GPU.

The sharing model

As mentioned above, every Bot on your account uses the same computer, and the docs enumerate exactly what that means.

Browser cookies and signed-in sessions are shared. Files are visible to every Bot. Command-line credentials are shared. One Bot can continue from work another Bot saved.

This means that a research Bot can save a file, and a writing Bot picks it up with no upload or additional login. The handoff costs nothing because nothing has to be re-established.

Under this shared setting, you should not place a credential or a file on the computer if another Bot on your account should not use it.

The action layer

Grok Bot has two ways to interact with a service.

A connector gives a Bot a structured path into a supported service. Connectors appear as Plugins in the current app, and they are account-wide rather than per Bot.

The browser handles everything else like vendor portals, ad managers, older internal tools, anything without a clean programmatic interface.

Ideally, prefer a connector when one is available, because it is often more reliable than clicking through a website.

Since programmatic coverage of business software can be limited, it falls back to pixel-level control, which also means that it inherits every fragility of the underlying site.

So a layout change, a new consent prompt, or a session timeout can break a workflow that ran fine earlier.

Authentication and the takeover flow

Sessions persist on the shared browser, so you usually do not sign in again for each task. Signing in once for one Bot makes that session available to your other Bots.

For anything sensitive, the Bot hands you the machine, like in case of passwords, passkeys, two-factor codes, CAPTCHAs, payment or identity checks, and sites that explicitly require a human.

Practically, you open the Agent Computer, take control, complete only the blocked step, return control, and tell the Bot to continue.

For supported connections, there is a secure secret request instead. The value is masked, excluded from the transcript, and not shown to the model.

Never paste a password or a one-time code into ordinary chat. The takeover flow exists precisely so those values stay out of the transcript.

Agent Memory

Memory is maintained per Bot.

Duplicating a Bot copies its profile, settings, enabled skills, routines, and avatar, but not its conversation history or learned memory.

​Read the rest of the issue here →​

6 types of contexts for Agents​

A poor LLM can possibly work with an appropriate context, but even a SOTA LLM can never make up for an incomplete context.

That is why production-grade LLM apps don’t just need instructions but rather structure, which is the full ecosystem of context that defines their reasoning, memory, and decision loops.

And all advanced agent architectures now treat context as a multi-dimensional design layer, not a line in a prompt.

Here’s the mental model to use when you think about the types of contexts for Agents:

• Instructions: This defines the who, why, and how:

  • Who’s the agent? (PM, researcher, coding assistant)

  • Why is it acting? (goal, motivation, outcome)

  • How should it behave? (steps, tone, format, constraints)

• Examples: This shows what good and bad look like:

  • This includes behavioral demos, structured examples, or even anti-patterns.

  • Models learn patterns much better than plain rules

• Knowledge: This is where you feed it domain knowledge.

  • From business processes and APIs to data models and workflows

  • This bridges the gap between text prediction and decision-making

• Memory: You want your Agent to remember what it did in the past. This layer gives it continuity across sessions.

  • Short-term: current reasoning steps, chat history

  • Long-term: facts, company knowledge, user preferences

• Tools: This layer extends the Agent’s power beyond language and takes real-world action.

  • Each tool has parameters, inputs, and examples.

  • The design here decides how well your agent uses external APIs.

• Tool Results: This layer feeds the tool’s results back to the model to enable self-correction, adaptation, and dynamic decision-making.

These are the exact six layers that help you build fully context-aware Agents.

Btw, this is exactly how systems like Claude Code, real-world agents, and effective memory tools are already working today.

Context engineering is becoming the core skill for anyone building long-horizon, multi-step agents.

We did a crash course to help you implement reliable Agentic systems, understand the underlying challenges, and develop expertise in building Agentic apps on LLMs, which every industry cares about now.

Here’s everything we did in the crash course (with implementation):

• ​​​In Part 1​​​, we covered the fundamentals of Agentic systems, understanding how AI agents act autonomously to perform tasks.

• ​​​In Part 2​​​, we extended Agent capabilities by integrating custom tools, using structured outputs, and we also built modular Crews.

• ​​​In Part 3​​​, we focused on Flows, learning about state management, flow control, and integrating a Crew into a Flow.

• ​​​In Part 4​​​, we extended these concepts into real-world multi-agent, multi-crew Flow projects.

• In ​​​Part 5​​​ and ​​​Part 6​​​, we moved into advanced techniques that make AI agents more robust, dynamic, and adaptable, like Guardrails, Async execution, Callbacks, Human-in-the-loop, Multimodal Agents, and more.

• In ​​​Part 7​​​, we covered Knowledge of agentic Systems.

• In ​​​Part 8​​​ and ​​​Part 9​​​, we primarily focused on 5 types of Memory for AI agents, which help agents “remember” and utilize past information.

• In ​​​Part 10​​​, we implemented the ReAct pattern from scratch.

• In ​​​Part 11​​​​​​​, we implemented the Planning pattern from scratch.

• In ​​Part 12​​, we implemented the Multi-agent pattern from scratch.

• In ​​Part 13​​ and ​​Part 14​​, we covered 10 practical steps to improve Agentic systems.

• In ​Part 15​, ​Part 16​, and ​Part 17​, we covered practical ways to optimize the Agent’s memory in production use cases.

Of course, if you have never worked with LLMs, that’s okay. We cover everything in a practical and beginner-friendly way.

👉 Over to you: Have we missed any context layer in this?

Good day!

Every CI run of an AI app today sends real requests to providers like OpenAI or Anthropic.

Like any other LLM call, this too gets billed at actual API rates. So for teams with high commit volumes, this accumulates into a meaningful chunk of API spend.

One common hack devs use is that instead of invoking the LLM API, the test calls a fake local server that speaks the same API and returns a dummy response.

The catch is that the dummy response is a copy of what the provider returned on the day it was saved, and providers keep adding fields and changing types.

So the tests keep passing against a schema that’s no longer valid, while the real integration breaks in production.

A smart approach is now actually implemented in CopilotKit’s recently open-sourced aimock project.

Every day, the repo’s own CI sends a handful of requests to the real API and the same requests to the fake server, then compares both against the official client library’s type definitions.

Those are the only real API calls in the whole setup, and they run on the repo’s own keys, not in anyone else’s CI.

A single team can push hundreds of commits a day, and thousands of teams are already doing that with coding agents.

All of those runs stay offline, because one repo checks against the real API on everyone’s behalf.

When a check fails, a coding agent updates aimock’s built-in response schema, the full test suite has to pass, and a patch version ships to npm.

By simply upgrading the package, the corrected schema gets reflected in every project using it.

The capability is not just limited to a single provider.

The same server works for Claude, OpenAI, Gemini, Bedrock, Azure, Ollama, plus MCP tools, A2A agents, AG-UI event streams, vector DBs like Pinecone and Qdrant, and search, speech, image, and video endpoints.

Here’s the repo: https://github.com/CopilotKit/aimock

(don’t forget to star it ⭐)

How a GPU actually works

A GPU rated for close to a thousand trillion operations per second returns a few dozen tokens per second when it serves a 70B model, with the utilization monitor reading high the whole time.

Nothing is broken in that setup, and a bigger chip barely improves the token generation speed.

Our latest issue explains why, starting from the asymmetry the entire design rests on. Doing arithmetic is cheap, and fetching the numbers to do arithmetic on is expensive.

That asymmetry explains why a GPU carries thousands of simple units sharing one controller, and why it hides the wait for data rather than shortening it.

The same mechanism is why the utilization number means so little, since a chip starved for data looks identical to one running at full throughput.

We walk the memory ladder next, from the register file through shared memory and L2 down to HBM, and show where each level sits relative to an SM.

The number that comes out of all this is arithmetic performed per byte fetched.

Current hardware breaks even near 300 operations per byte at 16-bit precision, and anything below that line is limited by memory rather than compute.

Generating a token performs two operations on every weight it reads, so the ratio lands near one. A single request runs three hundred times below break-even, which is exactly why the arithmetic units go idle.

The token rate follows from the same division. 140 GB of weights over 3.3 TB/s of bandwidth is 42 milliseconds per token, or roughly 24 tokens per second.

Every optimization in common use turns out to be one of two moves, either increasing the work done per fetch or decreasing the bytes fetched. Batching, quantization, fusion, and FlashAttention stop looking like separate tricks once it is clear which move each one makes.

Knowing that a neural network does matrix multiplications, and that weights are numbers sitting in memory, covers the prerequisites.

The full issue is completely free.

Read it here →

​Prefix caching vs CacheBlend​

If your system prompt and tool definitions are stable, prompt caching is the single highest-leverage optimization available today.

Cached input tokens get up to 90% cheaper, and hit rates of 60 to 85% are realistic.

But it comes with one rigid rule. The cached portion must be an exact, byte-for-byte prefix of the new request. Change a single character in that region, and you get a full cache miss.

That rule breaks in three situations you hit constantly:

→ RAG with multiple documents. You cached document A alone and document B alone. A query now needs both. Document B’s cached state was computed without any awareness of A, so it’s invalid and gets recomputed from scratch.

→ Document order changes. The same three documents appear in a different order across requests. Every permutation is a cache miss, even though the content is identical.

→ Growing conversation history. Each new turn changes everything after the stable prefix, so earlier cached states beyond it become useless.

Alibaba Cloud’s production data puts numbers on this. 10% of KV cache blocks serve 77% of all hits. The rest sits in storage and never gets reused, because prefix matching won’t allow it.

CacheBlend, a research paper from the ​LMCache​ (EuroSys 2025 Best Paper Award), attacks exactly this.

In modern transformers, tokens overwhelmingly attend to their own local context, and only a small fraction of tokens carry real connections across document boundaries.

So instead of recomputing everything after the first cached document, CacheBlend reuses every document’s cache as-is and selectively recomputes just those few boundary tokens.

This results in 2 to 4x faster processing on multi-document queries with no quality loss.

The order problem disappears with it. Shuffle the same documents however you like, and every permutation stays cached, where prefix caching recomputes all of them every time. The bottom of the diagram shows that side by side.

The shift is from caching prefixes to caching knowledge. Every document in your knowledge base becomes a reusable cached asset, regardless of what order it appears in or what sits next to it.

CacheBlend ships inside LMCache, the open-source cache management layer that runs outside the inference engine and integrates with vLLM, SGLang, and TensorRT-LLM, on both NVIDIA and AMD GPUs.

Check it out on GitHub: ​https://github.com/LMCache/LMCache​

(Don’t forget to star 🌟)

We wrote the full breakdown of the architecture, including why cache management should never live inside your inference engine.

​Read it here →​

Stay tuned for more on this!

• How to build your own AI company (100% local).

• A cheaper model does not imply a cheaper turn.

• ​The anatomy of a Claude prompt.

How to build your own AI company (100% local):

Multi-agent orchestration is not new. Plenty of frameworks already let agents hand off tasks, run in parallel, and talk to each other.

So the key question is not whether agents can collaborate but rather what structure you use to make them collaborate.

The common approach is to wire a graph of nodes and edges and reason about the plumbing yourself. It works, but you are learning a new abstraction just to describe who does what.

There is a coordination structure we have trusted for a hundred years already: an organization.

Every company runs the same way. People have roles, roles have reporting lines, and work moves up and down that chart without anyone relaying each message by hand.

Map that onto agents and the whole thing gets intuitive. You lay out an org chart, each agent fills one role, you talk to the person at the top, and the org sorts out the work between them.

You already know how a company works, so you already know how to run one here. There is no new abstraction to learn.

Alook (GitHub Repo) is an open-source, self-hosted platform that turns your coding agents into a real org chart.

Alook GitHub Repo

Each agent is a live Claude Code or OpenCode session with a defined role, a reporting line, and its own email inbox.

The agents coordinate over email, the same way a team would. And it all runs locally through a runtime on your own machine, so nothing leaves your setup.

You bring your own agent too. Claude Code and Codex both work, and if you would rather stay fully open source and local, OpenCode works the same way.

To show how this feels in practice, we set up three agents as a small sales team.

• Atlas (CEO) is the point of contact for the human. It delegates tasks to Mara.

• Mara (PM) turns Atlas’s briefs into specs and routes them to Theo or Ren. She’s the sole router on the chart.

• Theo (engineer) builds and maintains the scrapers for competitive intelligence.

• Ren (Ops and Customer-facing) notifies the human when a tracked change is detected.

We never relay a message between them.

The whole thing is open source and self-hosted, so it runs on your machine with your own agents.

Here’s the repo →

A cheaper model does not imply a cheaper turn

Model routing is one of the few cost levers in an agent stack that works without touching your prompts, your tools, or your model quality targets.

The reason devs find it handy is that API pricing is spread widely.

Anthropic currently charges $5 per million input tokens for Opus 5 and $1 per million for Haiku 4.5.

And for the kind of agent workloads we run today, it’s an important thing to consider.

A single coding task can push 400K to 2M cumulative input tokens through the API, because every turn re-sends the full conversation so far.

Most of those turns are not hard. For instance, deciding to run the test suite, reading a file, or checking whether a command succeeded do not necessarily need frontier reasoning.

Technically, a router can sit in front of the model call, score the incoming prompt for difficulty, and then send the easy ones to a cheaper model.

On a workload where 70% of turns are trivial, this can provide huge cost savings:

But the problem is that the cost reduction only holds for single, independent prompts.

It does not work when you are inside a long session. Let’s understand more.

How prompt caching works

When you send a prompt to an LLM, the model has to process every token of that prompt before it can generate anything.

This is the prefill phase, and the intermediate results it produces (the key and value vectors for each token) form the KV cache.

Providers let you keep that cache alive between requests with prompt caching.

When your next request starts with the exact same tokens as the last one, the provider reuses the stored state instead of recomputing it, and bills those tokens at 10% of the base input rate.

The only problem is that the cache belongs to one model. Opus 5 and Haiku 4.5 have different weights and separate cache stores. A prefix that is warm on one of them does not directly transfer onto the other.

So a router that switches models mid-session is discarding the warm prefix and forcing a full cold prefill of the entire conversation on the new model.

This means you don’t pay the cheap model’s rate on the new instruction but rather the whole transcript.

An example

Say an agent has had 14 turns with the user and it’s carrying 60,000 tokens of history, including the system prompt, the tool schemas, a few files it has read, and the back and forth so far.

The new instruction (about 200 tokens) tells it to run a test, which is easy.

Option A, stay on Opus 5:

The 60,000 tokens of history are already warm, so they bill at the cache rate of $0.50 per million instead of $5.00. The 200 new tokens bill at the full rate.

• 60,000 cached tokens at $0.50/M comes to $0.030

• 200 new tokens at $5.00/M comes to $0.001

• Total input cost for the turn: $0.031

Option B, route to Haiku 4.5:

Haiku has never seen this conversation, so all 60,200 tokens are new to it and bill at its full input rate.

• 60,200 tokens at $1.00/M comes to $0.060

The model with 5x cheaper tokens produced a bill roughly 2x larger.

The router did its job, the cheap model also handled the prompt fine, and the whole turn still cost double.

The general rule

Call the history already in the conversation H, and the new tokens in this turn N. Call the strong model’s input price Ps and the cheap model’s Pc.

• Staying on the strong model costs 0.1 × Ps × H for the warm history plus Ps × N for the new tokens.

• Switching costs Pc × (H + N), because nothing is warm. Setting those equal gives the condition where switching is cheaper:

• The numerator on the right denotes savings per new token by using the cheap model.

• The denominator denotes the loss per history token by re-prefilling it from cold rather than reading it warm.

Together, they tell you how many tokens of history you can afford to re-prefill for each new token you save on.

For Opus 5 and Haiku 4.5, that works out to be 8.

That means switching pays only while the history is under 8x the size of the new input. With a 200-token instruction, the conversation has to be shorter than 1,600 tokens.

An agent session with a system prompt and tool schemas is past that even before the first user turn.

Where routing is still important

In the example, switching cost an extra $0.029, but it saves $20 per million on output, so the cheap model needs to produce more than about 1,450 output tokens before the turn breaks even. That splits agent traffic along a clean line:

• Tool calls, short instructions, and control-flow turns emit a few hundred tokens and lose money on a switch.

• Turns that write a full file, draft a long document, or produce a large diff can benefit from a switch.

Short sessions follow a different rule.

With little accumulated history to re-prefill, the penalty nearly vanishes, and per-prompt routing actually works.

These are also the workloads routers were first measured on, and long agent sessions just took over their conclusion without understanding the conditions.

Where to place the switch

The cheapest moment to switch models is right after a compaction or a context reset. The prefix has already changed at that point, so the cache is invalid whichever model you use next, and the switch is free.

That suggests a different decision procedure.

Rather than scoring each prompt for difficulty and switching whenever one looks easy, find the points where the session’s cache already breaks and make every model change land on one of them.

A router scores the next prompt on its own, while the bill is computed over a prefix that grows for the entire session, and the savings disappear into the gap between those two units of measurement.

Further reading:

• We covered KV caching in detail here →

• The full LLM inference stack, prefill and decode phases, batching, and serving tradeoffs are in the LLMOps course →

• And 72 techniques to optimize LLMs in production →

👉 Over to you: has your router ever made a session more expensive than the model it was routing away from?

The anatomy of a Claude prompt​

The difference between a mediocre Claude output and a great one almost always comes down to how you structure your prompt. This involves a clear, repeatable structure that gives Claude exactly what it needs to do the job well.

Here’s how a well-built Claude prompt breaks down into 8 building blocks, each doing one job:

1) Role

Tell Claude who it is before telling it what to do.

“You are a [ROLE] with expertise in [DOMAIN]. Your tone should be [TONE]. Your audience is [AUDIENCE].”

Setting a role in the system prompt changes how Claude reasons, what it prioritizes, and how it communicates. A “senior backend engineer” writes differently than a “technical copywriter,” and Claude picks up on that distinction immediately.

2) Task

State what you want and what success looks like, in the same breath.

“I need you to [SPECIFIC TASK] so that [SUCCESS CRITERIA].”

The “so that” part is what people skip, and it’s the part that matters. It gives Claude a way to evaluate its own output. Without it, Claude is guessing what “good” means.

Be direct, skip the preamble, and cut the fluff.

3) Context

This is where you feed Claude everything it needs to do the job well.

Wrap it in XML tags like <context> and </context>, then paste your documents, data, or background inside.

One thing that dramatically improves quality: put long documents at the top of your prompt and your actual query at the end. Anthropic’s own testing shows this can improve response quality by up to 30%, especially with complex, multi-document inputs.

4) Examples

Nothing steers output quality like showing Claude what “good” looks like.

Provide 3-5 input/output pairs. Cover normal cases AND edge cases. Wrap them in <examples> tags so Claude doesn’t confuse them with instructions.

Claude pays extremely close attention to examples. If your example has a quirk you didn’t intend, Claude will replicate it. So make sure every example models the behavior you actually want.

5) Thinking

For anything requiring reasoning, analysis, or multi-step logic, ask Claude to think before answering.

“Before answering, think through this step by step. Use <thinking> tags for your reasoning. Put only your final answer in <answer> tags.”

This separates the messy reasoning from the clean output. You get to see how Claude arrived at its answer without that reasoning cluttering the final result.

6) Constraints

Every good prompt has guardrails.

“Never [thing to avoid]. Always [thing to ensure]. If you are about to break a rule, stop and tell me.”

That last line is underrated. It turns Claude into a collaborator instead of a blind executor. Instead of silently violating a constraint, Claude flags the conflict and lets you decide.

7) Output Format

Don’t leave the format to chance.

“Return your response as [JSON / markdown / table / prose]. Use this exact structure: [structure template].”

If you want JSON, show the exact schema. If you want markdown, show the heading structure. If you want a table, define the columns. The more specific you are about shape, the less time you spend reformatting afterward.

8) Prefill

This one is API-specific, but incredibly powerful.

You can pre-fill the start of Claude’s response to skip preamble and lock in the format. Claude will continue from exactly where you left off. No “Sure, I’d be happy to help!” opening, no throat-clearing, just clean output from the first token.

Here’s the thing people get wrong about prompting: they think it’s about finding the right words. It’s actually about giving Claude the right structure.

If you want to go deeper, we wrote a detailed article covering the anatomy of the .claude/ folder, a complete guide to CLAUDE(.)md, hooks, skills, agents, and permissions, and how to set them all up properly.

​You can read it here →

Good day!​

The above line sounds counterintuitive, but MCPMark V2 benchmarks confirmed this across 21 backend tasks.

And it’s not a minor 3-5% difference.

But 54% higher token usage.

The reason has nothing to do with the model itself.

Instead, it has to do with what the agent needs to know before it can start building.

When you’re building a full-stack app, CC must understand the entire backend, like:

• what tables already exist

• what RLS policies are active

• what storage buckets are available

• which auth providers are configured

• and what edge functions are deployed

Most backends don’t hand over this info cleanly.

For instance, with Supabase, asking for OAuth setup via MCP returns the entire auth docs, including sections on email/password, magic links, phone auth, SAML, and SSO.

That’s 5-10x more tokens than the agent actually needed. And this happens on every MCP call across every domain.

The agent then discovers the state through separate calls to list_tables, execute_sql, and list_extensions, each returning a partial view.

Some info, like which auth providers are configured, isn’t queryable through MCP at all.

And when something breaks, Supabase returns the same error code whether the rejection came from the platform layer or from the function code.

The agent has no way to infer accurately, so it cycles through code-level fixes for a problem that might not be in the code at all.

A better model does not have a magical way to skip these gaps.

In fact, it tries even harder to fill them, which means more discovery queries, more reasoning, and more retries. That’s why the token cost went up with a better Claude model.

A smarter approach is actually implemented in ​InsForge, an open-source backend (self-hostable via Docker) that offers the same primitives as Supabase but structures everything around the assumption that an agent is operating the backend, not a human on a dashboard.

Before writing any code, a single CLI call returns the full backend topology in ~500 tokens.

The agent sees every table, auth provider, storage bucket, and available AI models in one structured response.

Instead of one broad skill like Supabase that triggers on everything, it has four narrowly scoped skills.

• Creating tables only activates the CLI skill.

• Debug skill only activates when code breaks.

• Building frontend only activates the SDK skill.

• Wiring third-party auth only activates the integrations skill.

This keeps the agent’s cognitive load lean since it only loads what matches the current task.

The CLI returns structured JSON with semantic exit codes on every operation, so the agent always knows whether something succeeded or failed and why. There are no ambiguous 401s that may indicate three different things.

We tested both backends on the same full-stack RAG app and recorded the full sessions.

Supabase:

• consumed 10.4M tokens

• needed 10 manual interventions

InsForge:

• consumed 3.7M tokens

• completed the entire build without any errors

This isn’t a Supabase-specific problem. Most backends were designed for humans who can see dashboards and interpret raw errors.

When an agent operates the backend instead, every missing piece of context needs a discovery call, and every ambiguous error enters a retry loop.

Fixing this requires giving agents structured backend context before they start writing code.

InsForge is an open-source implementation of exactly this, and you can self-host it via Docker.

GitHub repo (14k+ stars): ​https://github.com/InsForge/InsForge​

(don’t forget to star it ⭐ )

​You can read our walkthrough on building the full-stack RAG with Supabase and InsForge in this newsletter issue →

How production LLMs reason better at inference time

There are largely just two ways to make an LLM think harder at inference:

• Parallel scaling samples the same prompt several times and selects one result.

• Sequential scaling extends one trajectory further before emitting an answer.

Tree methods compose both, so they inherit both failure modes:

None of them alter the weights. It all runs in the prompt and in the orchestration around the call, which is why it is the first thing teams try when accuracy falls short.

But it is also the layer where the cost recurs.

Essentially, training cost is paid once and amortized across every future call, while inference-time compute is charged per request, forever.

The additional cost is often helpful though since several papers have found that under a FLOPs-matched comparison, test-time compute outperformed a 14x larger model on problems where the smaller model already had non-trivial success rates.

Let’s cover the techniques under each mechanism below:

1) Chain of thought

The model is prompted to think step by step, so it writes out intermediate steps before committing to an answer.

Reasoning models already do this on their own, so prompting for it separately mostly adds tokens.

2) Majority voting

Run the same prompt several times at nonzero temperature, then return whichever final answer appears most often.

It does not need any reward model since agreement directly acts as a signal.

Voting fixes random mistakes and does nothing about consistent ones. If the model misreads the problem the same way every time, all the samples agree, and the vote returns that answer with higher confidence.

It also needs answers that can be checked for equality, which rules out open-ended text.

3) Best-of-N

Generate N complete answers, score each one with a reward model, keep the highest scorer.

A paper measured that true reward rose, peaked, then declined as optimization pressure on the proxy increased. So past some N, the search starts finding answers the reward model likes rather than answers that are right.

The reward model also has to judge better than the model generates, which is the harder half of the setup.

4) Extended thinking

The model thinks for a set number of tokens before answering, with the caller setting the budget through budget_tokens, thinkingBudget, or reasoning_effort.

R1-Zero went from 15.6% to 71.0% on AIME 2024 through RL alone, with no external scaffolding at inference.

That said, more thinking is not always better. Anthropic’s inverse scaling work built tasks where accuracy drops as traces lengthen, including counting problems seeded with irrelevant numbers and constraint puzzles where the model reopens deductions it had already solved.

5) Self-refinement

The model writes an answer, critiques its own answer, then rewrites it, and the loop repeats.

However, this has the weakest evidence in this lift because in GSM8K, GPT-3.5 corrected 7.6% of its wrong answers and changed 8.8% of its correct ones, making the loop net negative.

6) Tree of Thought

While Majority voting varies the final answer, Tree of Thought varies the steps of reasoning at each point and then picks the best path overall.

At every reasoning step, the model explores multiple possible directions. These branches form a tree, and a separate process evaluates which path seems the most promising at a particular timestamp.

Think of it like a search algorithm over reasoning paths, where we try to find the most logical and coherent trail to the solution.

It’s more compute-intensive, but in most cases, it significantly outperforms basic CoT.

7) Beam search with a process reward model

Keep K partial solutions alive and score every step as it is written, instead of waiting for the final answer.

A paper found it ahead of best-of-N at small budgets and below it as budgets grow, with over-optimized runs producing repetitive steps or collapsing to one or two steps.

It works well on medium-difficulty problems, over-optimizes on easy ones, and no method works on the hardest tasks.

8) MCTS

Pick a promising partial path, extend it, run it out to a full answer, push the score back up the path, and repeat many times.

DeepSeek tried this for R1 and dropped it. The number of possible next tokens is far larger than the number of legal moves in a board game, and training a value model good enough to guide that search became a blocker.

They dropped PRMs for related reasons, since step correctness is hard to define, labeling does not scale, and the policy exploits the scorer.

DeepSeek solved this by placing the search into the weights with rule-based rewards instead.

To dive deeper, we covered all of this in depth across 13 parts of our RL series:

• Part 1 covers the foundations, the agent-environment loop, and the reward hypothesis →

• Part 2 covers MDPs and value functions →

• Part 3 covers Bellman equations and dynamic programming →

• Part 4 covers model-free learning, Q-learning, and SARSA →

• Part 5 covers function approximation and what breaks when states stop being enumerable →

• Part 6 covers deep RL and DQN →

• Part 7 covers policy gradients and actor-critic →

• Part 8 covers PPO and why it became the default for LLM post-training →

• Part 9 covers RLHF, preference data, and reward model training →

• Part 10 covers verifiable rewards and GRPO, which is the rule-based path R1 took →

• Part 11 covers the reward signal problem for agents →

• Part 12 covers environments, trajectories, and the training loop →

• Part 13 covers how AI teams use RL in production, including Cursor’s five-hour checkpoint loop →

Good day!

The tooling to build agents is quite mature, so most of the work in shipping an agent isn’t writing the agent anymore.

It’s everything after, including scaffolding it, deploying it to a runtime, locking down its identity and network, evaluating it, and publishing it somewhere people can use.

Each of those has traditionally lived in its own console, its own config, its own separate tool.

Google’s Agents CLI + skills implements procedures to condense the entire lifecycle into the coding agent itself, by prompting in plain English.

A setup command injects the lifecycle skills, so a single coding agent can carry an idea from an empty folder to a governed, published enterprise asset.

We mapped the full lifecycle in the diagram below. Here’s what each stage does.

> Setup installs the skills into any coding agent (Claude Code, Cursor, Codex, Antigravity) from one command.

> Build scaffolds the agent and its deterministic tools from a prompt, then you run it locally in the playground.

> Deploy pushes it onto Agent Runtime with Sessions and Memory Bank, so it holds state across runs.

> Govern is the security stage, and Agents CLI drives all of it from prompts. It provisions a dedicated least-privilege identity, screens untrusted text for prompt injection through Model Armor, and confines the agent to an egress allow-list of hosts you approve.

> Evaluate checks grounding and hallucination, and then optimizes the prompt while proving no regression.

> Publish registers the agent into Gemini Enterprise for the whole org to use.

Every stage requires just natural-language prompts.

Agents CLI GitHub repo → https://github.com/google/agents-cli

(don’t forget to star it ⭐)

Thanks to Google Cloud for partnering with us today!

Continuous batching in LLMs

In traditional ML inference, a batch is usually represented as a matrix.

If there are inputs of varying lengths, then each input is padded or truncated to the same length, stacked into one tensor, and processed through a single forward pass to generate one prediction per row.

Every row costs the same, every row finishes at the same moment, and the full shape of the work is known before the pass starts. Batching there is a tensor-packing problem.

LLM decoding does not work on those principles.

Under the hood, one forward pass produces one token per sequence, so a request needs as many passes as it has output tokens, and nobody knows that count beforehand until the model emits a stop token.

Padding can’t fix it, because the mismatch isn’t in the input width but rather in how long each request occupies the GPU.

So a batch fixed at the start runs at the pace of its slowest member.

Continuous batching is an approach implemented in most serving engines to avoid this problem. vLLM, SGLang, TGI, and TensorRT-LLM...all run it by default.

The core idea is to decide batch membership at every forward pass instead of once when the batch starts. A finished request leaves at the next pass and a waiting request takes its slot right there, so no slot sits reserved for work that’s already done.

The scheduler underneath continuous batching is at the core of handling this process end-to-end, i.e., the loop that decides at every forward pass which requests get tokens and how many.

So let’s build it up from the problem, walk one scheduling step in order, and look at the measured gains once the mechanism is clear.

To dive deeper into the full LLMOps lifecycle, we have covered every bit of this in the LLMOps course, starting from fundamentals to production.

Start here →

Scheduler in traditional ML inference

A classifier takes a fixed-width input and returns a label.

Sequence models pad to a maximum length and mask the padding. Either way, the tensor going in has a known shape and the tensor coming out has a matching row count.

The reason to batch at all is that loading model weights out of HBM is a fixed cost per forward pass.

So if you stack 64 rows/inputs into a single pass and all those 64 rows share one weight read, the throughput rises:

Serving that in practice is straightforward since you can simply fill a batch, run it, return every result, and start the next one.

No request carries state into the following pass, and no request outlives the batch it arrived in.

Nature of LLM inference

The decoding process in LLMs breaks three assumptions discussed above:

• Each pass generates a single token per sequence.

• The KV cache carries state forward into the next pass

• The number of passes a request needs is only known when it produces a stop token.

So a request that finishes in 30 tokens sits in the same batch as one running to 400, and under a fixed batch it holds its slot for all 400 steps while producing nothing.

The GPU keeps paying the full weight read for a mostly empty batch.

The waste scales directly with how much the output lengths vary, which in real traffic can be a lot!

The solution

To solve this, we need to alter where the serving system and the execution engine talk to each other.

Essentially, instead of handing over a batch and waiting for it until all requests complete, the scheduler runs one iteration, gets control back, and decides again.

This is called iteration-level scheduling, and it describes the mechanism continuous batching uses under the hood.

A finished sequence leaves at the next iteration boundary rather than at the end of the full batch run. And a waiting request enters at that same boundary. The batch is rebuilt every forward pass.

This, in itself, is not sufficient since rebuilding the batch every pass creates a problem of its own.

For instance, a request that’s prefilling 4,096 tokens together and a request decoding its 900th token don’t share a tensor shape, so there’s nothing to stack in the usual way.

Selective batching handles this, and it ensures that batching is applied only to the operations that can take it.

Every token scheduled in the step, whatever request it came from, is flattened into one long sequence of shape (total tokens*hidden size).

Layer norm, the QKV projections, and the feed-forward blocks all operate on each token independently, so they neither know nor care which request a token belongs to.

They run once across the whole flat stream at full efficiency.

But the attention operation can not work like this. A token may only attend to earlier tokens from its own request, and every request has a KV cache of a different length.

So the flat tensor is split at the attention boundary, attention runs separately for each request against that request’s own cache, and the outputs merge back into the stream before the next batched operation.

One scheduling step, clearly explained:

Between two forward passes the scheduler answers one question. Which requests run next, and how many tokens does each one get.

It answers that in four steps and we’ll use vLLM’s V1 scheduler as the reference implementation:

1. The scheduler fixes a budget for the step:

• max_num_batched_tokens caps the total tokens the step may issue

• and max_num_seqs caps how many sequences can be in flight at once.

2. Running requests get to claim compute budget first:

Each request carries a count of tokens already computed and a target count. The scheduler hands out tokens to reduce the gap between where a request currently is and where it wants to get to, spending from the budget as it goes.

A scheduler could reasonably keep two code paths here, one for prefill and one for decode, since prefill processes a whole prompt and decode produces a single token.

But in practice, there’s no such split. Prefill and decode look like different jobs, but the scheduler sees the same number.

It takes the target, subtracts what’s already computed, and hands out that many tokens if the budget allows:

So for a fresh 4,096 token prompt, that number is 4,096. For a request mid-generation, it’s 1. It’s the same line of code either way.

# simplified from vllm/v1/core/sched/scheduler.py

tokens_this_step = min(
    request.num_tokens_with_spec - request.num_computed_tokens,
    remaining_token_budget,
)

Chunked prefill and prefix caching need no extra handling either.

If a 4,096 token prompt doesn’t fit in the budget, the scheduler hands out 2,048 now and the other 2,048 in the next step.

If 3,000 of those 4096 tokens were already cached from an earlier request, only 1,096 are left to hand out.

3. The scheduler claims KV blocks for the tokens it just assigned.

It reserves that memory while deciding, not after.

A step is only valid if the cache can hold what it produces, so when there aren’t enough free blocks, the scheduler takes them from the newest request in the running list.

4. Whatever budget is left goes to waiting requests

Requests that haven’t started yet get scheduled with whatever tokens are left over. If the running requests used the whole budget, nothing new starts this step.

The scheduler then hands the GPU a list of how many tokens each request runs and which KV blocks to write them into.

Token budget decides latency and throughput

A step takes as long as the tokens inside it, which makes max_num_batched_tokens a latency setting as much as a throughput one.

To recall, max_num_batched_tokens is the total number of tokens the scheduler may put into one forward pass, added up across every request in the step. It isn’t a per request limit. A decoding request contributes one token, since one token is all it produces. A prefill contributes however many prompt tokens the scheduler hands it.

At around 2048 tokens, no step runs long, so inter-token latency stays tight and the GPU is often under-fed.

At around 16384, the GPU stays saturated and throughput rises, but every request in the batch waits out a longer step.

Size isn’t the only thing this budget setting controls.

With chunked prefill on, which is the V1 default wherever possible, the scheduler fills the budget with pending decodes first and gives what’s left to prefills, chunking a prefill that doesn’t fit.

Preemption

When block allocation fails, the scheduler frees a running request’s blocks, marks it preempted, resets num_computed_tokens to zero, clears its speculative tokens, and puts it back at the front of the waiting queue.

The reset to zero is an actual cost because, let’s say a request had already prefilled 3,900 tokens, and it then comes back with none of that saved, it must compute all 3,900 again.

V1 made recompute the default preemption mode and dropped the older swap path entirely.

From the outside, this looks like the GPU running out of headroom. Latency climbs as traffic rises, and adding replicas seems like the fix.

What’s actually happening is the same prefill being computed two or three times. Requests get preempted, readmitted, and preempted again, and every one of those cycles repeats work the GPU had already finished.

vLLM counts this in Prometheus as total_cumulative_preemption_cnt, and it’s the first metric to check when p99 climbs without a traffic change.

For context, p99 is the latency your slowest 1% of requests see. Averages hide preemption because most requests are never preempted, so problems show up in the tail.

Three settings improve it.

• Raise gpu_memory_utilization so more VRAM becomes KV cache

• Lower max_num_seqs so fewer sequences compete for it

• Raise tensor_parallel_size so weights shard across GPUs and leave more room per device.

Wrapping up

Everything above comes down to one loop.

The scheduler rebuilds the batch at every forward pass, hands out tokens against a fixed budget, and reserves KV blocks while it decides, and those three moves determine what your GPU actually produces.

Anyscale’s benchmark on OPT-13B found that as output lengths varied, static batching fell to around 81 tokens/s while vLLM reached 23x the throughput of naive Hugging Face serving on the same A100. None of that came from a faster model:

So if throughput or p99 is your problem, the scheduler is the layer to look at before the model.

The preemption counter is usually the cheapest place to start, since it tells you in one number whether your KV pool matches the concurrency you configured.

To dive deeper into the full LLMOps lifecycle, we have covered every bit of it in the LLMOps course, starting from fundamentals to production:

• Read Part 1 on fundamentals of LLMOps here →

• Read Part 2 on understanding the core building blocks of LLMs →

• Read Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixture-of-experts, and the fundamentals of pretraining and fine-tuning →

• Read Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applications →

• Read Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and different prompting techniques →

• Read Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more →

• Read Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techniques for building high-signal inputs →

• Read Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic context injection, and common failure modes in agentic applications →

• Read Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong understanding of the fundamental concepts →

• Read Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for evaluation of LLM apps →

• Read Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming →

• Read Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO →

• Read Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model parallelism →

• Read Part 14 on the fundamentals of LLM serving, including API-based access, inference with vLLM, and practical decisions.