• LLM routing can cost more than not routing at all.

• Agent framework vs. agent harness, clearly explained.

• A new technique just made attention 23x faster.

LLM routing can cost more than not routing at all

The basic idea behind LLM routing sounds simple, i.e., send easy requests to cheaper models and save frontier models for hard tasks.

Production routing is rarely that simple.

Every classification adds latency. General-purpose models can misread intent. Routing rules go stale.

And switching models during an agent session can wipe out your prefix-cache savings, raise costs, and make the agent behave inconsistently.

In this video, we cover:

• Why DIY routing often breaks in production

• How prefix caching changes the economics

• Why agents need session pinning and model affinity

• How Plano and Arch-Router handle intent resolution

• How to build, test, and evaluate a router with DigitalOcean

Routing works, but only when the infrastructure accounts for the full session, not just the latest request.

So the question worth asking is how much of your AI traffic has been overpaying this whole time.

Get started with DigitalOcean Inference Router →

Thanks to DigitalOcean for partnering today!

Agent Framework vs. Agent Harness, clearly explained!

Giving an agent tools doesn’t solve the problem of keeping it productive across a long task. It still needs to track progress, recover from failed actions, and preserve useful context when its conversation grows.

Those responsibilities help explain the distinction between an agent framework and an agent harness.

A framework gives you components for building agent behavior. A harness packages an agent loop with an environment and operating conventions. The engineering decision is how much of that surrounding system you want to assemble and maintain.

A framework gives you control over the design

An agent framework provides abstractions for models, tools, state, and orchestration. You use those components to define how your application behaves.

Consider an invoice-processing system. It might extract fields, validate them against purchase orders, route discrepancies for review, and submit approved records.

Some steps need model judgment. Others should follow deterministic rules, especially when money moves or records change.

Explicit orchestration lets you enforce those boundaries. A model can suggest an action, while application code decides whether the required checks have passed.

However, the left side of the visual needs one qualification: frameworks aren’t limited to workflows you can draw upfront.

LangGraph, for example, supports both deterministic steps and model-driven decisions, along with persistence and human intervention. A graph can contain an agent loop whose next tool call depends on what it discovers.

The useful distinction is how much agent behavior comes assembled, and how much you define yourself.

A harness supplies an operating environment

An agent harness makes choices about how the model works inside a task.

Depending on the implementation, that can include file access, command execution, context management, progress tracking, and delegation to subagents.

Deep Agents is a concrete example. It builds on LangChain and LangGraph, adding filesystem tools, context offloading, and subagent support around the tool-calling loop. The framework and harness occupy different layers of the same system.

For a coding task, this changes your starting point.

Instead of implementing file inspection, editing, and execution behavior individually, you configure an existing environment with your repository instructions, tools, and access boundaries.

The model can inspect a failing test, read the relevant implementation, make a change, and run another check. Its intermediate actions depend on evidence gathered during execution.

You still own the definition of success. A working tool loop doesn’t establish that the requested feature works.

Long tasks

Consider a task spans multiple sessions or context windows.

A conversation summary might preserve the broad objective while losing a detail such as which migration ran or which test still fails. Persisting the conversation alone doesn’t guarantee that the next session can resume correctly.

Anthropic documented this problem in its work on long-running coding agents. Agents sometimes attempted too much at once, left incomplete work, or declared completion prematurely.

Its approach combined an initial setup phase with incremental coding sessions. Progress files, version history, and explicit feature checks helped subsequent sessions reconstruct the state of work.

The practical lesson is to make progress inspectable outside the model’s conversation.

A repository commit records what changed. A test result records what passed. A task file records what remains.

These artifacts answer different questions, so compressing everything into one narrative summary can remove information the next session needs.

Choosing what to build yourself

Start with the task’s control requirements.

For invoice processing, the required sequence and approval rules may be clear before execution. Explicit orchestration makes those rules visible and testable.

For investigating a failing build, the next useful action may only become clear after reading logs or running a test. A configurable harness can reduce the amount of execution machinery you need to build.

Many applications need both.

An outer workflow can enforce authorization and record final outcomes. Inside one step, an agent can investigate an issue using a harness, then return evidence for the workflow to validate.

That boundary also improves debugging. You can distinguish a failed business rule from an agent choosing an unhelpful tool or losing context.

Evaluate the behavior around the model

When comparing options, inspect what happens after a tool fails, what stays a restart, and how the system decides that work is complete.

Also check which operations require approval and whether those restrictions apply to delegated agents.

Test those behaviors with a deliberately interrupted task. Resume it and inspect whether the agent repeats completed work, misses an unresolved failure, or claims success without verification.

Frameworks give you building blocks. Harnesses package decisions about execution. Your application still needs evidence that those decisions produce the behavior you intended.

If you want to build both sides of this yourself, our AI Agents course has a full LangChain and LangGraph track. It starts with a single production-ready agent and works up to harness middleware, planning and verification loops, multi-agent orchestration, and durable agents:

Start the AI Agents with LangGraph course →

• Part 1: First Steps Towards a Production-Ready Agent with LangChain →

• Part 2: Middleware-Driven Context Engineering in LangChain →

• Part 3: Agent Harness with LangChain Middleware and Jev →

• Part 4: Planning and Verification Loops with LangGraph and Jev →

• Part 5: Multi-agent orchestration with LangGraph →

• Part 6: Durable Agents with LangGraph and LangSmith →

A new technique just made attention 23x faster

Full attention compares every query against every earlier key, so prefill costs O(T²). At 10M tokens, that cost dominates everything else.

Sparse attention reads only k keys per query. But something still has to pick those k keys.

That picker is now the bottleneck. DeepSeek’s DSA uses a lightning indexer that scores every key. GLM-5.3-Flash pools 4 keys into 1 before scoring. NSA, MoBA, and Quest score blocks of keys. In each case the indexer stays O(T²), and at 10M tokens it takes 98% of the layer’s prefill time.

Blocks also group keys by position, so one relevant key sitting among unrelated ones can get averaged away.

Latent Sparse Attention (LSA) groups keys by meaning instead:

• Keys are projected into a 64-dim index space, and every 4 are pooled into 1.

• Each 512-key block is clustered into buckets of about 8 keys, each with a centroid. Finished blocks are never rebuilt.

• A query scores only the centroids, keeps the top 1,024 buckets, and expands them to 4,096 keys, plus 4 sink tokens and the 512 most recent keys.

• Exact FlashAttention then runs over those keys with an unchanged kernel.

If you’ve worked with vector search, this will look familiar. It’s cluster-based search running inside attention, and the paper’s kernel design draws on FAISS.

The results at 10M tokens:

• One layer’s prefill runs 72.4x faster than full attention on Qwen-3.8-27B, and 22.7x faster than GLM-5.3-Flash’s sparse attention.

• End-to-end time to first token is 9.2x faster on Qwen and 2.2x faster on GLM.

• Below roughly 512k tokens, the end-to-end gain is about break-even.

So the 23x is a per-layer prefill number. The end-to-end gain is smaller and shows up only at very long contexts.

One thing LSA doesn’t change is storage. It makes finding the right keys cheaper, but the KV cache still grows linearly with context.

Read the LSA paper →

We covered the storage side in detail in our KV cache engineering deep dive:

Read it here →

👉 Over to you: Which part of your agent system has required more engineering, the workflow itself or keeping the agent productive during execution?

Good day!

• Your AI coding agents should get smarter every time you use them.

• 12 optimizers for training neural networks, clearly explained.

Your AI coding agents should get smarter every time you use them

AI coding agents write good code now, but they still don’t learn from experience.

Say you spend 30 minutes helping Claude Code debug a tricky issue. It explores your codebase, tries several approaches, hits dead ends, and eventually finds the fix.

Tomorrow you start a new session, and the agent makes the exact same mistakes.

Switch to Codex or Cursor and it gets worse. They have no idea what your previous agent figured out.

Every session produces useful knowledge: debugging breakthroughs, failed approaches, architectural decisions, and patterns specific to your codebase.

But most of it disappears when the session ends.

Beacon, built by Asymptote Labs, is an open-source memory layer that lets your coding agents learn from past sessions.

It works across Claude Code, Cursor, Codex, OpenCode, Cline, and 20+ other agent harnesses.

Beacon launched on Product Hunt today.

Check out the launch and show your support with an upvote →

Instead of treating every session as an isolated conversation, Beacon captures what your agents actually do and turns that experience into reusable knowledge.

Here’s how it works:

↳ Capture every session. Beacon records execution traces across your coding agents, including commands, file changes, errors, and debugging paths.

↳ Extract reusable knowledge. It picks out successful fixes, recurring workflows, and lessons worth remembering.

↳ Improve future sessions. Those lessons become skills and context your agents can use, even in a different coding tool.

↳ Close the feedback loop. Later traces show when saved knowledge stops working, so it can be updated or retired.

Beacon Cloud continuously finds ways to improve your agents

Capturing sessions is only the first step.

Every hour, Beacon Cloud reviews your agents’ new activity and turns it into findings in three categories:

↳ Security: risky things your agents did. Secrets pasted into prompts, destructive commands, permission bypasses, or connections to unknown MCP servers. Beacon flags the action, shows which session it happened in, and suggests a fix.

↳ Skill: workflows worth reusing. When you keep walking agents through the same debugging steps, Beacon notices and drafts a reusable skill. You download it and make it available to all your coding agents.

↳ Behavior: patterns in how you work with agents. Retry loops, abandoned sessions, unnecessary approvals, or expensive models used for simple tasks. Beacon points out the waste and recommends faster, cheaper ways to work.

The findings keep updating as your agents generate new traces.

One memory layer for every coding agent

Beacon sits across the agent harness layer.

So your Claude Code debugging lessons can help your next Codex session, and your Cursor discoveries can become skills in OpenCode.

With beacon handoff, you can also resume work in a different agent when you hit a rate limit or need to switch tools. You don’t re-explain the repo or retrace failed approaches.

It matters even more across a team. One engineer spends an hour debugging an infrastructure issue, Beacon captures the approach that worked, and the next engineer can reuse it as a skill in whichever coding agent they use.

The loop looks like this:

Run agents → Capture traces → Generate findings → Reuse skills → Improve → Repeat

Beacon runs locally without a hosted backend. Beacon Cloud adds hosted trace storage, cross-harness search, session analytics, and the findings above.

A smarter model only gets you so far. The coding agent that keeps getting better is the one that learns from experience.

Support Beacon on Product Hunt →

Star Beacon on GitHub →

Stop teaching your agents the same lessons twice.

Thanks to Asymptote Labs for partnering today!

12 optimizers for training neural networks, clearly explained!

An optimizer can reduce the number of training steps and still make your run slower. Each update has its own compute and memory cost, which becomes part of the training budget.

That tradeoff gets easier to reason about once you look at what each optimizer stores and how it transforms the gradient.

The 12 optimizers in the visual approach this differently. Let’s break them down through the engineering decisions they change, starting with how much gradient history survives each update.

Smoothing gradients and adapting step sizes

SGD with momentum keeps a running accumulation of gradients. Directions that persist across batches build up, while inconsistent directions tend to cancel.

This helps reduce oscillation, but momentum alone doesn’t give each parameter its own adaptive step size.

AdaGrad adds that adaptation by accumulating squared gradients for each parameter. Frequently updated parameters receive progressively smaller effective steps, which can help with sparse features.

Its limitation follows directly from that accumulation: the denominator keeps growing, so effective learning rates can become too small.

RMSprop replaces the full accumulation with an exponential moving average of squared gradients. Older observations fade, allowing the scaling to respond to more recent gradients.

Adam combines momentum with this second-moment scaling. One moving average tracks gradient direction; another tracks squared gradient magnitude.

That second quantity isn’t a direct curvature measurement. Adam adapts to gradient statistics without estimating the Hessian.

Changing regularization, memory, and gradient history

AdamW separates weight decay from the adaptive gradient update. Adding an L2 penalty to Adam’s loss lets its adaptive scaling affect that penalty; decoupled weight decay applies parameter shrinkage separately.

That distinction matters when you’re tuning regularization. Weight decay becomes easier to reason about independently of the gradient scaling.

Memory introduces another constraint. Standard Adam maintains two parameter-sized state arrays, in addition to the model weights and gradients.

With one billion parameters and FP32 moment buffers, those two arrays alone require about 8 GB before sharding or compression.

Adafactor reduces the second-moment storage for matrices using row and column statistics. For an n × m matrix, that replaces nm stored second-moment values with roughly n+m.

You’re trading a full per-parameter estimate for a factored approximation. That’s useful when optimizer state limits the model or batch size you can fit.

Lion keeps momentum without a second-moment buffer. Its gradient update uses the sign of a momentum-gradient blend, giving coordinates the same update magnitude within a parameter group.

That changes the update scale, so copying AdamW’s learning rate into Lion isn’t a fair comparison. The Lion paper specifically calls for smaller learning rates.

AdEMAMix takes another approach: it combines fast and slow moving averages of gradients alongside second-moment scaling. Recent gradients remain influential while older directional information survives longer, at the cost of additional state.

Using matrix structure and curvature

Adam scales coordinates individually. Shampoo uses matrix-valued statistics to transform updates across parameter dimensions, capturing relationships that elementwise scaling misses.

Those richer statistics bring additional storage and matrix computation.

SOAP connects this approach back to Adam. It rotates gradients into the eigenbasis of Shampoo’s preconditioners, applies Adam there, and rotates the update back.

The coordinate system changes how Adam’s elementwise scaling acts on the update. Computing and refreshing that basis also adds overhead.

Muon approximately orthogonalizes momentum updates for matrix parameters using Newton–Schulz iterations. It changes the matrix update’s singular-value structure rather than scaling every coordinate independently.

Parameter selection matters here. Muon is typically applied to hidden weight matrices, with another optimizer handling embeddings, output layers, and vector parameters.

Sophia uses a diagonal curvature estimate to scale momentum, then clips updates elementwise. Its curvature estimate is refreshed periodically, limiting the average overhead of obtaining that information.

Comparing optimizers on your workload

I’d use AdamW as the reference run for a transformer experiment, then choose alternatives against a specific constraint.

For memory pressure, investigate Adafactor or Lion. For matrix updates, evaluate Muon, Shampoo, or SOAP while accounting for their parameter coverage and implementation costs.

Give each optimizer a learning-rate search. Keep the data, model, and evaluation protocol consistent, and report both validation quality at a fixed compute budget and time to a target quality.

Also measure peak memory and step time. An optimizer that reaches the target in 20% fewer steps but makes each step 30% slower takes about 4% longer overall.

The useful result is a reproducible improvement under your training constraints, including the cost of tuning it.

If you want to build these foundations properly, our Classical ML and Deep Learning roadmap covers the full stack in order, from the assumptions behind linear models to neural network training and the architectures built on top:

Start the Classical ML and Deep Learning roadmap →

👉 Over to you: Which optimizer has improved your training runs enough to justify changing your default?

Good day!

• Harness Engineering 2.0 is here!

• Why other requests change your LLM’s output.

• RAG vs. retriever-free RAG, clearly explained.

Harness Engineering 2.0 is here!

Tools got MCP, agent-to-agent communication got A2A, and agent interfaces got AG-UI. Now harnesses have a protocol too.

Developers just built UHP (Unified Harness Protocol), a standard interface for running agent tasks on any agent harness. HarnessRouter is its open-source server implementation.

Here’s how it works:

↳ Your backend sends a task with a harness_id.

↳ HarnessRouter starts the selected harness in its own session.

↳ The harness runs its native agent loop and streams back progress.

↳ Files, artifacts, failures, and the final output come back through the same interface.

Without UHP, every harness needs its own handling for task execution, sessions, streaming, files, recovery, and output formats.

With UHP, Codex, Claude Code, Gemini CLI, Hermes, Pi, and other harnesses plug into the same product backend. Each one keeps its native tools, skills, models, and agent loop.

So developers can:

↳ Switch harnesses without rebuilding the product integration.

↳ Run the same task on different harnesses.

↳ Add new harnesses without maintaining another execution stack.

HarnessRouter supports 30 harness bases today.

The harness also changes what a task costs. In HarnessRouter’s published benchmark, keeping the model fixed and switching only the harness cut credits by 32% to 53% on the same healthcare prep task. Across all eight harness and model setups, the cheapest used 99.8% fewer credits than the most expensive.

That’s one task, so your numbers will differ. Harness Arena lets you run your own workload across configurations and compare cost and latency. Alibaba’s Qoder has also adopted UHP and is available as an agent backend through the HarnessRouter API.

GitHub repo: https://github.com/HarnessRouter/harnessrouter

(don’t forget to star it ⭐)

Why other requests change your LLM’s output

Your LLM’s output depends on requests that aren’t yours.

The same prompt can produce different logits depending on how many other requests share its batch on the GPU.

Thinking Machines Lab measured this. They sampled 1,000 completions of “Tell me about Richard Feynman” from Qwen3-235B at temperature 0 and got 80 unique completions.

All 1,000 matched for the first 102 tokens. At token 103, 992 continued with “Queens, New York” and 8 with “New York City”.

Here’s what happens inside the GPU.

Why the order of additions matters

Most of an LLM’s forward pass is spent accumulating sums. Every matmul, RMSNorm, and attention step reduces thousands of floating-point values into one output, so the order of those additions affects the result.

The catch is that floating-point addition isn’t associative. Every addition rounds to the nearest representable value, so (a + b) + c and a + (b + c) can differ in the low-order bits.

fp16 has an 11-bit significand, so between 1024 and 2048, representable values are 1.0 apart.

↳ Add 1024 + 0.4 + 0.4 left to right, and each step rounds back to 1024.

↳ Add the small values first, and 0.4 + 0.4 gives 0.8, then 1024 + 0.8 rounds to 1025.

Why batch size changes the order

GPU kernels pick the parallelization strategy that keeps the GPU busiest, and that choice depends on batch size.

An H100 SXM has 132 SMs, its compute units. A small batch doesn’t give each of them enough work.

So the kernel splits each long sum into chunks, adds the chunks on different SMs at the same time, then adds the partial results together. This is a split reduction, called split-K in matmuls and split-KV in attention.

A larger batch has enough rows to keep every SM busy, so each row’s sum stays on one SM and gets added in a single pass.

Different batch size, different split, different order of additions, slightly different logits.

This isn’t random noise. Run the exact same batch twice and you get bitwise identical results. The output only changes when the batch around your request changes.

How a tiny difference flips a token

Usually the difference is negligible. It matters when two tokens are nearly tied.

Say that in a batch of 8, “Queens” scores 12.3417 and “New” scores 12.3415. In a batch of 32, rounding shifts them to 12.3414 and 12.3416.

At temperature 0, argmax picks “Queens” in the first case and “New” in the second. Every later token is conditioned on a different prefix, so the whole completion diverges.

So temperature 0 doesn’t make inference deterministic. The sampler is deterministic, but it reads logits from a forward pass whose numerics depend on traffic you don’t control.

The fix: batch-invariant kernels

The fix is to make every reduction batch invariant. Each row gets reduced in the same fixed order, no matter how many rows share the batch.

This costs throughput. The kernel can no longer pick the best split for each batch size, so it uses one fixed strategy even when that leaves SMs idle.

The major inference engines now expose this as a flag:

↳ vLLM has VLLM_BATCH_INVARIANT=1, still in beta.

↳ SGLang has --enable-deterministic-inference, with most slowdowns between 25% and 45% in the LMSYS benchmarks.

↳ DeepSeek’s DeepInfer has DEEPINFER_BATCH_INVARIANT=1, on NVIDIA GPUs only.

# vLLM
VLLM_BATCH_INVARIANT=1 vllm serve Qwen/Qwen3-8B

# SGLang
python3 -m sglang.launch_server --model-path Qwen/Qwen3-8B \
  --attention-backend fa3 --enable-deterministic-inference

With batch-invariant kernels on, Thinking Machines got all 1,000 Feynman completions identical.

If you run evals, RL rollouts, or anything else that needs reproducible outputs, turn it on and treat batch composition as part of your experimental setup.

RL gains the most. When the sampler and the trainer compute bitwise identical numbers, training stays truly on-policy. In the Thinking Machines run, the KL divergence between sampler and trainer stayed flat at 0.

Why production makes this worse

In production, the batch around your request never stays fixed. With continuous batching, the server adds and removes requests at every decoding step.

So your request can run at a different batch size for every token it generates.

RAG vs. retriever-free RAG, clearly explained!

In standard RAG, similarity search decides what the LLM can ever see.

Documents are split into chunks, embedded, and stored in a vector index. When a query arrives, the system embeds it and pulls the top-k chunks with the closest vectors. A reranker reorders those candidates, and the best few go to the LLM.

If the right passage never made the top-k, nothing later in the pipeline can bring it back, not even a strong reranker.

Yesterday we covered vectorless RAG, where an LLM reads a document tree and picks sections. Retriever-free RAG, from a new Cornell and ASAPP paper, also walks a tree. But it uses a decision model to do it, not a generative LLM.

What a decision model does

A decision model doesn’t generate text. It reads one shared context and returns a probability for every yes/no question about it, in one forward pass.

The paper uses Jev 1.13. Put the query next to every candidate document, ask “Is this relevant?” for each, and a single pass scores all of them.

How a query moves through the tree

The documents are organized as a tree, like a table of contents, and the decision model walks down it:

↳ Top documents: the model reads the query next to every candidate document, asks “Is this relevant?” for each, and keeps the best few.

↳ Top passages: it opens only those documents and asks the same question for every passage inside them.

↳ Most relevant? (optional): a single multiple-choice question reorders the top 20 passages, or the top 5 on Natural Questions.

Dropped documents are never opened, so their passages are never scored.

For Wikipedia, the tree was built offline. The 2.68M paragraphs from 108,626 pages were grouped into 1,000 clusters using e5-base-v2 embeddings and k-means, and Qwen3-8B wrote the summaries.

The results

On Natural Questions, its top pick contained the answer 65.4% of the time. Embedding search plus an 8B reranker got 48.0%, and GPT-5 reranking the dense top 20 got 60.8%.

The gain comes from ranking, not coverage. Embedding search’s top 100 holds the answer more often than the navigated shortlist, 98.4% vs. 94.0%. But when both contain it, navigation puts it first 70.1% of the time vs. 61.9%.

Median retrieval takes under 2 seconds per query. Each method was timed in a different setup, though, so this isn’t a controlled speed comparison.

There are limits:

↳ It needs a sensible hierarchy.

↳ A document pruned early is gone for good.

↳ It skips embedding search only at query time, since the Wikipedia tree was built offline with embeddings.

If you’re building a RAG system, this is worth benchmarking against your current setup.

Here’s the paper: Retriever-Free Retrieval: Navigating Documents with Decision Models →

Whichever retrieval you pick, a RAG system only works when every stage holds up: chunking, embeddings, retrieval, reranking, evaluation, and latency. Our full RAG crash course covers RAG from basics to beyond:

• Part 1: Build the complete RAG workflow, including chunking, embeddings, retrieval, and reranking →

• Part 2: Evaluate retrieval and generation quality with RAG-specific metrics →

• Part 3: Reduce RAG latency with faster retrieval and generation paths →

• Part 4: Extend RAG beyond text to multiple data types →

• Part 5: Understand CLIP embeddings, multimodal prompting, and tool calling →

• Part 6: Build a multimodal RAG system over real-world data →

• Part 7: Use Graph RAG for relationships that ordinary chunk retrieval misses →

• Part 8: Improve retrieval with ColBERT and ColBERTv2 →

• Part 9: Build vision-driven retrieval with ColPali →

• Part 10: Apply the first set of techniques for production RAG systems →

• Part 11: Complete the production RAG techniques and implementation patterns →

• Part 12: Reduce prefill latency by reusing KV caches without a shared prefix →

• Part 13: Preload a corpus before queries arrive and examine the production constraints →

• Part 14: Compress preloaded caches and measure where the approaches break →

• Part 15: Implement Block-Attention, Cartridges, and production preloading →

👉 Over to you: Do you turn on batch-invariant mode for your evals, or accept a little drift?

Good day!

• Speculative decoding, now 2.6x faster on a MacBook.

• RAG vs. Vectorless RAG, clearly explained.

Speculative decoding, now 2.6x faster on a MacBook:

We ran Qwen3.5 9B on an M5 MacBook Pro with the same prompt, three ways:

• No speculative decoding: 22.0 tokens/s, done in 42.2s

• Regular speculative decoding: 34.7 tokens/s, done in 30.4s

• DFlash + Weaver in Uzu: 92.1 tokens/s, done in 11.0s

That is about 4x faster than plain decoding and 2.6x faster than regular speculative decoding. The video shows all three runs side by side.

Speculative decoding lets a cheap drafter guess the next few tokens, and the main model checks them all in one pass.

The fastest drafters, like DFlash, guess every position in parallel. But each guess is made on its own, so position 5 never sees what was picked at position 4.

One early mismatch, and everything after it gets thrown away.

Weaver fixes this. It is a tiny adapter, a single Transformer layer with 56.7M parameters, that sits on top of DFlash and:

↳ picks among DFlash’s top candidates left to right, so each pick depends on the last one.

↳ grows a tree of likely continuations instead of a single chain.

↳ lets the main model verify the whole tree in one pass.

On the MacBook, every pass of the 9B model produced 7.5 tokens instead of 1.

On a B200 GPU, the Weaver paper reports a 4.08x speedup over standard decoding, measured across chat, math, and code benchmarks.

Mirai open-sourced the whole stack in Uzu, its inference engine built for Apple Silicon. Trying it takes two commands:

brew install mirai
mirai --model trymirai/Qwen3.5-9B-M -m \
 "Write a python function to merge two sorted lists"

GitHub repo: https://github.com/trymirai/uzu

(don’t forget to star 🌟)

RAG vs. Vectorless RAG, clearly explained!

The chunk closest to a question is often not the chunk that answers it.

In a standard RAG setup, documents are split into chunks, embedded, and stored in a vector DB. The query gets embedded too, and the top-k closest chunks go to the LLM.

This works when the closest text also holds the answer. On long, structured documents, that assumption breaks in two ways.

Problems with similarity search

Five annual reports from one company usually follow the same template every year, and only the numbers change.

So a question about FY2022 can land just as close to an FY2025 chunk as to the FY2022 one. The embedding captures what a chunk is about, and both chunks are about the same thing.

The LLM then answers from the wrong year. Nothing in the pipeline flags it, because the retrieved chunk really is on topic.

Cross-references cause the second failure. A chunk might say “see Appendix G”, while the table in Appendix G holds the actual numbers.

That table looks nothing like the query, so it never makes it to the top-k. The pointer gets retrieved, but the answer it points to does not.

Fixes on the vector side do help. Anthropic’s contextual retrieval adds document context to each chunk before embedding, and with a reranker it cut top-20 retrieval failures from 5.7% to 1.9%.

But these fixes still rank text by how similar it looks to the query.

How vectorless RAG works

Vectorless RAG drops the embedding model and the vector DB, and replaces similarity search with LLM reasoning.

There are three parts to the process.

First, a preprocessing step parses each document’s layout into a ToC tree index. Every node holds a section title, its page range, and a short summary.

Nothing gets chunked, so a table never gets split across two chunks. And the whole tree is small enough to fit in the LLM’s context. In PageIndex, the most popular open-source implementation, the structure comes from the layout itself, and a basic model only writes the summaries.

Next, at query time, the LLM gets the query along with the tree. It reads the section titles and summaries and picks the few sections most likely to hold the answer.

For example, for a question about debt, it might pick the “Financial Summary” section and also “Appendix G”, because its summary says the debt details are there.

That is how an analyst reads a 10-K. They open the table of contents, go to the financial summary, and follow the note to the appendix.

Finally, the system pulls those full sections and checks whether the evidence is enough.

↳ If it is, the LLM answers from the full sections, with page citations.

↳ If not, it goes back to the tree and opens more nodes.

The LLM also sees the chat history. So if you ask about assets and then follow up with “what about liabilities?”, it goes back to the same report.

What it costs

The trade-off is that every query now runs LLM calls over the tree.

PageIndex tested this on 63 questions over NVIDIA’s FY2022 to FY2026 annual reports. Note that the benchmark comes from the PageIndex team itself.

• Tree retrieval answered all 63 correctly, at about $110 per 1,000 queries.

• The best vector setup reached 92.1%, and only by sending the top 100 chunks, at about $134.

• At top-10, vector search cost about $15 per 1,000 queries, but no embedding model got past 76.2%.

So vector search is cheaper per query at a normal k, but it can be less accurate on documents like these.

Latency follows the same pattern. Every hop through the tree is another LLM call, and PageIndex has not published latency numbers.

The index costs more too. One independent benchmark across about 1,000 documents skipped the tree entirely, because building one took 2 to 5 minutes of LLM calls per document.

And the tree only helps when documents have real structure. A pile of support tickets has no table of contents to reason over.

So the two suit different data:

↳ Use a tree for long, structured documents like filings, contracts, and manuals.

↳ Stick with a vector DB for lots of short, unrelated text.

Vector search asks which text looks like the question. Vectorless RAG asks where in the document the answer should be.

If you want to dive deeper, our full RAG course covers RAG from basics to beyond:

• Part 1: Build the complete RAG workflow, including chunking, embeddings, retrieval, and reranking →

• Part 2: Evaluate retrieval and generation quality with RAG-specific metrics →

• Part 3: Reduce RAG latency with faster retrieval and generation paths →

• Part 4: Extend RAG beyond text to multiple data types →

• Part 5: Understand CLIP embeddings, multimodal prompting, and tool calling →

• Part 6: Build a multimodal RAG system over real-world data →

• Part 7: Use Graph RAG for relationships that ordinary chunk retrieval misses →

• Part 8: Improve retrieval with ColBERT and ColBERTv2 →

• Part 9: Build vision-driven retrieval with ColPali →

• Part 10: Apply the first set of techniques for production RAG systems →

• Part 11: Complete the production RAG techniques and implementation patterns →

• Part 12: Reduce prefill latency by reusing KV caches without a shared prefix →

• Part 13: Preload a corpus before queries arrive and examine the production constraints →

• Part 14: Compress preloaded caches and measure where the approaches break →

• Part 15: Implement Block-Attention, Cartridges, and production preloading →

👉 Over to you: Do your documents have enough structure for a tree index, or does a vector DB still fit better?

Good day!

• Training LLMs without one correct answer.

• Prefix-aware LLM routing, clearly explained.

• Regular ML Inference vs. LLM Inference

Training LLMs without one correct answer

Suppose a model must translate questions from finance analysts into SQL.

A basic verifier might run each generated query and return a reward of 1 when it executes without an error.

Due to this, the model will learn to produce executable SQL, but it will not necessarily learn to produce correct SQL.

A query can select the wrong columns, omit a filter, duplicate rows after a join, or calculate revenue over the wrong period. The database will execute all of them without complaint.

This is the part of RLVR that deserves more attention. A verifier inherently does not discover what correctness means, which is why the engineer must encode it.

Amazon SageMaker AI makes it simple to test that definition through its serverless model customization workflow.

In SageMaker Studio, the engineer can select a base model, prepare the examples, attach a custom reward function, and submit an RLVR job without managing the training infrastructure.

SageMaker AI provisions the compute, runs and tracks the job, stores the resulting model artifacts, and tears the infrastructure down afterward.

That makes it easier to iterate on the model and reward logic as separate parts of the experiment.

For text-to-SQL, you can split the verifier into layers.

• The first layer checks whether the query parses, executes, and remains read-only.

• The second compares the returned rows with the reference result. This catches queries that are valid SQL but answer a different question.

• The final layer can score operational properties such as latency or scanned data, but only after the query has passed the correctness checks.

The order is important because a fast query that returns the wrong rows should never outrank a slower query that returns the right ones.

During customization, the model generates several candidates for each prompt. The verifier scores them, and GRPO updates the model toward the candidates that satisfy the full specification.

This also gives the evaluation stage a clear purpose. A higher training reward is not enough. The customized model must improve on held-out execution accuracy without increasing unsafe queries or weakening result correctness.

SageMaker AI agent skills can help with model selection, data preparation, job configuration, evaluation, and deployment through Claude Code, Codex, Kiro, or another compatible coding agent.

AWS has published an interactive workshop for anyone who wants to get hands-on with serverless model customization in SageMaker AI:

Try the SageMaker AI model customization workshop →

Thanks to AWS for working with us on today’s issue!

Prefix-Aware LLM Routing, clearly explained

The same request on the same GPUs can wait 92 seconds or half a second for its first token, depending only on where the router sends it.

Production prompts start with the same system prompt, tool definitions and earlier turns. Only the last part of each request is new.

So the GPU that served the previous turn still holds the KV cache for that shared prefix.

But a cache-blind router may send the next request to a different GPU, which then recomputes the whole prefix before it can write a single token.

A prefix-aware router avoids this by tracking which GPU holds which part of each prompt.

Let’s walk through how it works.

1. Why routing decides the cache hit

Inside one engine, prefix caching already works well. In llm-d’s tests, sending a roughly 10,000-token prompt to a Qwen3-32B vLLM instance a second time cut time to first token from 4.3s to 0.6s.

The trouble starts when we scale out. Each replica keeps its own KV cache, and no replica can see another’s.

With 8 replicas and random routing, the next turn lands on the warm GPU about 1 time in 8.

The other 7 times, a GPU recomputes a prefix that already sits in memory somewhere else in the cluster.

So in a cluster, the router decides whether the engine’s prefix cache gets used at all.

2. Hash the prompt block by block

The engine stores the KV cache in fixed-size blocks. vLLM uses 16 tokens per block by default and only caches full blocks.

Each block gets a hash built from its parent block’s hash plus its own token IDs. So the hash of block D covers A, B, C and D together.

Here is a simplified version of what vLLM does:

def block_hashes(token_ids, block_size=16):
    hashes, parent = [], None
    for i in range(0, len(token_ids) - block_size + 1, block_size):
        block = tuple(token_ids[i:i + block_size])
        parent = hash((parent, block))
        hashes.append(parent)
    return hashes

This chaining turns a block lookup into a prefix lookup. If block D’s hash matches, everything before it matched too.

The router computes the same hashes for each incoming request, so it can compare prompts without touching any KV tensors.

3. Look up which worker holds those blocks

The router keeps a block-to-worker index. It maps each block hash to the workers that hold that block, and it never stores the KV tensors themselves.

That keeps the index tiny. llm-d ran the numbers for DeepSeek-R1 in FP8 on 8 H200s, where each 128-token block takes about 8.6 MB of VRAM but only an 8-byte hash in the index.

So a 365 GB KV cache pool needs only about 339 KB of index.

For a request with prefix blocks A to D and a new suffix E, the router walks the hashes in order:

↳ Worker A holds 2 matching blocks.

↳ Worker C holds 3.

↳ Worker B holds all 4, so Worker B gets the request.

If no worker has a match, the request gets a full prefill on whichever worker the router picks.

4. Reuse the cache and report back

Worker B reuses the cached KV for A to D, computes KV only for E, and starts decoding.

Once E is cached, Worker B emits a cache event and the router records that E now lives on Worker B. So the next turn of this conversation lands on Worker B too.

Evictions emit events as well. In llm-d, each vLLM pod publishes these KV events over ZMQ every time a block is created or evicted, so the index tracks what each GPU actually holds.

That is the difference between precise and approximate routing. An approximate router guesses cache locations from its own routing history instead.

5. Weigh load against cache hits

If the router only chased cache matches, the warmest GPU would end up with the longest queue, because every matching request wants it.

So real routers score load too. NVIDIA Dynamo’s KV router adds the prefill a worker would still owe after cache credit to that worker’s decode load, then picks the lowest total.

Here is the example from Dynamo’s docs, for a request with 10 prefill blocks:

cost = (prefill blocks - cached blocks) + decode blocks

Worker 1: (10 - 2) + 10 = 18
Worker 2: (10 - 5) + 5  = 10   <- picked
Worker 3: (10 - 8) + 9  = 11

Worker 3 has the best match, yet Worker 2 wins. The router accepts recomputing 3 extra blocks to avoid the busier GPU.

Dynamo exposes this tradeoff as a knob. More cache credit lowers time to first token, and less credit spreads load more evenly and lowers inter-token latency.

llm-d compared four routing strategies on 8 vLLM pods across 16 H100s.

The workload simulated 150 customers, each with a 6,000-token shared context and 5 users sending 1,200-token questions, at 3 to 60 QPS. Those shared prefixes needed about 73% of the cluster’s KV cache, roughly 6x what one pod could hold.

Here are the p90 times to first token:

• Random: 92.6s

• Load-aware only: 94.9s

• Approximate prefix-aware: 31.1s

• Precise prefix-aware: 0.54s

That is about 170x sooner than random routing. Output throughput also roughly doubled, from 4,429 to 8,730 tokens/s.

Load-aware routing did no better than random. Spreading requests evenly doesn’t help when each one lands on a GPU that has to recompute its prefix.

A later llm-d rerun on a newer vLLM, against a plain Kubernetes round-robin Service, showed a smaller gap, but p90 time to first token still dropped by more than 99%.

All of this depends on the prefix cache itself working.

A small change early in the prompt, like a timestamp at the top of the system prompt, changes every block hash after it, and no router can match those blocks.

So keep the stable parts first, like the system prompt, tool definitions, and documents. Put whatever changes per request at the end.

A router can only find a cached prefix if the prompt still produces the same hashes.

We covered how prefix caching works, how it differs from KV, prompt, and semantic caching, and which prompt changes cause cache misses, in a detailed article.

Read it here →

👉 Over to you: Does your serving stack route requests by cache location, or does it still round-robin across replicas?

​Regular ML inference vs. LLM inference​

LLM inference presents unique challenges over regular ML inference, due to which we have specialized, high-performance LLM inference engines, like vLLM, LMCache, SGLang, and TensorRT-LLM.

Let’s understand these challenges today and how we solve them!

Continuous batching

Traditional models, like CNNs, usually deal with a fixed-size image input and a fixed-length output (like a label). This makes batching easy.

LLMs, however, deal with variable-length inputs (the prompt) and generate variable-length outputs.

So if you batch some requests, all will finish at different times, and the GPU would have to wait for the longest request to finish before it can process new requests. This leads to idle time on the GPU:

Continuous Batching solves this.

Instead of waiting for the entire batch to finish, the system monitors all sequences and swaps completed ones (<EOS> token) with new queries:

This keeps the GPU pipeline full and maximizes utilization.

Prefill-decode disaggregation

LLM inference is a two-stage process with fundamentally different resource requirements.

• The prefill stage processes all the input prompt tokens at once, so this is compute-heavy.

• The decode stage autoregressively generates the output, and this demands low latency.

Running both stages on the GPU means the compute-heavy prefill requests will interfere with the latency-sensitive decode requests.

Prefill-decode disaggregation solves this by allocating a dedicated pool of GPUs for the prefill stage and another pool for the decode stage.

In contrast, a standard ML model typically has a single, unified computation phase.

GPU memory management + KV caching

Generating a new token uses the key and value vectors of all previous tokens. To avoid recomputing these vectors for all tokens over and over, we cache them (​we covered KV caching in detail here​):

This KV Cache grows linearly with the total length of the conversation history.

But in many workflows, inputs like the system prompts are shared across many requests. So we can avoid recomputing them by using these KV vectors across all chats:

That said, KV cache takes up a significant memory since it’s stored in contiguous blocks. This wastes GPU memory and leads to memory fragmentation:

Paged Attention solves this problem by storing KV caching in non-contiguous blocks and then using a lookup table to track these blocks. The LLM only loads the blocks it needs, instead of loading everything at once.

We will cover Paged Attention in another issue.

Prefix-aware routing

To scale standard ML models, you can simply replicate the model across multiple servers/GPUs and use straightforward load-balancing schemes like Round Robin or routing to the least-busy server.

Since each request is independent, this works fine.

But LLMs heavily rely on caching (like the shared KV prefix discussed above), so requests are no longer independent.

If a new query comes in with a shared prefix that has already been cached on Replica A, but the router sends it to Replica B (which is less busy), Replica B has to recompute the entire prefix’s KV cache.

Prefix-aware routing solves this.

Different open-source frameworks each have their own implementations for prefix-aware routing.

Generally, prefix-aware routing requires the router to maintain a map or table (or use a predictive algorithm) that tracks which KV prefixes are currently cached on which GPU replicas.

When a new query arrives, the router sends the query to the replica that has the relevant prefix already cached.

Model sharding strategies

There are several strategies to scale a dense ML model (​discussed here​):

LLMs, like Mixture of Experts (MoE), are complicated.

MoE models use a specialized parallelism strategy called expert parallelism, which splits the experts themselves across different devices, and the attention layers are replicated across all GPUs:

So each GPU holds the full weights of only some experts, not all. This means that each GPU processes only the tokens assigned to the experts stored on that GPU.

Now, when a query arrives, the gating network in the MoE layer dynamically decides which GPU it should go to, depending on which experts are activated.

This is a complex internal routing problem that cannot be treated like a simple replicated model. You need a sophisticated inference engine to manage the dynamic flow of computation across the sharded expert pool.

If you want to learn the full LLMOps lifecycle, we have covered every bit of it in the LLMOps course, starting from fundamentals to production:

• ​Read Part 1 on fundamentals of LLMOps here →​

• ​Read Part 2 on understanding the core building blocks of LLMs →​

• ​Read Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixture-of-experts, and the fundamentals of pretraining and fine-tuning →​

• ​Read Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applications →​

• ​Read Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and different prompting techniques →​

• ​Read Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more →​

• ​Read Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techniques for building high-signal inputs →​

• ​Read Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic context injection, and common failure modes in agentic applications →​

• ​Read Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong understanding of the fundamental concepts →​

• ​Read Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for evaluation of LLM apps →​

• ​Read Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming →​

• ​Read Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO →​

• ​Read Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model parallelism →​

• ​Read Part 14 on the fundamentals of LLM serving, including API-based access, inference with vLLM, and practical decisions →​

👉 Over to you: What are some other differences between LLM inference and regular inference?

Good day!

• An open-source engine for faster local LLMs on Mac

• Why DeepSeek stores its KV cache on disk

• What is Early Exaggeration in tSNE?

An open-source engine for faster local LLMs on Mac

This open-source engine generates tokens 2.7x faster than Ollama on a MacBook:

I tested it on a base M5 MacBook Pro with Qwen3.5 4B at 4-bit and the same 9.2k-token prompt. Ollama ran its MTP build, so both had speculative decoding.

• Prefill: Ollama 464 tok/s, Magnitude 755 tok/s (1.6x)

• Decode: Ollama 22.6 tok/s, Magnitude 60.0 tok/s (2.7x)

• End-to-end, Magnitude finished in half the time

The video above shows both runs side by side.

The Ollama numbers don’t mean it’s badly built. Its kernels were simply tuned in advance.

A kernel is the small GPU program behind every matrix multiply and attention step. Its speed depends on choices like how work is split across GPU cores and how data moves through memory.

The best choices change with every chip. Apple Silicon alone spans 8 to 80 GPU cores and under 70 to over 800 GB/s of bandwidth, and every model has its own layer sizes.

A prebuilt engine has to pick one set of settings for all of them, so most local setups run on a compromise.

Magnitude replaces that compromise with tuning on your machine:

↳ When you download a model, it compiles that model’s kernels locally.

↳ It tries different kernel configurations and keeps whichever runs fastest on your chip.

↳ It pairs each model with the fastest speculative decoding drafter it has found, so each pass of the model produces more than one token.

This matters most for agents. Every file and tool output has to be processed before the model writes anything, and that prefill step is where kernel efficiency shows up. Concurrent sessions also share prefix caches, so parallel agents don’t slow each other down.

Magnitude is an open-source (Apache 2.0) inference engine written in Rust. It plugs into Claude Code, Codex, OpenCode, Pi and other agents with one click.

GitHub repo: github.com/magnitudedev/magnitude

(don’t forget to star it 🌟)

Why DeepSeek stores its KV cache on disk

They just open-sourced DeepInfer, the inference stack behind cache hits that return the first token up to 26x faster.

When a request repeats a long prefix, like a system prompt or earlier chat turns, the server can reuse the KV cache it already computed instead of running prefill again.

Ideally, you’d expect that cache to sit in GPU memory, since disk is supposed to be far too slow for anything on the request path.

In practice, DeepSeek keeps it on SSDs.

When DeepSeek launched this in 2024, a 128K-token prompt that hit the cache got its first token in 500ms instead of 13s.

And in one day of production traffic last year, 56% of all input tokens were served straight from disk.

So disk isn’t the bottleneck here, as long as what you store is small enough and the storage is wide enough.

Let’s understand how this works.

1. Why reuse the KV cache at all

Before generating anything, an LLM reads the whole prompt and computes a key and a value for every token in every layer.

This is prefill, and on long prompts it’s the slow part.

Many requests start the same way.

For instance, a chat resends every earlier turn, and an agent resends the same system prompt and tool definitions on every call.

If those KV are saved, the next request with the same prefix loads them and only computes the new tokens.

2. Why GPU memory is the default

Every token stores KV in every layer, so on a standard model a long prompt’s KV cache runs into many GBs.

Reading that much from disk can take longer than recomputing it on the GPU.

So engines keep it in GPU memory.

But GPU memory is small, so old prefixes get evicted fast and many repeats miss.

3. DeepSeek made the KV cache small

DeepSeek-V2 introduced Multi-head Latent Attention (MLA).

Instead of storing separate KV for every attention head, MLA stores one small compressed vector per token per layer. The keys and values are rebuilt from it during attention.

In the V2 paper, this cut the KV cache by 93.3% compared to DeepSeek’s earlier 67B model.

That made the KV cache small enough to store on low-cost disks.

Fewer bytes per cached token means fewer bytes to read on every hit, and that made loading from SSD faster than recomputing.

4. 3FS reads it from many SSDs at once

One SSD would still be too slow, so the cache lives on 3FS, DeepSeek’s own distributed file system.

3FS splits files into chunks and spreads them across thousands of SSDs on many storage nodes. A read pulls chunks from many drives in parallel, so their bandwidth adds up.

When a request arrives, the system matches its prefix against cached blocks and fetches those blocks from 3FS into host memory.

Then it copies them to the GPU layer by layer while the model is already running.

And since every prefill machine can reach every cached prefix, a request doesn’t have to go back to the machine that served it last.

Disk is also cheap, so the cache can stay for days. As per the DeepInfer docs, cached prefixes are retained for 3 days for agent traffic and 7 days for chat.

Disk caching works when the KV cache is small and the storage is wide. DeepSeek got the first from its attention design and the second from 3FS.

This also shows up in the price. On deepseek-flash today, a cache hit costs 50x less than a miss.

If you use the API, keep the stable parts of your prompt, like the system prompt, tools, and documents, at the front. Only an exact match from the very first token counts as a hit.

That said, prefix caching is only one of the ways to manage the KV cache in LLM serving.

We wrote an open-access KV Cache Engineering article covering quantization and other ways to reduce cache memory, with code examples and their trade-offs.

Read it here →

👉 Over to you: Do you order your prompts for cache hits, or does every request start from scratch?

​What is Early Exaggeration in tSNE?​

tSNE algorithm has an early_exaggeration parameter:

What is it, and how does it help?

Let’s understand this today, along with a strategy to improve it.

Let’s begin!

A brief about tSNE

To understand early exaggeration, we need to briefly overview how tSNE works.

The steps discussed below have been discussed in a mathematical manner here: ​Formulating and Implementing the t-SNE Algorithm From Scratch​.

The core objective of tSNE is to project a given high-dimensional data into low-dimensions (mostly 2D) for visualisation.

This is how we do it:

Step 1) For every point (i) in the given high-dimensional data, plot Euclidean distances to all other points (j), which gives you a Gaussian-like probability distribution. For instance, consider the dark red point marked on the left below. Other red points that are closer to it have a higher probability of being i‘s neighbor than those that are far:

The above step assigns a neighbourhood distribution to every data point.

Step 2) Next, for every data point xᵢ in the given data (which is in high-dimensions), randomly initialize its counterpart yᵢ in 2D space. These will be our projections.

Step 3) Just like we created a distribution of Euclidean distances using high-dimensional data in Step 1, we create a distribution of the random data as well.

Step 4) At this point, every data point (i) has a high-dimensional distribution and a random low-dimensional distribution. The objective is to match these two distributions so that the low-dimensional representation preserves the neighborhood structure of the high-dimensional data.

• Using KL divergence as a loss function helps us achieve this. It measures how much information is lost when we use a distribution Q to approximate distribution P.

• Ideally, we want to have the minimum loss value (which is zero), and this will be achieved when P=Q.

The model can be trained using gradient descent, and eventually, it generates projections that preserve the local neighborhood structure of the original high-dimensional data.

But having early exaggeration makes it even better.

Early exaggeration

The step exaggerates (inflates) the values of all high-dimensional probabilities (P) during the initial stages of the optimization by multiplying a constant factor greater than 1:

Here’s why this is useful:

Recall that the KL divergence is minimized when the distributions P and Q are identical.

By increasing P, the optimization process learns larger values of Q since it must match them to get minimum loss.

As a result, the low-dimensional clusters created will be very tight, and all individual clusters will be widely separated from each other.

This is why we use early exaggeration.

Improving early exaggeration

Early exaggeration multiplies all high-dimensional probabilities (P) by a constant factor greater than 1.

While this inflates the probability of data points that are close, the downside is that it also inflates the probability of data points that are far (likely in different clusters).

A better strategy is to set a threshold (λ) for inflating values:

• Only exaggerate the joint probabilities if they exceed a certain threshold.

• Otherwise, understate (or reduce) them.

This makes intuitive sense as well.

This is how early exaggeration works and why it is so useful in tSNE.

We went into much more detail about tSNE in a beginner-friendly manner in the full article: ​Formulating and Implementing t-SNE From Scratch →​

This article covers:

• The limitations of PCA.

• The entire end-to-end working of the SNE algorithm (an earlier version of t-SNE).

• What does SNE try to optimize?

• What are local and global structures?

• Understanding the idea of perplexity visually.

• The limitations of the SNE algorithm.

• How is t-SNE different from SNE?

• Advanced optimization methods used in t-SNE.

• Implementing the entire t-SNE algorithm by only using NumPy.

• A practice Jupyter notebook for you to implement t-SNE yourself.

• Understanding some potential gaps in t-SNE and how to possibly improve it.

• PCA vs. t-SNE comparison.

• Limitations of t-SNE.

• Why is perplexity the most important hyperparameter of t-SNE?

• Important takeaways for using t-SNE.

Similar to this, we also formulated the PCA algorithm from scratch here: ​Formulating the PCA Algorithm From Scratch →​

👉 Over to you: Can you identify a bottleneck in the t-SNE algorithm?

Good day!

We ran Qwen3.5 9B on a base M5 MacBook Pro with three local inference engines, one at a time.

• llama.cpp reached 22.0 tokens/s

• MLX reached 25.1 tokens/s

• uzu reached 92.1 tokens/s

The video shows all three runs side by side.

The first two numbers don’t mean llama.cpp or MLX are badly built. Both sit right under a hard limit of local decoding.

Every new token needs one full pass through the model, and that pass reads all of the model’s weights from memory.

A base M5 reads memory at about 153 GB/s, and a 4-bit 9B model weighs about 5.2 GB.

So at one token per pass, the ceiling is roughly 153 ÷ 5.2, or about 29 tokens/s, however fast the GPU computes.

uzu gets past that ceiling by producing several tokens per pass instead of one.

↳ A small drafter (DFlash) proposes candidates for the next 16 positions at once.

↳ A tiny adapter (Weaver, 56.7M params) picks among them left to right so the guesses form coherent text, and keeps a few alternative branches as a tree.

↳ The 9B model checks the whole tree in a single pass and keeps the longest path it agrees with.

The main model still decides every token that ends up in the output.

In our run, each pass of the 9B model produced 7.5 tokens on average. Ordinary decoding produces exactly 1.

The weights and kernels are built around this path too. uzu ships its own 4-bit checkpoints, and on M5 chips it runs the verification step on the GPU’s int8 matrix path.

uzu is an open-source inference engine by Mirai Labs, written in Rust with Metal kernels. It has Swift, Python, TypeScript, and Rust bindings, so the same engine runs inside a Mac or iOS app.

We covered why local inference is memory-bound and how uzu’s speculative decoding gets past it in a full engineering deep dive, with a benchmark client you can point at your own prompts.

Read it here →

Find the GitHub repo here →

(don’t forget to star it 🌟)

Which OCR solution works best for your documents?

The OCR model with the best average score can still be the wrong pick for your documents.

We ran DeepSeek-OCR, PaddleOCR-VL and Datalab Surya on the same 150 pages, from handwritten notes and rotated scans to dense tables and a filled tax form.

The video shows every page, what each parser returned, and why each test passed or failed.

Most OCR comparisons end with one number per parser.

That number tells you who’s ahead on average. It doesn’t tell you where a parser breaks on documents like yours.

And nobody parses an average document. An insurance team parses scanned claim forms, a research team parses two-column papers full of equations, and a finance team parses dense tables.

So instead of one score, we checked every page with simple yes/no tests:

• Is this line present in the output?

• Is this cell under the right headings?

• Does A come before B?

Each test passes or fails, and every failure comes with a reason, like “text missing” or “marked as table, not heading”.

The scoring runs as open code. No LLM judges the output.

What we found

Here’s the overall scoreboard from our 150-page run:

→ Datalab: 88.2

→ DeepSeek-OCR: 77.5

→ PaddleOCR-VL: 73.4

If we had stopped at the overall column, the answer would look obvious. It isn’t.

DeepSeek-OCR was the best on tables at 87.2, ahead of Datalab at 78.9 and PaddleOCR-VL at 78.0.

So a team whose documents are mostly dense tables would pick a different parser than the average suggests.

On handwritten notes, the gap went the other way. One parser (Datalab) passed 5 of 6 tests, while the other two passed 0.

The filled 1040 tax form showed the starkest failure.

PaddleOCR-VL returned only the form’s title, so every test looking for a filled-in value failed with “text missing”.

On the same page, Datalab passed 8 of 8 tests and DeepSeek-OCR passed 6 of 8.

A parse that returns only a title is a different risk from one with a few wrong cells. A single average folds both into the same number.

No single parser won on every type of page. The right pick depends on your documents, not on who has the best average.

The benchmark behind it

The tool we used is OmniParse Bench, and it’s 100% open-source.

It has 16,288 tests across 2,937 pages in 92 languages, drawn from 2,343 source documents.

It builds on what olmOCR-bench showed, that pass/fail unit tests are a simple proxy for parsing quality that anyone can check. Datalab’s view is that olmOCR-bench has saturated, and what now separates the top systems there is mostly output conventions and overfitting rather than reading.

Every test belongs to one of three families: text, tables, and layout. A family’s score is the share of its tests that pass, and the headline is the mean of the three.

A page where the parser call fails counts as 0, so a parser can’t look better by quietly dropping the hard pages.

The tags are where it gets useful.

Each test is tagged by what makes its content hard, like handwriting, rotation, tiny text, degraded scans, forms, tables, math and multi-column layouts, along with its script and language.

Tags never change whether a test passes. They only select tests, so you can score a parser on handwriting alone, or on French tables alone.

Every score also comes with the number of tests and documents behind it, so a thin slice tells you it’s thin.

Each test also records how its answer was established, such as the publisher’s own markup, a form the authors filled in themselves, or agreement across a panel of independent models.

One thing to note is that Datalab built this benchmark, and Surya is Datalab’s own model.

That’s exactly why the open scorer matters. You don’t have to trust anyone’s leaderboard, since you can rerun every test yourself and read why each one passed or failed.

So you can plug in any OCR solution and see exactly where it holds up and where it breaks.

Before you commit to a parser, run it on the slices that match your documents and read the failures yourself.

GitHub repo: https://github.com/datalab-to/omni_parse_bench

(don’t forget to star 🌟)

The No. 1 Deep Researcher Beats Claude and ChatGPT Using a Counterintuitive Trick

The No. 1 deep researcher beats Claude and ChatGPT with a trick neither uses.

We studied the open-source architecture behind it.

A counterintuitive thing we found is that the orchestrator agent that runs the entire research strategy has no search access.

It can’t query the web or open URLs.

This looks wrong at first glance. Every other deep research system gives its coordinator far more capability.

For instance:

• OpenAI’s approach trains a single model for many consecutive tool calls. It searches, reads, reasons, and writes the report in a long sequential chain.

↳ The researchers behind the No. 1 system (Onyx; GitHub Repo) observed that this causes the model to spend cycles on low-value searches instead of maintaining a high-level research strategy.

• Anthropic and Google use an orchestrator-researcher pattern similar to Onyx’s system. The key difference is how aggressively Onyx constrains the orchestrator.

Most orchestrators have access to search and retrieval tools alongside dispatch capabilities. And the moment an orchestrator can search, it will.

So instead of decomposing a query into focused research threads, it starts answering the question itself.

It pulls a few results, skips proper task decomposition, and produces a surface-level report from whatever it found first.

Stripping search from the orchestrator forces it to write self-contained and coherent task briefs for each research agent.

The researchers also kept the architecture only two levels deep. When info passes through multiple agents, each one subtly distorts it through summarization/reinterpretation. Keeping it to two levels prevents this.

These two constraints sit inside a larger three-phase pipeline (the visual below maps this):

• Phase 1 decomposes the query into up to 6 research directions. No tool access prevents the model from prematurely answering.

• Phase 2 dispatches 3 isolated research agents. Each runs up to 8 sub-cycles of search, read, and think, to produce an intermediate report with citations. The agents can also search internal enterprise docs (Confluence, Slack, 100+ connectors) with document-level permissions enforced, unlike proprietary solutions.

• Phase 3 runs a deterministic step that renumbers and deduplicates to produce a report with a unified citation map.

This pattern has been ranked No. 1 on DeepResearch Bench. The whole implementation is available on GitHub, and you can try it yourself.

Here’s the Onyx Repo → (don’t forget to star ⭐ )

Onyx GitHub Repo

👉 Over to you: Which page type breaks your current OCR pipeline?

Good day!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

• A voice agent that gets the details right.

• Why temperature 0 isn’t deterministic in LLM inference.

A voice agent that gets the details right

Speech accuracy matters on both sides of an LLM. Speech-to-text must preserve what the caller says, while text-to-speech must preserve the response the LLM generates.

That becomes especially important with medication names, dosages, appointment times, and alphanumeric reference codes.

Deepgram's Flux TTS handles the outbound speech layer. It recorded a 3.4% word error rate in an evaluation containing drug names, tracking numbers, technical strings, and alphanumerics.

We tested it through a pet-hospital conversation about a dog’s prescription refill. The agent retained the medication, dosage, prescription number, and a collection time that changed during the call.

The details remained correct, the voice sounded natural, and the interaction felt like a call I would be comfortable having with an actual pet hospital.

Try Flux TTS here →

Thanks to Deepgram for partnering today.

Why temperature 0 isn’t deterministic in LLM inference

At temperature 0, the model does not sample. At every step, it picks the token with the highest logit, so the same prompt should give the same output every time.

In practice, it doesn’t.

Thinking Machines sent “Tell me about Richard Feynman” to Qwen3-235B 1,000 times at temperature 0 and got 80 different completions.

Picking the highest logit is perfectly deterministic. The variation comes from computing the logits themselves, which happens from scratch on every request through billions of additions.

If those numbers come out slightly different, argmax can pick a different token.

Let’s understand why the logits change.

1) The order of addition changes the result

GPUs store numbers with limited precision. fp16 keeps only 11 significant bits (10 stored mantissa bits plus 1 implicit bit).

Between 1024 and 2048, that leaves no room for anything after the decimal point. Representable values are spaced exactly 1 apart, so 1024.4 gets rounded back to 1024.

Now consider 1024 + 0.4 + 0.4 in two orders. The diagram below shows both.

Left to right, 1024 + 0.4 rounds back to 1024, and adding 0.4 again rounds back to 1024 once more. The answer is 1024.

If the small numbers go first, 0.4 + 0.4 gives 0.8, and 1024 + 0.8 rounds up to 1025. The answer is 1025.

You can verify this yourself:

import numpy as np

a, b = np.float16(1024), np.float16(0.4)
print((a + b) + b)   # 1024.0
print(a + (b + b))   # 1025.0

In other words, floating-point addition is not associative. An LLM performs billions of such additions in every forward pass, so if their order changes, the logits change slightly too.

2) What changes the order?

The popular answer is that parallel GPU threads finish in an unpredictable order.

That isn’t the main reason. The kernels in a typical forward pass compute each sum in a fixed order, and the same matrix multiplication on the same inputs gives bit-for-bit identical results every time you run it.

So the forward pass itself is run-to-run deterministic. Something else must be changing between requests.

3) The real reason is the other requests on the server

An inference server groups requests from many users into a batch and runs them together.

The kernels decide how to split their work based on the batch size. With a large batch, every GPU core is busy, so each core computes its sum in one go.

With a small batch, many cores would sit idle. So the kernel cuts each long sum into chunks, spreads them across cores, and combines the partial results at the end. This shows up in matmuls (split-K), in RMSNorm, and in attention during decoding (split-KV, as in FlashDecoding).

A different split means a different order of addition. The batch size, in turn, depends on how many other people are using the server at that moment.

Since you have no control over that, the output looks random from your side. Thinking Machines call the missing property batch invariance: a request’s numbers should not depend on what else is in the batch.

4) How a tiny difference changes the entire answer

A difference in the fourth decimal place sounds harmless, and most of the time it is.

Consider two runs of the same prompt. The numbers below are illustrative, but the mechanism is real.

→ In a batch of 8, the logits come out as “Queens” = 12.3417 and “New” = 12.3415, so argmax picks “Queens”.

→ In a batch of 32, the rounding works out slightly differently, giving “Queens” = 12.3414 and “New” = 12.3416, so argmax picks “New”.

The output only flips at such near ties. Most of the time, the top token wins by a comfortable margin and rounding noise can’t touch it.

The problem is what happens after a flip. Generation is autoregressive, so every token after that point is computed from a different prefix.

That’s exactly what Thinking Machines saw. In the Feynman test, all 1,000 runs matched for the first 102 tokens. At token 103, 992 runs wrote “Queens, New York” while 8 wrote “New York City”. From there, the completions diverged into 80 unique outputs.

5) The fix and its cost

The fix follows directly from the cause. Every kernel should split its work the same way regardless of batch size, even if that leaves some cores idle on small batches.

For attention, Thinking Machines fix the split size rather than the number of splits, so a given token always lands in the same chunk. They also update the KV cache before the attention kernel runs, so cached and freshly computed tokens are reduced in the same order.

With these batch-invariant kernels, all 1,000 Feynman runs were identical.

The cost is speed. On Qwen3-8B with 1,000 requests, default vLLM took 26 seconds. The first deterministic version took 55 seconds, and an improved attention kernel brought it down to 42 seconds.

Both major open-source engines now ship this:

• vLLM: the VLLM_BATCH_INVARIANT=1 environment variable (beta).

• SGLang: the --enable-deterministic-inference flag, with a reported average slowdown of 34.35%.

If you just chat with a model, none of this matters much. It matters a lot for evals that need to reproduce, for replaying a bad production output while debugging, and for RL, where the sampler and the trainer must compute the same probabilities for training to stay truly on-policy.

So temperature 0 removes the sampling randomness, but it doesn’t remove the batch.

The visual below depicts the entire process.

If you’re shipping LLMs to production, issues like this one rarely show up in a demo. They show up later as evals that won’t reproduce, latency that spikes under load, or a serving bill that grows faster than traffic.

That’s the gap the LLMOps course fills. It walks you through the full lifecycle, from how LLMs work under the hood to prompting, context, evaluation, fine-tuning, inference optimization, and serving, so you can reason about these problems before they hit your users.

If you want to learn the full LLMOps lifecycle, we have covered every bit of it in the LLMOps course, starting from fundamentals to production:

• Read Part 1 on fundamentals of LLMOps here →

• Read Part 2 on understanding the core building blocks of LLMs →

• Read Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixture-of-experts, and the fundamentals of pretraining and fine-tuning →

• Read Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applications →

• Read Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and different prompting techniques →

• Read Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more →

• Read Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techniques for building high-signal inputs →

• Read Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic context injection, and common failure modes in agentic applications →

• Read Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong understanding of the fundamental concepts →

• Read Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for evaluation of LLM apps →

• Read Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming →

• Read Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO →

• Read Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model parallelism →

• Read Part 14 on the fundamentals of LLM serving, including API-based access, inference with vLLM, and practical decisions →

👉 Over to you: Have you ever had an eval score move between two identical runs?

Good day!

Finally, you can connect Grok Bot to different agent harnesses and run tasks through their native runtimes.

Devs just open-sourced HarnessRouter, the infrastructure layer that lets Grok Bot run tasks across:

• Codex

• Hermes

• Claude Code

• DeepSeek Harness

• Pi, and more (including System 1 harnesses)

GitHub repo: github.com/HarnessRouter/harnessrouter

(don’t forget to star it ⭐)

With HarnessRouter, each external specialist gets to use its own harness, model, tools, skills, and agent loop while Grok coordinates the project from one conversation.

We tested this by asking Grok Bot to turn a support-ticket CSV into an interactive dashboard.

Grok Bot assigned the analysis to DeepSeek Harness with Claude Sonnet 5.5, passed the output to Codex with GPT-5.5, and checked the finished dashboard in its browser.

This differs from simply creating more Bots.

These workers are not additional Grok Bots, and Grok Bot itself does not switch harnesses.

Instead, it remains the coordinator while the delegated tasks run inside separate HarnessRouter sessions with the harness of your choice.

So for another task, you can use Claude Code, Hermes, Pi, etc., without requiring a separate integration for task execution, sessions, streaming, and generated files.

HarnessRouter standardizes all those operations through the Unified Harness Protocol (UHP).

UHP handles sessions, streaming, files, cancellation, and failures through the same interface, regardless of which harness runs the task.

You can find the repo here →

How CUDA graph replay cuts inference overhead

A GPU can finish its work faster than the CPU can tell it what to do next.

This happens because a model forward pass is not one giant GPU operation.

Instead, it expands into many smaller operations for attention, matrix multiplication, normalization, sampling, and memory movement.

The CPU normally submits these operations one at a time.

It prepares a kernel launch, sends it to the GPU, and then prepares the next one. Every launch passes through the framework, CUDA runtime, and driver.

That overhead is small for one operation.

But an LLM executes many short kernels during every decoding step, then repeats a similar sequence for the next generated token.

If the GPU finishes a kernel before the CPU submits the next one, it sits idle.

A profiler shows this as small gaps between GPU operations.

CUDA Graphs were introduced to change how the sequence is submitted.

Basically, a CUDA Graph is a saved execution plan for GPU work.

It stores which operations need to run, the arguments they use, and the dependencies between them.

For example, a graph might record this sequence:

Attention → matrix multiplication → normalization → sampling

Instead of rebuilding and submitting that sequence one operation at a time, the application prepares the complete plan once.

CUDA Graph replay is how that saved plan gets used repeatedly.

There are three parts to the process.

First, the application runs a normal warm-up pass.

Warm-up performs any one-time initialization, selects kernels, allocates buffers, and establishes the tensor shapes and memory addresses the workload will use.

Next, the application captures one execution.

During capture, CUDA watches the operations sent to a stream and records them in a graph. It stores the launch sequence and dependencies, not the output produced by that run.

Finally, the application replays the graph.

Before each replay, new input data is copied into the same input buffers used during capture.

The CPU then makes one graph-launch call. The GPU reruns the recorded operations on that new data and writes the new result into the graph’s output buffers.

So replay does not mean returning an earlier result.

It means rerunning the same GPU workflow without rebuilding and submitting every kernel separately.

This removes much of the repeated work around the kernels.

Framework dispatch, argument setup, driver calls, and individual kernel submissions move out of the steady-state path.

The model still performs almost the same arithmetic. Its weights remain unchanged, and every replay runs the kernels again.

The speedup comes from reducing the time spent starting that work.

This fits LLM decoding well because every decoding step follows a largely repeated path.

One token is generated, the inputs and cache state are updated, and the model runs again for the next token.

When the execution shape stays compatible, the inference engine can reuse the same captured graph across many decoding steps.

The diagram below shows the process.

Warm-up prepares stable buffers. Capture saves the launch plan. Replay runs that plan on new input data through one graph-launch call.

The main tradeoff is flexibility.

A captured graph expects compatible kernels, arguments, tensor shapes, control flow, and memory addresses on every replay.

Production inference servers receive different batch sizes and execution cases, so one graph cannot cover everything.

They often capture several graphs for common batch sizes. A batch can use a matching graph, be padded to a captured size, or fall back to normal execution.

More captured graphs improve coverage, but require additional memory and startup time.

CUDA Graph replay works best when kernels are short, the execution path repeats, and launch overhead creates visible gaps in the GPU timeline.

In this case, the model is not doing less work but rather the system is submitting that work more efficiently.

https://www.dailydoseofds.com/llmops-crash-course-part-13/

In Part 13 of the LLMOps course, we covered the entire mechanics of LLM inference, from prefill and decode phases to KV caching, batching, and optimization techniques that improve latency and throughput.

Read it here →

RAG vs. Graph RAG vs. Agentic RAG

Standard RAG embeds documents into vectors and retrieves the most similar chunks via similarity search. For direct factual lookups, this works well.

But it breaks down when a query needs to connect facts spread across multiple documents. Similarity search retrieves individual chunks, not the relationships between them.

Graph RAG adds a knowledge graph layer on top.

• During indexing, an LLM extracts entities and relationships from the documents.

• During retrieval, the system traverses these connections instead of relying on embedding similarity alone.

This is what enables multi-hop queries.

Say a vector DB stores three facts about internal services:

↳ “The checkout service uses payments API.”

↳ “The payments API runs on cluster-3.”

↳ “Cluster-3 is scheduled for maintenance on Friday.”

Someone asks: “Will the checkout service be affected by Friday’s maintenance?”

Vector search can likely retrieve facts 1 and 3 because the query mentions “checkout service” and “Friday maintenance.”

But it will miss fact 2, which connects the payments API to cluster-3.

That middle fact sits too far from the query in embedding space. It mentions neither “checkout” nor “maintenance,” so it never makes it into the retrieved context.

A knowledge graph connects these as linked entities, and graph traversal finds the full path in one query.

Agentic RAG takes a different approach entirely.

Instead of a fixed retrieval pipeline, an LLM agent decides at query time which tools to invoke, which sources to query, and in what order.

Check the visual below again to understand the three architectures thoroughly.

One thing to note here is that these three aren’t levels of sophistication that you need to graduate through.

Instead, they solve different query types.

↳ Single-hop factual lookups → standard RAG

↳ Multi-hop relationship queries → Graph RAG

↳ Dynamic multi-source tasks with tool use → Agentic RAG

If you want to dive further, our full RAG crash course discusses RAG from basics to beyond:

• ​​​​RAG fundamentals​​​​

• ​​​​RAG evaluation​​​​

• ​​​​RAG optimization​​​​

• ​​​​Multimodal RAG​​​​

• ​​​​Graph RAG​​​​

• ​​​​Multivector retrieval using ColBERT​​​​

• ​​​​RAG over complex real-world docs ft. ColPalida

• 16 techniques to build RAG systems for production

• Why RAG latency is a prefill problem, not a retrieval problem​​​​

• Preloading knowledge instead of retrieving it

👉 Over to you: Which RAG architecture are you running in production?

Thanks for reading!

The real moat in 2027 won’t be anything to do with the agent.

Instead, it’ll be about capturing full-stack trajectories.

Imagine two teams ship the same agent, on the same model, on the same day.

Your competitor ships it and moves on.

But you ship agents that learn from every session: where users stepped in, what they did by hand, what they keep asking for. Every week it picks up a few new skills, evals, and datasets.

Six months later, you aren’t really competing on the same product anymore.

Because the advantage isn’t in the model but rather it’s in your users’ data.

CLiF is a new learning platform built into CopilotKit that sees every trajectory:

• The trace: every step the agent took

• The product: everything the users did

The AG-UI protocol already sits between the agent and the UI, so it sees both sides and then turns real sessions into exportable fullstack trajectories, evals, skills and fine-tuned models.

You can self-host it and access it here: http://go.copilotkit.ai/get-clif

Our suggestion if you’re building user-facing agents: start collecting this data now.

CLiF already implements the entire continual learning infrastructure you need to achieve that.

We are working on a hands-on demo of this and will share it soon. Stay tuned!

6 KV cache quantization approaches, explained visually:

Quantizing model weights doesn’t automatically shrink the KV cache.

Weights store the model’s learned parameters. The KV cache stores keys and values computed while processing a request. Reducing the precision of one doesn’t automatically change the other.

The cache grows as requests retain more tokens, and concurrent requests need room for their own entries. KV quantization reduces the bytes used to store those entries.

For instance, moving from 16-bit to 8-bit halves the raw KV data. The total saving is smaller once scales, other metadata, and any high-precision buffers are included.

The six approaches below describe different choices within KV quantization. Several can work together, such as 4-bit integers, small quantization groups, and a high-precision buffer for recent tokens.

1. FP8 cache

A cache using FP16 or BF16 stores each number in 16 bits. FP8 stores it in 8 bits, so the same number of entries takes half the raw space.

A scaling factor adjusts the numbers to fit the FP8 range. The implementation retains the scale so the attention kernel can interpret the stored numbers correctly.

For example, 1 GiB of 16-bit KV data becomes 512 MiB of FP8 data, before scales and other overhead. The model weights occupy the same space unless they are quantized separately.

The serving engine and GPU need a compatible attention kernel. FP8 storage alone doesn’t establish how fast that kernel will run.

2. INT8 / INT4 cache

These methods represent cached numbers with 8-bit or 4-bit integers. A scale, and sometimes a zero-point, tells the kernel how those integers map back to approximate values.

INT8 has 256 possible integer levels. INT4 has 16, so it uses less storage but offers fewer choices when rounding a number.

For example, two nearby values might round to the same level in INT4. After conversion back, attention can no longer distinguish them exactly.

The kernel unpacks and dequantizes the stored entries for attention, often within the attention computation itself. This work can offset some of the benefit of reading fewer bytes.

3. Quantization granularity

Granularity determines how many numbers share a scale. One scale can cover a whole tensor, or separate scales can cover channels, tokens, or smaller groups.

Suppose most numbers in a group lie between 0 and 1, but one is 20. A shared scale that includes 20 can leave fewer rounding levels available for distinguishing the smaller numbers.

Smaller groups limit how many other numbers that outlier affects. But every group needs its own scale, which also occupies memory.

Two implementations can both use 4-bit values and still have different total memory use and accuracy because they share scales differently.

4. KIVI-style quantization

KIVI uses different grouping rules for keys and values. Its analysis found that unusually large key values tend to appear in particular channels, while values have a different pattern.

A channel is one coordinate of a key or value vector. KIVI groups keys from the same channel across a block of tokens, giving that channel its own scale within the block.

For values, it groups numbers within each token instead. A token’s value range therefore doesn’t set the scale for every other token.

The original method targets 2-bit KV storage without fine-tuning the model. Its grouping rules help control rounding error, but accuracy still needs checking on the model and workload being served.

5. Mixed precision

One mixed-precision approach keeps recent keys and values in a high-precision buffer while storing older entries in a quantized cache.

New entries enter the buffer first. As it fills, the implementation quantizes a block of entries and moves it into the compressed portion.

During decoding, attention reads both portions. Recent entries avoid quantization error while they remain in the buffer.

A larger buffer keeps more entries at higher precision, but reduces memory savings. This approach can also help collect enough tokens to quantize keys in blocks, as KIVI does.

6. Non-uniform / outlier-aware quantization

Uniform quantization rounds numbers to evenly spaced levels. Non-uniform quantization changes that spacing, assigning levels to better represent the value distribution or preserve sensitive values.

For example, if many numbers cluster near zero, more levels can be placed there rather than spread evenly across a wide range.

Outlier handling addresses a related problem. KVQuant separates unusually large values into sparse storage, keeping their positions so the kernel can account for them during attention.

The remaining values can then use a narrower range. The separate outliers and their positions add overhead, so the memory cost includes more than the low-bit data alone.

Check the saving against the cost

A smaller cache can fit longer contexts or more requests on the same GPU. Faster decoding depends on whether reduced memory traffic outweighs the extra quantization work.

Compare against an unquantized cache using the same model, prompts, output lengths, and concurrency. Check total memory use, decoding latency, and answer quality on the long-context tasks that matter for the application.

We have covered every bit of it in the LLMOps course, starting from fundamentals to production:

• ​​​​Read Part 1 on fundamentals of LLMOps here →​​​​

• ​​​​Read Part 2 on understanding the core building blocks of LLMs →​​​​

• ​​​​Read Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixture-of-experts, and the fundamentals of pretraining and fine-tuning →​​​​

• ​​​​Read Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applications →​​​​

• ​​​​Read Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and different prompting techniques →​​​​

• ​​​​Read Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more →​​​​

• ​​​​Read Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techniques for building high-signal inputs →​​​​

• ​​​​Read Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic context injection, and common failure modes in agentic applications →​​​​

• ​​​​Read Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong understanding of the fundamental concepts →​​​​

• ​​​​Read Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for evaluation of LLM apps →​​​​

• ​​​​Read Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming →​​​​

• ​​​​Read Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO →​​​​

• ​​​​Read Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model parallelism →​​​​

• ​​​​Read Part 14 on the fundamentals of LLM serving, including API-based access, inference with vLLM, and practical decisions.

And we also wrote ​an open-access KV Cache Engineering article​ covering quantization and other ways to reduce cache memory, with code examples and their trade-offs.

Read it here →

​The evolution of the retrieval layer​

Google and Anthropic agree on one thing about RAG that retrieval is no longer a pipeline you build once.

Instead, it is a tool that something else calls.

• Anthropic’s MCP exposes retrieval as a tool an agent can call.

• Google now ships retrieval the same way, with its RAG Engine sitting under the Gemini agent platform alongside MCP servers.

That shift sounds small, but it changes the entire architecture underneath.

Today, let’s break down why classical RAG isn’t ideal in production, what a standing retrieval layer looks like, and how agents consume it.

Classical/Naive RAG

Naive RAG runs as a one-time pipeline inside a single app. A knowledge base gets chunked, embedded, and written to a vector DB.

At query time, the user’s query hits that vector DB for a similarity search, and the retrieved chunks go to the LLM.

This works when the answer lives in one corpus that rarely changes. It breaks on two fronts that show up immediately at scale.

The first is staleness. The embeddings reflect the source at the moment of indexing. When a doc changes in Notion or a row updates in Postgres, the vectors still point at the old content until someone reruns the pipeline.

The second is duplication. The retrieval logic is welded to one application. A second app, or an agent, that needs the same data has to rebuild the same connectors, chunking, and embedding from scratch.

Retrieval as a standing layer

The fix is to stop treating retrieval as a step inside the app and start treating it as a standing layer underneath it. A retrieval layer separates ingestion from query, and runs ingestion continuously instead of once.

On the ingest side, connectors pull from many sources, handle auth and extraction, chunk into entities, embed, and write to a vector store with a metadata store alongside it for source links and versions.

A sync process watches for changes and updates only what changed, using content hashing so unchanged data is not re-embedded.

On the query side, a request gets expanded, run through hybrid search that combines neural and keyword retrieval, then reranked before results come back with their source attribution.

The key property is that this layer is owned by no single app. It is exposed over an API, so a RAG chatbot and an autonomous agent can both query the same index.

How agents alter this scenario

An agent does not run retrieval once at the start.

It reasons, decides it needs context, calls search, reads the result, and decides whether to search again with a refined query. Retrieval becomes a tool inside a loop, not a fixed first step.

And the tool the agent calls is the retrieval layer.

The agent’s search function and the RAG app’s retrieval are the same query path into the same synced store.

They were never separate systems. Instead, one was just drawn inside the app, and the other was pulled out and shared.

👉 Over to you: Is moving retrieval out of the app and into a shared layer the right call for your stack?

Good day!

We served an embedder, reranker, entity extractor, and generator concurrently on just one GPU through a single inference server.

The server processed 2,700 requests with 32 requests in flight and zero failures.

vLLM supports several inference tasks, but each server instance still runs one model and one task. This pipeline would require separate instances, task-specific endpoints, and another layer to route requests between them.

SIE handled the entire pipeline through one server. It loaded models when requested, kept active models resident, and evicted them when GPU memory was needed.

The same system can serve embedding, reranking, extraction, OCR, and generation workloads.

GitHub repo: github.com/superlinked/sie

(don’t forget to star it ⭐)

We recently used the same architecture with five specialized models on one GPU and explained how dynamic loading, shared GPU memory, and compute-aware batching make it work.

Read the full walkthrough here →

Build a Jev-powered Agentic RAG

Agentic RAG systems have two problems that prompt tuning alone cannot solve.

The first is knowing when retrieved evidence is insufficient. The second is acting on that decision.

Decision models, like Jev, address the first problem.

Instead of generating text, they return structured judgments, such as relevance scores or probabilities that a passage contains the answer. The application uses these judgments to generate a response or escalate the question to a human.

Multimodal RAG adds another challenge here. Evidence may lie within images as well as text, so the system must retrieve and evaluate both.

Building this system also means connecting multiple services and coordinating what happens when the evidence is insufficient.

InsForge (open-source) provides the entire backend for that work.

It brings Postgres with pgvector, authentication, storage, edge functions, real-time updates, email, and a model gateway into one place.

So today, we’ll use it to connect model decisions to application logic and human review.

We’ll build a support desk with a customer-facing chat and a dashboard for support representatives.

The system will retrieve relevant documents, check whether they contain enough evidence to answer, and either generate a response or send the question for human review.

We’ll use four components:

1. InsForge for the backend, database, auth, and human-review workflow

2. EmbeddingGemma 2 multimodal embeddings for retrieving text and images

3. Liquid AI’s d1 family of open-weight decision models to score passages and assess answerability

4. Lightning AI Studio to host the embedding & decision models.

The goal is an agentic RAG system that can assess evidence, decide what to do, and carry out that decision.

Step 1: Set up Claude Code with InsForge

InsForge (open-source) provides the database, auth, storage, compute, hosting, and model gateway. We’ll use Claude Code with InsForge’s skills and CLI to set up the backend.

Install the skills:

npx skills add insforge/insforge-skills

The package installs four skills.

1. insforge for SDK patterns

2. insforge-cli for infrastructure commands

3. insforge-debug for diagnosing failures

4. insforge-integrations for third-party authentication providers

Next, authenticate the CLI and link the project.

npx -y @insforge/cli login --user-api-key <your-api-key>
npx -y @insforge/cli link --project-id <project-id>

Linking also creates an AGENTS.md file with instructions for Claude Code to consult the InsForge skills instead of guessing API patterns. From here, Claude Code can configure the backend and build the app.

Step 2: Serve the models on Lightning AI

InsForge handles the backend, but the embedding and decision models need their own compute. So we serve them through a FastAPI server in Lightning AI Studio, and the backend calls it through a public URL and an API key.

The server exposes endpoints for embedding queries, reranking candidates, and indexing documents.

EmbeddingGemma 2 for multimodal search

EmbeddingGemma 2 maps text and images into the same 768-dimensional space, which enables cross-modal search. So, a text question can retrieve a PDF page whose answer sits in a diagram or screenshot.

Decision model for reranking and answerability

Decision models handle structured judgments rather than generating free-form text.

TypeSafe AI’s Jev is one example where it takes a state plus one or more typed questions and returns structured answers:

• Choice selects one option from a defined set.

• Score places the state on an ordered scale.

• Noul returns the probability that a yes-or-no statement is true.

We use Liquid AI’s open-weight decision model d1-3B, which processes text and images. It first reranks retrieved pages by relevance, and then checks whether they contain enough evidence before anything reaches the generation model.

Both models fit on a single NVIDIA L4 and use about 7.2 GB of GPU memory.

Step 3: Build the backend

The support desk needs more than a vector database. It needs data isolation, private document storage, and a way to index both text and images.

First, ask Claude Code to configure the InsForge backend.

Set up the InsForge backend for a multi-company support desk: tables for
documents, chunks (768-dim vectors), conversations, tickets and a decision
log, all scoped to a company with RLS. Add private storage buckets and store
the model server URL and API key as secrets.

Index each document as text and images

Next, we’ll build the document-indexing pipeline.

When an admin uploads a PDF or DOCX, store it in InsForge storage and
index it on the model server: each page's text chunks and a rendered image
of the whole page, both tagged with the page number. Show progress live
in the dashboard.

The pipeline splits extracted text into chunks of about 220 words and renders each page as a PNG. It embeds both and stores them with the document and page numbers.

A text query can therefore retrieve either a passage or a full page containing a diagram. We index full pages instead of individual figures because diagrams are often vector drawings, and pulling them out can lose captions and context.

Track indexing progress

An InsForge edge function checks the uploader’s permissions, creates an indexing job, and hands it off to background processing. As each page finishes, the backend sends a real-time update to the dashboard.

A 20-page help PDF indexes in about 27 seconds.

Step 4: Add a decision gate to RAG

Once documents are indexed, the next step is deciding whether the retrieved evidence is enough to answer.

The query pipeline brings retrieval and the decision model together in an InsForge edge function.

Create an edge function for customer questions. Embed the question and any
screenshot, search text chunks and page images separately, and send the
candidates to /rerank. If nothing clears the threshold, retry once with a
wider search, then open a ticket instead of calling the LLM. Log every
decision.

When a customer asks a question, the InsForge edge Function embeds it, along with any attached screenshot. It then runs two searches over the company’s documents, one across text chunks and one across page images.

Results from the same page are merged and sent to the model server’s /rerank endpoint.

Liquid AI’s d1-3B scores each candidate for relevance and answerability. The text and image thresholds are separate and stored in a database table, so you can tune them without code changes.

If no candidate passes, the pipeline retries with a wider search. If the evidence still falls short, it opens a support ticket instead of calling the generation model.

Step 5: Connect answers to human review

Next, we generate answers from the retrieved evidence:

When candidates pass the gate, send the surviving text and page images to a
vision model through the InsForge Model Gateway, and return an answer that
cites page numbers.

The gateway is configured to use anthropic/claude-haiku-5.5. The model answers only from the supplied evidence and cites the sources. The model ID is stored as a secret, making it easy to switch models by changing one value.

For example, the error code E-4012 in this screenshot appears on page 17. The answer returns to the customer with citations and page thumbnails.

When the system cannot answer

Questions that fail the gate become tickets.

A database trigger publishes each new ticket to the company’s real-time channel, so it appears on the dashboard.

Build a dashboard where reps see new tickets in real time, with the question
and the rejected candidates. When a rep replies, email the customer, and let
the rep save the reply as knowledge. Deploy the frontend to InsForge.

The representative sees the customer’s question alongside the rejected pages and their scores. They can then write a reply and send it to the customer by email, which is useful when the customer has already left the chat.

Save to knowledge base embeds the question and the rep’s reply as a new chunk. The next time a similar question comes in, retrieval can find that answer.

Finally, deploy the frontend through the InsForge CLI.

npx -y @insforge/cli deployments deploy web

That completes the full build.

The system answers when the evidence is sufficient, escalates when it isn’t, and human replies improve retrieval for next time.

Final evaluation

We ran three RAG pipelines on the same candidate set, using 75 labelled questions across four cases:

• Answerable from text

• Answerable only from screenshots

• Questions that include a screenshot

• Should be escalated

The following table summarizes the results:

The decision gate had the highest accuracy, and it was the only pipeline that held full performance on image-only questions.

The BGE reranker avoided wrong answers but struggled with image-based questions, escalating most of them.

The takeaway is that a decision model can do more than rank passages. It can also judge whether the evidence supports an answer, even when that evidence is visual.

Retrieval, answerability checks, and response generation are only part of the app.

The core challenge is handling questions the docs can’t answer, routing them to a human, notifying the support team, and keeping the workflow connected.

We built the app in a single Claude Code session using prompts, with InsForge’s skills and CLI handling the backend setup.

Starting from an empty project, it became a support desk that answers from the docs when there’s enough evidence and hands unanswered questions to a person.

You can find InsForge on GitHub here →

Here are the InsForge skills →

👉 Over to you: what does your RAG system do when the docs don’t have the answer?

Good day!

Onyx is a self-hostable AI chat platform that works with any LLM, like Claude, GPT, Gemini, Llama, or any open-weight model you want.

Onyx GitHub Repo

Here’s what it ships with:

• Agents that chain multiple tools in sequence

• RAG with full indexing across 40+ connectors (Slack, Drive, Confluence, Jira, GitHub, email, call transcripts)

• Deep research ranked #1 on DeepResearchBench, above every proprietary alternative

• MCP support for connecting to external systems

• Code interpreter for data analysis and file generation

• Self-host on your own infrastructure via Docker in a few mins

Unlike Claude’s MCP-based connectors that query your tools at runtime, Onyx actually indexes all your data. That means faster, more reliable search across everything your team has ever written.

The entire code is open-source under the MIT license, so you can see the full implementation on GitHub and try it yourself.

Find the GitHub repo here → (don’t forget to star it ⭐️)

Give Claude Code, Codex, and Cursor a self-improving shared memory

Coding agents work in silos.

Your corrections in Cursor don’t carry over to Claude Code, and skills built in Codex don’t automatically transfer to other harnesses.

So switching tools often means starting from scratch.

What if every coding agent could learn from everything you’ve already done?

We invited Justin, co-founder of Asymptote Labs, to show us how to make this possible with Beacon, a 100% open-source memory layer for coding agents.

Beacon GitHub Repo →

We set it up live, explored real coding sessions, and watched Beacon turn past work into reusable knowledge.

Here’s what we learned:

1. One memory layer for 20+ coding agents.

Beacon captures activity across Claude Code, Cursor, Codex, and 20+ harnesses using OpenTelemetry, hooks, and local session files. Everything gets normalized into a single format, regardless of which agent you’re using.

2. Your entire coding history becomes searchable.

Every prompt, tool call, command, file change, and response is captured in a structured trace. Instead of losing valuable context when a session ends, you build a searchable history of everything your agents have done.

3. Switch agents without starting over.

If you hit your usage limit in Cursor, you can simply open Claude Code and ask it to continue your latest Cursor session. Beacon retrieves the relevant context in milliseconds so your next agent can pick up where the last one left off.

4. Your agents learn from your past work.

This is where things get interesting.

Beacon identifies repeated workflows and turns them into reusable skills. In our demo, it discovered a recurring UI development workflow and generated a skill containing the exact dev server commands, screenshot steps, and deployment instructions.

You teach one agent something once. Every other agent can benefit.

5. Stop accumulating skills nobody uses.

Have 100 skills but only use five? Beacon analyzes actual usage across thousands of sessions and identifies skills worth keeping, combining, or removing.

Your agent’s memory gets more useful, not just bigger.

6. Security rules that follow you across every harness.

Beacon includes hundreds of detection rules that work across agents. It can identify risky behavior, including sequences where an agent reads sensitive credentials and then makes an outbound network request.

7. It even caught a real credential leak.

While reviewing Justin’s actual Cursor history, Beacon surfaced an incident where part of a production database password had been sent to a model provider.

That’s the power of having visibility into what your agents are actually doing.

8. One prompt to get started. Completely open source.

Paste a setup prompt into your coding agent and you’re off. Keep everything local on your machine, or use Beacon Cloud for free.

Beacon on GitHub: http://github.com/Asymptote-Labs/agent-beacon

(don’t forget to star 🌟)

Enjoy the episode.

Good day, and thanks to Asymptote Labs for partnering today!

By the time approval arrives (that could take a few hours), the Python process would be gone, and the container may be gone with it too.

Re-running the request looks harmless, but it creates a new o…

Read more

• ​This inference engine runs LLMs 4x faster than llama.cpp.

• ​Agents 2.0 is here!

• Continuous batching in LLMs, explained visually.

This inference engine runs LLMs 4x faster than llama.cpp:​

We ran Qwen3.5 9B on a base M5 MacBook Pro using three local inference engines, one at a time.

• llama.cpp reached 22.0 tokens/s

• MLX reached 25.1 tokens/s

• Uzu reached 92.1 tokens/s

The video shows all three runs side by side.

The first two results do not mean llama.cpp or MLX are poorly optimized. Both are approaching a physical limit of ordinary local decoding.

Generating each new token normally requires another pass through the model. That pass streams roughly 5.2 GB of weights through memory.

A base M5 provides about 153 GB/s of memory bandwidth.

At one token per pass, the theoretical ceiling is roughly 153 ÷ 5.2, or 29 tokens/s, however fast the GPU computes.

Uzu gets past that by optimizing speculative decoding for Apple Silicon.

Speculative decoding uses a smaller drafting system to propose several future tokens. The full 9B model then checks those proposals together instead of generating every token through a separate pass.

Uzu pushes this further in two ways.

→ It drafts up to 16 positions and uses Weaver to turn independent guesses into coherent sequences.

→ It verifies several possible sequences as a tree, then commits state only for the path the full model accepts.

The 9B model remains in control of the final output. Uzu simply gives it more useful work to approve in each pass.

In our run, every pass of the 9B model produced 7.5 output tokens on average.

Ordinary decoding produces exactly 1.

That is how Uzu reached 92.1 tokens/s on hardware whose one-token-per-pass ceiling is roughly 29.

The rest of the engine is designed around this path too. Uzu ships its own quantized checkpoints, and on M5 chips its Metal kernels run verification through the GPU’s accelerated int8 matrix path.

It is an open-source inference engine with bindings for Rust, Swift, Python, and TypeScript.

The repository is here: ​https://github.com/trymirai/uzu​

(don’t forget to star 🌟)

Agents 2.0 is here!

Anthropic. OpenAI. xAI.

Are all racing to build an AI coworker.

These are agents that can work alongside humans, talk to other agents, and coordinate tasks in a shared workspace.

But the catch is that each product ties the model, runtime, memory, and orchestration to one company’s stack.

Most teams won’t use one provider for every job. They may use Claude Code, Codex, OpenCode running locally, and custom agents for internal systems. Making them collaborate with each other and your team usually means building the glue yourself.

That is vendor lock-in.

In the video at the top, we have shown how to bring humans and agents from different providers into one shared workspace using an open-source coordination layer you can host yourself.

Humans can talk directly to agents. Agents can talk to each other. And people can join the conversation, add context, or give further instructions as the work unfolds.

The project is called Switch, and it brings this collaboration into Slack, Teams, Discord, Telegram, or Mattermost.

You give each agent a role and can work through a Manager. The Manager creates rooms, brings in the relevant agents, carries context between them, and returns the result.

In this video, we gave the Manager two incidents.

→ The inference API started returning 500s after a morning deploy

→ A recommendation-model training job had been frozen for six hours

From one message, it created two linked rooms, brought in a Software Engineer, Architect, and QA Lead, and ran both investigations in parallel.

Then the engineer investigating the training failure checked the linked inference room and spotted the same config push behind both failures.

The Manager returned one consolidated RCA to the original channel.

Throughout the run, we could follow the conversations, give additional instructions, and inspect the underlying agent sessions in Switch Console.

That is what we mean by Agents 2.0. Humans and agents working together, sharing context, and contributing as the work unfolds.

We build everything locally with Docker and Mattermost and follow the run through to the final handoff.

Models will keep changing, so the coordination layer should be able to run separately from any one provider.

GitHub: http://github.com/sandbox-quantum/switch

(don’t forget to star 🌟)

Thanks to Switch for partnering today!

Continuous batching in LLMs, explained visually:

In traditional ML inference, a batch is a matrix.

Every input is padded to the same length, one forward pass runs, and every row finishes at the same moment.

LLM decoding does not work that way.

One forward pass produces one token per sequence, so a request needs as many passes as it has output tokens, and nobody knows that count until the model emits a stop token.

Under static batching, membership is fixed when the batch starts.

A request that finishes in 30 tokens holds its slot until the slowest request in the same batch finishes at 400.

The GPU keeps paying the full weight read for a batch that is mostly empty.

Loading model weights out of HBM costs the same whether four slots are producing tokens or one.

Continuous batching moves the decision boundary.

Instead of scheduling once per batch, the scheduler runs a single forward pass, gets control back, and decides again.

A finished request leaves at the next iteration boundary, and a queued request takes its slot right there. No slot stays reserved for work that is already done.

Anyscale benchmarked both OPT-13B on a single A100.

With uniform generation lengths, the two policies came out about level (as expected), and as output length variance rose, static batching fell to around 81 tokens per second while vLLM reached 23x the throughput of naive Hugging Face serving.

Variance drives the entire gap.

Production traffic mixes 30-token replies with 400-token ones, which is exactly the condition static batching handles worst.

None of this alters the model.

vLLM, SGLang, TGI, and TensorRT-LLM all run it by default, and NVIDIA implements the same mechanism under the name in-flight batching.

The animation below depicts the continuous batching process:

To dive deeper into continuous batching specifically, we wrote a full breakdown of the scheduler underneath it.

It covers what happens between two forward passes, how tokens get handed out against a fixed budget, why the scheduler needs no separate path for prefill and decode, and what preemption costs you when the KV cache fills up mid-generation.

Read it here →

Good day!

Agents burn a lot of tokens on decisions that are really just picking one option from a short list, like matching a client request to the right GitHub issue.

So we built a weekly client follow-up agent that does exactly that. We tested it on 40 requests, and it was as accurate as letting Opus make every match, at 150x lower decision cost.

Here’s the tech stack we used:

• TrueForge as the agent harness

• Jev for the matching decisions

• Daytona as the sandbox for code execution

Jev is a decision model from TypeSafe AI. It takes the options from your app and returns a structured choice with a calibrated confidence score, without generating any text.

Low-confidence matches go back to Opus for review, and in our test, that review caught 5 wrong matches that would otherwise have gone through.

TrueForge is TrueFoundry’s harness, and it keeps context small. It loads tool definitions only when needed, writes large tool outputs to files, and lets the agent process data in scripts instead of the conversation.

On DevRev’s Enterprise Bench, it solved about as many tasks as Claude Managed Agents on roughly a third of the tokens. It also keeps credentials outside the sandbox and holds every write for approval.

If you want to try this out, TrueForge is fully open source (MIT-licensed) and runs on your own machine with a single command.

TrueForge GitHub: https://github.com/truefoundry/trueforge

(don’t forget to star 🌟)

Lastly, to help you reproduce my work and understand all the integration details, we wrote a detailed article.

You can read it here →

How to turn an LLM into a decision model?

Say we have a support ticket that says, “I was charged twice for invoice 4411. Please refund.”

Our application needs to make two decisions. Which team should handle the ticket, and whether to issue a refund.

A language model would then generate text token by token, while a decision model would score the available choices directly.

Here’s how to turn an LLM into one:

Define the questions and choices:

Give the model the ticket alongside the decisions to make. For the team, the choices are billing, technical, and sales. For the refund, they are yes and no.

Keep the backbone and freeze its weights:

The backbone processes the input and builds internal representations of it. Add LoRA, small trainable components that adapt those representations while the original weights stay frozen.

Replace the language model head:

The language model head predicts the next token. Replace it with a decision head that gathers evidence from the ticket for each option, lets the questions and options interact, and produces one score per choice.

Convert scores into probabilities:

Apply softmax, which converts scores into probabilities, separately for each question. You get one probability distribution over the teams and another over the refund choices.

Train on the correct answers:

For this training example, the labels are billing and yes. Cross-entropy loss measures how well the predictions match those labels, and training updates only the decision head and LoRA components.

Calibrate on held-out examples.

After training, use examples excluded from training to adjust the model’s confidence. Its predicted probabilities should reflect how often those predictions are correct.

At inference time, one forward pass scores all the choices. Your application gets probabilities it can use to route the ticket and make the refund decision.

We trained a JEV-style decision model locally using Qwen3.5 0.8B. Accuracy went from 37% to 65% in just 60 steps, with 10 minutes of training on 4GB of VRAM.

You can watch our full video walkthrough below, from setup and training to evaluation and inference:

Good day!

And this one is built for Agent harnesses:

UHP (Unified Harness Protocol) is a standardized interface for running agent tasks across any agent harness, regardless of which one you use.

HarnessRouter is its open-source server implementation.

Here’s the GitHub repo →

Here’s how it works:

• Your product backend sends a task + harness_id to HarnessRouter.

• HarnessRouter routes it to whichever configured harness should run it.

• The harness streams progress and artifacts back through HarnessRouter.

• HarnessRouter returns the output to your product.

Three core components handle this:

• Gateway: Responses API, harness config, your model keys.

• Runner: Isolated workspaces, one event stream, turn checkpoints.

• Built-in tools: MCP servers, browser, database, GitHub, media.

This means your agent infrastructure becomes fully interchangeable.

• Swap Codex for Claude Code without touching your product code.

• Run the same task across different harnesses and compare outputs.

• Add new harnesses as they emerge without any rewrites.

Here’s the repo: github.com/HarnessRouter/harnessrouter

(don’t forget to star it ⭐)

A Full Walkthrough on Multimodal Jev

Jev can see now!

We recorded a full video showcasing how to use it:

Chapters:

• 00:00 - Intro

• 00:28 - Multimodal Jev demo

• 02:13 - How Jev works

• 03:49 - Jev with images

• 06:51 - Demo app walkthrough

• 10:21 - Using multiple images

• 11:16 - Parallel questions

• 12:32 - Caveats and danger zone headers

• 12:59 - Outro

If you’re new to JEV and want to understand how decision models work, we’ve written a detailed article about it.

Read here →

A tricky LLM interview question:

You move your LLM to a faster GPU.

Profiling confirms its kernels now finish sooner, but the time between generated tokens barely changes.

The model, precision, and batch size are identical.

Where did the expected speedup go?

(answer below)

An LLM forward pass is not one giant GPU operation.

It launches a sequence of operations for attention, matmul, normalization, sampling, data movement, and other work.

Without CUDA Graphs, the CPU prepares and submits these operations individually.

Each launch adds only a small amount of overhead, but token-by-token decoding repeats a similar sequence for every generated token.

This creates a strange bottleneck.

Some kernels finish in only a few microseconds. The GPU can complete one kernel before the CPU has submitted the next one.

That leaves empty spaces in a profiler trace. These gaps are not model computation. They show the GPU waiting for its next operation.

CUDA Graphs change how this work reaches the GPU.

The runtime captures a repeatable sequence of operations once, including their dependencies. Later iterations replay the captured sequence through one graph launch instead of submitting every operation separately.

To be clear, the kernels themselves do not become substantially faster.

Also, the model does not get smaller, and the forward pass performs almost the same computation.

Most of the gain comes from removing CPU launch work between kernels.

This is especially useful during LLM decoding.

Each decoding step processes one new token per sequence. This produces many short, repeated GPU operations where launch overhead can become a meaningful part of total latency.

Prefill often behaves differently.

A large prompt gives the GPU more parallel work, producing longer-running kernels. When those kernels already keep the GPU busy, reducing launch overhead matters less.

There is also a tradeoff.

A captured graph expects the same kernel sequence, stable arguments, fixed memory addresses, and a compatible execution structure on every replay.

Inference servers handle varying batch sizes and different prefill, mixed, and decode workloads. One captured graph cannot cover every execution path efficiently.

Engines such as vLLM and TensorRT-LLM therefore capture graphs for several common execution cases and batch sizes.

When a live batch does not match a captured size, the engine may use eager execution or pad the batch to a captured size.

Capturing more sizes improves graph coverage, but it also consumes more GPU memory and increases startup time.

So CUDA Graphs are not a universal inference speed switch.

They trade flexibility and setup cost for lower steady-state launch overhead.

The visual below shows the core mechanism.

• Without a graph, the CPU submits each kernel independently and may leave idle gaps on the GPU timeline.

• With a graph, the sequence is captured once and replayed as a unit.

CUDA Graphs are one example of why LLM inference needs a specialized serving engine rather than a standard prediction API.

Good day!

• Building a durable multi-agent system with LangGraph and LangSmith.

• Jev for Agent Memory, clearly explained.

• ​The AI engineering master stack for 2026!​

Building a durable multi-agent system with LangGraph and LangSmith

Parts 5 and 6 of our hands-on Agent Engineering course are now available.

• Part 5: Multi-agent orchestration with LangGraph →

• Part 6: Durable Agents with LangGraph and LangSmith →

Part 5 turns the verified agent loop into a team of parallel specialists. Part 6 adds the controls needed to keep that team running across restarts, approvals, graph changes, and releases.

Together, they cover:

• Parallel specialist workers with scoped, read-only tools

• Reducer-backed shared state, deterministic merging, and independent review

• Postgres checkpoints, expiring approvals, and a queryable audit trail

• Graph versioning, LangSmith evaluation gates, and an authenticated streaming API

Why care?

Building an agent that can call tools is one engineering task. Operating several agents as one reliable system requires a broader set of skills.

Once workers run in parallel, shared state becomes a concurrency problem. Each worker needs private loop state, concurrent writes need reducers, and the final merge must remain deterministic. A failed worker must become an explicit partial result instead of blocking the entire investigation.

Durability introduces a different class of problems. A run may pause for approval, lose its process, and resume elsewhere hours later.

That requires synchronous checkpoints, persistent interrupts, expiring approvals, and an atomic decision so approval, rejection, and expiry cannot resume the same thread twice.

Deployment adds change-management and security concerns. A paused thread created on yesterday’s graph must remain compatible with today’s code.

Prompt changes need behavioral evaluations because deterministic unit tests cannot detect every regression. Identity must come from authentication rather than the request body, and tenant boundaries must hold across every endpoint.

These are the same system-design concerns found in backend, platform, and agent-infrastructure roles: concurrency semantics, durable workflows, authorization, observability, compatibility, and release safety.

For a developer who already understands tool calling and basic LangGraph workflows, Parts 5 and 6 clarify the difference between framework familiarity and production system design.

Being able to explain why a state key needs a reducer, how a paused run survives a deployment, or why an evaluation gate catches failures that unit tests miss is a stronger signal than another agent demo.

• Learn to orchestrate parallel specialists safely in Part 5 →

• Learn to make agent runs durable and deployable in Part 6 →

👉 Over to you: Which production guarantee is still missing from the agent systems being built today?

Jev for Agent Memory, clearly explained:

An agent-memory system often passes conversations, tool results, and task outcomes through an extractor.

The extractor identifies possible memories such as user preferences, facts, and procedures.

Consider a travel assistant booking a flight.

The user says:

• I usually prefer an aisle seat.

• For this flight, book a window seat.

• The flight is delayed by 25 minutes.

An extractor may identify all three as facts. Saving all three to long-term memory would be a mistake.

• The aisle-seat preference could help with future bookings.

• The window-seat request applies only to this flight. Saving it as the user’s latest seat preference could make the next booking wrong.

• And the delay is useful during the trip. It has no value once the flight is over.

The extractor can identify what the user said, but it does not necessarily know how long the fact should live or where it should apply.

Jev can run here before the application writes anything to memory.

For each candidate, the application gives Jev the proposed fact, its source evidence, and the current user, trip, and session state.

The application can ask Jev typed questions such as:

• Does the evidence support this fact?

• Does it apply outside the current trip?

• Should it be stored for the user, the trip, or only the current session?

• How likely is incorrect reuse to cause a bad booking?

Jev returns choices, scores, and probabilities.

Application code then applies explicit rules. For example:

↳ Save the aisle-seat preference for the user.

↳ Attach the window-seat request only to this trip.

↳ Keep the delay in the current session and remove it when the trip ends.

Only the approved candidate memories are sent to Graphiti as episodes.

Graphiti keeps the original episode, extracts entities and relationships, resolves new facts against the existing graph, and records when facts become valid or invalid.

GitHub repo: https://github.com/getzep/graphiti

(don’t forget to star it ⭐ )

In code, Jev runs before the call that adds an episode to Graphiti.

If a candidate passes the threshold, the app writes it to Graphiti with the appropriate scope and source. If it fails, the application can keep it in the current session or discard it.

Graphiti still handles ingestion, entity resolution, temporal updates, and retrieval. Jev only evaluates the proposed memory before that ingestion begins.

This way, fewer temporary instructions and expired updates enter the set of facts that later agent runs can retrieve.

The diagram shows the write flow:

• Agent activity produces candidate memories.

• Jev evaluates each candidate using its evidence and scope.

• Application rules decide whether and where to store it.

• Graphiti maintains the accepted memory.

That said, Jev becomes more useful when the same typed decisions sit inside a complete agent loop. Parts 3 and 4 of our Agent Engineering course implement that system end to end:

• Part 3: Build routing, guardrails, and approval controls with LangGraph and Jev →

• Part 4: Add planning, verification, replanning, and stopping logic →

​The AI engineering master stack for 2026!​

We prepared this AI engineering master stack that covers ten layers from the model itself to running it safely in production:

Here’s what each layer comprises:

1) ​Foundations​ set how the model represents input through tokens, embeddings, transformer, attention, context window, positional encoding, and mixture of experts.

2) ​Model behavior​ covers how a trained model responds through pretraining, post-training, sampling, temperature, reasoning models, multimodality, and test-time compute.

3) ​Prompt engineering​ shapes the output through the prompt alone using system prompts, few-shot, chain-of-thought, structured outputs, prompt caching, self-consistency, and meta-prompting.

4) ​Retrieval​ feeds the model data it was never trained on through chunking, vector databases, hybrid search, reranking, retrieval eval, query rewriting, and GraphRAG.

5) ​Agents​ let the model take actions instead of only answering through function calling, ReAct, planning, reflection, multi-agent, computer use, and human-in-the-loop.

6) ​Context engineering​ controls what fills the context window across steps through context management, compaction, memory, MCP, agent harness, just-in-time retrieval, and structured note-taking.

7) ​Fine-tuning​ changes the weights when prompting and context fall short through SFT, LoRA, RLHF, DPO, distillation, GRPO, and RLVR.

8) ​Inference optimization​ makes the model cheap and fast to serve through quantization, KV cache, batching, speculative decoding, vLLM serving, FlashAttention, and PagedAttention.

9) ​Deployment​ turns models into services through serving engines, model routing, semantic caching, streaming, load balancing, autoscaling, fallbacks, and on-device inference.

10) ​Evaluation​ measures whether the system is actually correct through benchmarks, LLM-as-judge, golden datasets, hallucination detection, regression tests, trajectory evaluation, and red teaming.

11-12) ​LLMOps and safety​ keep the system reliable and safe in production through observability, cost tracking, guardrails, PII redaction, feedback loops, prompt injection defense, and model routing.

13) ​Frontier​ patterns include decision models, harness engineering, loop engineering, recursive and contrastive LMs, verifier engineering, WebMCP, and forkable agent runs.

This grid above is the overview, but each layer is deep enough to be its own field with dedicated tooling.

We have covered every bit of it in the LLMOps course, starting from fundamentals to production:

• ​Read Part 1 on fundamentals of LLMOps here →​

• ​Read Part 2 on understanding the core building blocks of LLMs →​

• ​Read Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixture-of-experts, and the fundamentals of pretraining and fine-tuning →​

• ​Read Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applications →​

• ​Read Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and different prompting techniques →​

• ​Read Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more →​

• ​Read Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techniques for building high-signal inputs →​

• ​Read Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic context injection, and common failure modes in agentic applications →​

• ​Read Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong understanding of the fundamental concepts →​

• ​Read Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for evaluation of LLM apps →​

• ​Read Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming →​

• ​Read Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO →​

• ​Read Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model parallelism →​

• ​Read Part 14 on the fundamentals of LLM serving, including API-based access, inference with vLLM, and practical decisions.​

👉 Over to you: What else would you add to the master tree?

Good day!

• ​4x faster LLM inference engine than llama.cpp.

• ​A model decision is only as safe as the system that executes it.

• How LLM inference uses GPU memory.

4x faster LLM inference engine than llama.cpp:

We ran Qwen3.5 9B on a base M5 MacBook Pro using three local inference engines, one at a time.

• llama.cpp reached 22.0 tokens/s

• MLX reached 25.1 tokens/s

• Uzu reached 92.1 tokens/s

This frame shows all three runs side by side:

The first two results do not mean llama.cpp or MLX are poorly optimized. Both are approaching a physical limit of ordinary local decoding.

Generating each new token normally requires another pass through the model. That pass streams roughly 5.2 GB of weights through memory.

A base M5 provides about 153 GB/s of memory bandwidth.

At one token per pass, the theoretical ceiling is roughly 153 ÷ 5.2, or 29 tokens/s, however fast the GPU computes.

Uzu gets past that ceiling by optimizing speculative decoding for Apple Silicon.

Speculative decoding uses a smaller drafting system to propose several future tokens. The full 9B model then checks those proposals together instead of generating every token through a separate pass.

Uzu pushes this further in two ways.

• It drafts up to 16 positions and uses Weaver to turn independent guesses into coherent sequences.

• It verifies several possible sequences as a tree, then commits state only for the path the full model accepts.

The 9B model remains in control of the final output. Uzu simply gives it more useful work to approve in each pass.

In our run, every pass of the 9B model produced 7.5 output tokens on average.

Ordinary decoding produces exactly 1.

That is how Uzu reached 92.1 tokens/s on hardware whose one-token-per-pass ceiling is roughly 29.

The rest of the engine is designed around this path too. Uzu ships its own quantized checkpoints, and on M5 chips its Metal kernels run verification through the GPU’s accelerated int8 matrix path.

It is an open-source inference engine from Mirai, with bindings for Rust, Swift, Python, and TypeScript.

The repository is here: https://github.com/trymirai/uzu

(don’t forget to star 🌟)

A model decision is only as safe as the system that executes it

Jev reached nearly 13% of paid teams on Vercel AI Gateway in its first 24 hours, more than double the share for any previous model launch.

Unlike other models, Jev does not write an open-ended response, and instead selects from a defined set of choices. You provide the choices and the current application state, and it returns a selected choice and a probability for each option.

The interesting part here starts after that response returns. For an incident-response agent, that could mean deciding whether to page a team, gathering more evidence, or handing off control to a human.

We used Claude Code to build the agent. It produced 30 files and 3,383 lines covering the model client, policy engine, agent loop, tools, API, dashboard, fixtures, and tests.

The application ran, all 46 tests passed, and the feature looked complete at first glance. Before merging, we still needed to review how those pieces worked together.

That’s where CodeRabbit comes in. As the governance and control layer for agentic software development, it helps developers understand and validate what their agents build so they can ship with confidence.

It organizes a large pull request into its logical architecture, identifies the paths that deserve attention, and reviews behavior and security across the full change rather than one file at a time.

Before looking at what it found, let’s look at the agent Claude Code built.

The incident-response agent

Incident response gives the model a constrained set of choices, each with consequences.

Paging the wrong team interrupts people and can delay the actual response.

We wanted the application to record each dev decision, its confidence, the available evidence, and any handoff to a human.

We gave Claude Code a single prompt with those requirements and let it build the application.

The generated dashboard showed every Jev answer, its probabilities, the tools executed, and the final outcome.

That makes the boundary between model judgment and application policy visible during the run.

The agent starts with an alert. Jev receives the current incident state and answers six questions in one request:

• Which team owns the incident?

• How severe was the incident?

• Whether customers are affected?

• Whether more evidence is needed?

• Whether a human should take over

• What should the agent should do next (next_action)?

The last answer, next_action is restricted to six values: inspect logs, check a deployment, check service health, page a team, request human review, or stop.

When the agent gathers evidence, it adds the result to the incident state and sends the updated state back to Jev.

It reads from local JSON fixtures, which keeps each run reproducible and prevents it from touching a production service.

Every pass through this loop reaches the same boundary, where Jev proposes next_action and the application decides whether that action can become a tool call.

The policy gate

The policy uses the probability assigned to the selected action:

• At 0.80 or above, it permits the action.

• From 0.55 to 0.79, it gathers more evidence.

• Below 0.55, it transfers control to a human.

A page_team recommendation also requires confirmation that customers are affected.

We ran three fixture-backed incidents through the dashboard. The payments outage made the boundary easiest to see.

Jev selected page_team with 0.62 confidence, so the policy refused to page and ran inspect_logs instead.

It then evaluated the updated evidence and returned 0.50 confidence, which fell below the human-review threshold. The run stopped without executing another tool.

The dashboard and 46 tests covered what the application did with expected responses. They did not cover whether next_action and its confidence would stay paired in every response the application could receive.

Tracing the decision through the pull request

Reviewing the change meant opening 30 diffs and reading them one by one.

Each diff was understandable on its own.

The problem was that the safety guarantee did not live in any single file. It depended on how the selected action and its probability moved between them.

We opened the same PR in CodeRabbit. It reviewed the change automatically, rated the merge risk as High, and posted six actionable comments ranked by severity.

CodeRabbit’s Change Stack grouped the 30 changed files into six logical layers based on their dependencies, giving me a path from the Jev response contract to the resulting tool call.

The first four layers covered the code that ran during an incident including the decision, the confidence check, and the loop that chose what to do next.

The remaining two covered the API, the dashboard that exposed the result, and the setup files that supported it.

That grouping gave me a reading order that matched the application’s runtime dependencies.

We could start in decisions.py, where the selected action was paired with its probability, then follow the value into the policy and agent loop.

CodeRabbit attached each finding to the relevant layer and ranked them by severity. The first one we opened was in decisions.py, before the confidence value reached the policy.

The confidence bug 46 tests missed

In decisions.py, when the selected choice was absent from the returned probability map, the code used the largest probability from any option as the selected choice's confidence.

That fallback can combine two valid values into an invalid decision.

choice = page_team
page_team probability = missing
inspect_logs probability = 0.90

normalized confidence for page_team = 0.90

Both values are valid on their own, but their association is false. The policy now sees a high-confidence paging decision even though Jev never assigned page_team that score.

The dashboard did not expose the failure because its live responses always included the selected option’s probability. The tests missed it for the same reason.

Every fixture preserved the expected relationship between choice and probabilities.

The safe behavior is to assign zero confidence or reject the response and route the run to human review. CodeRabbit flagged the untested response and explained how the fallback could authorize a tool call.

Tracing security from the API to the dashboard

The confidence bug was one of six findings. The others tracked where external input entered the system and what it could reach from there.

Change Stack’s security blast-radius view mapped four findings across five changed components.

The highest-impact path began with incident text submitted through the Run API. The application retained that text in a shared run state.

The dashboard later inserted it into HTML without encoding, creating a stored XSS vulnerability. Anyone able to reach the API could store content that executed when another user viewed the run.

Three other boundaries had their own exposure:

• The API exposed shared incidents and evaluation records without authorization or ownership checks.

• The Jev request URL could be changed without restricting its scheme or destination.

• Fixture mode could fall through to a live request when an incident did not match a fixture.

The vulnerable line was in the dashboard, but the exploit path began at the API and crossed the shared state. The blast-radius view connected those components instead of treating the rendering line as an isolated defect.

Validating the agent-generated change

Generating code doesn't make a change ready to merge. Agentic Change Management helps teams understand how the change fits the architecture, identify its riskiest paths, and validate behavior and security across the full change.

That control continues after merge, helping keep the codebase secure and maintainable.

In the example, CodeRabbit reconstructed the architecture and reading order, automated review found the confidence failure, and the security blast-radius view followed external input across the API, shared state, and dashboard.

The agent ran successfully, and all 46 tests passed. Review still found an untested response that could bypass the intended confidence gate, along with a security flaw that crossed the API, shared state, and dashboard.

We built the agent in one session, then reviewed how it worked and checked for security flaws before deciding whether it was ready to merge.

You can start a free 14-day CodeRabbit trial, with no credit card required.

Ship agent-built code with confidence →

Thanks to CodeRabbit for partnering today!

How LLM inference uses GPU memory

Loading an LLM does not guarantee inference!

The model may load successfully and still run out of GPU memory when the first requests arrive.

This happens because model weights are only one of four VRAM allocations:

1) Model weights

The parameters are loaded into VRAM before inference begins.

Ignoring quantization metadata and implementation overhead, a 70B parameter model requires approximately:

→ 140 GB in FP16 or BF16

→ 70 GB in INT8

→ 35 GB in INT4

This allocation remains mostly constant during inference.

Quantization reduces it by storing each parameter with fewer bits. However, kernels may use higher-precision values for some operations.

2) KV cache

Each Transformer layer generates KV tensors for every processed token.

The inference engine retains these tensors so new tokens can attend to earlier tokens without recomputing their key and value projections.

The approximate KV cache memory per token is:

2 × layers × KV heads × head dimension × bytes per value

The factor of two represents the key and value tensors.

This allocation increases with context length and the number of active requests.

Grouped-query attention and multi-query attention reduce it by sharing fewer key-value heads across multiple query heads.

3) Activations and workspace

A forward pass creates intermediate tensors for attention, MLP operations, normalization, and matrix multiplication.

GPU kernels may also require temporary workspace for computation.

The inference engine reuses much of this memory across layers and generation steps.

Its peak size depends on the model architecture, number of tokens processed together, selected kernels, and serving configuration.

Prefill processes multiple prompt tokens in parallel.

Decode usually processes one new token per active request.

The two phases therefore have different activation and workspace requirements.

4) Runtime overhead

CUDA contexts, loaded kernels, memory allocators, metadata, communication buffers, and serving-engine state also consume VRAM.

The amount varies across runtimes and deployment configurations.

Memory fragmentation can also leave enough total free VRAM but no sufficiently large contiguous region for a new allocation.

The total memory requirement is:

model weights + KV cache + peak activations and workspace + runtime overhead

• Model weights remain mostly constant.

• KV cache memory increases with context length and concurrency.

• Activation and workspace usage changes during execution.

• Runtime overhead depends on the software stack.

This is why model size alone cannot determine whether an LLM can serve a given workload.

The visual below shows when each allocation is created, how long it remains in memory, and which workload parameters increase it.

Good day!

• Finally, an OpenRouter for agent harnesses!

• 6 parallelism strategies for LLMs in production.

• ​Schema-guided agent memory.

Finally, an OpenRouter for agent harnesses!

Devs just open-sourced a plug-and-play infrastructure layer that lets you run any harness under a single interface, like:

• Codex

• Hermes

• Claude code

• DeepSeek Harness

• System One, powered by Jev

• And 9 more agent harnesses

This means you can bring Jev into the same product that already uses Codex, Claude Code, or another supported harness, without writing another implementation for sessions, streaming, files, cancellation, and failure handling.

Here’s the repo: github.com/HarnessRouter/harnessrouter

(don’t forget to star it ⭐ )

The harnesses run locally, and the Unified Harness Protocol (UHP) defines the common task interface with an OpenAI Responses-compatible API.

6 parallelism strategies for LLMs in production

Running a model across four GPUs does not merge their memory into one shared pool.

Each GPU has its own memory, so data stored on another GPU must cross the interconnect before a layer can use it.

This is the central trade-off behind parallelism. It removes local memory or compute limits but adds communication overhead.

The visual below explains the six common strategies for parallelizing LLM inference:

• Data parallelism runs complete model replicas on separate GPU groups. It increases throughput, but cannot make an oversized model fit.

• Tensor parallelism splits matrix operations inside each transformer layer. It reduces weight memory per GPU, but the GPUs must combine partial results throughout the layer stack.

• Pipeline parallelism assigns layer groups to different GPUs. Activations move between stages, while insufficient micro-batching leaves some stages idle.

• Context parallelism splits sequence positions in two ways. Prefill context parallelism divides a long input across GPUs to target lower TTFT. Decode context parallelism divides the stored KV history to increase cache capacity, but every decoder layer must merge partial attention results.

• Expert parallelism distributes MoE experts across GPUs. Token routing and uneven expert load add network cost and idle time.

• Hybrid parallelism combines multiple strategies in one deployment. For example, tensor parallelism can operate within a node, pipeline parallelism across nodes, and data parallelism across model replicas. This matches each communication pattern to the available links, but adds more coordination paths.

Production deployments often combine methods, but each one should fix a measured bottleneck.

• If the weights do not fit, use tensor or pipeline parallelism.

• If long-prompt prefill or KV-cache capacity is the limit, use the corresponding form of context parallelism.

• MoE models may need expert parallelism to place expert weights.

• If one replica meets its latency target but requests still queue, add data-parallel replicas.

Keep tensor parallelism on the fastest links. After each change, measure memory, TTFT, TPOT, and other metrics again.

Add another method only when measurements reveal another bottleneck.

We wrote a detailed article explaining how each strategy works, where communication occurs, and how to choose a layout for a real deployment.

Read it here →

If you want to learn the full LLMOps lifecycle, we have covered every bit of it in the LLMOps course, starting from fundamentals to production:

• Read Part 1 on fundamentals of LLMOps here →

• Read Part 2 on understanding the core building blocks of LLMs →

• Read Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixture-of-experts, and the fundamentals of pretraining and fine-tuning →

• Read Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applications →

• Read Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and different prompting techniques →

• Read Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more →

• Read Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techniques for building high-signal inputs →

• Read Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic context injection, and common failure modes in agentic applications →

• Read Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong understanding of the fundamental concepts →

• Read Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for evaluation of LLM apps →

• Read Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming →

• Read Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO →

• Read Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model parallelism →

• Read Part 14 on the fundamentals of LLM serving, including API-based access, inference with vLLM, and practical decisions →

​Schema-guided agent memory​

Most agent memory systems optimize for recall. The harder problem is what to forget, or more precisely, what to never store in the first place.

The default agent memory pipeline hands an LLM raw text and asks it to extract entities and relationships. The model decides the types, labels, and attributes all on its own.

The result is a knowledge graph that behaves like an expensive vector store. Entity types collapse into generic labels. Relationships flatten into a single “RELATES_TO.”

The graph has the data, but no query can reach it with precision.

This is not a retrieval problem but rather a structure problem. And the fix is the same pattern that already works everywhere else in the AI stack, i.e., constrain the output space before generation, not after.

• Entities define what the agent is allowed to remember. Pydantic models with typed fields and descriptive docstrings replace the LLM’s guesswork with domain vocabulary it was never trained on.

• Edges define how things connect. Source/target constraints on relationship types mean the graph can only form valid connections. If your schema has no edge connecting Project to Competitor, that relationship cannot exist in memory.

• Temporal resolution handles what was true versus what is true. Fact resolution invalidates outdated edges while preserving history, so the graph never silently serves a stale state.

The schema guides extraction at two points in the pipeline (entity extraction and fact extraction) while resolution and temporal processing run automatically downstream.

You define what to look for. The system handles deduplication, contradiction detection, and time-windowing without additional configuration.

A useful constraint is to have just 10 entity types, 10 edge types, and 10 fields per type.

That forces you to model the 80% that matters rather than attempting completeness. Start with 3-4 of each and expand only when retrieval fails.

​Zep Graphiti​ does all of this as a fully open-source temporal knowledge graph library.

It includes Pydantic-based ontology definition, schema-guided extraction, entity resolution, fact resolution, and temporal windowing.

If you are building agent memory with any kind of domain specificity, schemas are a must.

​Here’s the GitHub repo →​

(don’t forget to star 🌟)

​We also covered this topic in more depth (with code) in this article →​

Good day!

For a math problem, the training record only needs the question and its final answer. The model samples several solutions. A reward function checks each result. GRPO then increases the probability of responses that score higher within the group.

This gives us a practical way to customize reasoning behavior without curating a reasoning trace for every example. It also introduces a hard engineering constraint. The model will optimize exactly what the verifier rewards, including any shortcuts hidden in that verifier.

We will test this on Qwen3 0.6B using serverless RLVR in Amazon SageMaker AI. The target is higher exact-match accuracy on a fixed set of unseen arithmetic problems.

One evaluation set stays fixed throughout the experiment. We will test the reward function, run one customization job, repeat the evaluation, and deploy the resulting model to a SageMaker AI real-time endpoint.

Training against an executable reward

Supervised fine-tuning (SFT) learns from target completions. Each training record contains an input and the response that the model should imitate. This works well when high-quality completions already exist.

RLVR needs a different asset. It needs a verifier that can score generated responses. The training set can contain a problem and its reference answer without prescribing the full reasoning trace.

For a math problem, the verifier can extract the final number. For generated code, it can run unit tests. For structured extraction, it can validate a schema and compare required fields.

Reasoning traces are expensive to curate. When correctness has a programmatic definition, a verifier can replace them.

RLVR does not remove supervision. It moves supervision into the reward function. That reward function becomes part of the task specification.

A verifier only measures what its code expresses. It cannot distinguish genuine reasoning from a shortcut unless the test exposes that shortcut.

Choosing the optimization signal

Proximal Policy Optimization (PPO) is a general policy-gradient method. RLHF systems often combine it with a learned reward model and a separate value model. That stack is flexible, but it adds training and operational complexity.

Group Relative Policy Optimization (GRPO) removes the separate value model. It samples a group of responses for one prompt. Their relative rewards provide the baseline used during the policy update. The DeepSeekMath paper introduced GRPO for mathematical reasoning.

Direct Preference Optimization (DPO) avoids online reinforcement learning. It learns directly from preferred and rejected response pairs. DPO is useful when pairwise preference data already captures the desired behavior.

RLVR is not universally better than these methods. It fits tasks with reliable checks. A weak verifier can produce a model that scores well while failing the real task.

RLVR replaces ideal traces with executable feedback, which puts more responsibility on reward design.

Experiment design and evaluation contract

The experiment uses the custom-reward RLVR path from the Amazon SageMaker AI serverless model customization workshop. AWS configures this lab around Qwen3 0.6B and 180 GSM8K samples.

The setup is fairly direct. Just deploy the provided Studio environment, clone the repository, and run five Jupyter notebooks in sequence.

We will preserve the workshop’s Qwen3 model and training path while treating the evaluation protocol as our own experiment. Qwen3 is an open-weight model, so this workflow is not limited to Amazon-developed models.

You can find the model on HF here →

Opening SageMaker Studio

You do not create a separate SageMaker account. You need an AWS account with permission to create SageMaker, IAM, VPC, S3, Lambda, and MLflow resources.

Sign in to the AWS Management Console and open the workshop’s own-account setup page. Launch its Deploy SageMaker Studio domain CloudFormation template.

The template creates the networking environment, SageMaker execution role, Studio domain, user profile, and managed MLflow tracking server. It defaults to us-west-2. If you choose another region, update the Availability Zone parameters before creating the stack.

Wait until the CloudFormation stack reaches CREATE_COMPLETE. Then open the workspace through the AWS console:

1. Search for SageMaker AI.

2. Select Domains from the left navigation.

3. Open the Studio domain created by CloudFormation.