Every dependency in a modern codebase shows up in a manifest, except the AI ones.

Open-source models, LLM APIs, MCP servers, and agents enter production through the same pull requests as ordinary packages, but no lockfile lists them and no dependency scanner sees them.

Checkmarx found that 70% of teams expect AI components in production, while 43% have zero governance over them.

Checkmarx AI Inventory (part of Checkmarx One) catalogs every model, SDK, and MCP server deterministically, traces each one to the exact file and line, and exports an AI-BOM for compliance audits.

Gartner named Checkmarx a Leader in its inaugural Magic Quadrant for Software Supply Chain Security.

Learn more about Checkmarx AI Inventory here →

Thanks to Checkmarx for partnering today!

5 LLM Quantization Techniques

A 70B parameter model in FP16 requires 140GB of memory for weights alone.

That exceeds any single GPU available today:

• An H100 carries 80GB

• An RTX 4090 carries 24GB

So serving the model in half precision means multi-GPU setups and tensor parallelism before the first token is generated.

Quantization reduces this footprint by storing weights in lower bits.

The standard scheme maps each weight from FP16 onto a small integer grid: compute the dynamic range of the tensor, divide it into 2^b levels (16 levels at 4-bit), round each weight to the nearest level, and store a scale factor to dequantize during inference.

At 4-bit, the weight memory drops 4x. The 140GB model fits in 35GB, within the range of a single 40GB or 48GB card.

That said, if rounding were lossless in practice, one method would suffice, and the diagram below would have had one row:

But it has five because weights and activations in large transformers are not uniformly important.

The LLM.int8() paper quantified this. Beyond roughly 6.7B parameters, every transformer layer develops a handful of hidden dimensions carrying values 20x to 100x larger than the rest.

These dimensions make up about 0.1% of the model’s features, but zeroing them out nearly destroys the model’s ability to predict text.

These outliers are also what break naive rounding. The quantization grid spans the range of the tensor, so a single activation of magnitude 60 among values near 1 inflates the scale factor and collapses the remaining values into a few levels, destroying their precision.

Each method in the visual shown earlier solves this saliency problem differently.

1) RTN (round to nearest) does not handle it at all. It applies the scale-and-round procedure directly with no calibration data, which makes it the cheapest baseline and the weakest at low bit widths, where accumulated rounding error is unrecoverable.

2) GPTQ corrects rounding damage as it happens. It quantizes the weights of a layer a few at a time, measures how much error the rounding just introduced, and nudges the not-yet-quantized weights to cancel out that error before moving on.

The adjustment is not uniform. GPTQ uses statistics from calibration data to work out which remaining weights can absorb the error with the least effect on the layer’s output, which is what separates it from blind rounding.

This runs fast enough to matter in practice. A 175B model quantizes to 4-bit in around four GPU hours. The known weakness is that fitting the calibration data this closely can tune the model to it, and the AWQ authors showed this hurts accuracy on inputs that look different from the calibration set.

3) AWQ protects the important weights before rounding instead of repairing damage afterward. It runs calibration samples through the model and watches which weights get multiplied by the largest incoming values, because a rounding error on those weights gets amplified the most in the layer’s output.

Only around 1% of weight channels turn out to matter this much. Rather than storing them at higher precision, AWQ multiplies them by a scaling factor before rounding, so they spread across more levels of the grid and lose less detail, and the math is rearranged so a matching inverse scale elsewhere keeps the layer’s output unchanged.

Every weight still ends up in plain INT4, with no special-case handling at inference time. AWQ also depends far less on which calibration samples were used than GPTQ does, which is a big part of why it became the standard choice in serving engines like vLLM.

4) LLM.int8() isolates the outliers instead of fighting them. When the model is loaded, it finds the dimensions carrying extreme values and splits every matrix multiplication into two:

• The outliers run in full FP16

• The other 99.9% of values run in INT8, and the two results are added back together.

Nothing needs to be computed ahead of time. That is why it is also available as the load_in_8bit flag in bitsandbytes.

The cost is speed. Splitting and merging every matrix multiplication is slower than running one optimized 4-bit kernel, so this path is common for local development and QLoRA fine-tuning, not for serving traffic at scale.

5) QAT modifies training rather than the quantization procedure. During a short fine-tuning run, every forward pass rounds the weights to INT4 before using them, so the model experiences the exact damage that quantization will cause, and its predictions are scored under that damage.

The weight updates still happen in full precision behind the scenes, since rounding itself cannot be trained through directly. Over the fine-tuning run, the weights drift toward values that keep the loss low even after rounding, so when the real quantization is applied at the end, there is almost nothing left to break.

Google produced its Gemma 3 QAT checkpoints with roughly 5,000 fine-tuning steps guided by the original model’s outputs, and the int4 versions hold close to full-precision quality at about 3x lower memory.

All five methods described above produce the same artifact, i.e., a model at a fraction of its trained precision.

They differ in where the outlier problem gets addressed, at rounding time, after rounding, before rounding, at inference, or during training itself.

To dive deeper, we have already written a full deep dive on Quantization, specifically, which covers several of these methods with their simplified mathematics.

Learn how Quantization optimizes LLMs to run them on tiny hardware here →

Good day!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

Part 13 of the Reinforcement Learning course is available now.

Here, we look at what actual AI labs and companies are actually doing with everything we learned across twelve chapters.

Read Part 13 of the RL course here →

RL Nanodegree Part 13

It covers four case studies, each showing a different role RL plays in the industry:

• Cursor’s real-time RL loop that ships improved checkpoints every five hours

• How frontier labs turned verifiable rewards into reasoning and agentic capabilities

• Scale AI’s enterprise results, where a 4B model beat GPT-5 on domain-specific tasks

• The emerging market where RL environments are built, priced, and sold as products

Read Part 13 of the RL course here →

RL Nanodegree Part 13

Why care?

Across all twelve chapters of this series, we built the full mechanics of RL piece by piece.

We covered bandits, value functions, Bellman equations, policy gradients, PPO, GRPO, judges, trajectories, and environments.

One real test of understanding RL is whether you can read a company’s blog post about their training pipeline and immediately identify the environment, the reward source, the trajectory structure, and the failure modes they are likely facing.

That is what this chapter exercises.

We cover stories of very different companies, each using RL in a way that maps directly onto concepts from this series.

The vocabulary you built is the same vocabulary these teams operate in, like reward hacking, credit assignment, environment design, and the tension between learned and verifiable rewards.

This is the final chapter, and this is what we have covered in the full course:

• Part 1: Foundations of Reinforcement Learning →

• Part 2: MDPs and Value Functions →

• Part 3: Bellman Equations and Dynamic Programming →

• Part 4: Model-Free Learning →

• Part 5: Function Approximation →

• Part 6: Deep RL and DQN →

• Part 7: Policy Gradients and Actor-Critic →

• Part 8: Proximal Policy Optimization →

• Part 9: RLHF →

• Part 10: Verifiable Rewards and GRPO →

• Part 11: Reward signal problem for Agents →

• Part 12: Environments, trajectories, and the training Loop →

• Part 13: How AI teams use RL in production →

👉 Over to you: What topics would you like us to cover in this RL series?

LLM Routing vs Mixture of Agents

When you’re building a system that needs to handle diverse queries, two architectural patterns are considered.

One picks the best model for each task, and the other gets multiple models to collaborate on the same task.

Both involve multiple LLMs, but what they do with those models is completely different.

The visual below explains how each one works:

In LLM routing, an intent classifier reads the prompt, and a cost ranker weighs latency, token price, and capability, so only one model gets the call while everything else stays idle.

• The prompt is classified by domain and action.

• A cost ranker picks the best model given latency and price constraints.

• One model responds, and the rest never see the query.

In Mixture of Agents, a set of consulting models each analyze the query independently and hands their output to a single aggregator, which uses every perspective to write the final answer and make the tool calls.

The consulting models never see each other’s analysis, only the aggregator does.

On HermesBench, Nous Research reports a MoA preset scored higher than either model alone by roughly six points and eight to eleven percent, and no individual model in that setup reached the same result on its own.

LLM routing and MoA are not competing approaches but rather two different tools for two different problems.

Routing is about picking the right model for the task, while MoA is about combining multiple models to exceed what any one can do alone.

We wrote a full deep dive on LLM routing with Plano, including a working demo, in a recent article.

Here’s the Plano GitHub repo: https://github.com/katanemo/plano (Don’t forget to star it ⭐️)

And here’s the article we wrote →

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

Anyone can write that loop in thirty lines. But a bare version of it falls apart on a real codebase, reading the wrong files, losing the goal midway, and filling its context with output it no longer needs, while Claude Code finishes the same task cleanly.

The easy conclusion is that Anthropic simply has a better model. The actual difference is the harness, the ordinary code wrapped around the model that handles planning, tool execution, memory, and safety, while the model only decides the next step.

Here’s what a fully harnessed agent looks like when you draw it out:

• Memory feeds the model its working context plus facts it has learned across sessions.

• Skills encode how the agent should operate, meaning the procedures, constraints, and heuristics it follows.

• Protocols connect the agent to users, tools, and other agents.

• The harness core ties it all together with sub-agent orchestration, a sandbox, an evaluator, an approval loop, observability, and context compression.

Anthropic describes this split as the brain and the hands. The model is the brain that picks each action, and the harness is the hands that carry it out and keep the run on track.

So the gap between your agent and Claude Code isn’t the model, it’s the machinery around the model.

Claude Code is one of the most capable harnesses in production today, and it’s built from a surprisingly small set of layers in that illustration. To see how much of that machinery you’d have to build yourself, I rebuilt it in CrewAI, an open-source framework for orchestrating agents.

More of it maps onto built-in features than I expected, and the part that doesn’t is where the real engineering lives.

Let’s build it layer by layer, starting with the core loop and stacking planning, subagents, sandboxing, and memory on top. At each step, we’ll mark where the framework ends and where your job begins.

How Claude Code's harness works

At the center of Claude Code is a plain agent loop. You send it a message, the model decides what to do next, and it either responds directly or requests a tool. If it requests one, the tool runs, the result goes back into the conversation, and the model decides again.

This repeats until the model returns a final answer with no further tool calls.

Inside that loop, the model reads files, edits code, runs shell commands, and executes tests. These aren’t separate modes. They’re just different tool calls inside the same loop.

The loop alone isn’t enough for a reliable coding agent, though. Claude Code adds planning, file tools, subagents, memory, and a permission and sandbox system around it. These layers don’t replace the loop, they make it safe and reliable enough for real work.

That’s the architecture we’ll rebuild, the core loop first, then each layer on top, mapping every layer to the CrewAI feature that handles it.

The core agent loop

The loop runs the same sequence until the task is done:

1. Ask the model to perform the task.

2. The model responds directly or requests one or more tools.

3. If tools are requested, run them and return the results to the model.

4. Repeat with the updated conversation.

5. When the model responds without requesting any tools, the task is complete.

while True:
    reply = model(messages, tools)
    calls = [b for b in reply if b.type == "tool_use"]
    if not calls:            # plain text, no tool call: the job is done
        return reply.text
    messages += [reply, run_all(calls)]

Each tool call completes one step, gives the model new information, and feeds into the next decision. A simple question might finish in one iteration, while fixing a complex bug or refactoring a large codebase can take dozens of iterations before the model has enough to produce a final answer.

CrewAI provides this execution loop automatically as soon as you create an agent. You don’t implement the while loop yourself, you define the agent and assign it a task.

Building the first agent

Let’s create a simple Bug Fixer agent.

from crewai import LLM, Agent, Crew, Task

bug_fixer = Agent(
    role="Bug Fixer",
    goal="Find and describe the fix for the reported bug in the codebase.",
    backstory="You read directories and files to build an accurate picture of the code.",
    llm="claude-sonnet-4-6",
)

task = Task(
    description="Find the fix for {objective}.",
    expected_output="A short description of the fix and which file it belongs in.",
)

result = Crew(agents=[bug_fixer], tasks=[task]).kickoff(
    inputs={"objective": "the overdraft bug in account.py"}
)

Three concepts to understand here:

• An Agent defines who does the work, through its role, goal, LLM, and tools.

• A Task describes the assignment.

• A Crew brings agents and tasks together. Calling kickoff() runs the same execution loop described above, regardless of whether the underlying model is Anthropic, OpenAI, Google, or something else.

Giving the agent tools

Tools are what let a model that only generates text actually work on a codebase. They read files, write them, run shell commands, and call external APIs.

CrewAI ships file system tools out of the box:

• FileReadTool reads files.

• DirectoryReadTool lists directories.

• FileWriterTool writes files.

from crewai_tools import DirectoryReadTool, FileReadTool, FileWriterTool

read_file = FileReadTool()
write_file = FileWriterTool()
list_dir = DirectoryReadTool()

filesystem_tools = [read_file, write_file, list_dir]

These also double as external memory. Instead of holding a large search result in the model’s context window, the agent can write it to a file, keep only the filename, and read it back when needed.

This keeps the context window smaller and the model more focused.

Built-in tools cover common workflows only. For anything more specific, you expose a Python function as a tool with the @tool decorator.

The docstring acts as the instruction manual, telling the model what the tool does, when to use it, and what it expects as input.

from crewai.tools import tool
import subprocess

@tool("run_tests")
def run_tests(path: str = "tests/") -> str:
    """Run the pytest suite at the given path and return the result."""
    result = subprocess.run(
        ["pytest", path, "-q"], capture_output=True, text=True, timeout=120
    )
    output = result.stdout + result.stderr
    return output[-4000:] if len(output) > 4000 else output

Adding planning capability

As tasks get more complex, a plain execution loop starts losing track of the original goal. After enough tool calls, file reads, and intermediate results, the context fills up and the objective gets crowded out by everything that came after it.

This slow degradation is what people call context rot.

Planning addresses it directly. The agent builds a step-by-step plan before doing any work and keeps that plan in context throughout execution.

The plan doesn’t do the work. It’s a roadmap that keeps the model connected to the original goal, which is the same job Claude Code’s to-do list does.

CrewAI adds this at the crew level with planning=True. It generates a plan before execution and keeps it available as the task progresses.

from crewai import Crew, LLM

crew = Crew(
    agents=self.agents,
    tasks=self.tasks,
    planning=True,
    planning_llm=LLM(model="gpt-4o-mini"),
)

Note: By default, CrewAI uses gpt-4o-mini for planning, and you can swap in any LLM you prefer for that step.

Individual agents can also reason about their own work with reasoning=True:

from crewai import Agent

bug_fixer = Agent(
    role="Bug Fixer",
    goal="Find and describe the fix for the reported bug in the codebase.",
    backstory="You read directories and files to build an accurate picture of the code.",
    tools=[FileReadTool()],
    reasoning=True,
    max_reasoning_attempts=3  # Optional: Set a maximum number of reasoning attempts
)

Planning and reasoning solve different problems. Planning builds a high-level roadmap for the overall task, while reasoning gives one agent time to think through its own approach before acting.

When reasoning is enabled, the agent:

1. Reflects on the task and drafts an execution plan.

2. Evaluates whether the plan is ready.

3. Refines the plan if necessary, until it’s satisfied or hits max_reasoning_attempts.

4. Injects the finalized reasoning plan into the task before execution.

Together, they keep the agent anchored on long-running tasks and reduce drift from the original goal.

Subagents to maintain main agent’s context

Planning keeps the agent focused, but it doesn’t reduce how much information the model has to hold. On a large codebase, even a well-planned task can exceed a single context window.

Finding one bug may require reading dozens of files, and the main agent doesn’t need to keep all of them in memory.

Subagents solve this through delegation. The main agent hands a specific task to a helper agent, which works in its own context and returns a short summary. The main agent sees the conclusion, not the intermediate steps.

CrewAI supports this through hierarchical workflows, where a manager agent delegates to specialist agents and combines their results.

In our earlier setup, one Bug Fixer agent did all the heavy lifting. Let’s split the work across a manager and three specialists:

• Codebase Explorer explores the code and maps the repository.

• Software Engineer implements the requested change.

• Test Runner runs the tests in the sandbox and reports pass or fail.

• Engineering Lead oversees the three specialists.

from crewai import Crew, Agent, Task, Process

explorer = Agent(
    role="Codebase Explorer",
    goal="Map the repository and surface the files relevant to the task.",
    backstory="You read directories and files to build a picture of the code.",
    tools=[read_file, list_dir],
    llm=llm,
)  # Same for other two specialist agents

manager = Agent(
    role="Engineering Lead",
    goal="Break the request into steps and delegate each to the right specialist.",
    backstory="You decide who does what, review tests, finish once change is done.",
    llm=llm,
    allow_delegation=True,
)

crew = Crew(
    agents=[explorer, coder, tester],
    tasks=[task],
    manager_agent=manager,
    process=Process.hierarchical,
)

One thing to note is that allow_delegation is disabled by default, so it must be explicitly enabled on the manager.

Safety comes from a sandbox and approvals, not from prompts

An agent with shell access can run a destructive command, and telling the model not to do something isn’t a safeguard.

Real protection comes from two layers:

1. A permission system that requires approval for sensitive actions.

2. A sandbox that isolates execution, so even approved commands can’t touch the host machine.

Anthropic uses the same approach. Moving code execution into a sandbox cuts down how often a user needs to approve actions while still protecting the host system.

Sandboxing in CrewAI

Executing code inside a sandbox rather than on the host machine applies that second layer. In this setup, code runs inside E2B, which spins up a fresh VM per session and destroys it afterward.

Shell commands and Python run entirely inside that isolated environment.

from crewai_tools import E2BExecTool, E2BPythonTool
sandbox_tools = [E2BExecTool(), E2BPythonTool()]  # run tests / run code

Human-in-the-loop

Setting human_input=True on a Task pauses the crew after it generates an answer. You review the output, then approve it or send it back for another iteration.

When execution reaches that task, CrewAI waits for your feedback through standard input.

from crewai import Task

task = Task(
    description=(
        "In the working directory ./workspace, {objective}. "
        "Explore the code first, make the change, then run the tests and report."
    ),
    expected_output="A summary of the files changed and the final test output.",
    human_input=True,
)

If your crew runs behind a web app or chat interface rather than a terminal, CrewAI’s webhook-based human-in-the-loop system handles the same review step.

Memory and checkpointing

By default, an agent forgets everything once a run ends. So if you come back tomorrow to fix another bug in the same project, and it starts from zero.

Two mechanisms let an agent carry information across runs, and each serves a different purpose:

• Checkpointing saves the agent’s state during a run, so it can resume after an interruption or continue from the same point along a different path.

• Persistent memory stores facts across separate conversations, including project preferences like “always format the final code before finishing.”

Adding memory

CrewAI provides a unified Memory interface rather than separate short-term, long-term, entity, and external memory types. When saving, it uses an LLM to identify important details, organize them, and make them retrievable later.

Setting memory=True on the crew gives it memory across runs. After each task, CrewAI extracts useful facts from the output and stores them, and in future runs, it retrieves relevant memories and adds them to the task prompt.

from crewai import Crew

crew = Crew(
    agents=[explorer, coder, tester],
    tasks=[task],
    memory=True,
)

All agents in a crew share its memory unless an agent is given its own.

Adding Checkpoints

A checkpoint is a snapshot of an agent’s progress, including its configuration, task state, memory, intermediate results, inputs, and execution history.

By default, CrewAI creates a checkpoint whenever a task finishes, allowing the workflow to resume from that point if it’s interrupted.

Checkpoints can live in one of two built-in stores:

• JsonProvider saves each checkpoint as a separate JSON file, which is easy to read and inspect manually.

• SqliteProvider stores all checkpoints in a single SQLite database, which holds up better under frequent checkpointing and larger workloads.

from crewai import Crew

crew = Crew(
    agents=[explorer, coder, tester],
    tasks=[task],
    checkpoint=True,
)

Crew, Flow, and Agent all accept a checkpoint argument, and children inherit from their parent unless they set their own value.

All six layers on one bug-fix task

Here’s the full harness on one task, with the execution loop, tools, planning, subagents, sandboxing, and memory working together:

from crewai import Agent, Crew, LLM, Process, Task
from crewai.tools import tool
from crewai_tools import (DirectoryReadTool, FileReadTool, FileWriterTool, 
E2BExecTool, E2BPythonTool)

llm = LLM(model="anthropic/claude-sonnet-4.6")

list_dir = DirectoryReadTool(directory="./workspace")
filesystem_tools = [FileReadTool(), FileWriterTool(), list_dir]
sandbox_tools = [exec_tool, E2BPythonTool()]

@tool("run_tests")
def run_tests(path: str = "tests/") -> str:
    """Sync ./workspace into the sandbox, then run pytest there."""
    return E2BExecTool().run(command=sync_and_test_command(path))

explorer = Agent(role="Codebase Explorer", goal="Map repo, surface relevant files.",
    tools=[read_file, list_dir], llm=llm)
coder = Agent(role="Software Engineer", goal="Implement requested change.",
    tools=filesystem_tools, reasoning=True, llm=llm)
tester = Agent(role="Test Runner", goal="Run tests in sandbox, report pass/fail.",
    tools=sandbox_tools + [read_file] + [run_tests], llm=llm)
manager = Agent(role="Engineering Lead", goal="Delegate steps, finish once tests pass.",
    allow_delegation=True, llm=llm)

task = Task(
    description="In ./workspace, {objective}. Explore, edit, test, report.",
    expected_output="Summary of changes and test output.", human_input=True,
)
crew = Crew(
    agents=[explorer, coder, tester], tasks=[task],
    manager_agent=manager, process=Process.hierarchical,
    planning=True, memory=True, checkpoint=True,
)
result = crew.kickoff(inputs={"objective": "fix failing tests in account.py"})

Agent harnesses are easiest to evaluate when success can be checked automatically. A test suite gives the agent a concrete objective, so it can plan, edit, test, and repeat until everything passes.

So this was tested against a small codebase, a BankAccount class with two real bugs and five tests, three of which failed. The rule was to fix only the implementation, not the tests.

This mirrors how Anthropic evaluates coding agents internally. One published example has Claude rebuilding a clone of the claude.ai interface against a large suite of failing tests.

Here, the harness took the project from 3 failing and 2 passing to all 5 passing, with the implementation-only rule closing off the shortcut of editing or removing the failing tests.

Beyond prompts, environment, and tool choice

Some parts of the system aren’t things the framework builds for you:

• The prompts. Each agent’s behavior comes from its role, goal, and backstory. Getting those right takes testing and iteration, and no configuration flag substitutes for it.

• The execution environment. The sandbox, whether E2B or a self-managed VM, has to be set up and wired in.

• Tool selection. Which tools each agent gets, and which agent should have access to what, is a design decision the framework doesn’t make.

There’s also a cost to the harness itself. Planning, subagents, and looping all add API calls, so a complex agent setup can end up more expensive than a task a single model call would have solved directly.

And there’s a longer-term limitation worth keeping in mind. As models improve, some of the scaffolding stops being necessary, because some of what gets built into a harness today is a workaround for today’s model limits rather than a permanent requirement.

Anthropic originally used context resets to keep Claude Sonnet 4.5 from ending tasks too early, and they weren’t needed anymore with the more capable Claude Opus 4.5.

That’s the whole finding. A coding agent’s capability lives mostly in the harness, and an orchestration framework hands you more of that harness than you’d guess.

The loop, planning, delegation, sandboxing, and memory all arrive as configuration, while the prompts, the execution environment, and the tool choices remain yours.

If you want to run this against your own codebase, the CrewAI docs cover every feature used here, and the framework is fully open source.

Check out CrewAI Docs →

Check out CrewAI GitHub (100% Open Source) →

Find the code for this project in this repo →

Good day!

When Claude Opus 4.8 shipped on May 28, it was the most capable model available to the public.

Yesterday, Moonshot AI released Kimi K3, and it beats Opus 4.8 at max effort on four of five major benchmarks: BrowseComp, DeepSWE, GPQA Diamond, FrontierSWE, and HLE with tools.

And this is not just an Opus story. K3 matches Claude Fable 5 and GPT-5.6 Sol on several benchmarks and beats them on some, while its API is priced at roughly a third of Fable 5’s.

K3 is open weights. Moonshot says it will ship all 2.8 trillion parameters on July 27, which would make it the largest open model released so far.

Every closed frontier model is rented intelligence, accessed through an API the provider can reprice, deprecate, or shut off.

Last month proved why rented intelligence comes with risks. A US export directive forced Anthropic to take Fable 5 offline globally for 18 days.

Weights on a disk don’t have that failure mode since they run on private infrastructure, fine-tune on private data, and cannot be switched off by anyone.

Do note that the weights aren’t public yet, so until July 27, this is an API model like any other.

Here’s the official release →

RLHF vs. DPO vs. GRPO in RL

RLHF, DPO and GRPO are often grouped together as variations of the same algorithm.

But they take completely different approaches to teaching a model to behave the way you want, and those differences show up in the training setup, the data you need, and what it costs you at scale.

The visual below summarizes how they work:

Let’s understand this in detail today!

To dive deeper into these topics (with hands-on implementation and understand actual production tradeoffs), we have already covered every bit of it in the Reinforcement Learning Nanodegree:

• Part 1: Foundations of Reinforcement Learning →

• Part 2: MDPs and Value Functions →

• Part 3: Bellman Equations and Dynamic Programming →

• Part 4: Model-Free Learning →

• Part 5: Function Approximation →

• Part 6: Deep RL and DQN →

• Part 7: Policy Gradients and Actor-Critic →

• Part 8: Proximal Policy Optimization →

• Part 9: RLHF →

• Part 10: Verifiable Rewards and GRPO →

• Part 11: Reward signal problem for Agents →

• Part 12: Environments, trajectories, and the training Loop →

RLHF (Reinforcement Learning from Human Feedback) is the simplest formulation, which also went into building ChatGPT.

You sample prompts, run them through the policy, and score the outputs with a reward model. Then you compute advantages, apply a KL penalty to keep the policy close to the reference, and update via PPO.

This requires four live models running simultaneously:

• The policy you’re training

• A frozen reference copy

• A reward model

• And a critic.

This means running four sets of forward passes per training step.

At production scale, this is expensive to run and hard to stabilize. And the reward model needs its own human-preference training before any of this can even begin.

DPO (Direct Preference Optimization) provides a better solution.

Instead of training a separate reward model, DPO derives the reward signal implicitly from the policy itself.

It works from preference pairs, a winner and a loser response to the same prompt. It computes log-probability ratios between the current policy and a frozen reference, and uses that ratio as the implicit reward.

This removes the explicit reward model and the critic entirely, bringing the pipeline down to just policy and frozen reference.

The tradeoff is that you need labeled preference pairs upfront. And since there’s no online exploration, DPO can be brittle if the preference data doesn’t cover the failure modes you care about.

GRPO (Group Relative Policy Optimization), introduced by DeepSeek in 2024, takes a different path.

It keeps the RL framing but eliminates the critic by replacing the value baseline with group statistics.

For each prompt, the model generates a group of outputs, and a verifier scores each one. Advantages are then computed from the mean and standard deviation of those scores within the group.

The critic is a key bottleneck in RLHF since it’s slow to converge, adds instability, and requires a regression head on top of the policy.

GRPO removes all of that by using the group itself as the baseline, keeping the RL loop intact but at lower overhead than PPO.

Here’s a useful way to understand all three at once.

• RLHF uses external scores and an internal critic to measure advantage.

• DPO encodes preference implicitly and skips both.

• GRPO generates its own baseline from a group and keeps the RL loop without the critic.

All three produce aligned models, but they differ on how much machinery the alignment requires.

To dive deeper into these topics (with hands-on implementation and understanding actual production tradeoffs), we have already covered every bit of it in the Reinforcement Learning Nanodegree. Start here →

​Paged Attention in LLMs​

When you’re serving LLMs at scale, memory becomes your bottleneck before compute does.

The problem isn’t a lack of GPU memory; it’s how inefficiently we use it.

Let’s understand why this happens and how Paged Attention solves it elegantly!

KV Cache

During LLM inference, the model stores the key and value vectors of all previously generated tokens in memory. This is the KV cache (​we covered KV caching in detail here​).

Simply put, instead of recomputing attention over all previous tokens every time a new token is generated, the model just looks up the cached values and computes attention with them.

This is crucial for efficiency. Without it, generating a 1000-token response would require recomputing attention for all previous tokens at each step. That’s O(n²) operations instead of O(n) and the speed difference is evident below:

But there’s a problem.

The memory inefficiency problem

Traditional KV cache implementations pre-allocate large, contiguous memory blocks for each request.

Here’s why this is wasteful:

Imagine serving 100 concurrent users. You don’t know how long each response will be, so you pre-allocate space for the maximum possible length, say, 2048 tokens per request. But the average response turns out to be only 200 tokens.

You’ve just reserved 10x the memory you actually need.

And it gets worse. Each request gets its own isolated block. So even if 80 of those 100 users share the same system prompt, you’re storing 80 duplicate copies of the same KV cache entries. None of that memory can be shared or reclaimed.

As a result, you’re often using only 20-30% of your allocated GPU memory effectively. The rest is reserved but unused, and because it’s scattered across fragmented contiguous blocks, you can’t even fit new requests into the gaps.

This is the core inefficiency that kills throughput.

Paged Attention

Paged Attention solves this by borrowing the concept of virtual paging memory from operating systems.

If you’ve studied OS fundamentals, you know that programs don’t get one big contiguous chunk of RAM. Instead, the OS breaks memory into small, fixed-size pages, scatters them anywhere in physical memory, and uses a page table to map each program’s virtual addresses to physical locations.

Paged Attention does the same thing for the KV cache. Here’s how:

Block-level allocation: Instead of reserving one large contiguous block per request, the KV cache is divided into small, fixed-size blocks (typically 16 tokens each). These blocks can live anywhere in GPU memory and don’t necessarily need to be next to each other.

Block table (page table): Each request maintains a block table which is a simple mapping from logical block index to physical block location in GPU memory. When the model needs the KV cache for token 33, it looks up which physical block holds it, fetches it, and computes attention. The LLM doesn’t care where the block is physically sitting.

Shared prefixes: Here’s where it gets really clever. Multiple requests that share the same system prompt don’t need separate copies of the KV cache for that shared prefix. Their block tables simply point to the same physical blocks. Only when their responses diverge do new blocks get allocated for each request individually.

In a production setting, where every request typically shares a system prompt, the memory savings via shared prefix become massive.

Impact

In the original Paged Attention paper, the authors measured that existing systems only utilized 20-38% of the allocated KV cache memory effectively. The rest was wasted due to fragmentation and over-reservation.

Paged Attention can achieve:

• 2-4x higher throughput compared to state-of-the-art systems at the same level of latency

• Near-zero memory waste, as it only allocates memory for what’s actually used

Better batching leads to more concurrent requests on the same GPU hardware

This is why vLLM, which implements Paged Attention as its core algorithm, has become the go-to inference engine for production deployments. Other frameworks like TensorRT-LLM and SGLang have also adopted similar paging mechanisms for faster inference.

This optimization becomes even more crucial when running inference servers at scale, especially with variable request patterns, shared prompts, and cost-per-request constraints.

👉 Over to you: What other OS concepts do you think could be applied to optimize LLM inference?

​To dive deeper into how LLM systems are deployed, start with the practical and hands-on LLMOps course here →

Good day!​

Coding agents collapsed the cost of writing code, but validation capacity stayed as is. Tests still encode only the failure modes someone anticipated, and staging still cannot reproduce production traffic.

AI-generated code makes this worse because it fails on unknown-unknowns. Nobody formed a mental model while writing it, so nobody can predict which inputs will break it.

The same holds for prompt changes, which can pass every offline eval and still degrade for one user segment because eval sets rarely match production traffic.

That leaves production as the only environment where this code gets validated, and it demands telemetry that answers questions nobody predicted.

Dashboards only answer the questions someone thought to set up in advance. Catching an unknown failure means keeping raw, high-cardinality events and slicing them by user ID, prompt version, or any other attribute after the fact.”

This is also why Observability Engineering, the 2022 O’Reilly book that became the standard reference on the topic, just got a near-complete rewrite.

The authors rebuilt it around this exact shift, with 27 net-new chapters on instrumenting LLM apps, feeding production telemetry back into evals, and debugging with agentic AI.

Honeycomb partnered with us on this issue and has made the early-release chapters free to download →

Download book for free!

Grab the free copy today to learn about observability engineering!

Agents need a new kind of web search

Andrej Karpathy once said LLMs are a bit like a coworker with anterograde amnesia. That’s the condition where you keep your old memories but can’t form new ones.

An LLM lives with that condition in two ways. It doesn’t remember you across conversations, and it never learns anything that happened after its training ended.

Memory features exist to patch the first gap. This piece is about the second one, because agents mostly get hired to deal with the present.

Markets move, people change roles, prices update, news breaks. Web search is how the model sees any of it, and how well it works ends up deciding how useful the whole agent is.

So you bolt a search tool onto your agent and assume it can now read the web. Then you check the first few traces, and the result is disappointing.

Nearly all the tokens you paid for went into raw page text the agent had to dig through. Only a thin slice went into answering your question.

The reason is what a search call actually returns. A typical search API hands back links and thirty-word snippets, nothing more.

So the agent isn’t reading the web, it’s reading a table of contents. Fetching the actual pages becomes its job, which means pulling the HTML, stripping the markup, and digging out the usable text before any real work starts.

On a single query, you barely notice the overhead. In a task that takes several searches, like a research run or a briefing pipeline, you pay it again on every single one.

The problem isn’t that agents search badly. It’s that each search call returns pointers to content instead of the content itself, and the agent pays to close that gap on every call.

We ran the same question through three retrieval setups and counted tokens. The most expensive setup costs roughly 4x the cheapest retrieval option.

This piece walks through those numbers and shows what a search response should look like when an agent is the one reading it. It also covers a class of questions that only full documents can answer.

Let’s start with where the cost actually comes from.

Every loop iteration pays a retrieval cost.

That repeated fetch-and-clean work has a name. Call it the retrieval tax, the tokens an agent burns preparing content before it can reason about your problem.

One looping agent is enough to feel it. A research agent searches, reads what came back, decides what to look up next, and searches again, and every pass pays the tax on top of the last one.

Here’s what that costs on a single question. We picked “What was Y2K?” on purpose, because the model already knows the answer from training.

We answered it three ways, straight from memory, through a web search loop, and through an owned index (a search service that crawls and cleans pages ahead of time and stores its own processed copy of the web), counting the total tokens billed each time.

Picking a question the model already knows is what keeps the experiment clean. The thinking cost is identical in all three runs, so any tokens above it come purely from the retrieval.

• From memory, the whole thing took about 600 tokens. That’s the baseline, the cost of just answering with zero retrieval.

• An owned index is a search service that has already crawled and cleaned the pages, so one call returns the full document. Going through one took roughly 6,900 tokens.

• A single web search hop cost about 3,750 tokens. But snippets are often too thin to answer from, so the agent refetches and refines, and three hops climbed to roughly 28,700, more than 4x the owned index.

The gap between the loop and the index is the tax made visible. Every hop re-billed the entire growing context window, so by the third pass, the agent was mostly paying to carry pages it had already read.

The owned index skipped all of that. The full document was already there when the query arrived.

What makes that possible is what comes back from the search call itself, so let’s look at that next.

Good retrieval returns documents, not directions

The fix isn’t a smarter agent or a better prompt. The fix is what comes back from the search call.

• A SERP, the plain list of results a search API returns, hands you URLs and asks the agent to go read the web itself.

• A scraper-first tool gets you closer by returning raw HTML or markdown, but the agent still has to clean it before it can use anything.

• An owned index does all of that work before the query ever arrives, so what comes back is already readable.

Here’s the same lookup across all three. We’ll search for Christoph Molnar, the author of the Interpretable Machine Learning book.

1) SERP

# query
results = search("Christoph Molnar interpretable ML")

# response
[
  {"title": "Interpretable Machine Learning", "url": "christophmolnar.com/books/...", "snippet": "A guide for making black box models explainable..."},
  {"title": "Christoph Molnar - Google Scholar", "url": "scholar.google.com/...", "snippet": "Cited by 12,847..."},
  # 5 more results across different domains
]

The agent got a list of tabs, not an answer.

2) Neural search

# query
results = neural_search("Christoph Molnar", type="person")

# response
[
  {"highlight": "Christoph Molnar is a statistician and machine learning interpretability researcher..."},
  {"highlight": "Author of Interpretable Machine Learning, an open access book with 400k+ readers..."},
  # one stray paragraph about a different researcher
]

It found the right person, but as highlight fragments. Getting the full record still needs another call.

3) Owned index

# query
result = client.search("Christoph Molnar", scope="people")

# response
{
  "name": "Christoph Molnar",
  "roles": [
    {"title": "Research Scientist", "org": "Mindful Modeler", "start": "2021"},
    {"title": "Author", "work": "Interpretable Machine Learning", "editions": 3}
  ],
  "education": [{"degree": "PhD Statistics", "institution": "LMU Munich"}],
  "websites": ["christophmolnar.com", "github.com/christophM"],
  "languages": ["German", "English"]
}

One call returns the complete record, and the agent goes straight to reasoning.

Seltz is a web index built exactly for this. It crawls the web ahead of time, processes every page into a structured document, and returns finished content through a single API call.

It comes with three scopes, and each one returns a different kind of finished document depending on what your agent needs.

• The People Scope returns full structured profiles, with every role and its dates, education, websites, and the organizations someone has worked at.

• The News Scope returns complete article text, date-windowed, so you can pull only what was published in a specific period.

• The Wiki Scope returns clean Wikipedia documents, useful for building context around companies, industries, or topics before going deeper.

One caveat is worth knowing before you build on the people scope. It works best at the director and regional-leader layer.

For the most senior executives, start with open web search and use Seltz to enrich everyone below them.

Some answers only exist across records

Full documents unlock something that goes beyond cutting the retrieval tax.

Some questions don’t live in any single search result. The answer only exists once you read across multiple records and find where they intersect.

Neither a SERP nor a neural search tool can do this on the first pass. They don’t hold enough of each record to join them, so the work lands back on the agent.

Consider a question a GTM team actually runs. You want every target account that hired someone into a data or AI leadership role in the last quarter.

Answering it needs two things together. Full role histories to find who started a new position and when, plus recent news to confirm the timing and surface the trigger event.

You query the people scope to find who moved into a relevant role recently, then cross-reference the news scope for the trigger event.

What comes out is a ranked list of warm accounts with enough context to write a relevant first outreach line for each one.

That answer doesn’t exist anywhere on the web as a page. It only exists once you join the records, and the join only works when you hold the full records to begin with.

Chain discovery and depth, and every iteration gets cheaper

Seltz isn’t a replacement for open web search. It’s the right tool for a specific class of questions.

Open web search handles discovery well. Finding who currently holds a title, confirming something launched last week, reading a page that went live this morning.

Once you know what you’re looking for, Seltz returns the complete record in one hop rather than a headline and a link.

Many pipelines use the same search tool for every query, regardless of what the question needs. Discovery queries go through the same call as deep-dive lookups, and the loop pays the retrieval tax on both.

Chain the two instead, and each loop iteration gets the right retrieval shape for what it’s actually trying to do.

That’s the core argument here. The cost of an agent loop is set less by the model and more by what each search call hands back, and returning finished documents is how you stop paying for the same pages twice.

If you’re building loops that hit these query shapes, you should give Seltz a try.

Get started here →

Stay tuned as we dig more into efficient web search for Agents and share what we learn along the way.

Thanks to Seltz for working with us on this article.

RAG, Agentic RAG, and AI Memory​

RAG was never the end goal.

Memory in AI agents is where everything is heading.

Let’s break down this evolution in the simplest way possible.

RAG (2020-2023):

• Retrieve info once, generate response

• No decision-making, just fetch and answer

• Problem: Often retrieves irrelevant context

Agentic RAG:

• Agent decides *if* retrieval is needed

• Agent picks *which* source to query

• Agent validates *if* results are useful

• Problem: Still read-only, can’t learn from interactions

AI Memory:

• Reads AND writes to external knowledge

• Learns from past conversations

• Remembers user preferences, past context

• Enables true personalization

The mental model is simple:

• RAG: read-only, one-shot

• Agentic RAG: read-only via tool calls

• Agent Memory: read-write via tool calls

Here’s what makes agent memory powerful:

The agent can now “remember” things, like user preferences, past conversations, and important dates. All stored and retrievable for future interactions.

This unlocks something bigger: continual learning.

Instead of being frozen at training time, agents can now accumulate knowledge from every interaction. They improve over time without retraining.

Memory is the bridge between static models and truly adaptive AI systems.

But it’s not as easy as it sounds.

Memory introduces new challenges RAG never had: memory corruption, deciding what to forget, and managing multiple memory types (procedural, episodic, and semantic).

Solving these problems from scratch is hard. If you want to give your agents human-like memory, check out ​Graphiti​, an open-source framework for building real-time knowledge graphs.

​You can find the GitHub repo here →​

Good day!

The move is correct, but using SLMs alone doesn’t imply savings since those models still have to be served. And serving them without wasting the GPU underneath is a separate problem to be solved.

Today, let’s understand why that is the case, what serving many small models well actually takes, and how the SIE open-source inference engine solves it end-to-end.

Let’s begin!

Workflow of a traditional AI pipeline

A production AI system is usually a stack of individual tasks, and each task is handled by the model that happens to be best at it.

For instance, a single request might:

• parse a document or an image into clean text

• embed that text into vectors for retrieval

• rerank the retrieved results by relevance

• extract the specific fields, entities, or labels that matter

• and check the output against a safety policy before sending it to user

Until recently, the default instinct was to use the largest general-purpose model you could and make it do as much of this as possible, paying frontier rates for every call.

That reflex is now reversing toward using small models fine-tuned for each task, which can run the same step at a fraction of the cost since most tasks don’t always need frontier reasoning.

But the catch teams usually miss is that a cheaper model does not imply a cheaper system.

This is because the cost of a model isn't only what it charges per call. It's also the hardware you keep running to answer that call.

A pipeline that replaced one large model with multiple small ones now has multiple models to serve, each occupying a GPU of its own.

Why don’t we fit multiple models on one GPU?

The obvious objection is that we should just put them all on one card.

This will help because no GPU provider bills for the work you do on them. They do not meter FLOPs.

What you rent is a whole GPU for a span of wall-clock time, and the meter runs whether the card is busy or sitting idle.

So every GPU you hold adds another term to the full bill:

Four models on four cards means four terms in that sum. If one card can carry all four models’ traffic, three of those terms disappear. Nothing about the work has changed, and yet the bill drops by 75%.

The objection holds up on the memory front too. A small embedding model, a cross-encoder reranker, and an entity extractor together take a few gigabytes.

An L4 has 24GB, so they can fit with room to spare, and nothing in the hardware stops you.

The reason almost nobody does it is that every layer of the stack, from the serving framework down to the driver, was built around one model owning one GPU.

vLLM and TEI are two common servers.

• TEI is Hugging Face’s embedding and reranking server and takes a single --model-id at launch.

• vLLM’s engine is built around one model as well.

So putting four models on one card requires running four separate server processes on it. Nothing in either tool coordinates those processes, and you must own that job.

In fact, each process claims its memory upfront and cannot see its neighbours.

vLLM’s --gpu-memory-utilization defaults to 0.9. The setting pre-allocates that fraction of the card at startup rather than adjusting it as per demand.

vLLM’s own documentation says this is a per-instance limit that ignores any other vLLM instance on the same GPU, and that running two means setting the value to 0.5 for each of them, by hand.

So the memory split is a constant you compute yourself, before any traffic arrives.

It must cover each model’s peak, because activation memory grows with batch size and sequence length.

Four worst-case slices pinned onto one card is the same over-provisioning you were trying to escape, and now they have been moved inside a single GPU.

Even if you size them optimally and two models spike together later, they will take the whole card down.

For instance, if the extraction model hits an unusually long document and asks for more memory than its slice allows, it does not just fail on its own. Instead, it takes OCR, embedding, and reranking with it.

This explains why one model per GPU has not been a hardware requirement but a path of least resistance through a stack where every tool was built to serve one model well, and none of them was built to share.

Sharing needs a different kind of server. One server that holds all four models, instead of four servers holding one each.

• It owns the card’s memory, so nothing has to be split by hand.

• It sees every request, so it can decide what runs first.

• And it loads a model when a request needs it, then drops it when memory runs short, so a model nobody is calling is not sitting on memory it does not need.

You do not get that by running four servers side by side. This requires an entirely different stack.

The three ways teams try to avoid this

At this point, one common reaction teams follow is to not to run GPUs the naive way. There are three common escape routes, and each one trades the problem for a different one.

• The first is a managed API. Invoke OpenAI or Cohere to avoid thinking about GPUs at all.

  

  • It is a good way to start, but catches fast when you have got scale.

  • The bill increases in a straight line with usage while the work stays routine.

  • You also run whatever the provider hosts, so the model you fine-tuned for your own task usually cannot go there, and your prompts and documents leave your environment on every call.

• The second is serverless GPUs, where you deploy your own model and pay only while it runs.

  

  • This fixes the idle-time problem on paper.

  • The problem is the cold start. A model that has scaled to zero has to pull gigabytes of weights into memory before it can answer, which adds seconds to the first request, and a reranker in the middle of a search cannot spend seconds waking up.

  • You avoid that by keeping instances warm, and now you are paying for idle GPUs again, which is the exact thing we came here to escape.

• The third is the one that is actually right for cost and for control. You self-host the models on GPUs you rent, in your own environment, where the data stays inside your walls, and the bill is a flat hourly rate rather than a per-token meter.

  

  • This is the path most serious teams end up on.

  • So you self-host, which requires you to reach for the standard serving tools. And that is where the infrastructure problem gets specific. Let’s understand more problems about them below.

Problems with standard serving tools in a multi-model pipeline

The tools devs use here were built for a different job than the one you have, and it shows up in two ways.

Start with the fact that each tool serves one shape of model.

• vLLM is built to serve LLMs.

• TEI is built to serve embedding and reranking models.

• Neither one does OCR, vision, or document parsing.

This is not a gap in the models themselves. The models for every one of these tasks already exist and are excellent, like docling and PaddleOCR for documents, SigLIP and Florence for vision, GLiNER for extraction.

What does not exist is a single server that runs all of them.

So when the agent needs to parse a document and embed it and rerank and check a policy, you need vLLM for one part, TEI for another, and some FastAPI wrapper somebody wrote for the rest.

This gives you four servers + four APIs + four deployments, all behind one agent.

Now, notice what this setup does to the GPU by going back to the reason each server holds a whole card.

vLLM will occupy 90% of the GPU at startup. TEI will grab its part. The OCR wrapper also needs something.

Each was written on the assumption that it owns the card, and none of them will give memory back to the others, because none of them knows the others exist.

There is no shared queue across them and no way to say that the live search matters more than the background indexing job. We already walked through what that looks like on one card, and it does not get better when the processes come from four different tools instead of four copies of one.

You moved to small models to save money.

But the discussion so far proves that the savings will only show up if you can put several of them on one GPU.

The standard tools cannot make that happen since they force one model per server, and one server per card, which puts you right back on one GPU per model.

The fragmentation in tooling re-creates the exact waste we were trying to avoid.

What should the serving stack for small models look like

Let’s step back from the tools and understand the things we need the ideal tooling to do.

• It should run every kind of model an agent needs, embeddings, reranking, OCR, vision, extraction, and generation, behind one API. That is the only way to get off four servers and onto one.

• It should fill the GPU instead of padding it. And that turns out to require more than a good scheduler, because to pack requests of different lengths into one pass without wasting compute, the engine has to control the batching and the attention path for every architecture it serves.

• It should load and evict models as traffic moves, keeping the busy ones loaded and letting the idle ones move out, so that a model nobody is calling is not holding memory the way a cold serverless worker or a padded vLLM instance does.

• It should ship the production layer with routing, autoscaling, monitoring, and GPU pools. This is important since a bare runtime like vLLM is only the engine. On its own, it cannot spread load across replicas, add or remove GPUs as traffic changes, or place models across a mixed fleet of cheap and large GPU cards. Today, devs need to write all of that themselves, which explains why self-hosting is more work than it sounds.

• And adding a new model should be a config change, not a redeployment.

All this is hard with months of engineering since different model families do not run the same way underneath.

• A BERT and a Qwen handle positions and attention differently.

• A ColBERT returns a vector for every token.

• A reranker returns a single score and no vector at all.

Building one engine that can hold all of those shapes, and pack any of them into a full batch, is critical work, and it is the reason this did not already exist as an open-source package.

Open-source solution: Superlinked Inference Engine

The solution to all of this is actually implemented in the open-source Superlinked Inference Engine (SIE):

It runs as one cluster inside your own cloud, and it’s built for exactly the kind of pipeline we have been describing, i.e., several small models of different kinds running back to back on shared GPUs.

• It runs every kind of model through one API with four calls, so document parsing, vision, content-safety checks, and open-model generation all run on the same server:

  • encode turns text or images into vectors

  • score reranks a query against a set of documents (returns a number)

  • extract pulls structured fields, entities, or markdown out of a raw document (return span of text)

  • and generate runs an open LLM on a prompt (returns text and token usage)

  • TEI can hand back vectors and relevance scores, but one TEI process serves exactly one model, so embedding plus reranking already means two separate deployments. And TEI does not cover extraction, OCR, vision, or generation, so each of those still needs a server of its own.

  • A single SIE cluster can serve all four shapes in one server.

• It packs those requests without wasting space.

  

  • Even inside a full batch, there is a second kind of waste. Requests come in different lengths, and the simple way to run them together is to pad the short ones out to match the longest, then process the padding as if it were real.

  • SIE batches by compute cost instead of item count. It groups requests of similar length and pads only to the longest item in that batch, never to a global maximum, so almost none of the card’s time goes to padding.

  • One key difference is where the batch gets formed. Standard stacks route each request to a worker first, and every worker batches only the slice of traffic it happens to receive, so queues run unevenly, and mixed sizes pack poorly.

  • SIE inverts this order. The gateway publishes all work into one shared queue, and each worker pulls from that pool and forms a full batch before touching the GPU.

• It loads and evicts models based on traffic:

  

  • A model loads into GPU memory the first time a request needs it, and the least recently used one is dropped when memory runs short, the same idea as a browser cache.

  • This is also the concrete answer to the reserved-but-idle problem. Several public vLLM issues report that models reserve about 80% percent of a card’s memory on load, while utilization during embedding stays under forty percent.

  • Optimized load-and-eviction strategies let you run several models in the whole pipeline.

• It comes with the production layer already built.

  • A gateway sits in front and routes each request to a worker, and when there is a burst, it accepts the request and holds it while more workers start up.

  • The workers scale up under load and back down to zero when traffic drops, so you are not paying for idle GPUs.

  • It spreads work across whatever cards you give it and comes with the dashboards to watch it and the Terraform to deploy it on AWS or GCP, so the same setup runs on your laptop and on a full cluster.

  • It can also run fully offline, with the model weights downloaded ahead of time, for environments where data cannot leave.

• Every model is tuned and ready-to-use.

  

  • With vLLM or Triton, each model has to be tuned by hand, meaning the batch size, memory fraction, precision, maximum sequence length, and attention implementation all set for that specific architecture on that specific GPU.

  • In SIE, every one of the 85+ models in the catalog carries a config file with those values already filled in, so you reference a model by name, and the engine loads it with settings known to work in production.

  • Adding or swapping just requires pointing to a different config rather than starting the tuning over.

Implementation

Below, let’s look at the pipeline that involves three calls to one server.

We'll build a small retrieval pipeline that embeds some text, reranks it, pulls named entity fields out of the context, and then generates the final answer, all through a single SIE server.

This will involve four different model architectures, all hitting one endpoint. The first three calls run on CPU, so you can follow along on a laptop.

The generation step is the one exception, since it runs on SGLang underneath, which needs a GPU.

Start by installing and starting the server. The local extra pulls the CPU runtime, and serve command starts the server on port 8080.

pip install "sie-server[local]"
sie-server serve

It is ready when the health check answers.

curl http://localhost:8080/readyz     # ok

Now, instantiate the client and point it at the running server:

from sie_sdk import SIEClient
from sie_sdk.types import Item

client = SIEClient("http://localhost:8080")

query = "Who issued the invoice and what is the amount due?"
passages = [
    "Invoice INV-4471 was issued by Northwind Logistics on 14 March 2026.",
    "The total amount due is $18,240, payable within 30 days.",
    "The cafeteria menu changes every Monday and Thursday.",
]

Every call in the rest of the pipeline goes through this one object and this one endpoint, no matter which model it hits.

First we invoke encode.

This turns each passage into a dense vector using an embedding model.

docs = client.encode(
    "sentence-transformers/all-MiniLM-L6-v2",
    [Item(text=p) for p in passages],
)

print(docs[0]["dense"].shape)
# (384,)

Passing a list gives you back a list of results, one per passage, and each carries its vector under the dense key.

Next, we invoke score that uses a cross-encoder to read the query against each passage together and returns a relevance score for each one, a different kind of output than the vectors above:

from pprint import pprint

scores = client.score(
    "cross-encoder/ms-marco-MiniLM-L-6-v2",
    Item(text=query),
    [Item(id=str(i), text=p) for i, p in enumerate(passages)],
)
pprint(scores["scores"])
# [{'item_id': '0', 'rank': 0, 'score': 3.009},
#  {'item_id': '1', 'rank': 1, 'score': -1.061},
#  {'item_id': '2', 'rank': 2, 'score': -11.425}]

The scores are cross-encoder logits, so the absolute numbers do not mean much on their own. Instead, the order is important. The invoice line scores above the payment terms, and the cafeteria has the lowest score, as expected.

The results come back already ranked, so the top passage is the best match:

best_id = scores["scores"][0]["item_id"]
best = passages[int(best_id)]
# 'Invoice INV-4471 was issued by Northwind Logistics on 14 March 2026.'

Finally, we invoke extract which runs an extraction model to generate the fields we ask for, returning each as a labelled span with a confidence score.

result = client.extract(
    "urchade/gliner_multi-v2.1",
    Item(text=best),
    labels=["organization", "date", "amount"],
)
print(result["entities"])

[
  {
    'text': 'Northwind Logistics', 'label': 'organization', 
    'score': 0.979, 'start': 31, 'end': 50, 'bbox': None
  },
  {
    'text': '14 March 2026', 'label': 'date',
    'score': 0.951, 'start': 54, 'end': 67, 'bbox': None
  }
]

This is a third architecture that has a different output shape, but is invoked via the same client and the same server.

Finally, we invoke generate to compose the answer from the top-ranked context. This call needs the GPU build of the server, since generation runs on SGLang, which has no CPU path. Everything else about the call is identical.

context = " ".join(
    passages[int(s["item_id"])] for s in scores["scores"][:2]
)

answer = client.generate(
    "Qwen/Qwen3-0.6B",
    f"Answer in one sentence using only this context.\n\n"
    f"Context: {context}\n\nQuestion: {query}",
    max_new_tokens=64,
)

print(answer["text"])
# 'The invoice was issued by Northwind Logistics, and the amount due is $18,240.'

print(answer["usage"])
# {'prompt_tokens': 61, 'completion_tokens': 17, 'total_tokens': 78}

This is a fourth output shape, free text with token usage, produced by a generative model, and it comes back through the same client as the vectors, the scores, and the spans.

That is the whole pipeline involving four model families with four different output types, running under one server and client, with nothing to deploy beyond the single process started at the top.

Each model is pulled from Hugging Face on its first call and stays active for the next request.

Using small, specialized models for the narrow tasks is the right approach, and it is not really controversial. They are accurate enough on the task they are trained for, and they keep your data in your own environment.

But switching to small models does not make inference cheaper by itself. It moves the cost from a per-token bill to the GPUs you rent, and if you put each model on its own GPU, most of that hardware sits idle, and you are paying for it.

The saving only appears when the models share GPUs, and that requires a single engine that can run all of them, because the moment you serve each model with a different tool, each one takes a GPU of its own again.

SIE (open-source) already implements that, and it lets you run the different models an agent needs on one shared cluster, keeps the GPUs busy instead of idle, and includes the routing and autoscaling you would otherwise have to build.

It also plugs into the retrieval stack you already run, with integrations for Chroma, Qdrant, Weaviate, and LanceDB.

You can find the repo here: github.com/superlinked/sie

(don’t forget to star it ⭐️)

👉 Over to you, how many separate models does a single request in your stack call, and how many of them are holding a dedicated GPU right now?

Good day!

We built a Hugging Face fine-tuning studio that lets you fine-tune any LLM directly from Claude:

The app connects to the HF Hub for model and dataset search. It handles chat template formatting for the training data, and lets you configure LoRA rank, quantization, batch size, and learning rate directly from Claude.

Training runs on HF’s GPU infra via AutoTrain.

Once training finishes, you can also chat with your fine-tuned model (or any other LLM on HF) directly from Claude.

The studio’s built with the mcp-use SDK, an open-source full-stack framework to build MCP Apps for Agents.

In mcp-use, any MCP tool can be associated with a UI.

You define a tool handler, create a React component, and the mcp-use framework handles the tool registration, prop mapping between server and widget, bundling, and hot reload during development.

The widgets follow the MCP Apps standard, inspired by OpenAI’s Apps SDK.

You can find the mcp-use GitHub repo here →

And you can find the code for this fine-tuning studio here →

The four types of agent loops

Loop engineering keeps getting talked about as one thing, when it’s actually a choice between four structures, each fitting a different kind of task.

It means designing the system that steers the agent, instead of steering it by hand, move by move.

That system always answers two questions: what starts a run, and what decides the work is done.

In a hand-run session, the human answers both, every single time. Each loop type moves more of that into the system.

Here’s each type, what triggers it, and when to reach for it.

#1) Turn-based loops, triggered by a user prompt.

The agent gathers context, acts, and checks its work inside a single turn. Then a human reviews the output and writes the next prompt.

Use this when requirements are still forming and every output changes what the next prompt should ask for.

#2) Goal-based loops, triggered by a /goal command that carries success criteria and a budget, like “get the homepage Lighthouse score to 90, stop after 5 tries.”

When the agent tries to stop, an evaluator model checks whether the goal is met, and it sends it back to work.

Use this when the outcome is measurable but the path there doesn’t need human attention.

#3) Time-based loops, triggered by a clock.

An interval fires, the agent runs a fixed prompt like “check the PR, fix CI,” then waits for the next tick. The /loop command runs on the local machine, and /schedule moves it to the cloud so it survives a closed laptop.

Use this for recurring work where the task is known in advance and only the timing repeats.

#4) Proactive loops, triggered by an event or schedule with no human present.

A routine watches a channel and spawns a workflow when something needs handling. That workflow runs a triage agent, a fix agent, and a reviewer who adversarially judges the work before the task closes.

Use this for standing responsibilities where nobody can predict what will come in, only that something will.

Each type hands off one more job than the last. Turn-based keeps both jobs with the human, goal-based automates the checking, time-based automates the trigger, and proactive automates both while deciding the workflow shape at runtime.

So the mapping question isn’t which loop is most advanced. It’s whether the task is exploratory, measurable, recurring, or standing.

The more you hand off, the less you monitor yourself.

We wrote a full breakdown on loop engineering.

You can read it here →

NVIDIA researchers built a new transformer variant

One small change to the layers made:

• decoding 1.7x faster

• long-reasoning accuracy up 6.5 points

In a typical transformer architecture, every attention layer computes Q, K, and V.

NVIDIA’s tweak adds a fourth projection, which predicts what the next layer will need.

To understand why they did this, let’s first see what happens in a Transformer architecture during inference right now.

Sparse attention was an attempt to handle long-context inference. Instead of attending to every cached token, modern designs score the KV cache in blocks, keep the top-k, and attend only to those.

That cuts attention compute and bandwidth, but this still leaves us with two problems.

> First, the KV cache still grows with every generated token.

At 100K+ context, it no longer fits in GPU memory and gets offloaded to CPU RAM.

Now every layer must first copy its selected KV blocks from CPU memory back to the GPU. That copy is slow, the GPU sits idle while it waits, and the stall repeats at every layer of every decode step.

> Second, the selection step itself is not free.

Standard selectors score every candidate block with every query head in a GQA group (grouped-query attention, where several query heads share one KV head), then softmax each head’s scores and sum them across the group.

During decode, the sparse attention itself is cheap because there is only one query token.

But the expensive part is deciding which blocks to attend to, and that cost keeps growing with context length.

Both problems trace back to the same design in today’s sparse attention methods, i.e., the attention query drives the block selection.

Selection needs the query vector Q, and Q only exists once its layer is already running. By then, it’s too late to fetch anything early.

The query also drags its multi-head layout into selection, so all that scoring computation runs just to make one top-k decision.

A recent paper from NVIDIA and MIT called SparDA breaks this coupling with one architectural change.

Each layer now emits four projections instead of three:

↳ Q, K, V, and a Forecast.

The Forecast from layer L predicts which KV blocks layer L+1 will need.

Layer L+1’s own query performs the sparse attention over those selected blocks.

This one change fixes both problems.

Since the next layer’s block set is known while the current layer is still computing, the runtime fetches those blocks from CPU memory on a separate CUDA stream.

The copy overlaps with the current layer’s compute, so the GPU no longer waits for it.

And since the Forecast is separate from the attention query, it doesn’t need one score per query head.

SparDA uses one Forecast head per GQA group, which removes the per-query-head scoring loop and skips the softmax step entirely.

DeepSeek did something similar in DSA, where a small indexer picks important tokens instead of the query doing it.

SparDA applies the same idea to blocks and adds the prefetch angle that DSA doesn’t touch.

The cost of the change is small.

The Forecast adds just 33.5M parameters on an 8B model (0.41%), and only those projections are trained, using a KL loss that matches the original selector’s block distribution.

On MiniCPM4.1-8B and NOSA-8B, accuracy matches or beats the sparse baseline, with NOSA-8B gaining +6.5 on long reasoning.

Prefill runs up to 1.25x faster and decode up to 1.7x faster than the sparse offload baseline.

There’s one more benefit.

Because prefetch hides the offload cost, most of the KV cache can live in CPU RAM, and the freed GPU memory fits much bigger batches, pushing decode throughput up to 5.3x over the non-offload sparse baseline.

That said, this lookahead will only pay off during decode with CPU offload. During prefill, all keys already live on the GPU, so the gain there comes purely from the cheaper selection.

Here’s the paper: ​https://arxiv.org/abs/2606.04511​

We wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching.

​Read it here →​

Good day!

Most GPU clouds sell you a VM and hope the abstraction holds. Lightning AI just shipped something that inverts that.

Their AI Cloud is based on the idea that instead of leasing hardware from someone else and building software on top, own the entire stack: GPUs, datacenter fabric, hypervisor, and scheduler as one system.

Why that matters mechanically:

• Topology stops being a guess. Because Lightning controls the hypervisor, the guest sees the real hardware topology instead of a flattened abstraction. NCCL and PyTorch work correctly without manual tuning.

• Multi-node placement becomes deterministic. Because they own the datacenter, the scheduler knows the physical switch fabric in advance and will land jobs on well-connected machines instead of scattered leased capacity.

• Provision has no leased layer sitting between a request and the hardware, so there’s one operator accountable for the machine actually showing up.

The AI Cloud is now available as guaranteed on-demand or spot capacity, on the same platform behind PyTorch Lightning.

Infrastructure ownership determines whether your placement, your performance, and your provisioning are things you can actually reason about, whereas when you rent your infra, you’re hoping someone (or a series of vendors) get it right in their cloud.

Request access to the Lightning Cloud now →

Thanks to Lightning for partnering today!

Agentic RL: Environments, Trajectories, and the Training Loop

Part 12 of the Reinforcement Learning course is available now.

The previous chapters covered GRPO, verifiable rewards, and LLM-as-a-judge. This one extends all of that into a working training loop for an agent that takes multi-step actions with tools.

Read Part 12 of the RL course here →

RL Nanodegree Part 12

It covers:

• What an RL environment means for an LLM agent

• Trajectories as the unit of training

• The environment spectrum from state-changing to static worlds

• Outcome vs. process-informed scoring

• RULER as a packaged judge with prefix deduplication

• The credit assignment problem in long-horizon episodes

• Hands-on: training a SQL agent with ART and RULER on free Colab

Read Part 12 of the RL course here →

RL Nanodegree Part 12

Why care?

Right now, most agents in production are shaped entirely through prompt engineering. The system prompt tells the agent what tools it has, how to reason, and when to stop.

All the intelligence comes from the base model’s pre-existing capabilities and whatever structure the prompt imposes.

That works up to a point, but they cannot reward the behavior you want and penalize the behavior you do not want, across thousands of rollouts, the way RL can.

This chapter covers the complete RL loop.

It defines what an RL environment looks like when the agent is an LLM with tools, formalizes the multi-step trajectory as the training object, introduces RULER as the scoring layer, and runs the full rollout-score-update cycle end to end.

The hands-on section trains a 3B parameter model to answer database questions using only tool calls, on a free Colab GPU, with no hand-written reward function beyond a one-line correctness bonus.

Here’s what we have covered so far:

• Part 1: Foundations of Reinforcement Learning →

• Part 2: MDPs and Value Functions →

• Part 3: Bellman Equations and Dynamic Programming →

• Part 4: Model-Free Learning →

• Part 5: Function Approximation →

• Part 6: Deep RL and DQN →

• Part 7: Policy Gradients and Actor-Critic →

• Part 8: Proximal Policy Optimization →

• Part 9: RLHF →

• Part 10: Verifiable Rewards and GRPO →

• Part 11: Reward signal problem for Agents →

• Part 12: Environments, trajectories, and the training Loop →

👉 Over to you: What topics would you like us to cover in this RL series?

​Skill bundles in Hermes agent​

We found skill bundles to be the most underrated feature in Hermes.

Real workflows need clusters of skills together, not one at a time.

For example, writing code might need a code review skill, a testing skill, and a PR workflow skill.

Every time, you’re loading the same group manually with three slash commands in sequence, or asking the agent in natural language and hoping it picks the right ones.

A skill bundle fixes this with a single YAML file that groups multiple skills under one slash command.

When you invoke it, every listed skill loads at once, plus any custom instruction you’ve baked in.

The image below shows the anatomy of a bundle file and how it expands at invocation with one command that has a shared instruction baked in.

But the design choices underneath are what make it practically useful:

1) Missing skills are skipped, and it’s not fatal.

→ If one skill in the bundle is uninstalled, the rest still load. You don’t lose the whole workflow because of one missing piece.

2) Bundles beat skills on name collisions

→ If a skill and a bundle share a name, the bundle wins since you opted into it.

3) It works everywhere

→ CLI, TUI, dashboard, Telegram, Discord, Slack. one YAML definition, all platforms.

4) No cache invalidation

→ Bundles generate a fresh user message at invocation, the same as individual skill loading, and at no performance cost.

Once your bundles are stable, the natural next step is team sharing.

Bundle YAMLs are just file so you can just put them in a Git repo and have each team member symlink into the ~/.hermes/skill-bundles/ folder.

If you update the repo, everyone gets the update.

This is what turns a personal agent into something a small team can standardize around.

We wrote ​a full deep dive covering Hermes agent’s self-evolving skills, three-tier memory, GEPA optimization, and setting up multiple specialized agents.

​Read it here →​

Good day!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

Transformer Lab is an open-source ML platform that orchestrates GPUs across any cloud and runs any training or eval workflow you define:

• supports LoRA, QLoRA, DPO, ORPO, SIMPO

• use it from a GUI, CLI, or agent skill.

• works with MLX, vLLM, Ollama, and HF Transformers

• one-click convert between HF, GGUF, and MLX

• diffusion training and inference built in

• LLM-as-a-judge evals plus the EleutherAI harness

• submits jobs to Slurm and SkyPilot from the same UI

• runs on Apple Silicon, NVIDIA, and AMD

The same interface a solo researcher uses on a MacBook also drives a 64-GPU cluster. Nothing else does that today.

​Find the GitHub repo here →​ (don’t forget to star it ⭐️)

Transformer Lab GitHub Repo

How LLM routing actually works in production

A tricky LLM interview question:

Your agent runs everything on a frontier LLM, so you add a routing layer that sends several easy calls to a 15x cheaper LLM.

The router is working as expected, but the bill is still the same as before.

Where did the savings go?

On a high-level, model routing works as follows:

• Add a classifier in front of the agent

• Send easy prompts to the cheap model

• Send hard ones to an expensive model

While this looks simple, under the hood, a lot more engineering must go into this layer before you can actually realize any savings.

And the reason is in how agents consume tokens.

A typical agent task is never one LLM call. Instead, one prompt fires several LLM calls in sequence, like planning, tool use, and analyzing results.

Each call sends the accumulated context back through the model, so the input grows every step.

To avoid redundantly processing the same tokens on every call, providers built prompt caching.

Essentially, the input a model has already seen costs about 90% less than fresh input.

The problem is that this cache is specific to a model.

So every time the router switches models mid-task, the accumulated warm cache gets thrown away, and the full context is re-billed at cold rates.

This means a cache-hot expensive model can still cost less than a cache-cold cheap model.

Production routers solve this by making the routing decision once per task instead of once per call.

The first call routes normally, and the chosen model gets pinned to a session ID. Every call after that goes to the same model, so the cache stays warm and the context stays coherent.

When the task changes, the pin resets, and routing starts fresh.

The savings don’t disappear, but rather they move one level up.

Each new task still gets routed by difficulty, so easy tasks land on a cheap model and hard ones on an expensive model.

The pin only stops switching inside a task, which is exactly where switching costs more than it saves.

This is called model affinity, and it’s the last layer of a 4-stage production routing layer that runs on every request.

A) Guardrail filter

The prompt goes through a safety check. Jailbreaks and unsafe inputs get caught before they reach the router, and every call is logged and traced from this point.

B) Router model

A small routing model reads the prompt, infers the domain and action of the request, and matches it against candidate models.

The model has to be tiny, because a routing decision that costs as much as the call it’s routing saves nothing.

C) Selection policy

A cost policy picks the winner from the candidates, preferring the cheapest or the fastest, depending on the config.

Prices are picked from pricing catalogs, so the choice adapts when providers reprice, and the runner-up stays on standby as the fallback if the winner fails.

D) Model affinity

After selection, the winning model gets pinned to the session ID. Every later call in the task goes to it, so the cache never resets and the context never splits.

To see this in practice, the exact pipeline is already implemented in Plano, an open-source proxy that runs locally between your agent and your model providers.

Here’s the GitHub repo →

The config for all four layers is defined in one YAML config file, and swapping any of it never touches agent code.

The routing model in stage 2 uses Arch-Router, a 1.5B model available on HF. It’s trained on human preference data, i.e., what developers actually pick for each task type.

So routing reflects real-world preference instead of benchmark rank, and you define those preferences in plain English in the config.

It also implements an observability console with a per-request cost column, so you can see exactly which model answered each request and what it cost.

Here’s the repo: https://github.com/katanemo/plano

(don’t forget to star it ⭐ )

We put this exact pipeline in front of my Hermes agent, and the usage dropped 2x, without changing a line of agent code.

Here’s the full article we wrote →

Good day!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

MongoDB put together an AI Skill Badges program on MongoDB University with hands-on learning tracks for developers building production-grade AI applications.

Three standout AI Skill Badges you should start with:

1. Auto-embedding AI Models: Learn how to generate vector embeddings directly inside MongoDB without a separate embedding pipeline.

2. Agentic Memory with MongoDB: Design short and long-term memory for AI agents using MongoDB, LangGraph, vector search, and Voyage AI.

3. Voyage AI for Semantic Search and RAG: Build a two-step retrieval pipeline using Voyage AI and MongoDB’s auto-embedding to improve relevance while controlling latency and cost.

Beyond the three, MongoDB University also covers AI Skill Badges like:

↳ AI Data Strategy with MongoDB

↳ Building an App with Code Agents and MongoDB

↳ Vector Search Fundamentals

↳ RAG with MongoDB

↳ AI Agents with MongoDB

Every badge focuses on building working systems, not walkthroughs of abstract concepts.

Start learning for free here →

Thanks to MongoDB for making this public and partnering with us today.

RAG & Fine-tuning in LLMs

If you’re building real-world LLM apps, you can rarely use a model out of the box without adjustments.

Devs typically treat RAG and fine-tuning as interchangeable options, but in reality, they are not.

RAG and fine-tuning solve fundamentally different problems. One controls what the model knows at runtime. The other changes how the model behaves by default.

This visual breaks it down for you:

For RAG, look at the top half of the visual.

RAG operates at inference time. When a user sends a query, the retriever searches your knowledge base (PDFs, vector DBs, APIs, documents), pulls relevant context, and passes it to the LLM along with the query. The model weights never change. You’re giving the LLM a “cheat sheet” at runtime.

Fine-tuning is different. To understand, look at the bottom half of the visual.

It happens offline, before deployment. You train the model on domain-specific data, and the weights actually update. The model now behaves differently by default.

Fine-tuning is for changing how the model behaves. Its tone, vocabulary, response structure, or specialized reasoning patterns.

Two questions guide which one you need:

• How much external knowledge does your task require?

• How much behavioral adaptation do you need?

If you need the model to reference specific documents, product catalogs, or anything that updates frequently, that’s mostly a RAG territory.

If you need the model to adopt internal vocabulary, match a specific writing style, or follow domain-specific reasoning patterns, that’s mostly a fine-tuning territory.

For instance, an LLM might struggle to summarize company meeting transcripts because speakers use internal jargon the model has never seen. Fine-tuning fixes this.

That said, in production systems, you might often need both. A customer support bot might need to pull answers from documentation (RAG) while responding in your brand’s voice (fine-tuning).

The simple takeaway:

• RAG → What should the model know?

• Fine-tuning → How should the model behave

They’re not competing. They’re complementary layers in an LLM stack.

On a side note, we started a beginner-friendly crash course on RAGs recently with implementations, which covers:

• ​​RAG fundamentals​​​​​

• ​RAG evaluation​

• ​​​​​RAG optimization​​​​​

• ​Multimodal RAG​

• ​​​​​Graph RAG​​

• ​​Multivector retrieval using ColBERT​

• ​​​​​RAG over complex real-world docs ft. ColPali

Also, we have covered several LLM fine-tuning with code techniques below:

• Understanding LoRA-derived Techniques for Optimal LLM Fine-tuning.

• Implementing LoRA From Scratch for Fine-tuning LLMs.

• Implementing DoRA (an Improved LoRA) from Scratch.

​ANN search using inverted file index​

kNN performs an exhaustive search, which is inefficient at scale!

Approximate nearest neighbor search algorithms solve this.

The core idea is to narrow down the search space using indexing techniques, which improves the run-time performance.

Inverted file index (IFV) is one of the simplest and intuitive techniques to do this.

Steps for indexing:

• Partition the given data using techniques like k-means.

• Each partition gets a centroid, and each data point gets associated with only one partition.

• A map holds all the data points that belong to a centroid’s partition.

Indexing done!

Here’s how we search.

• First, find the closest centroid to the query:

• Next, find the nearest neighbor among only those data points that belong to the closest centroid’s partition:

This drastically reduces the run-time.

Consider:

• There are N data points.

• Each data point is D-dimensional.

• We create K partitions.

• Lastly, for simplicity, let’s assume that each partition gets equal data points.

In kNN, the query data point is matched to all N data points, which makes the time complexity → O(ND).

In IFV, however, there are two steps:

1. Match to all centroids → O(KD).

2. Find the nearest neighbor in the nearest partition → O(ND/K).

The final time complexity comes out to be the following:

…which is significantly lower than that of kNN.

Assume → N=10M and k=100:

• The search complexity of kNN will be proportional to 10M.

• With IFV, the search complexity will be proportional to 100 + 100k = 100100, which is nearly 100 times faster.

That said, ANN is not always accurate.

If some data points are actually close to the query data point but not in the same partition, they may still get missed:

We willingly accept such trade-offs to reduce latency.

In the deep dives on vector databases (open access), we discussed 4 such techniques, along with an entirely beginner-friendly and thorough discussion on Vector Databases.

Check it out here if you haven’t already: ​A Beginner-friendly and Comprehensive Deep Dive on Vector Databases.

Good day!

The voice agent stack is one of the few areas in AI where the demand is enormous, and the worked examples barely exist.

Speechmatics Academy has open-sourced a collection of runnable examples across batch, real-time, voice, and TTS, each standalone enough to clone a single folder and have a working pipeline in minutes.

Speechmatics Academy GitHub Repo

The integrations have complete loops with LiveKit, Pipecat, Twilio, and VAPI, covering WebRTC capture, turn detection, speaker focus, interruption handling, and function calling.

The use cases it teaches cover production territory, including SRT captioning, call-center topic detection, and HIPAA-friendly medical microbatching with Silero VAD chunking.

While everyone debates voice as the next interface, this is the repo that shows how to actually build for it.

You can find the GitHub repo here → (don’t forget to star it ⭐️)

Thanks to Speechmatics for partnering today!

[Hands-on] How to build an RL environment

Andrej Karpathy summarized the entire history of LLM training in three nouns:

• text

• conversations

• and environments

Pretraining ran on internet text, supervised fine-tuning ran on curated conversations, and the current era of reinforcement learning runs on environments.

OpenAI’s o1 proved that framing by training on math and coding problems with verifiable answers, and DeepSeek-R1 published the recipe openly.

The industry now treats environments as the scarce resource. Anthropic reportedly discussed spending over $1 billion on them in a single year.

Meanwhile, someone is giving them away for free through a Hugging Face-style hub hosting 2,500+ open-source environments.

And today, we’ll use their framework to build one of our own from the ground up.

Before we build anything, let’s understand what an environment actually is. Every RL setup is a loop with four moving parts, and they map onto anything the model interacts with:

• State → what the model sees right now

• Action → the choice it makes from that state

• Reward → a number that says how good that choice was

• Environment → the thing that holds the state, accepts the action, and returns the reward

The hard part of this loop was always the reward, which traditionally required training a separate model on human preferences.

DeepSeek-R1 replaced that entire model with a plain Python function through GRPO (Group Relative Policy Optimization). Score several answers to the same prompt and push the model toward the ones that beat the group average.

Here is an illustration of how GRPO works:

That leaves the environment as the barrier still standing. A reward function can score a final state, but something has to present the task, accept the model’s actions, enforce the rules, and produce that final state in the first place.

Frontier labs guard this part as a proprietary asset, which is why few practitioners have ever seen a working environment from the inside.

That team giving environments away is Prime Intellect, and their library, Verifiers, is the framework we’ll build with (100% open-source).

The design is model-agnostic, the rewards are fully verifiable, and the same skeleton works for any turn-based task you want to train on.

How to read this issue

The goal here is simple. We’re building a complete RL environment, and we’re covering it in a way that stays easy to digest even if this is your first one.

Every idea comes with the code that implements it, kept short enough to read in the flow of the article. The complete, runnable version is shared at the end.

And the game we’ll build around is just the working example. By the time you finish, you’ll be able to take this exact structure and adapt it to your own use case.

The Othello RL env: Why a board game

Othello is a two-player game on an 8x8 grid. You place a disc so it traps opponent discs in a straight line, every trapped disc flips to your color, and whoever owns more discs when the board fills up wins.

That flipping rule is why a single move can swing the score hard, and why corners matter so much (a corner disc can never be flipped back).

Every RL part has a home here. The board is the state, the disc placement is the action, the final result feeds the reward, and the game engine that validates moves, flips discs, and plays the other side is the environment.

The LLM plays Black and a built-in engine plays White. After each Black move, White responds, the updated board goes back to the model, and this repeats until the game ends.

Tech stack

Three tools make this work, each handling one layer:

• Verifiers: the RL framework. It defines the environment, runs the turn loop, and handles evals.

• Lightning AI: an OpenAI-compatible inference API, so the same code calls hosted models like Claude or DeepSeek without provider-specific rewrites.

• vLLM: serves open-weight models locally behind that same OpenAI-compatible endpoint.

The shared interface is what makes the environment model-agnostic. A local Ministral-3B and a hosted GPT-4.1 swap in one line, and you change the model name, nothing else.

The game loop

Here’s what the model sees on each turn:

The board, the score, and the list of valid moves are the entire state. Everything the model knows about the game comes from this text.

It must respond with a <think> section followed by a <move> tag, which forces it to reason through the position before committing to a move.

The environment validates the move against the valid list, applies it to the board, lets White respond, and sends the updated board back. An invalid move gets an error message and a retry, with a penalty applied to the reward.

All of that lives in one method. OthelloEnv inherits from MultiTurnEnv in Verifiers, which handles the loop, turn tracking, and termination, and the base class calls your env_response every time the model sends a move:

class OthelloEnv(MultiTurnEnv):
    def env_response(self, model_output, state):
        move = parse_move(model_output)

        if not is_valid(move, state.board):
            state.penalty += INVALID_MOVE_PENALTY
            return error_message(move), state          # same turn, retry

        state.board = apply_move(state.board, move, player="black")

        if not game_over(state.board):
            white_move = opponent_engine(state.board, state.difficulty)
            state.board = apply_move(state.board, white_move, player="white")

        return render_board(state.board), state

This is the shape of the logic; the real version also handles board parsing and edge cases that the snippet skips. I’ll share the full working code later once we understand the setup end-to-end.

The built-in opponent engine

The built-in engine plays white and has two modes:

• Random picks any legal move. Useful for early training, since the model can win just by making reasonable decisions.

• Minimax simulates future moves, scores each resulting position by corner control, board position, and available moves, and picks the move with the best worst-case outcome. Depth-3 means it looks three moves ahead, enough to set traps and avoid obvious blunders.

A randomness parameter controls how often White ignores its strategy and picks randomly instead. Lower randomness means more consistent, punishing play.

White’s moves are generated from the current board state and a fixed game seed, so the same position always produces the same response. Determinism lets you compare different models under identical conditions.

Here’s how the code for the same looks like:

def opponent_engine(board, randomness, depth):
    if random.random() < randomness:               # random mode
        return random.choice(legal_moves(board))
    best_move = None                                 # minimax mode: try every move,
    best_score = float("-inf")                       # keep the one with the best score
    for move in legal_moves(board):
        score = minimax(apply_move(board, move), depth - 1, my_turn=False)
        if score > best_score:
            best_move, best_score = move, score
    return best_move

The reward functions

At game end, four signals combine into a single reward:

Each one is a function that reads something off the final state and returns a number, and the win signal is the simplest:

def win_reward_func(state):
    result = state.get("result")
    if result == "black":   # the model's color
        return 1.0
    if result == "draw":
        return 0.5
    return 0.0              # loss, or game never finished

The other three follow the same shape, and all four combine into one score:

def total_reward(state):
    return (
        win_loss_score(state)
        + piece_advantage(state)
        + format_compliance(state)
        - invalid_move_penalty(state)
    )

The reason for four signals instead of a single win/loss bit is resolution. Early in training, most games are losses, and to a pure win/loss reward, they all look identical, so the model has nothing to optimize.

• Piece advantage separates a close loss from a blowout, giving the model a gradient before it starts winning.

• Format compliance carries a low weight, so clean formatting never outweighs good play.

• The invalid move penalty is capped, so one broken game can’t drown out everything the model did right.

Every score comes directly from the game state and rules, with no judge model or LLM evaluator involved, so the reward is fully deterministic and reproducible.

Wiring it together

The environment generates games across different starting positions and opponent difficulties, the model plays each one through the loop above, and at game end the four rewards are computed and combined. During evaluation, rewards, token usage, and turn counts are aggregated across all games into a results table.

One function wires everything up, and it’s what the prime eval command calls behind the scenes:

def load_environment(min_random_move_prob, max_random_move_prob, parse_think):
    dataset = generate_games(min_random_move_prob, max_random_move_prob)
    parser = XMLParser(fields=["think", "move"])
    rubric = Rubric(funcs=[...], weights=[1.0, 1.0, 0.2, 1.0])
    return OthelloEnv(dataset, parser, rubric)

What the numbers show

Running an evaluation is one command, with the opponent settings passed as arguments:

prime eval run othello -m openai/gpt-4.1 -n 100 \
  -a '{"min_random_move_prob": 0.0, "max_random_move_prob": 0.0, "minimax_depth": 3}'

Swap the model name to test something else, whether it’s hosted through Lightning AI or running on a local vLLM server. Here’s how two models fared against two opponents:

From evaluation to training

Evaluation tells you where the model struggles, and training fixes it in three stages. The same environment supports all of them:

Data generation: have your strongest model play a few hundred games and save the results. The same eval command writes straight to a dataset:

prime eval run othello -m openai/gpt-4.1 -n 200 \
  --save-to-hf-hub --hf-hub-dataset-name your-username/othello-data

Filter it down to wins and draws before training on it, so you’re not teaching the model your strongest player’s mistakes alongside its format.

Supervised fine-tuning: teach format and valid moves first. The Ministral-3B turn counts make the point directly, since unreliable formatting and illegal moves are noise RL can’t train through.

RL training: this is where strategy improves. Multiple games are played from the same starting position, each is scored with the same reward functions from evaluation, and the model is updated toward the higher-scoring rollouts.

Stripped to its core, that’s the GRPO loop from the top of the article:

for prompt in batch:
    rollouts = [play_game(model, prompt) for _ in range(group_size)]
    rewards = [total_reward(r.final_state) for r in rollouts]
    advantage = rewards - mean(rewards)     # relative to the group
    update_model(model, rollouts, advantage)

Each rollout is scored against the average of its own group, so a move that wins 6 of 10 games from a given position is rewarded more than the weaker attempts beside it.

Adapting this to your own task

Once you remove the Othello specifics and the same MultiTurnEnv gives you a skeleton that fits any turn-based task:

class TaskEnv(MultiTurnEnv):
    def env_response(self, model_output, state):
        action = parse_action(model_output)          # your task's syntax
        if not is_valid(action, state):
            return error_message(action), penalize(state)
        state = apply_action(state, action)          # your task's rules
        if not task_complete(state):
            state = environment_step(state)          # tool, API call, or opponent
        return render_state(state), state            # your task's display

This isn’t specific to games. A coding agent swaps apply_action for running a test suite against generated code, a support agent swaps it for checking whether a tool call retrieved the right record, and a research task swaps it for verifying a claim against a cited source.

Adapting it comes down to four swaps:

• Task logic: your domain’s rules for what counts as a valid action and how the state changes

• Response engine: whatever the model reacts to, from a rule-based simulator to a live API to another model

• Reward functions: keep the pattern (outcome signal, partial credit, format, penalty) and replace the domain logic

• State rendering: whatever the model needs to see, whether that’s a file diff, a conversation transcript, or a tool’s response

The structure underneath stays the same regardless of domain: parse, validate, apply, respond, score. The rubric is the design, and if you get the components right, the training signal takes care of itself.

Try it yourself

All the code, setup instructions, and ready-to-use GPUs to reproduce these results are in the Lightning AI Studio template:

​Build a Custom RL Environment →​

​Check out Lightning AI Inference →​

Good day!

Karpathy defined agentic engineering as the discipline that separates production agent work from vibe coding. The core skills he listed were spec design, eval loops, and security oversight.

The problem is that production agents don’t run on the agent code alone. They need a model provider, a retrieval layer, an eval system, a deployment target, and observability.

And normally, these are scattered across different dashboards, config files, and tools, each with its own setup and learning curve.

The solution to production-grade Agentic Engineering is now actually implemented in Google’s Agents CLI.

It covers the entire lifecycle in a single interface, covering scaffolding, evaluating, and deploying ADK agents.

The setup command injects 7 ADK-specific skills into a coding agent’s context, which lets it handle scaffolding, evals, deployment, and enterprise registration through natural language.

We mapped the whole thing end-to-end in the diagram below. Here is what each layer is doing:

• Agent Orchestration defines the agent using the ADK, and the A2A protocol handles coordination when one agent needs to invoke another. ADK is model-agnostic, so the models cover Gemini, Gemma, and third-party models like Claude, via Model Garden.

• The data layer handles retrieval. Vector Store stores the embeddings for RAG, and the ingestion pipeline loads and chunks docs into it. Monitoring and storage happen alongside it.

• The evaluation layer runs the agent against test scenarios with LLM-as judge scoring. It runs before deployment.

• For deployment, you can choose between Agent Runtime, Cloud Run, etc., depending on how much control you need. Agents CLI configures across all targets.

• Finally, teams that need staging and production environments, the IaC and CI/CD layer handles infrastructure as code and deployment pipelines.

Every one of these layers is a service that a dev would otherwise configure manually.

But Google’s Agents CLI ties them together to make the full agentic engineering lifecycle practical.

Here’s the Agents CLI GitHub repo: https://github.com/google/agents-cli

(don’t forget to star it ⭐ )

Thanks to Google Cloud for partnering today!

Build your own 100% local AI second brain

Andrej Karpathy said:

“There’s room for an incredible new product in the second brain space”

And after that, everyone is suddenly building a second brain.

Karpathy’s LLM wiki pattern went viral, and many people are now hand-wiring Obsidian to Claude Code so an agent maintains their notes for them.

The idea is simple, wherein you stop making the AI re-read raw notes on every question. Let it build a wiki that compounds.

As Karpathy put it, “LLMs don’t get bored, they don’t forget to update a cross-reference (backlinks), and can touch 15 files in one pass.”

But if you start doing it manually, it becomes a project in itself. You wire up the vault, the agents, the schedules, the integrations, etc,

So we sat down with Arjun, who actually built the open-source version of this, and we broke down what it looks like when the whole thing already works out of the box:

It just crossed 15K stars on GitHub.

It’s like Claude’s desktop app but open-source, with two things layered on top:

• A work brain: background agents index your emails, meetings, and notes into a living knowledge graph that updates itself as you work.

• Work surfaces: chat is not the best interface for real work, so you get an email client, a meeting note taker, a browser, and a code mode where you and the AI actually collaborate.

You can bring your existing Obsidian vault, connect Slack, X, and Fireflies, and let it run your day.

Here’s the full breakdown of what we covered in this session:

• 00:00: Intro

• 01:08: What is Roboat (an open source AI co-worker)

• 02:42: The second brain (a knowledge graph of your work)

• 04:01: Bringing your existing Obsidian vault in

• 04:46: Work surfaces

• 05:29: Meetings and automatic note-taking

• 06:53: Connecting Slack, X, and other sources

• 07:55: Background agents that run your day

• 09:24: Code mode (Claude Code and Codex)

• 10:18: Demo: from an email to written code

• 14:28: Guardrails: approvals and agent workspaces

• 17:15: Scheduling agents on a cron

• 18:52: The browser work surface (browser use)

• 20:42: Wrapping up: automating your whole day

• 22:44: Outro

Checkout Rowboat’s GitHub repo: https://github.com/rowboatlabs/rowboat

(don’t forget to star it 🌟)

We recently also wrote an article on the same idea, and we highly recommend reading it as well.

You can read it here →

Good day!

Code review assumes the author understood the codebase. An agent doesn’t carry that assumption.

It writes code that compiles and clears the obvious checks, enough to land a commit.

What’s actually wrong with it surfaces in CI later, after you’re already building on top of it.

Gitar, an AI-native code review and fix offering, now part of Sonar, is built for finding and fixing the issues in the pull request itself.

It reads the change with codebase context, not just the diff, catching bugs a syntax scan would miss.

When it finds something, it writes a patch and runs it against your CI. It doesn’t mark the job done until the build passes.

Sonar calls the full loop the Agent Centric Development Cycle, or AC/DC.

The agent writes, the Sonar Vortex engine verifies inside that same session, and Gitar fixes what’s wrong at the PR stage.

No human has to act on a comment first.

Teams running this combination see 44% fewer outages tied to AI-generated code.

Token usage drops too, by up to 36%, since agents spend less time re-parsing a codebase that isn’t piling up mess.

Start with AI-native code validation with Gitar from Sonar here →

Use AI-native code validation today

Thanks to Sonar for partnering today!

Rethinking KV caching for production inference

Researchers at Stanford studied how AI agents actually spend their inference budgets.

One key finding was that ~62% of what gets sent to an agent on every call is just repeated content, i.e., the same system prompts, tool definitions, and documents, which are fed in again and again.

So every time the agent takes a single step, you hand it everything from scratch, even if it just processed the exact same info one turn ago:

Per-token prices dropped 80% between 2023 and 2026, with GPT-4 class models falling from $30/M to $0.40/M tokens. But agentic workflows consume 5 to 30x more tokens per task than a standard chatbot query, because every step re-sends all that context.

So even though each token got cheaper, the total bill went up, since volume outran the price cuts.

Uber shared a similar story recently. After rolling out Claude Code across their engineering org burned through their entire 2026 AI budget in just 4 months. Gartner now forecasts that 40% of AI agent projects will be cancelled by 2027 because of cost overruns alone.

The industry is optimizing the wrong variable. Making tokens cheaper doesn’t help if most of those tokens shouldn’t exist in the first place.

So, in this article, we’ll learn about a new open-source architecture called LMCache that moves cache management out of the inference engine entirely.

Teams running it are seeing up to 14x faster time-to-first-token, so understanding it now puts you ahead of nearly everyone running inference today.

What happens when we prompt a model in a naive setup

Every time you prompt a model, it runs every token through the attention mechanism. For each token, the model computes a Key vector and a Value vector across every attention layer. These vectors capture how the model understands each token’s relationship to every other token in the context.

This collection of K and V vectors is called the KV cache, and the computation scales quadratically with input length.

One MI300X GPU generates roughly 15 TB of KV cache per day. Most of it gets thrown away after each request.

The KV cache for your system prompt is identical every time you send it. The KV cache for a document you uploaded is identical every time a user asks about it. But the model re-derives that same understanding from scratch, every single time.

Think of it like re-reading a textbook from page 1 every time someone asks a follow-up about chapter 7. You already understood chapters 1 through 6, but you have no way to save and reuse that understanding.

What prefix caching solves and what it doesn’t

The industry noticed the above problem and built a technique called prompt caching to deal with it.

If two consecutive requests share the same opening tokens (the “prefix”), the provider stores the KV cache from the first request and reuses it on the second. The model skips recomputing those tokens and only processes what’s new.

This is incredibly helpful. Anthropic's own implementation gives a 90% cost reduction on cached input tokens. Hit rates of 60 to 85% are achievable for stable workloads. For teams with stable system prompts and tool definitions, this is the single highest-leverage optimization available today.

But prefix caching has a hard ceiling.

The cached portion must be an exact, byte-for-byte prefix of the new request. If you change anything in the cached region (even a single character), it leads to a full cache miss, and it happens in three common scenarios:

• RAG with multiple documents → You cache document A alone and document B alone. If a new query needs both documents, the 2nd document’s cached KV state is invalid since it was computed without awareness of the first document.

• Document order changes → The same three documents appear in different orders across requests. Every permutation is a cache miss, even though the documents themselves are identical.

• Growing conversation history → Each new turn changes the full context after the prefix. Earlier cached states beyond the stable prefix become useless.

Alibaba Cloud’s production data validates these limitations. 10% of KV cache blocks serve 77% of all hits. Most cached content never gets reused because the rigid prefix-matching rule prevents it.

Prefix caching is a meaningful optimization, but it only helps when your context has a long, unchanging beginning, and many real-world workloads don’t look like that.

Another performance bottleneck with caching

Every KV cache library runs inside the inference engine’s process. That means cache operations (storing, loading, moving KV tensors around) and the actual inference computation share the same resources.

They can’t run at the same time, so when the engine is busy managing cache, it stops doing inference, and vice versa.

Google's TurboQuant shows this effect. It is a recent KV cache quantization technique that compresses the cache to 3 bits per value with zero accuracy loss. But when it runs inside the inference engine, it causes 20%+ inference slowdown.

Cache management and inference serving are fundamentally different workloads.

One is I/O-heavy (moving large tensors between GPU, CPU, and storage). The other is compute-heavy (matrix multiplications on GPU).

LMCache and the disaggregated approach

LMCache is an open-source project (10k+ stars) that takes a fundamentally different approach. Instead of running cache management inside the inference engine, it runs as a completely separate process alongside it.

LMCache GitHub Repo

In practice, LMCache connects to the inference engine through shared GPU memory. The engine just tells LMCache “here are the block IDs I need” (tiny messages, almost no data).

All the heavy work of actually moving KV tensors between GPU, CPU, and storage happens inside LMCache’s own process. The inference engine doesn’t even notice it’s happening.

This separation produces three benefits:

• No resource contention → Cache I/O never blocks inference, and inference never blocks cache I/O. The 20% throughput loss from running optimization techniques inside the engine disappears.

• Zero-copy sharing across GPUs → In the traditional setup, sharing cached data between two GPUs requires multiple memory copies. LMCache lets both GPUs read and write the same memory region directly, skipping those copies entirely.

• Multi-tier parallel loading → Cached data can live across GPU memory, CPU RAM, local SSD, and remote storage. Traditional approaches check these one by one, bottlenecking on the slowest tier. LMCache checks all of them simultaneously and streams data from wherever it finds a match, in parallel.

The performance difference is significant. On H200 GPUs with the Qwen3-235B model and 50 concurrent users, LMCache delivers 14x faster time-to-first-token and 4x faster decoding compared to in-process caching. Startup time drops from over 3 minutes to about 30 seconds.

Also, LMCache integrates with all major inference engines (vLLM, SGLang, TensorRT-LLM) and supports both NVIDIA and AMD GPUs.

Solving the prefix problem with CacheBlend

LMCache’s architecture solves the performance side of caching.

But recall another problem we discussed above, where a query needed 2 documents.

The LMCache team’s research paper CacheBlend, which won the EuroSys 2025 Best Paper Award, directly addresses this limitation.

The observation is that in modern transformer models, most tokens primarily attend to their own local context. Only a small fraction of tokens have strong connections across document boundaries.

CacheBlend exploits this by identifying just those few tokens and selectively recomputing only them. Everything else gets reused as-is from the independent caches.

This gives 2 to 4x faster processing for multi-document queries (the kind you see in RAG apps) without any quality loss. Instead of recomputing everything from scratch when documents are combined, CacheBlend recovers the missing cross-document understanding at a fraction of the cost.

Using LMCache in production

LMCache isn’t a research prototype but rather ships with the infrastructure that production teams expect.

• Prometheus and OpenTelemetry integration for tracking cache hit rates and I/O performance.

• Kubernetes operator for deployment

• CLI for debugging and benchmarking.

If the inference engine crashes, LMCache preserves all cached data on CPU and storage, so recovery doesn’t start cold.

If LMCache itself crashes, the inference engine enters a downgrade mode where caching is disabled, but inference continues normally, and it reconnects automatically when the cache process recovers.

Neither failure takes the whole system down.

You can find the LMCache GitHub repo here →

LMCache GitHub Repo

(don’t forget to star it ⭐️)

Good day!

Part 11 of the Reinforcement Learning course is available now.

It tackles what is arguably the hardest open problem in agentic RL right now. GRPO needs a reward for every response. For math and code, a verifier supplies one. But what about the majority of real agent tasks where no answer key exists?

Read Part 11 of the RL course here →

RL Nanodegree Part 11

It covers:

• Why GRPO is agnostic to reward source

• Verifiable vs. non-verifiable tasks

• Why hand-written reward functions break

• Constitutional AI and universal verifiers

• LLM-as-a-judge as the core solution

• Known judge biases and when not to use one

• Hands-on LLM-as-judge scorer on a RAG task

Read Part 11 of the RL course here →

RL Nanodegree Part 11

Why care?

The biggest bottleneck in training agents with RL right now is not the optimization algorithm. GRPO/PPO/DPO all work well.

Instead, the bottleneck is the reward signal.

For math and code, this was solved cleanly since a verifier can check correctness, and that becomes the reward.

But when you move to tasks like RAG, customer support, summarization, or any agent that produces free-form output, there’s no verifier.

And without a usable reward signal, GRPO has nothing to optimize against.

The current best answer is LLM-as-a-judge.

A capable model scores agent outputs relative to each other within each group, and GRPO normalizes those scores into advantages.

Research shows that strong judge models agree with human preferences at over 80%, which is roughly the rate at which two humans agree with each other.

This chapter covers the full landscape of covering hand-written functions and Constitutional AI to LLM judges, along with a hands-on implementation that builds a judge scorer from scratch and maps it into signals GRPO uses.

• Part 1: Foundations of Reinforcement Learning →

• Part 2: MDPs and Value Functions →

• Part 3: Bellman Equations and Dynamic Programming →

• Part 4: Model-Free Learning →

• Part 5: Function Approximation →

• Part 6: Deep RL and DQN →

• Part 7: Policy Gradients and Actor-Critic →

• Part 8: Proximal Policy Optimization →

• Part 9: RLHF →

• Part 10: Verifiable Rewards and GRPO →

• Part 11: Reward signal problem for Agents →

Over to you: What topics would you like us to cover in this RL series?

Good day!

Strands Agents: The open source agent harness SDK

Most SDKs give you tools to build an agent. Strands gives you tools to build an agent harness so you can control what it does, when it does it, and how far it goes.

Build an agent harness. Control it end-to-end.

Get Started →

Thanks to AWS for partnering today!

Build a 3Blue1Brown video generator using Strands

To learn about how to use Strands in practice, continue reading...

The challenge with most agent frameworks today is that they still force devs to hardcode workflows, decision trees, and failure paths.

That works until the agent sees something you did not anticipate.

As a solution, instead of over-engineering orchestration, newer agent systems lean into the reasoning capabilities of modern models and let them drive the workflow.

You define the goal, provide tools, and allow the agent to decide how to get there.

Strands Agents from AWS is a good example of this shift.

It is not about adding more rules or abstractions, but about letting the model plan, act, and recover dynamically.

Today, let’s do a hands-on walkthrough on this to learn how you can build Agents with a model-driven agent loop.

Everything will run 100% locally!

Let’s begin!

Implementation

Start by setting a uv environment and installing the required frameworks:

Make sure you have Ollama installed and its server is running locally.

Now building and running an Agent is as simple as these few lines of code:

You can print metrics, like token usage with result.totalTokens, run-time with result.latencyMs, or run result.get_summary() get a detailed nested JSON summary with tool calls, traces, etc.

While this is a minimal example, notice that there are no system prompts, no routing logic, no hardcoded steps, and no predefined workflow.

It simply defines three things: the model, the tools to use, and the task.

From there, the agent loop takes over.

At the time of Agent invocation, the model decides whether it needs a tool, which tool to use, how to structure the input, and when to stop.

Everything is model-driven.

So instead of encoding decision trees like “if math then calculator” or “if API fails then retry,” the model handles planning and execution dynamically. The workflow emerges at runtime based on the goal and available tools.

Strands Agents shifts agent development away from brittle orchestration and toward letting modern LLMs do what they are already good at: reasoning, planning, and adapting step by step.

Let’s look at a more concrete and real-world example.

We’ll build an MCP-powered Agent that can create 3Blue1Brown-style videos for us by just taking a simple prompt.

Yet again, the process stays similar:

In the above code:

• An MCP client connects to an MCP server. This server has a tool that executes the Manim script generated by the model.

• We extract the tool via list_tools_sync() method and define the Agent with these tools.

• Finally, we execute the Agent!

This produces the following video output:

Notice that we defined the Agent with only tools. No system prompts or workflow rules. The model decides when and how to use Manim based purely on the task.

And a single natural language call that drives the entire flow. The model plans the steps, generates the Manim scene, and invokes the right tool autonomously.

Lastly, if needed, you can also deploy the Agent over the Amazon Bedrock AgentCore Runtime in a few lines of code!

It is a secure, serverless runtime purpose-built for deploying and scaling dynamic AI agents and tools using any open-source framework, including Strands Agents, LangChain, LangGraph, and CrewAI.

Here’s a simple example:

In the above code, we:

• Import the Bedrock AgentCore runtime alongside Strands

• Initialize a BedrockAgentCoreApp that handles scaling, security, and the request lifecycle.

• Define a Strands Agent exactly the same way as before.

• Register a single entrypoint function that receives user input, forwards it to the agent, and returns the response.

This example above shows what makes Strands Agents genuinely interesting. You are not assembling an agent out of prompts, routers, and special cases.

You are defining capabilities and letting the model reason its way through the task, which lowers the barrier to building useful agents while still leaving plenty of room for control, customization, and experimentation.

A minimal setup can already produce surprisingly capable behavior, and as you add better tools or models, the agent naturally becomes more powerful without rewriting the workflow.

👉 Over to you: What would you like us to build next with Strands Agents?

Good day!

Coinbase is already running on that belief. They built a routing layer in front of every request.

LLM routing sits between the app and model providers, reads each prompt, classifies the task, and forwards it to the right model automatically.

Coinbase used it to match each prompt to the right model and default to cheaper options where the task allowed it. Cache hit rate went from 5% to 60% and spend ended up nearly cut in half while usage kept growing.

Plano (here’s the open-source GitHub repo) is an open-source implementation of exactly that routing layer, the one that decides which model answers which request, without your application code ever specifying one.

Plano GitHub Repo

The way it works is that you define routing preferences in plain English, and Plano orchestrator infers the domain and action of each incoming prompt, matches it to the right preference, and picks the model automatically.

Next, let’s set up Plano in front of Hermes Agent, and watch it route requests to different models based on what each prompt actually needs.

Before the setup, let’s understand one key concept.

The problem with routing

When an agent runs a task, it doesn’t send a single request. One user prompt triggers several LLM calls in sequence through tool use, each building on what came before.

With a hardcoded model, that’s never a problem. But once routing is in place, each of those calls gets classified independently.

Consider an agent debugging a function. It makes a planning call, runs a tool, reads the output, then makes another call to analyze what came back.

Call 1 routes to claude-sonnet-4.5 for code generation. Call 4 gets classified as analysis and routes to gpt-4o-mini instead.

Two problems occur when that happens.

• The model on call 4 has no memory of decisions made in call 1.

• And every model switch invalidates the cache, so you’re paying full token rate again from scratch.

Plano solves this with model affinity.

The first call in a session routes normally, and Plano pins that model to a session ID passed in a header.

Every subsequent call with the same ID goes to the same model regardless of how it gets classified.

import uuid
from openai import OpenAI

client = OpenAI(base_url="http://localhost:12000/v1", api_key="EMPTY")
affinity_id = str(uuid.uuid4())

response = client.chat.completions.create(
    messages=messages,
    tools=tools,
    extra_headers={"X-Model-Affinity": affinity_id}
)

The pin lasts 10 minutes by default.

When the task changes, you generate a new ID, and routing starts fresh. For multi-replica deployments, backing it with Redis keeps the pin shared across instances:

routing:
  session_ttl_seconds: 600
  session_cache:
    type: redis
    url: redis://localhost:6379

Model affinity handles consistency within a task. Now let’s build the routing layer using Plano.

Setup

Plano runs as a local proxy on your local machine. The routing config, model providers, and listener settings…everything is defined in one YAML file.

Plano supports three routing methods:

• Model-based: the client specifies an exact model name on every call.

• Alias-based: you define a friendly name like fast-model or reasoning-model that maps to a real model behind it, so your app never hardcodes a provider, and swapping models is one line in the config.

• Preference-aligned: the client specifies no model at all. Plano-Orchestrator infers the domain and action of each prompt and picks the right model automatically based on plain-English descriptions you define.

For alias-based, the config looks like this:

model_aliases:
  fast-model:
    target: gpt-4o-mini
  reasoning-model:
    target: claude-sonnet-4.5

We’ll use preference-aligned routing in this walkthrough.

First, we install Plano. Also, we’re using OpenRouter as our model provider, so one key gives us access to every model we need without managing separate API keys:

pip install planoai
export OPENROUTER_API_KEY=sk-or-...

Note: OpenRouter is one option. Plano works with any OpenAI-compatible endpoint, so you can point it at a self-hosted model running on Ollama or vLLM just by swapping the base_url.

Then we create the config.

The routing preferences tell Plano which tasks go where. For complex reasoning and code, we use claude-sonnet-4.5, for everything routine gpt-4o-mini:

# plano_config.yaml
version: v0.4.0

listeners:
  - name: egress_traffic
    type: model
    address: 0.0.0.0
    port: 12000
    timeout: 30s

model_providers:
  - model: anthropic/claude-sonnet-4.5
    base_url: https://openrouter.ai/api/v1
    access_key: $OPENROUTER_API_KEY

  - model: openai/gpt-4o-mini
    base_url: https://openrouter.ai/api/v1
    access_key: $OPENROUTER_API_KEY
    default: true

routing_preferences:
  - name: code generation
    description: writing new code, implementing functions, or building features from scratch
    models:
      - anthropic/claude-sonnet-4.5

  - name: complex reasoning
    description: deep analysis, architectural decisions, debugging hard problems, multi-step reasoning
    models:
      - anthropic/claude-sonnet-4.5

  - name: general tasks
    description: answering questions, explaining concepts, quick lookups, casual conversation
    models:
      - openai/gpt-4o-mini
tracing:
  random_sampling: 100
  opentracing_grpc_endpoint: http://localhost:4317

Plano reads every incoming prompt and routes it to the right model based on these preferences.

Pointing Hermes at Plano

Hermes connects to any OpenAI-compatible endpoint, so we point it at Plano instead of a provider directly:

First, we install Hermes:

curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash

Then we configure it to use Plano as its endpoint:

hermes model
# Select "Custom endpoint (self-hosted / VLLM / etc.)"
# Enter URL: http://localhost:12000/v1
# Skip API key

From here, every prompt Hermes sends goes to Plano, which classifies it and forwards it to the right model through OpenRouter. Hermes just got a completion back.

We start Plano first, so the proxy is ready before Hermes begins sending requests:

planoai up plano_config.yaml
hermes

Hermes is now sending every request through Plano instead of directly to a provider.

Routing in practice

Now that both Plano and Hermes are up, we send two different kinds of requests.

First, a simple question:

> what does the __init__ method do in Python?

Plano classifies this as “general tasks” and routes it to gpt-4o-mini. Fast, cheap, more than capable enough for a conceptual question.

Then something that actually needs horsepower:

> implement a rate limiter middleware for a FastAPI app using Redis, with per-user and per-endpoint limits

Plano classifies this as “code generation” and routes it to claude-sonnet-4.5.

The Hermes session is hitting the same endpoint, but it’s picking a different model automatically based on what the prompt actually needed.

We wrote zero branching logic to make that happen. The routing decision lived in the YAML, not in the agent.

That covers routing by intent. The next case is when two models can both handle a task equally well, and the only difference is price.

Routing to the cheapest model that qualifies

When two models can both handle a task equally well, we can let Plano pick based on live pricing instead of the order we wrote them in.

routing_preferences:
  - name: code review
    description: reviewing, analyzing, and suggesting improvements to existing code
    models:
      - anthropic/claude-sonnet-4.5
      - openai/gpt-4o
    selection_policy:
      prefer: cheapest

Plano reads a pricing catalog before routing and reorders the candidate list by total token cost.

The cheaper model answers, but the expensive one stays as a fallback if the cheaper one fails.

Plano also ships an observability console, planoai obs, which shows a per-request USD cost column so we can see exactly which model answered each request and what it cost.

Model agility, without the rewrite

Armstrong's team built their routing layer from scratch, inside custom harnesses with engineers assigned to own it.

Most teams are making the same decisions without any of that infrastructure

Plano gives you three things out of the box without writing a single line of routing logic.

• Preference-aligned routing automatically sent complex tasks to claude-sonnet-4.5 and routine ones to gpt-4o-mini.

• Cost-aware selection reached for the cheaper option when two models could both handle the job.

• Model affinity kept the same model pinned across every turn of a multi-step task, so nothing broke mid-session.

All of it is configured in YAML instead of inside your coding agent.

When a cheaper model drops for routine tasks, you update one line. When a better one beats Claude on code, you point the preference at it. The agent never changes.

The routing decision just moves.

Plano’s GitHub repo →

(Don’t forget to star it ⭐️)

Good day!

Getting raw records into clean JSONL, deduped and correctly formatted, routinely takes longer than the training run itself.

Then comes hyperparameter tuning, where a bad configuration runs for a full step before the eval reveals it was bad.

Fireworks Training Agent collapses both into two inputs, a task description in plain English and a raw data upload.

Fine-tune with Fireworks Training Agent

It cleans the data, selects a base model, runs a hyperparameter sweep, generates evaluation criteria, and deploys the result to a live inference endpoint, running on the same infrastructure that Cursor and Vercel use in production.

Get started with Fireworks Training Agent here →

Thanks to Fireworks for partnering today!

Prompt, context, harness & loop engineering

At its core, an agent is a while loop:

• The model runs

• It requests tool calls

• The tool results return to the context

• The model runs again until it stops requesting tools

ReAct described this form of loop back in 2022-23, and almost every agent/framework runs a similar implementation of this (we implemented ReAct from scratch in pure Python here).

But this whole loop wraps four layers of engineering around it:

• Prompt engineering

• Context engineering

• Harness engineering

• Loop engineering

Each one wraps the last, and the model sits in the middle, so none of them compete with the others. Instead, they just zoom one level further out.

Prompt engineering:

This defines the input the model sees on one call, often composed of a role, instructions, examples, and an output format.

The techniques here alter the internal computation and reasoning the model goes through due to the wording it sees:

• Chain-of-thought makes it work in steps before answering

• Few-shot examples define the format and the edge cases

• A JSON schema or XML tags make the output parseable by code

• Self-consistency samples a few chains and takes the majority

Context engineering:

It’s everything the model sees on a turn, not just the prompt. That includes the query, retrieved docs, memory, prior turns, and tool outputs from earlier steps.

The window is finite and fills up fast, so the engineering work is to rank inputs and cut everything that isn’t pulling weight.

You do this by:

• Retrieving only the chunks relevant to the query, then reranking them

• Keeping key facts out of the middle, where accuracy drops

• Summarizing old turns, evict stale outputs, push big blobs to files

Harness engineering:

It’s the code around the model that defines the tools, parses the calls, retries on failure, and can route work to sub-agents so one handles retrieval and another handles code.

A verifier then grades the result by running tests, validating a schema, etc.

Prompt and context involve getting one call right. The harness involves everything that has to happen around that call for it to run in a real system.

Loop engineering:

In the usual setup, you manage the outer loop, i.e, you write a prompt, read the turns the agent runs, write the next prompt, and repeat, while catching failures.

This layer hands that job to the agent itself. It kicks off on a schedule or an event, and runs many turns with no prompt in between.

A loop inherently doesn’t know when it’s finished. An agent can report that it’s done and halt while the tests still fail. So the stop can’t be the agent’s word, but rather it has to be a real signal, like:

• A turn and token cap to stop stuck runs

• A no-progress detector to catch repeated calls

• A completion check to verify the goal with a separate model or a deterministic test

By this layer, you’re operating on the whole run, so the engineering moves from writing each prompt to setting the goal and the stop conditions up front and letting it run.

If you want to dive deeper into loop engineering, we wrote a full breakdown of loop engineering recently.

It goes from the basic while loop to a run that finishes on its own, with the code behind each part, and the parts that are hard to get right, like knowing when to stop, context rot over a long run, and keeping the checker separate from the maker.

Read it here →

11 most important plots in DS/ML

This visual depicts the 11 most important and must-know plots in DS:

Today, let’s understand them briefly and how they are used.

1) KS plot:

• It is used to assess the distributional differences.

• The idea is to measure the maximum distance between the cumulative distribution functions (CDF) of two distributions.

• The lower the maximum distance, the more likely they belong to the same distribution.

2) SHAP plot:

• It summarizes feature importance to a model’s predictions by considering interactions/dependencies between them.

• It is useful in determining how different values (low or high) of a feature affect the overall output.

• We covered model interpretability extensively in our 3-part crash course. Start here: A Crash Course on Model Interpretability →

3) ROC curve:

• It depicts the tradeoff between the true positive rate (good performance) and the false positive rate (bad performance) across different classification thresholds.

• The idea is to balance TPR (good performance) vs. FPR (bad performance).

4) Precision-recall curve:

• It depicts the tradeoff between Precision and Recall across different classification thresholds.

5) QQ plot:

• It assesses the distributional similarity between observed data and theoretical distribution.

• It plots the quantiles of the two distributions against each other.

• Deviations from the straight line indicate a departure from the assumed distribution.

6) Cumulative explained variance plot:

• It is useful in determining the number of dimensions we can reduce our data to while preserving max variance during PCA.

• Read the full article on PCA here for more clarity: Formulating the Principal Component Analysis (PCA) Algorithm From Scratch.

7) Elbow curve:

• The plot helps identify the optimal number of clusters for the k-means algorithm.

• The point of the elbow depicts the ideal number of clusters.

8) Silhouette curve:

• The Elbow curve is often ineffective when you have plenty of clusters.

• Silhouette Curve is a better alternative, as depicted above.

9) Gini-Impurity and entropy:

• They are used to measure the impurity or disorder of a node or split in a decision tree.

• The plot compares Gini impurity and Entropy across different splits.

• This provides insights into the tradeoff between these measures.

10) Bias-variance tradeoff:

• It’s probably the most popular plot on this list.

• It is used to find the right balance between the bias and the variance of a model against complexity.

11) Partial dependency plots:

• Depicts the dependence between target and features.

• A plot between the target and one feature forms → 1-way PDP.

• A plot between the target and two feature forms → 2-way PDP.

• In the leftmost plot, an increase in temperature generally results in a higher target value.

• We covered model interpretability extensively in our 3-part crash course. Start here: A Crash Course on Model Interpretability →

👉 Over to you: Which important plots have I missed here?

External services are peppered throughout your code: talking to APIs, auth providers, payment processors, etc.

Coding agents make it worse, because while the code works, nobody has a clear inventory of what it depends on or whether those dependencies are healthy.

And vendors never report outages fully nor in a quick fashion.

Gloria.dev’s first tool, Canary, solves this by providing you with a global monitoring layer.

It scans the codebase to discover every internal and external dependency, turns each one into an authenticated health check, and runs those checks on a schedule. Outages, error spikes, and cost anomalies surface before they reach users.

The dependency inventory stays current automatically, driven by the code itself rather than a manually maintained list.

And it’s free.

Learn more about Gloria.dev here →

Try Canary today for free

Thanks to the team for partnering today!

How to achieve 2.8x faster automatic speech recognition

Speech recognition systems typically spend more compute processing silence than they spend processing speech. And it’s not something a bigger model fixes.

Consider talking to a voice agent and pausing mid-sentence. “I want to book a... uhh... table for two, around... 7pm?”

A human listener tunes out the silence and the “uhh” instantly, but automatic speech recognition (ASR) systems usually process that dead air piece by piece, the same way they process actual speech.

If voice agents feel laggy, this is part of why.

For a 10-second audio clip, a typical speech decoder runs about 125 sequential calls, most of them spent confirming that nothing new happened.

That decoder, called an RNN-Transducer (RNN-T), is the dominant architecture in production ASR today.

Why RNN-T wastes most of its steps

A frame is a thin slice of audio, just a few milliseconds wide. RNN-T processes these one at a time.

At every frame, a small network called the joint network looks at that slice plus everything predicted so far, and outputs a probability distribution over the vocabulary plus one special token called blank.

If the model picks blank, the frame counter advances by one. If it picks a real token like “the” or “cat,” the token gets added to the output, and the frame counter stays put until the next blank.

So across those 125 steps, most of the calls land on silence, pauses, held phonemes, the “uhh” before “table for two,” stepped through one blank at a time.

The joint network itself is cheap, often just a linear layer. What’s expensive is that you can’t skip ahead.

Each call depends on the previous output, so the calls are strictly sequential and can’t be parallelized.

This compounds on longer audio, where a 60-second call has roughly six times the decoder steps of a 10-second clip, and most of those extra steps are still just blank.

What frame-skipping should look like

A bigger encoder or more training data is not the solution to this, and word error rate (WER) alone is the wrong thing to optimize for, because it says nothing about speed.

A model that’s marginally more accurate but takes 10 seconds to transcribe 1 second of audio is useless in a real-time system.

The fix is letting the decoder predict its own stride, how many frames to skip, not just what token to emit.

If the model already knows a stretch of audio is silence, it should jump past all of it in one step instead of confirming “still nothing” 6 times in a row.

Speechmatics (a real-time speech-to-text API used in production voice agents) actually implements exactly this approach, using their Token-and-Duration Transducer (TDT).

It’s the same architecture behind NVIDIA’s Parakeet TDT models, which led the Huggingface Open ASR Leaderboard on throughput despite using the same encoder size and training data as plain RNN-T models ranked below them on that same leaderboard.

The entire gap comes from this one decoder-level change.

How TDT works

The Token-and-Duration Transducer (TDT) keeps the encoder and prediction history network completely unchanged.

The only modification is a second output head on the joint network.

The original joint network has a single output, a distribution over the vocabulary plus blank. TDT’s joint network has two independent outputs:

• Token distribution: the same as before, predicting which token to emit

• Duration distribution: over a predefined set of values (commonly 0 through 4), predicting how many frames that step covers

So instead of always advancing by exactly 1 frame, the model now predicts its own stride.

If it predicts a duration of 4, the frame counter jumps forward 4 steps. The model learned during training that those 4 frames were going to be blank anyway.

Here’s what the decoding loop looks like:

# TDT Greedy Decoding (simplified)
t = 0
output = []
while t < T:
    token_logits, duration_logits = joint(encoder[t], predictor(output))
    token = argmax(token_logits)
    duration = argmax(duration_logits)

    if token == BLANK:
        t += max(duration, 1)  # blanks must advance at least 1 frame
    else:
        output.append(token)
        t += duration          # tokens can have duration=0 (no frame advance)

Blanks and tokens handle frame advancement differently.

Blanks must advance at least 1 frame, otherwise decoding would stall on silence. Tokens can have duration=0, meaning the model emits a token and stays on the same frame, useful for fast speech where multiple tokens align to a single frame.

Why are the two heads independent?

The obvious alternative is predicting token and duration jointly, but that explodes the output space.

A joint distribution over every token-duration combination would have vocabulary size × number of duration buckets possible outcomes.

Take a vocabulary of 1,000 tokens with a hypothetical 8 duration buckets, that’s 8,000 output classes instead of 1,008.

This leads to a larger output space, which is harder to train, and the added precision rarely helps because token identity and frame coverage are mostly unrelated.

Keeping them independent means each head has a small, clean output space without interfering with the other’s gradients.

Training still uses the same algorithm as RNN-T, with the lattice now allowing multi-frame transitions instead of single-frame ones.

The benchmarks

On English ASR, TDT matches or beats standard RNN-T on word error rate while decoding up to 2.82x faster.

The gains carry across tasks. On speech translation (MUST-C V2), TDT picks up 1.26 BLEU points, a standard translation accuracy metric, over RNN-T while running 2.27x faster.

The gains are largest on longer audio and silence-heavy recordings, exactly where blank-heavy decoding costs the most.

This is why Parakeet TDT models sit at the top of the Huggingface Open ASR Leaderboard on RTFx, the metric for how many seconds of audio a model processes per second of wall-clock time.

Putting it together

The encoder processes the whole audio clip at once, in parallel, before decoding even starts.

That part never changes with TDT.

All the savings are in the decoder, the part that has to run one step at a time. Each joint network call depends on the previous prediction, so those calls can’t be parallelized.

Cutting that call count by 2.82x translates directly to wall-clock time, which is why TDT-based models lead on RTFx while models with marginally better WER sit below them.

The full change from RNN-T to TDT is one extra output head on the joint network, with the encoder and loss structure staying identical.

Speechmatics runs exactly this in production. Their API scored 1.07% pooled WER on the Pipecat voice agent benchmark across 1,000 samples.

You can read the full technical breakdown from Speechmatics here →

AI Agent deployment strategies!​

Deploying AI agents isn’t one-size-fits-all. The architecture you choose can make or break your agent’s performance, cost efficiency, and user experience.

Here are the 4 main deployment patterns you need to know:

1) Batch deployment

You can think of this as a scheduled automation.

• The Agent runs periodically, like a scheduled CLI job.

• Just like any other Agent, it can connect to external context (databases, APIs, or tools), process data in bulk, and store results.

• This typically optimizes for throughput over latency.

• This is best for processing large volumes of data that don’t need immediate responses.

2) Stream deployment

Here, the Agent becomes part of a streaming data pipeline.

• It continuously processes data as it flows through systems.

• Your agent stays active, handling concurrent streams while accessing both streaming storage and backend services as needed.

• Multiple downstream applications can then make use of these processed outputs.

• Best for: Continuous data processing and real-time monitoring

3) Real-Time deployment

This is where Agents act like live backend services.

• The Agent runs behind an API (REST or gRPC).

• When a request arrives, it retrieves any needed context, reasons using the LLM, and responds instantly.

• Load balancers ensure scalability across multiple concurrent requests.

• This is your go-to for chatbots, virtual assistants, and any application where users expect sub-second responses.

4) Edge deployment

The agent runs directly on user devices: mobile phones, smartwatches, and laptops, so no server round-trip is needed.

• The reasoning logic lives inside your mobile, smartwatch, or laptop.

• Sensitive data never leaves the device, improving privacy and security.

• Useful for tasks that need to work offline or maintain user confidentiality.

• Best for: Privacy-first applications and offline functionality

To summarize:

• Batch = Maximum throughput

• Stream = Continuous processing

• Real-Time = Instant interaction

• Edge = Privacy + offline capability

Each pattern serves different needs. The key is matching your deployment strategy to your specific use case, performance requirements, and user expectations.

We’ll have already covered deployment extensively in the LLMOps course:

• ​Read Part 1 on fundamentals of LLMOps here →​

• ​Read Part 2 on understanding the core building blocks of LLMs →​

• ​Read Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixture-of-experts, and the fundamentals of pretraining and fine-tuning →​

• ​Read Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applications →​

• ​Read Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and different prompting techniques →​

• ​Read Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more →​

• ​Read Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techniques for building high-signal inputs →​

• ​Read Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic context injection, and common failure modes in agentic applications →​

• ​Read Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong understanding of the fundamental concepts →​

• ​Read Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for evaluation of LLM apps →​

• ​Read Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming →​

• ​Read Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO →​

• ​Read Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model parallelism →​

• ​Read Part 14 on the fundamentals of LLM serving, including API-based access, inference with vLLM, and practical decisions.​

👉 Over to you: What deployment pattern are you using for your AI agents? Or are you combining multiple approaches?

Good day!

Matthew Gallagher proved it by starting Medvi, an AI-run telehealth company, with $20,000 and no employees. It made $401 million in revenue in its first year.

Gallagher's stack was AI agents talking to each other. When two couldn't talk, he built more agents to handle the coordination. He tested the prompts himself and stepped in when something broke.

Alook (GitHub Repo) is an open-source, self-hosted platform that turns your coding agents into a real org chart.

Alook GitHub Repo

The way it works is that you create an agent, assign it a role, and give it a real email inbox.

Agents email each other and update you. They run as actual Claude Code or OpenCode sessions on your machine, with full access to your tools.

An org chart gives every agent a defined role and a reporting line. They coordinate with each other without you relaying a single message or fixing the wiring yourself.

Let’s set up Alook from scratch, build a four-agent org chart, and watch what happens once a real job gets handed to it.

Setup

Alook runs as a daemon on your own machine, and this one command connects it:

npx @alook/app onboard

It detects whatever coding agent runtime is already installed, Claude Code or OpenCode, and deploys the agent company.

This opens a local dashboard at http://localhost:15210.

From there, you start from a blank org chart or one of Alook’s pre-built templates, whichever’s closest to what you’re building.

Each agent on the chart is a real Claude Code or OpenCode session running on your machine, with full access to the same tools you use, and a real @alook.ai inbox.

The inbox is the coordination layer. Agents email each other, the same way a team would, instead of passing data through a trigger you wired by hand.

Building the company

Competitive intelligence usually means someone checking a pricing page, copying numbers into a spreadsheet, and doing it again tomorrow.

We’re going to replace that with four agents who build a price tracker, run it on a schedule, and email you the moment something changes.

First, we create the agents one by one, assigning them distinct roles and claiming their real @alook.ai inboxes:

• Atlas (CEO) is the single point of contact for the human. It delegates tasks to Mara.

• Mara (PM) turns Atlas's briefs into specs and routes them to Theo or Ren. She's the sole router on the chart.

• Theo (engineer) builds and maintains the scrapers for competitive intelligence.

• Ren (Ops and Customer-facing) notifies the human when a tracked change is detected.

Once the agents are live, we wire the reporting hierarchy: Atlas to Mara, and Mara to Theo and Ren.

Theo and Ren never talk to each other, or to Atlas directly. They only go through Mara.

This setup avoids building a chaotic AI group chat where every agent talks over each other and loses context.

Theo’s job touches the competitor’s site tracks, so he needs a way to scrape it reliably and schedule.

So we have given it access to the Bright Data CLI, which lets it scrape any website, provision custom scrapers if needed…all while avoiding IP blocks and CAPTCHAs that hit any agent scraping at real scale.

To build a custom scraper over any website, you can describe the page in plain English, and it builds the scraper that returns structured data:

Now let’s see how our AI company handles the real task.

Running the company

As mentioned above, we don't need to manage every agent ourselves. Instead, we just talk to Atlas, the CEO, and let the org chart handle the rest.

We ask him to track pricing on railway.app/pricing.

Atlas replies in the chat, and behind that reply, he's briefing Mara over email, and the thread shows up in the same window:

Mara turns the brief into a spec and hands it to Theo.

The spec covers a scraper for the page, timestamped snapshots, change detection, a daily run, and a report we can actually read.

Theo acknowledges the spec over email, the same way Mara and Atlas did, then builds it using the Bright Data CLI and reports back once it's running.

This is the scraper Theo just built, sitting in Bright Data's dashboard:

It's a real custom scraper that the agent provisioned itself by understanding the specified website, not a one-off CLI call that disappears after running. You can trigger it manually from this same screen, or call it straight using the API:

Mara relays the build-up to Atlas, and Atlas informs us in the same chat:

The company runs without you

Once Theo confirms the build is live, the job isn’t done. The schedule still needs to run, and someone still needs to watch what it finds.

The agent adds the scraper to the company calendar as a recurring 9am task, on its own.

This is Ren’s job. He watches the tracker’s output, and the moment the price on the page actually changes, he sends a note.

The whole loop runs unattended.

We handed Atlas one brief, and the org provisioned a scraper, scheduled it, and kept someone watching the output, with no further input from us.

Try it yourself

Every agent in this walkthrough ran as Claude Code, but Codex and OpenCode work the same way, as Alook lets you bring your own agent and gives whichever one you pick a role, an inbox, and a runtime that stays on.

Every completed task builds context for the next one, so the agents aren’t relearning the company from scratch on every run.

Every email between them gets logged the same way, so you can read back exactly how a decision got made.

Here’s the Alook’s Github repo →

And here’s the Bright Data CLI→

(don’t forget to star them 🌟)

Good day!

Teams usually validate their agents’ outputs by calling a frontier model as the judge. It works, but three problems stack up fast:

1. Cost: you’re hitting a frontier API on every turn, every tool call, every response. In production that burns millions.

2. Latency: bigger models, remote calls, slow reasoning on every check.

3. Blind spots: frontier models don’t actually know your domain. In finance, insurance, or healthcare, they miss the keywords and principles your work depends on.

In the video above, we walk through a different approach, which is to train your own small LLM judge.

Instead of a giant model, you start with a small one and let the system generate the training data for you. It decomposes your domain, samples synthetic examples, runs them through a debate arena where judges reach consensus, and then trains on the refined set.