• How to query billion+ rows on postgres without overhead.

• ​Cross-model KV cache transfer in LLM families​.

How to query billion+ rows on postgres without the overhead

Robert Cepa’s team at Cloudflare spent two years patching Postgres before they found a database that cut their query times by up to 35x.

Plain Postgres held up fine until the dataset hit billions of rows, and querying across longer time windows became the bottleneck.

They manually built precomputed aggregates with cron jobs, then evaluated ClickHouse, which required a full ingestion pipeline just to handle their write pattern cleanly.

The full Cloudflare case study is available here if you want to learn more →

Most teams face the same bottlenecks and end up spending months on partition management, aggregation pipelines, and retention infrastructure before the actual product gets any attention.

Tiger Cloud is built for exactly that layer. It is a managed TimescaleDB service, the same technology Cloudflare moved to, built on the Postgres you already know, with automatic time-based partitioning, continuous aggregates, and compression built in from the start.

To show how this feels in practice, I built a real-time earthquake intelligence dashboard on a 3D globe using Claude Code and Tiger Cloud in a single session.

But before that, let’s understand the problem.

The problem with time-series data

Most applications that deal with timestamped data start the same way.

You create a table, add a timestamp column, and plain Postgres handles it fine.

When the table grows into hundreds of millions of rows and queries start filtering across long time windows, Postgres slows down.

Postgres does have time-based partitioning, but nothing creates or retires those partitions for you, so splitting a growing table by day or month is a system you have to build and maintain yourself. The more data accumulates, the worse queries over longer time windows get.

Adding indexes on the timestamp column helps initially, but as the table keeps growing, those indexes become expensive to maintain, and the query planner starts making poor choices.

The cost of building it yourself

Most teams end up manually splitting data into child tables by day or month.

That means building tooling to create new partitions on schedule, handle edge cases at boundaries, and drop old ones when retention policies kick in.

Dashboards need event counts per hour, weekly averages, and trends across months. Computing those on raw data gets expensive fast, so teams build precomputed aggregate tables and refresh them with cron jobs.

Every schema change after that means updating the cron logic and coordinating across teams.

Cloudflare spent two years in exactly that cycle before the data outgrew it, and they had to rebuild the layer underneath.

How Tiger Cloud handles it

Tiger Cloud takes a different approach entirely.

TimescaleDB partitions data by timestamp automatically through hypertables.

Converting any Postgres table to a hypertable takes a single function call, and time-based partitioning runs from that point forward.

Queries hit only the chunks relevant to the time window being filtered, regardless of how large the full dataset gets.

Continuous aggregates replace the cron job layer. You define a rollup query once, and TimescaleDB refreshes it incrementally in the background, including the most recent data not yet rolled up, with no separate pipeline to maintain.

Tiger Cloud provisions all of this as a managed service. You connect through the standard Postgres wire protocol, and the infrastructure is ready from day one.

Setup

Tiger CLI is the bridge between Claude Code and Tiger Cloud. It runs as an MCP server that gives Claude Code direct access to your database infrastructure.

Start by signing up at tigerdata.com and creating an account.

Then install Tiger CLI with one command.

curl -fsSL https://cli.tigerdata.com | sh

Once installed, authenticate with your Tiger Cloud account. This opens a browser window, and the CLI stores your session locally after you log in.

tiger auth login

Now, connect Tiger CLI to Claude Code as an MCP server.

tiger mcp install claude-code

Restart Claude Code after that.

Tiger Cloud shows up as a connected tool in the session, and Claude Code can now provision databases, create hypertables, and run SQL directly against your Tiger Cloud instance without leaving the conversation.

Building the dashboard

I gave Claude Code a single prompt describing the full build and let it run.

Build a real-time earthquake intelligence dashboard on a 3D globe.
TimescaleDB backend on Tiger Cloud via Tiger MCP, hypertable for
earthquake events with event_time as the time dimension, continuous
aggregates for hourly counts and average magnitude per region, seeded
from the USGS catalog for all magnitude 4.0+ events from 1900 to
today. Next.js and Three.js frontend, dark theme, ripple animations
sized by magnitude and colored by depth, time slider from 1900 to today,
magnitude filter, side panel with event count and top quakes, popup on click.

The video below shows the final build in action.

Claude Code connected to Tiger Cloud through the Tiger CLI MCP server, provisioned the database, pulled the full USGS catalog, and assembled the complete frontend in the same session.

The globe plots every earthquake as a ripple animation sized by magnitude and colored by depth. The Ring of Fire, the seismic belt circling the Pacific, lights up naturally from the data.

Every slider position fires a live query.

Dragging it across 120 years of history, the hypertable partitioning means each query hits only the relevant time chunk, and the side panel pulls from continuous aggregates, so both update fast regardless of how far back you go.

The takeaway

Cloudflare spent two years getting to the infrastructure that Tiger Cloud provisions on day one.

The partition management, aggregate refresh, and retention tooling are all there from the start. The database layer should not be a project in itself.

Tiger CLI is open source under Apache 2.0 and works with Claude Code, Cursor, Codex, Gemini CLI, and VS Code.

→ Sign up for Tiger Cloud. New accounts get $1,000 in free credits, no credit card required.

→ Install Tiger CLI:

curl -fsSL https://cli.tigerdata.com | sh

→ Connect to Claude Code:

tiger mcp install claude-code

→ Give Claude Code a prompt and let it build.

Tiger Cloud uses the Postgres you already know, built for real-time analytics and time-series workloads, with autoscaling reads and writes and deep observability built in.

Get started with Tiger Cloud here →

Also, the full Cloudflare case study is here if you want to learn more →

Thanks to Tiger Data for working with us today!

Cross-model KV cache transfer in LLM Families

NVIDIA released a paper in which they shared a method to make the KV cache transferable between models.

The target model skips prefill entirely, and the conversion runs 2.7 to 25x faster than processing the context again.

Let’s understand why this is so important today.

During LLM token generation, every turn sends the entire conversation back to the model. The model reads all of it again before writing a single new token, and all of it is billed as input.

Prompt caching allows Anthropic and other providers to hold the KV cache for a stable prefix and bill a hit at roughly 10% of the base input rate, because the compute was already done once.

The 90% reduction is one of the largest levers in LLM serving, which is why so much production work goes into keeping prefixes byte-stable.

But the cache only works on the model that produced it. Keys and values are produced from that model’s weights, so no other model can read them.

In practice, the constraint shows up in LLM routing. If the traffic is shifted to a different model for cost/capability reasons, the accumulated KV cache becomes invalid.

As a result, the accumulated context has to be processed from scratch, and it’s billed at full rate.

NVIDIA’s recent paper treats this as a representation problem.

Prefill’s only output is the KV cache, so to move KV between models, we need to convert one model’s cache into the format the other expects.

They first checked whether the conversion has any structure worth exploiting.

They found that moving from Qwen3 14B to 32B, a plain linear regression from a single source layer reconstructed 56% of the variance in the target model’s keys.

The two models obviously may have different layer counts, so there is no natural one-to-one pairing between them.

For each target layer, they rank every source layer by how well it predicts that layer, then feed the top eight in together, which takes the reconstruction to 79%.

The mapper itself has three parts:

> Each target layer and head gets its own independent linear map, solved in one closed-form step rather than by gradient descent.

> The cross-layer selection described above is the second part, and their ablation shows it carries the most weight of the three.

> Keys also carry a position-dependent rotation from RoPE. They strip that rotation, fit the map in position-free space, then reapply the target model’s rotation at inference.

Across six pairs from Qwen3, Llama 3.1, and Ministral 3, four retain 73 to 98% of the receiving model’s standalone accuracy, and the conversion runs 3-25x faster than processing the context again.

Prior work on cross-model KV reuse exists, but it either trains a neural adapter per pair or requires both models to be architecturally identical.

This is probably the first version that is closed-form and training-free, so a lot of it is still open research.

Every pair tested belongs to one family, so it works on Qwen to Qwen and Llama to Llama.

Cross-family transfer is listed as future work.

All six pairs mentioned above also happen to share the KV head count and per-head dimension across scales. Mismatched head configurations are currently untested.

The researchers scoped this to dense full-attention only, so sliding-window and attention-recurrent hybrids still need work.

Here’s the paper: https://arxiv.org/abs/2608.03893

Plenty of work is yet to be done. Still, the constraint being solved is genuine.

Every model swap currently invalidates the full KV that was already paid for, and this is the first result showing that work might be recoverable without training anything extra.

To dive deeper into KV cache management specifically for running LLMs in production, we wrote a full 50 min deep dive that covers:

• Why prefill dominates RAG latency, not retrieval

• How the KV cache works and what it costs in memory

• Why prefix caching approaches zero hit rate for RAG workloads

• Three independent failures when you try to reuse cached chunks

• Six published approaches to fix them

• Hands-on implementation that covers every problem and fixes

• Best practices for production to assess what applies to your system

Read the full deep dive here →

Production RAG Deep Dive

Good day!

• ReAct vs Plan-and-Act pattern in agents.

• A 10-week roadmap to run LLMs in production.

• Phases of ML modeling.

ReAct vs Plan-and-Act Pattern in Agents

Every wrong turn a ReAct agent takes stays in its context for the rest of the run.

For more context, the ReAct pattern runs one model in a single loop.

The model writes a thought about what to do next, takes one action, reads the observation that comes back, appends all three to the same prompt, and repeats until it decides the task is done.

Nothing ever leaves the prompt.

So a failed search from step three is still retained and accessible at subsequent steps, competing with the original objective for the model’s attention.

Plan-and-Act splits the overall loop into two jobs.

The authors of the Plan-and-Act paper experimented on web navigation, so the observation coming back after every action is the raw HTML of the page the agent is currently on.

• A planner reads the user query and the initial page and writes high-level steps.

• An executor reads the plan, the task, its own past actions, and the current HTML, then emits one grounded action.

After each action, the executor strips the HTML it no longer needs before taking the next one, so the execution context does not grow the way a ReAct trace does.

Whether this is helpful is determined by plan granularity.

A good step covers one unit of work, like searching for the product in the search box.

An individual click is too small to be a step, and “analyze the search results” is not a step at all, because it pushes the reasoning back onto the executor.

A step must also name the actual values it needs.

The paper’s planner instructions ask for “input New York as the arrival city” instead of “input the arrival city”, because the second version leaves the executor to guess which city goes in the box.

The executor’s job is picking the right element and typing into it, not filling in the blanks the planner left open.

They also found that a badly trained planner makes things worse than no planner at all.

On WebArena-Lite, a ReAct-style executor with no planner scored 36.97%.

But the same executor with a naively finetuned planner scored just 20.60%.

The planner had never seen those sites, so it wrote steps that read fine but matched nothing on the page, and the executor followed them anyway.

A properly trained planner reached 43.63%.

A plan written once and never revised has its own problem. For instance, if the search for “library at CMU” gives no results, the executor will still hold a step that cannot work anymore, and it will keep trying it anyway.

Replanning after every action is necessary to recover from that.

For instance, in the paper, the planner saw the current state, the previous plans, and the actions taken, and rewrote the step to “libraries near CMU”, which increased the score to 53.94%.

As a result, the failed attempt got replaced in the plan rather than accumulated in the context.

The cost is one planner call per executor step. The authors flag this directly and suggest letting the executor decide when a replan is required.

Nearly all of this is harness design rather than model choice.

You can read the paper here → https://arxiv.org/pdf/2503.09572

A 10-week Roadmap to Run LLMs in Production

As an AI engineer, please learn:

• Learn the roofline model and why decode is memory-bound

• Deploy vLLM and SGLang, then read their schedulers

• Understand paged attention from the code, not the blog post

• Build observability before you optimize anything

• Track TTFT, inter-token latency, throughput, queue depth

• Use Grafana + Prometheus for inference dashboards

• Turn on prefix caching and find which workloads it helps

• Learn continuous batching and chunked prefill

• Run load tests with 1000+ concurrent requests

• Report p50, p95, p99, never just the mean

• Master quantization tradeoffs (FP8, INT4, AWQ, GPTQ)

• Learn speculative decoding and where it stops helping

• Set up KV cache eviction for long contexts

• Try disaggregated prefill and decode serving

• Learn Kubernetes for AI workloads and autoscale on queue depth

• Learn how inference costs break unit economics

• Build your own model router based on cost, latency, and quality

• Create a token budgeting system per request

• Build one inference service and benchmark it publicly

• Read inference research instead of model release news

• Start sharing your optimization benchmarks

We put together a 10-week plan that covers every one of these at 30 minutes a day.

It has 50 sessions, split between reading the theory and building on your own service, and all of them feed one artifact: an inference service you deploy, instrument, load test past 1000 concurrent requests, tune, and publish as a reproducible benchmark.

It is open on GitHub, and contributions are welcome, especially newer sources worth adding.

GitHub repo: http://github.com/patchy631/time-to-first-token

(don’t forget to star it 🌟)

​​Phases of ML Modeling​​

Most ML systems don’t jump straight to deep learning. They evolve.

A practical way to think about this evolution is in phases, starting from the simplest possible solution and gradually increasing complexity only when it’s justified. Because unnecessary complexity = low utility.

A staged approach reduces risk, improves debuggability, and aligns naturally with MLOps best practices.

Now, let’s walk through the different phases of ML model development:

Phase 1: Before ML (heuristics and rules):

If you’re solving a problem for the first time, resist the urge to start with a model. Begin with a non-ML baseline: a rule, a heuristic, or a simple deterministic strategy.

For example, in a movie recommendation system, a phase-1 solution could be as simple as recommending the top-10 most popular movies to every user.

This might sound naive, but such heuristics are often surprisingly strong.

These baselines are fast to build, easy to reason about, and set a minimum performance bar. If a complex ML model cannot beat a naive heuristic, something is wrong; either ML isn’t adding value, or there’s a bug in the pipeline.

Here is a conceptual diagram summarizing Phase 1:

Conceptually, Phase 1 looks like a direct mapping from input to output using rules, without any learning component.

Phase 2: The simplest ML model

Once a heuristic baseline exists (or once it’s clear that heuristics aren’t enough), the next step is not a deep model.

It’s the simplest possible ML model.

Think logistic regression, a shallow decision tree, k-nearest neighbors, or a basic linear model; something easy to train, interpret, and deploy.

The goal here is not peak accuracy. This phase answers foundational questions:

• Can we train on historical data and get sensible predictions?

• Are the features informative?

• Does the model generalize better than the heuristic?

This is where you validate the end-to-end ML pipeline: data ingestion, feature extraction, training, evaluation, and serving.

Conceptually, Phase 2 introduces learning, but keeps the model and serving logic minimal.

Phase 3: Optimizing the simple model:

Once the basic model works, there’s often significant performance left on the table, without changing the model class at all.

Phase 3 focuses on extracting as much value as possible from the existing approach.

Typical levers include:

• Feature engineering: creating better representations of the input data.

• Hyperparameter tuning: systematically searching for better configurations.

• More data: expanding the dataset or improving data quality.

This phase is where returns on investment are often highest.

You’re working with models that are easy to understand, cheap to train, and simple to serve, while still achieving meaningful gains.

Many real-world ML systems stop here. A well-tuned logistic regression, gradient boosted tree, or modest ensemble can meet production requirements without the complexity of deep learning.

Here’s the entire thing summarized as a diagram:

Hence, phase 3 overall looks like a refinement loop around the same model family, not a shift in paradigm.

Phase 4: Complex models:

Only after simpler approaches are exhausted should you move to fundamentally more complex models.

This includes deep neural networks, transformers, or large pretrained architectures, depending on the domain.

Complex models bring capacity, but also cost. The decision to enter Phase 4 should be evidence-driven.

Conceptually, Phase 4 introduces higher model expressiveness alongside increased engineering complexity.

A key point to keep in mind is that, at every phase, the previous phase’s best model becomes the baseline.

This phased approach encourages incremental progress and disciplined decision-making.

If you want to learn more about these real-world ML practices and start your MLOps, we have already covered MLOps from an engineering perspective in our ​18-part MLOps course.

It covers foundations, ML system lifecycle, reproducibility, versioning, data and pipeline engineering, model compression, deployment, Docker and Kubernetes, cloud fundamentals, virtualization, a deep dive into AWS EKS, monitoring, and CI/CD in production.

​Start with the MLOps course here →​

👉 Over to you: Which phase do most of your models actually live in today?

Good day!

• ​Build a semantic search inside your database without an embedding pipeline.

• 8 LLM precision formats.

• ​6 memory optimization techniques for Agentic systems.

[Hands-on] Build semantic search inside your database without an embedding pipeline

We typed a plain English query against 21,000 movie plots and got back semantically relevant results without writing a single line of embedding code.

The database is MongoDB Atlas, which added auto-embedding powered by Voyage AI directly into its vector search index configuration.

MongoDB Atlas Platform

When you add semantic search to an app, the standard move is to wire up an external embedding service, sync vectors to a separate store, and write glue to keep everything updated as data changes.

Most teams never question this setup because that’s just how the stack looked when vector search was new.

The real problem shows up later when your data changes, but your pipeline has already run, and search quality starts degrading in ways that are hard to pin down because nothing is explicitly broken.

MongoDB Atlas handles this with auto-embedding. You point an index at a text field, specify a Voyage AI model, and it generates and maintains the vectors inside the database.

When a document changes, it re-embeds automatically, so your search stays current.

Setting up auto-embedding

Load MongoDB’s sample dataset from your Atlas cluster and navigate to sample_mflix → movies. Then go to Search & Vector Search → Create Search Index.

On the index creation screen:

• Select Vector Search as the search type

• Scroll down and select Automated Embedding under “How do you want to set up your vector data?”

• Name your index autoembed_index

• Select sample_mflix → movies as the database and collection

• Choose JSON Editor as the configuration method

Atlas pre-populates the config. Just replace <field-name> with plot:

{
  "fields": [
    {
      "type": "autoEmbed",
      "modality": "text",
      "path": "plot",
      "model": "voyage-4"
    }
  ]
}

Once the index flips to Active, open the Aggregation tab on the movies collection and run:

[
  {
    "$vectorSearch": {
      "index": "autoembed_index",
      "path": "plot",
      "query": "dystopian future where machines control humans",
      "numCandidates": 50,
      "limit": 3
    }
  },
  {
    "$project": {
      "title": 1,
      "plot": 1,
      "_id": 0
    }
  }
]

The results come back with movies whose plots share none of those exact words with the query, which is the index doing semantic matching rather than keyword lookup.

One index config replaced the embedding service, the vector store, and the sync layer you used to maintain separately.

If you want to go deeper, MongoDB has a full AI Skill Badges program on MongoDB University covering everything from vector search fundamentals to agentic memory and RAG, and each badge earns a Credly credential you can share on LinkedIn.

Explore the AI Skill Badges →

MongoDB Atlas Platform →

Thanks to MongoDB for working with us on today’s issue!

8 LLM precision formats

A 12GB consumer GPU cannot hold a 7B model in FP32, which needs 28GB just for the weights.

But running ollama run llama3 on that card works anyway.

The reason is that Ollama does not ship weights in FP32.

Instead, it pulls a Q4_K_M GGUF by default, which averages around 4.8 bits per weight and puts a 7B model near 4.1GB.

We covered several different LLM quantization techniques here →

Every format below FP32 is doing the same thing, i.e., trading numeric detail for memory.

The visual below explains 8 such LLM precision formats:

Before we dive into them, some background:

A floating-point number splits its bits three ways.

One sign bit, then exponent bits that decide how large or small a value can get, then mantissa bits that decide how finely values can be told apart inside that range.

• Cutting exponent bits causes overflow, where large values saturate, and training blows up.

• Cutting mantissa bits causes rounding error, where nearby values collapse into the same number and the error accumulates.

Different formats focus on different sides.

1) FP32 has 8 exponent bits and 23 mantissa bits, reaching about 3.4e38.

Four bytes per parameter is expensive, but optimizer states and master weights still use FP32 even in low-precision training runs, because the gradient updates are small enough that FP16 storage might round them away entirely.

2) TF32 is 19 bits, made of FP32’s 8 exponent bits and FP16’s 10 mantissa bits.

It only exists inside the tensor core, so weights load and store as FP32, and memory usage does not drop at all.

PyTorch enables it on Ampere and later without any code change, and this small change results in roughly 3x faster matmuls in exchange for 13 fewer mantissa bits during the multiplication process.

3) BF16 keeps FP32’s 8 exponent bits and cuts the mantissa to 7.

An identical range means casting down from FP32 only rounds and never overflows, which is why it has become the pre-training default.

4) FP16 splits the same 16 bits the other way, 5 exponent bits and 10 mantissa bits.

It resolves values eight times more finely than BF16 but is limited to just 65504, so gradients above that become infinity, and training needs loss scaling to survive.

It is also the only 16-bit option on V100 and T4 class hardware, which predate BF16.

An RL fine-tuning paper from late 2025 found BF16’s rounding error accumulates across autoregressive sampling until the training and inference engines assign different probabilities to the same tokens, and that moving the whole pipeline to FP16 removed the divergence.

5) FP8 has two layouts:

• E4M3 reaches 448 and is ideal for weights and activations

• E5M2 reaches 57344 and is ideal for gradients, which span more orders of magnitude.

Throughput-wise, this is about 2x faster BF16, but since the range is narrow, scaling factors per tensor or per block are no longer optional.

6) INT8 has no floating point.

Weights map onto 256 evenly spaced integer levels between a calibrated minimum and maximum, which puts a 7B model in about 7GB with under 1% quality loss on many models.

Yes, even spacing is indeed a weakness, since a few outlier activations stretch the range and waste levels on values almost nothing uses.

7) INT4 gives each weight 16 possible values:

To compare, INT8 can take 256 values, so the gaps between allowed values are sixteen times wider, and every weight moves further when it snaps to the nearest one.

Rounding to the nearest value keeps that individual move as small as possible. But a layer multiplies thousands of weights against the input and adds the results, so the error that shows up in the output is the sum of thousands of these moves.

Rounding cannot see that sum. It treats every weight as equally important, when in practice, only some of them meaningfully change the output.

GPTQ works around this by quantizing a block one weight at a time and adjusting the weights not yet done to cancel the error already made.

AWQ finds the channels carrying the most signal and scales them up before quantizing, spending more of the 16 values where it counts.

Both need calibration data, a few hundred real inputs, since that is the only way to tell which weights matter.

8) NF4 also uses 16 levels but spaces them unevenly, with bin edges placed along a normal distribution because pretrained weights sit close to a zero-centered Gaussian.

It exists for QLoRA, where the base model stays frozen in 4 bits, and LoRA adapters train in BF16 on top, which puts 65B fine-tuning on one 48GB GPU at 16-bit task quality.

The weights get dequantized back to BF16 for the actual matmul, so it saves memory without adding speed.

As a result, quality degrades far less on larger models, so a 70B at 4-bit generally beats a 13B at 16-bit for the same memory.

And the sharp drop is between 3-bit and 4-bit, not between 4-bit and 8-bit, which is why 4-bit ended up as the default everywhere from Ollama to vLLM.

To dive deeper, we have already written a full deep dive on Quantization, specifically, which covers several of these methods with their simplified mathematics.

Learn how Quantization optimizes LLMs to run them on tiny hardware here →

​6 memory optimization techniques for Agentic systems​

We recently added ​Part 16​ and ​Part 17​ to our AI Agents crash course, where we use LangGraph to implement 6 production-grade memory optimization techniques in agentic workflows.

• ​You can read Part 16 here →​

• ​You can read Part 17 here →​

But what exactly is Memory, and why is it so powerful for Agentic systems?

To understand this, consider an Agentic system without Memory (below):

• In iteration #1, the user mentions their favorite color.

• In iteration #2, the Agent knows nothing about iteration #1.

This means the Agent is mostly stateless, and it has no recall abilities.

But now consider an Agentic system built with Memory (below):

• In iteration #1, the user mentions their favorite color.

• In iteration #2, the Agent can recall iteration #1.

Memory matters because if a memory-less Agentic system is deployed in production, every interaction with that Agent will be a blank slate.

It doesn’t matter if the user told the Agent their name five seconds ago, it’s forgotten. If the Agent helped troubleshoot an issue in the last session, it won’t remember any of it now.

With Memory, your Agent becomes context-aware and practically applicable.

But Memory isn’t an abstract concept.

If you dive deeper, it follows a structured and intuitive architecture with several types of Memory.​

• Short-Term Memory

• Long-Term Memory

• Entity Memory

• Contextual Memory, and

• User Memory

Each serves a unique purpose in helping agents “remember” and utilize past information.

To simulate memory, the system has to manage context explicitly: choosing what to keep, what to discard, and what to retrieve before each new model call.

This is why memory is not a property of the model itself. It is a system design problem that can also be optimized, and we covered them in these two parts:

• ​You can read Part 16 with LangGraph →​

• ​You can read Part 17 with LangGraph →​

Also, Part 8 and Part 9 of the crash course cover memory with CrewAI:

• ​AI Agents Crash Course Part 8 →​

• ​AI Agents Crash Course Part 9 →​

Both parts cover:

• 5 types of Memory from a theoretical, practical, and intuitive perspective.

• How each type of Memory helps an Agent.

• How an Agent retrieves relevant details from the Memory.

• The underlying mechanics of Memory and how it is stored.

• How to utilize each type of Memory for Agents (implementations).

• How to customize Memory settings.

• How to reset the Memory if needed.

• And more.

Happy learning!

Most databases that need both fast ingest and fast analytical queries end up split across two systems.

One table for writes, another for reads, and a pipeline stitching them together that you now have to maintain.

That split means more infrastructure to manage, more failure points to monitor, and query results that are only as fresh as your last pipeline run.

Tiger Cloud by Tiger Data ships with Hypercore, a hybrid storage engine built into TimescaleDB. New data lands in row storage for fast inserts. As it ages, Hypercore converts it automatically to columnar storage, compressing it by up to 95%.

The query engine then scans far fewer bytes. Summary queries like COUNT, MIN, MAX, FIRST, and LAST read directly from batch metadata without decompressing anything.

You write and query the same table, and Hypercore handles which storage format each chunk lives in based on age.

New accounts get $1,000 in free credits, no card required.

Get started with Tiger Cloud here →

Thanks to Tiger Data for working with us on today’s issue!

The missing piece of agent self-improvement

Most AI systems don’t get better on their own. A model ships, it fails on some input, someone notices, and someone fixes it. The system itself carries none of that experience forward.

Self-improvement loops are defined to improve that.

Every failure ending in a one-off patch is the old pattern. The new one uses the outcome of a run to change the system’s future behavior. So the same mistake becomes less likely to happen again.

This idea isn’t unique to agents. Search ranking systems adjust signal weights based on click behavior. Recommendation engines update their predictions based on what users engage with and what they ignore.

Agents make the problem more interesting because they don’t produce a single output. They don’t just produce one output; they make a chain of decisions. So the mistake that causes a failure happens steps before the final answer is wrong.

Can the agent turn those failures into something it acts on later, or does the failure get forgotten?

That’s the real test of self-improvement.

Why most agents don’t self-improve

In theory, a self-improving agent should learn from both its successes and its failures. In practice, most production systems still follow a manual loop.

The code gets changed, the bug gets fixed, but the agent itself doesn’t learn anything from the failure. If the same class of problem shows up again a month later, you will repeat the same debugging process.

Hermes is one of the few open-source agents built around the idea that useful experience should be reusable. Rather than restarting from zero on every task, it keeps what worked and reuses it later.

How Hermes learns

When Hermes discovers a useful way to solve a problem, it doesn’t discard that experience after the task finishes. It saves the successful approach as a reusable SKILL.md file.

The next time it encounters a similar problem, it can load that skill instead of rediscovering the solution from scratch.

That is the core runtime learning loop of solving a problem, capturing the successful strategy, and reusing it later.

As more tasks are completed, the number of saved skills grows. So Hermes also includes a background curator to manage that library.

The curator removes outdated skills, keeps the collection organized, and helps prevent the skill store from turning into an unmaintainable pile of snippets.

Hermes also has a second learning path through GEPA (Genetic-Pareto Prompt Evolution). This is not the same as the runtime skill loop.

GEPA works offline by analyzing execution traces from agent runs. It checks the results and keeps the variants that perform best.

The important distinction is that GEPA is an optimization process, not a training run. It doesn’t require GPU fine-tuning.

The Hermes self-evolution repository describes this as an evolutionary search that can improve prompts, skills, and tool descriptions without retraining the model itself.

Taken together, these pieces give Hermes a genuine self-improvement loop. Successful solutions become reusable skills, the curator keeps those skills healthy, and GEPA helps refine the system from what it has learned running.

Self-improvement has limits

Hermes’ self-improvement loop still has a few practical limitations:

• Self-bias. Hermes can rate its own work highly, which increases the risk of weak solutions being saved as reusable skills.

• Skill overwrite. Auto-generated skills can replace well-crafted manual ones with lower-quality alternatives.

• GEPA runs offline. Optimization happens outside the running agent, so improvements only happen after someone collects traces and reruns the optimization process.

Those are specific implementation details, but they point to a broader limitation. The loop has no automatic way to learn from failures that appear in production.

A production failure often looks very different from a development-time mistake. The agent may behave correctly in testing and fail only under real workloads, real user inputs, or unusual tool interactions. Those failures don’t automatically become part of what the agent learns.

Instead, someone opens the trace, figures out what went wrong, patches the code, and redeploys the agent. The fix lives in Git, but the learning stops there.

That is the gap between a system that can remember successful solutions and a system that can continuously learn from production failures.

Closing the production gap

Hermes’s missing piece shows up once the agent is live in production. Opik picks up there, covering what the internal loop can’t.

Opik is an open-source observability and optimization platform for AI agents. It integrates with dozens of agent frameworks and developer tools and now includes an official integration for Hermes as well.

Most observability platforms stop after showing you the trace. Opik treats that trace as the beginning of another learning loop instead of the end of one.

It closes the production gap with a workflow where each layer feeds into the next:

Trace → Ollie diagnoses → proposed fix → approval → rerun → regression test → future traces.

These aren’t separate features; they’re stages of the same debugging loop. Rather than treating those as separate features, think of them as stages of the same debugging loop.

Here’s each layer in practice.

Layer 1: Tracing

Everything starts with an agent trace, a record of what the agent did during a run.

Every LLM call, tool invocation, and retrieval step is recorded automatically. Opik works out of the box with LangGraph, CrewAI, Hermes, and more than fifty other frameworks, so the entire execution path is available when something goes wrong.

For example, I asked my Hermes agent for the latest news about Google’s Gemma 4 model family. The entire execution was captured automatically as an Opik trace.

Layer 2: Ollie

A trace tells you what happened, and your next logical question will be why.

That’s where most observability platforms stop, right after showing you the trace. Opik goes further with Ollie, a built-in coding agent that turns failures into fixes.

Ollie is the coding agent built into Opik. By reading the span tree, it follows the chain of events across every model call and tool invocation until it identifies the actual failure.

If you run a single opik connect command from your project root, Ollie can also inspect your local project.

From there it can:

• read the relevant source files

• identify the code responsible for the failure

• propose a fix as a Git-style diff

One detail I particularly like is that nothing happens automatically. Ollie never edits your project on its own. Every proposed change waits for your approval before it’s applied.

Here’s a real example using Hermes.

Install the pip-audit tool and run a security
vulnerability scan on the current Python environment.

The above task failed because the execution environment was configured incorrectly.

After connecting Ollie to the project, it inspected both the execution trace and my local configuration. It traced the failure back to where the execute_code tool had been configured with an unsupported cloud_runner environment.

Ollie proposed a small change to the configuration, explained why it would fix the problem, and waited for approval before making any modifications.

The Hermes agent’s execute_code tool is configured with
environment type cloud_runner, which is not a supported value.

The fix Ollie proposes:

The entire workflow becomes the following:

Bad trace → root cause → diff → approve → rerun → regression locked

Layer 3: Test Suites

Fixing a bug once isn’t enough. The real goal is making sure it never comes back.

Many evaluation systems start by asking you to build labeled datasets and numerical metrics. Opik gives you test suites to describe expected behavior in plain English instead of a dataset to maintain.

Opik instead gives you test suites to describe expected behavior in plain English.

suite = opik.TestSuite(”Hermes Agent Regression”)

suite.add_assertion(
    "Check if the ‘requests’ package is installed and print its version.”
)

suite.add_assertion(
    "List the files in the current directory and identify the project.”
)

suite.add_assertion(
    "Run Python and print the interpreter version together with the first ten installed packages.”
)

suite.run_tests()

Under the hood, Opik evaluates those assertions with LLM-as-a-judge, giving each case a straightforward pass or fail without requiring you to maintain labeled evaluation datasets.

So this workflow lets you turn real production failures into regression tests as part of the debugging loop.

Note: Everything above can be done manually with the Opik SDK, but Ollie automates the process for you.

Layer 4: Agent Sandbox

One question still remains unanswered. Did the fix actually solve the problem without breaking anything else?

Most playgrounds let you tweak a system prompt and rerun a single LLM call. Opik’s Agent Sandbox goes further; it executes the complete Hermes workflow end to end.

You can change a prompt, update a configuration, switch a model, or apply Ollie’s proposed fix, then rerun the same agent against the same input that originally failed.

After approving Ollie’s config.yaml change, I reran the original pip-audit request.

The first trace failed during environment validation. The second completed successfully using the updated configuration.

Rather than relying on a patch that “looks right,” you can verify the fix against the exact scenario that exposed the bug in the first place.

Why this matters in practice

What I like about this workflow is that it complements Hermes instead of replacing it.

Hermes gets better by remembering successful ways to solve problems.

Opik focuses on the failures that only appear after deployment. It traces them, helps diagnose the root cause, verifies the proposed fix, and turns that failure into a regression test. So the same issue is far less likely to slip through again.

Together, they cover both sides of the learning loop, where one learns from success and the other systematically learns from failure.

Running it yourself

The full setup runs on Linux, macOS, or WSL2. Python 3.11+ comes with the Hermes installer. 8GB RAM is enough for API-based usage.

Step 1: Hermes setup

curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash
source ~/.bashrc   # or ~/.zshrc

Run the setup wizard. It walks through provider, API key, model, and tools.

hermes setup
hermes

Step 2: Install the Opik plugin

Install it into the same Python environment as Hermes:

 ~/.hermes/hermes-agent/venv/bin/python3 -m pip install opik-hermes

This pulls in the Opik SDK automatically as a dependency; no separate install needed.

Step 3: Enable the plugin

Add it to your Hermes config:

# ~/.hermes/config.yaml
plugins:
  enabled: [opik]

The plugin registers itself through a Hermes entry point once installed, so this is the only step required to turn it on.

Step 4: Configure Opik

Set these in ~/.hermes/.env. If you’re running local open-source Opik, http://localhost:5173/api then no API key is needed.

For Comet-hosted Opik, add your workspace and key instead:

# ~/.hermes/.env
OPIK_API_KEY=...
OPIK_WORKSPACE=your-workspace
OPIK_PROJECT_NAME=Hermes Agent

Step 5: Restart Hermes and check the trace

hermes

Talk to Hermes from the terminal UI however you normally do; every turn gets traced automatically.

Using the CLI directly:

hermes chat -q “list the files here and tell me how many there are”

Step 6: Connect Ollie

Run this from your agent folder:

cd .hermes
opik connect --project “Hermes Agent”

Open Opik and look for a trace named after your message, with an LLM span (input, output, model, and token cost) and a span for each tool call.

From here, the loop runs on its own. Every failure that lands in Opik gets diagnosed by Ollie. Once you approve, that failure locks permanently as a test case. So the test suite grows with every cycle.

One thing worth knowing as you scale is that Hermes limits an agent to 90 turns. Opik’s traces show where those turns are being spent, making it much easier to spot unnecessary loops and wasted steps.

Putting it together

Hermes and Opik aren’t trying to solve the same problem, and that’s exactly why they work well together.

Hermes improves from what already works. Opik makes sure production failures don’t just get patched and forgotten, turning each one into a fix that’s diagnosed, verified, and locked in as a regression test.

Taken together, the two systems cover much more of the self-improvement loop than either one does on its own.

The whole Opik stack ships in the open with Tracing, Test Suites, the Agent Sandbox, a 6-algorithm Agent Optimizer, and 50+ framework integrations, with the project past 20.8K stars on GitHub.

If you’d like to run Opik yourself, you can self-host it in three commands:

git clone https://github.com/comet-ml/opik
cd opik
./opik.sh

If you’re already running agents in production, it’s worth exploring what that extra half of the learning loop looks like in practice.

Check out Comet ML and Opik →

(don’t forget to star 🌟)

👉 Over to you: Where does your debugging loop hand back to you, and is that failure gone for good or just gone for now?

Good day!

Production AI systems are moving from a single large model doing everything to several smaller models, each doing one job.

One parses the document, the next extracts fields, a third reranks search results, a vision model reads the image, and a final model handles generation.

At the model level, this usually brings the cost down quite a bit.

But the model is only part of the inference bill. You still need GPUs to run it, memory to keep it loaded, and a serving layer to batch and schedule the requests coming through it, which is why most teams don't experience the cost savings they expect.

Today, we understand why that is the case, what serving small models well actually takes, and how the ​SIE open-source inference engine solves it end-to-end.

Problem with standard serving tools

Once an agentic pipeline starts using several model types, the multi-framework serving problem becomes a hardware problem too.

Even though pipelines rely on a mix of vLLM, TEI, and custom servers, each tool is great at what it does. The challenge is deciding where and how those models should actually run.

That leaves two practical ways to arrange them. Give each serving stack its own GPU, or put several of them on the same card.

Unfortunately, with today’s standard AI serving tools, neither setup works cleanly.

1. Give every model its own GPU

The simplest setup is to give each serving stack its own GPU. There is no resource sharing to manage, and each service gets the hardware it needs.

vLLM handles the main LLM on one GPU, TEI gets another for embeddings and reranking, and your document processing, NER, and vision models each get their own dedicated GPUs.

While this setup works operationally, the issue comes down to GPU cost and utilization.

Take our earlier flood insurance claim pipeline, where work proceeds in the following sequence.

The catch is that a single claim moves through those stages sequentially, so each dedicated GPU spends much of that time waiting for its turn.

Across many claims, those workloads can overlap, but that still doesn’t mean every dedicated GPU stays well utilized all the time.

So if each stage has its own dedicated GPU, the GPU running the document parser sits idle while the reranker runs. Then the reranker waits while the LLM runs.

That idle GPU also doesn’t get freed up or handed to something else. The serving process is still running on it, so the hardware stays allocated to that one stage the entire time, whether it’s doing anything or not.

That’s a real budget concern because GPU infrastructure is paid for by the time you hold it, not by how many seconds the GPU is actually computing.

Furthermore, many of these specialized models are small enough that they do not need an entire dedicated GPU.

An extraction model, reranker, or small vision model will use only a fraction of an L4 GPU’s 24 GB of memory, while also spending much of its time waiting for work.

So adding more GPUs increases the bill while leaving capacity unused on each card.

Remember how your pipeline started with multiple specialized models? By the time you add extra steps, every new capability will demand its own serving process and another dedicated GPU before you know it.

You moved to smaller models to reduce the hardware needed per task. Now the hardware count grows with every new task you add.

2. Fit multiple models on one GPU

The other option looks much better from a cost perspective, where you put several of those models on the same GPU.

When the models fit within the GPU’s memory, there is no fundamental reason each needs its own GPU. A single L4, for example, has 24 GB of memory, which is enough to hold several small models at once.

The difficult part is not fitting the models onto the card. It is getting separate serving processes to share that card efficiently.

A serving process is usually built around the model it was started with. It manages that model’s memory, requests, and batches without knowing what the other processes on the same GPU are doing.

Take vLLM as an example. Its --gpu-memory-utilization setting defaults to 0.92, which defines how much of the GPU memory that instance is allowed to use.

Run another serving process beside it, and that second process does not automatically know what vLLM is using or what memory could safely be made available to it.

Now try packing vLLM, TEI, and your custom parsing, extraction, and vision servers all onto the same GPU.

You are deciding manually how much memory each process should be given to use before you even know what the actual traffic will look like:

• Give one process too little room or card memory. And a long document or sudden batch can push one model beyond its limit.

  When that happens, it doesn't just fail on its own; it can take every other model sharing that GPU down with it.

• Give one process too much room or card memory. That memory sits unavailable to every other model, even while it's sitting idle itself.

What a serving stack for small models needs

First, step back from searching for another framework and understand the things we need the ideal server to do.

• The first requirement is breadth. The server has to run embeddings, rerankers, OCR, vision, extraction, and generation behind one API.

• GPU utilization is the second problem. And that turns out to require more than a good scheduler.

  Because to pack requests of different lengths into one pass without wasting compute, the engine has to control the batching and the attention path for every architecture it serves.

• Model memory needs to follow traffic too. It should load and evict models as traffic moves, keeping the busy ones loaded and letting the idle ones move out.

  So that a model is not holding memory the way a cold serverless worker or a padded vLLM instance does.

• And the serving layer still has to behave like production infrastructure with routing, autoscaling, monitoring, and GPU pools.

  This is important since a bare runtime like vLLM is only the engine. On its own, it cannot spread load across replicas, add or remove GPUs as traffic changes.

• And adding a new replica would be a config change, not a redeployment.

Today, developers have to build all of this from scratch, spending months on custom engineering because different model families run completely differently under the hood.

• A Qwen model handles positions and attention differently.

• A ColBERT model returns a vector for every token.

• A reranker returns a single score and no vector at all.

Building one engine that can hold all of those shapes and pack any of them into a full batch is critical work. And it is the reason this did not already exist as an open-source package.

Open-source solution: Superlinked Inference Engine

The solution to all of this is implemented in the open-source Superlinked Inference Engine (SIE).

SIE is an open-source inference engine that runs as a production cluster for multi-model pipelines on shared infrastructure.

It supports 100+ models through a unified API, so different model types can run through the same serving layer instead of each needing its own deployment.

One API is useful, but the bigger advantage is coordination. SIE can manage those models around the same GPU pool.

It runs as one cluster inside your own cloud, and it’s built for exactly the kind of multi-model pipeline we have been describing, i.e., several small models of different kinds running back to back on shared GPUs.

For the kinds of workloads we have been discussing for the flood insurance claim use case, SIE exposes multiple core primitives as:

• extract does three different jobs in this pipeline:

  • turning the claim form and policy document into clean markdown (docling)

  • pulling labeled fields like name, policy number, and date out of text (gliner)

  • and finding labeled flood-damaged categories in the claim photo (grounding-dino)

• Three genuinely different tasks of parsing, entity extraction, and vision detection using the same extract interface.

  The models can be different underneath, but the serving interface doesn't have to be.

• score reranks the policy’s chunks against a query using bge-reranker, and returns the ranked list.

• generate takes everything gathered so far- parsed documents, extracted fields, policy language, and photo analysis- and produces the final review, using Qwen3.5.

Under the old approach, TEI could handle the reranking stage, but that still leaves four other stages to serve separately.

Parsing, entity extraction, vision detection, and generation would each need their own serving setup.

A single SIE cluster runs all five stages without a separate serving stack for any of them. It gives us five stages, three primitives, and one shared serving layer.

The API is only the visible part. The interesting work happens underneath it, where SIE has to actually coordinate those models on shared GPUs.

1. Models load only when they are needed

The first problem we had with standard serving tools was memory.

If several models share one GPU, we cannot keep every model loaded all the time and hope the memory works out. The serving layer needs to decide which models actually deserve space on the card.

SIE loads a model when a request actually needs it.

• When GPU memory becomes constrained, SIE evicts the least recently used model and makes room for another one.

• The GPU is no longer permanently attached to one model. It becomes a shared pool that different models can use as traffic moves through the pipeline.

2. One queue sees all the work

The second problem was coordination in a shared pool of resources.

With separate serving processes, each model sees only its own requests. The document server does not know what the reranker is waiting for, and the reranker has no idea what the extraction service is doing.

SIE puts the work behind a shared queue instead.

• The gateway publishes requests into that common pool, and workers pull from it when they are ready to run.

• That gives the serving layer a view of the workload across models instead of forcing every process to make scheduling decisions in isolation.

3. Batching follows compute cost

There is another source of waste once incoming requests start sharing the same GPU.

Requests are rarely the same size. If you batch a short input together with a much longer one, the shorter input is usually padded to the longer length.

The GPU then spends part of the computation processing padding rather than useful input.

• SIE groups requests by estimated compute cost instead of simply grouping a fixed number of requests together.

• Requests with similar compute costs can be batched together, so shorter inputs don't spend most of their GPU time being padded to match much longer ones.

4. The shared server scales with the workload

Sharing one GPU is useful locally, but production traffic does not stay constant.

• SIE puts a gateway and worker layer around the model-serving runtime so the same setup can scale beyond.

• It matters because the shared-GPU idea should not end when you move from a laptop to a production cluster. The same serving layer needs to handle both.

• The system can add workers as demand increases and scale back down when demand falls.

SIE also provides deployment and operational pieces for production environments, including Kubernetes-oriented infrastructure, monitoring, and cloud deployment (AWS, GCP) support.

5. Models come with their serving configuration

There is one more problem that is very easy to underestimate.

Supporting a new model is not just downloading its weights. Different model architectures have different memory requirements, batching behavior, precision settings, and runtime characteristics.

A production serving layer therefore needs to know how each supported model should run instead of making you tune every model from scratch with vLLM or Triton, etc.

• SIE’s model catalog packages supported models with their serving configuration, so adding or swapping a supported model does not mean rebuilding an entire serving stack around it.

• The current catalog covers 112 models. So you reference a model by name, and the engine loads it with settings known to work.

Proving it against a real document

To see what serving looks like in practice, let’s execute our flood insurance claim review workflow in SIE (Superlinked Inference Engine).

The claim workflow spans plain text, formatted PDFs, and images, bringing multiple modalities together into a single pipeline.

The documents are deliberately messy, pulled straight from real public-domain sources to stress-test the pipeline:

• a proof-of-loss user form

• a repair estimate cost

• an insurance policy document

• and a photograph of the actual flood damage

Under the fragmented approach, this is the same five-tool sprawl covered earlier. SIE runs the same five jobs through one shared cluster instead.

None of these models does the same job. That means five serving setups where SIE runs the same five jobs through one shared cluster.

One cluster executing five different jobs

Here’s what the pipeline looks like, with each stage showing the task, model, and SIE endpoint it uses:

First, start by installing and starting the server. The serve command starts the server on port 8080.

pip install "sie-server[local]"
sie-server serve

It is ready when the health check answers.

curl http://localhost:8080/readyz     # ok

Now, instantiate the client and point it at the running server:

from sie_sdk import SIEClient

client = SIEClient("http://localhost:8080")

Every stage of the rest of the pipeline now goes through this one object and this one endpoint, no matter which model it hits.

Let’s walk through them one by one.

1. Parsing the policy documents

The proof-of-loss form, the repair estimate cost, and the insurance policy all go through docling for parsing, turning documents into clean markdown.

Docling is an open-source document parser that reads PDFs and structured documents and outputs clean markdown while keeping tables and layout intact.

result = client.extract(
    "docling",
    Item(id=path.stem, document=path),
    options={"profile": "default"},
)
markdown = result["data"]["markdown"]

print(markdown)
# "# Proof of Loss\n\nPolicyholder: ...\n\nDate of Loss: ..."

Parsing calls go through the extract endpoint. extract is SIE’s general-purpose call turns raw input, text, or image into structured or clean output.

So a document parser, an entity extractor, and a vision model can all sit behind the same endpoint even though they’re doing different work underneath.

2. Pulling the claim identity

Once the proof-of-loss form is in Markdown, an extraction model pulls the structured fields actually needed. i.e., name, loss date, property address, etc.

GLiNER is doing named entity extraction here, which means it can identify fields from labels you provide instead of needing a fixed schema baked into the model.

labels = [
    "insured name",
    "date and time loss",
    "flood insurance policy number",
    "insured property address",
]
result = client.extract(
    "fastino/gliner2-large-v1",
    Item(id="claim-identity", text=markdown[:5000]),
    labels=labels,
)

print(result["data"])
# [
#   {"label": "insured name", "text": "...", "score": 0.97, "start": 0, "end": 0},
#   {"label": "date and time loss", "text": "...", "score": 0.96, "start": 0, "end": 0},
#   ...
# ]

3. Finding the policy language

The policy document is long, so a naive approach would rerank every chunk against the query.

The cross-encoder reads the query and a candidate together and outputs a relevance score. This means every chunk it evaluates costs a full pass, not a cheap lookup.

So we will filter first before actual reranking:

• It scores candidate chunks by keyword overlap, with terms like “proof of loss,” “signed,” and “60 days,” before sending only the strongest candidates to the reranker.

score_result = client.score(
    "BAAI/bge-reranker-v2-m3",
    Item(id="policy-requirements", text=POLICY_QUERY),
    [Item(id=str(i), text=text) for i, text in candidates],
)

print(score_result["scores"])
# [{'item_id': '0', 'rank': 0, 'score': 3.009},
#  {'item_id': '1', 'rank': 1, 'score': -1.061},
#  {'item_id': '2', 'rank': 2, 'score': -11.425}]

SIE has the score() method that uses a cross-encoder. The scores are cross-encoder logits, so the absolute numbers do not mean much on their own.

Instead, the order is important. The results come back already ranked, so the top passage is the best match:

best_id = score_result["scores"][0]["item_id"]

best = candidates[int(best_id)][1]
# 'Proof of loss must be signed and submitted within 60 days of the date of loss.'

4. Reading the damage photo

The damage photograph goes through zero-shot object detection, looking for specific damage categories with a confidence floor.

Zero-shot means the model was never trained on these exact labels of 'standing water’ and 'flooded room.’ So just text descriptions are handed to it at request time.

And it matches them against the image directly instead of needing a model fine-tuned to recognize this specific list of categories in advance.

labels = ["standing water", "flooded room", "water damaged wall", "damaged furniture"]
result = client.extract(
    "IDEA-Research/grounding-dino-tiny",
    Item(id="damage-photo", images=[photo_path]),
    labels=labels,
    options={"score_threshold": 0.05},
)

print(result["data"])
# [
#   {"label": "furniture", "score": 0.509, "bbox": [412, 890, 1180, 1640]},
#   {"label": "standing water", "score": 0.276, "bbox": [50, 1500, 2900, 2280]},
# ]

5. Writing the review

By this point, the pipeline has written out its own set of intermediate results as markdown for each parsed document, the structured claim identity, the ranked policy passages, and the photo analysis.

Those feed into one generation call, which produces the final review. The output is locked to a JSON schema, so the result has a fixed structure.

result = client.generate(
    "Qwen/Qwen3.5-4B:no-spec",
    generation_prompt,
    max_new_tokens=1500,
    temperature=0,
    top_p=1,
)

print(result["data"])
# {
#   "status": "manual_review",
#   "findings": [
#     {
#       "severity": "blocking",
#       "detail": "Proof of Loss lacks required signature and date."
#     }
#   ]
# }

For generation, SIE invokes generate() to compose the answer. This is free text with token usage, produced by a generative model, and it comes back through the same client as the entities, the scores, and the bounding boxes.

Run it yourself

Finally, we have a whole pipeline with five models, three shared primitives, extract, score, and generate, running through one client against one SIE cluster.

The important part is that these five stages do not require five separate serving stacks. How you place the models underneath that serving layer depends on the available GPU memory and the models you need to run.

For this example, you can also run two SIE servers:

• one for Docling, GLiNER2, and reranking

• a second for Grounding DINO and Qwen’s generation

Or, if you want to reuse a single GPU, you can load one model bundle, run its calls, release it, and then load the second bundle.

So the deployment can use multiple servers or reuse the same GPU. The serving layer stays the same.

Five models are pulled from Hugging Face on their first call and stay active for subsequent requests while their bundle is loaded.

Run the five calls end to end, inspect what each model returns, and see how the whole pipeline runs through SIE.

Want to try the pipeline without building it from scratch?

The repository includes notebooks that walk through the complete insurance-claim workflow, along with the full code and server setup.

Full example and notebooks →

(Don’t forget to ⭐ the repo.)

Using small, specialized models for the narrow tasks is the right approach, and it is not really controversial.

They are accurate enough on the task they are trained for, and they keep your data in your own environment.

But switching to small models does not make inference cheaper by itself. It moves the cost from a per-token bill to the GPUs you rent, and if you put each model on its own GPU, most of that hardware sits idle, and you are paying for it.

The saving only appears when the models share GPUs, and that requires a single engine that can run all of them. The moment you serve each model with a different tool, each one takes a GPU of its own again.

SIE (open-source) already implements that, letting you run the different models your agent needs on one shared cluster, with the routing and autoscaling for production you’d otherwise have to build yourself.

It also plugs into your existing stack, from vector databases (Chroma, Qdrant) to agent frameworks (LangChain, CrewAI). And it even has drop-in OpenAI compatibility, so existing embedding or chat can just point at a new URL.

Superlinked GitHub (100% Open Source) →

👉 Over to you: how many separate models does a single request in your stack call, and how many of them are holding a dedicated GPU right now?

Good day!

• Code verification that runs inside the agent’s session.

• Your Agent remembers everything and understands nothing.

• ​Automatic LLM routing in two lines of code.

Code verification that runs inside the agent’s session

GitGuardian tracked every public GitHub commit last year and found Claude Code-assisted commits leaked credentials at 3.2%, against a 1.5% human baseline.

Most teams catch this with a pre-commit hook that shells out to a scanner, or a script that pipes findings back into the agent’s context.

Both are glue you maintain, and both run outside the session where the code gets written.

SonarQube CLI (by Sonar) removes that glue. One command registers the hooks and configures the SonarQube MCP server in Claude Code:

sonar integrate claude -p <your-project>

Secrets detection and the MCP server run on the SonarQube free tier, with context augmentation and deeper code analysis on paid plans as part of Sonar Vortex.

After that, secrets detection runs on every file the agent reads or writes and every prompt submission, static analysis checks whatever the agent just wrote, and the MCP server puts those findings in Claude’s context so it can fix its own issues.

Because the hooks sit in the session rather than in CI, the agent gets the finding while it still has the context to fix it.

Find the SonarQube CLI GitHub repo here →

(don’t forget to star it ⭐ )

Thanks to Sonar for partnering today!

Your Agent remembers everything and understands nothing

Every agent memory system today solves the same problem. They store facts and retrieve the right ones at query time.

The part that’s missing is pattern recognition.

Consider a project management assistant that reads status updates from an engineering team. Three people filled out updates in the same week. Each one reports a blocker, and each report is accurate and specific.

The agent stores all three and can retrieve any of them on request.

What it can’t tell you is that all three blockers trace back to a single delayed task.

Think of it like this. A new project manager reads three status updates and sees three problems to solve. An experienced one reads the same three and sees one problem with three symptoms.

Nobody taught the experienced PM to connect them. They’ve sat through enough standups to recognize the shape.

Agent memory today gives you the new PM. It stores every update correctly, retrieves them accurately, but misses the structure connecting them,

Zep implements a feature called Observations that addresses this. It automatically detects cross-conversation patterns from the knowledge graph and surfaces them as durable, evidence-backed context the agent can use.

Let’s look at how it works.

How the knowledge graph stores data

Zep builds a knowledge graph from conversations and business data. The structure has three building blocks:

• Entities are the nouns: people, products, places, tasks, services.

• Facts are specific claims connecting two entities, stored as labeled edges.

• Episodes are the raw source material: the actual conversation messages, JSON payloads, or documents that Zep ingested. Every fact traces back to the episode it was extracted from.

Graphs can be scoped to a single user or shared across a team.

A user graph holds one person’s history. A shared graph holds everything about a project, a workspace, or an organization, with data from many people flowing into the same structure.

Here’s a concrete example of a shared project graph.

Three engineers file status updates with a PM assistant over one week:

• Alice (Monday): The API migration is blocked. She’s waiting on the auth service refactor, which the backend team owns.

• Bob (Wednesday): The payments integration is stalled. He’s also waiting on the auth service refactor from the backend team.

• Clara (Friday): The checkout flow can’t move forward because the payments integration isn’t ready. That work belongs to Bob.

Zep processes each update as an episode, extracts the entities and facts automatically, and builds the knowledge graph:

Every fact is accurate and individually retrievable. What the graph doesn’t contain is the shape those facts form together.

Why better retrieval doesn’t solve this

When the agent queries “what’s blocking the team,” it gets back the stored facts:

• Alice is blocked on the API migration

• The API migration is waiting on the auth service refactor

• Bob is blocked on the payments integration

• The payments integration is waiting on the auth service refactor

• Clara is blocked on the checkout flow

• The checkout flow is waiting on the payments integration

Every line is correct, but nothing in the result tells the agent that unblocking one task clears all three people.

You can tune the reranker, widen the search scope, or return more results.

None of it helps, because the insight the agent needs was never stored as a discrete piece of content. It exists in how the facts connect across three separate conversations.

The workaround most teams reach for is a rule: flag any task that three or more people mention as a blocker. It just tells you the team is stuck without telling you which task to unblock first, and it only catches blockers that people name directly.

Clara never mentions the auth service. Her update is two hops away from the actual cause.

At this scale, a person could connect the dots manually. But at thirty status updates a week, nobody does, and that’s exactly when the chain matters most.

What Observations produce

For this project graph, Zep generates:

Name: Auth service refactor blocking multiple workstreams

Summary: The auth service refactor, owned by the backend team, is the root blocker in a dependency chain affecting three workstreams. Alice’s API migration and Bob’s payments integration are both waiting on it directly. Clara’s checkout flow is blocked downstream of the payments integration. Unblocking the auth service refactor clears all three.

No individual fact in the graph contains this. It spans three people, four work items, and three separate conversations.

Three properties make observations different from individual facts:

• Cross-entity: A fact sits on one edge between two entities. An observation synthesizes across an entire cluster, connecting multiple entities and conversations.

• Evidence-backed: Every claim traces to specific facts and episodes in the graph.

• Read-only: You can’t manually create, edit, or delete observations. They follow the evidence. The mechanism below explains why.

How Observations are created

The mechanism is a two-stage pipeline:

• A deterministic algorithm first handles the clustering

• An LLM writes the summary afterward. The model never decides what gets grouped.

Reducing facts to signatures

Zep runs a background process that periodically checks for new data. When new conversations have been ingested and the graph has settled, it kicks off the analysis.

The first step reduces every fact to a signature: the two entities it connects plus the relationship type. The goal is to find which conversations reference the same relationships between the same entities.

After reduction, each episode (conversation) has a set of signatures:

• Episode 1 (Alice’s update): (Alice, API Migration), (API Migration, Auth Service), (Auth Service, Backend Team)

• Episode 2 (Bob’s update): (Auth Service, Backend Team), (Payments Integration, Auth Service), (Bob, Payments Integration)

• Episode 3 (Clara’s update): (Bob, Payments Integration), (Checkout Flow, Payments Integration), (Clara, Checkout Flow)

The overlaps become visible:

• (Auth Service, Backend Team) appears in Episode 1 and Episode 2

• (Bob, Payments Integration) appears in Episode 2 and Episode 3

• Episode 1 and Episode 3 share nothing

Alice never mentions Bob, Clara, the payments integration, or the checkout flow. Clara never mentions Alice, the auth service, or the backend team.

Building the episode graph

Here, Zep flips the perspective.

Instead of looking at entities connected by facts, it builds an episode graph where conversations are the nodes and shared signatures are the edges.

• Episode 1 links to Episode 2 through (Auth Service, Backend Team)

• Episode 2 links to Episode 3 through (Bob, Payments Integration)

• Episode 1 and Episode 3 don’t share a signature

But all three end up in the same connected component because Episode 2 bridges them.

Alice’s update and Clara’s update join the same cluster despite having no entity in common, because Bob’s update touches both sides and acts as the bridge between the two.

Worth noticing that the chain the algorithm follows, and the dependency chain in the project are the same structure. Zep finds the root blocker by following exactly the kind of transitive link it was built to follow.

This does not use embeddings, semantic similarity, and no ML model. It’s pure graph topology: which conversations share facts about the same entity pairs, and how those shared pairs chain them into connected groups.

The distinction matters because embedding-based clustering groups content that discusses similar topics. All three status updates would cluster together under that approach simply because they’re all about engineering work, along with every other status update that week.

Signature-based clustering groups episodes that reference the same relationships between the same entities, which is what makes the specific chain detectable.

Summarizing the cluster

Once the algorithm identifies the connected component, an LLM writes the summary.

A single, constrained LLM call receives the cluster’s key entities, the supporting conversations in chronological order, and the relationship types.

It then returns a name and summary steered toward durable signals like decisions, constraints, dependencies, and state changes, and restricted to facts the evidence explicitly supports.

Every structural property of the observation (which entities, which conversations, the time window) is decided by the algorithm. The LLM only turns that structure into readable prose.

This is why observations are read-only. They’re structural properties of the graph, described in natural language. When the evidence changes:

• New conversations that fit an existing observation cause it to be regenerated with the new evidence

• Contradicting evidence causes the old observation to be retired

• The graph always reflects the current state of what is known’

If the backend team ships the auth service refactor next week, the observation updates. Alice and Bob unblock, Clara’s dependency shifts, and the pattern reforms around whatever is actually blocking work now.

How does this help the agent

Retrieving observations through the SDK:

# List all observations for a project graph
observations = client.graph.observation.get_by_graph_id(graph_id="project-atlas”)

# Search observations by relevance
results = client.graph.search(
    graph_id="project-atlas”,
    query="blocked workstreams”,
    scope="observations”,
)

Now consider the PM asking the agent what the team should prioritize this week.

Without observations, the agent retrieves the blocker facts and lists them. Three people are blocked. Two mention the auth service. One mentions the payment integration. The PM gets an accurate summary and still has to work out the ordering.

With observations, the agent’s context includes the dependency chain. It reports that the auth service refactor is the root blocker, that unblocking it clears Alice and Bob directly and Clara downstream, and that the single highest-leverage action this week is getting the backend team the resources to finish it.

Same query, same data. One response describes the situation. The other identifies the action.

The progression of context in Zep:

• Episodes store raw conversation data

• Facts structure it into timestamped claims

• Entity summaries narrate the history of individual nodes

• Observations synthesize patterns across the cluster

Each layer answers a different type of question. Together, they move agents from recall toward understanding.

Alice, Bob, and Clara each filed an accurate status update. Every fact went into the graph correctly, and every one was retrievable.

What none of them could report, because none of them could see it, was the chain running underneath all three.

That’s the gap Observations closes. Not better retrieval of what people said, but structural analysis of what their combined data shows.

Resources:

• Zep Graphiti GitHub repo →

• Zep Observations docs →

👉 Over to you: have you built an agent where the memory system returned the right facts but still missed the structure connecting them?

​Automatic LLM routing in two lines of code!​

Teams usually hardcode the model directly into every LLM call, which means every task gets the same model regardless of what it actually needs.

But the prompt intent is already the signal you need to route differently.

A smart approach is actually implemented in ​Plano (open-source), which lets you do that with just one config change.

See the code snippet below:

Point base_url to Plano’s local endpoint, and your client starts routing based on prompt intent.

For instance:

• “Write a binary search” routes to `claude-sonnet-4.5` for code generation.

• “Capital of France?” routes to `gpt-4o-mini` for general tasks.

You don’t need to touch a single line of your agent code.

planoai obs is Plano’s observability command.

It gives you a live view of every routing decision, the model picked, the route taken, and the cost per request.

Plano is fully open-source, and you can self-host it natively or via Docker.

GitHub repo: ​https://github.com/katanemo/plano →​

(don’t forget to star it ⭐ )

To see this in practice, we put this exact pipeline in front of my Hermes agent, and the bill dropped 2x, without changing a line of agent code. I have also written an article with the complete walkthrough about it.

​Read it here →​

We have published a new deep dive in the RAG Systems course.

In a typical RAG request, embedding the query takes milliseconds, vector search takes tens of milliseconds, and reranking takes maybe a hundred milliseconds.

Then the model reads the retrieved chunks, and that step alone can take seconds. This deep dive is about that step.

Read the full deep dive here →

The full 50 min deep dive covers:

• Why prefill dominates RAG latency, not retrieval

• How the KV cache works and what it costs in memory

• Why prefix caching approaches zero hit rate for RAG workloads

• Three independent failures when you try to reuse cached chunks

• Six published approaches to fix them

• Hands-on implementation that covers every problem and fixes

• Best practices for production to assess what applies to your system

Read the full deep dive here →

Production RAG Deep Dive

Why care?

Processing 16,000 input tokens with a 14B parameter model on an NVIDIA L20 takes ~5.5 seconds to get the first token.

And in a RAG system, the vast majority of those tokens are retrieved chunks, not the user’s question.

Prefill is the root cause of this, and it scales quadratically with input length because every token must attend to every other token.

For production RAG systems serving thousands of requests per hour, this is the dominant line item on the inference bill.

The techniques covered in this deep dive (KV cache reuse, selective recomputation, prompt reordering, and more) directly solve that cost.

• CacheBlend, for instance, reuses precomputed chunk caches and recomputes only 10 to 15 percent of tokens, cutting time to first token by 2 to 3x while keeping answer quality within 0.01 to 0.03 F1 of a full recompute.

• TurboRAG reports up to 9.4x TTFT reduction by moving the entire prefill offline.

This is also an area the industry is actively investing in.

LMCache, the open-source package that implements CacheBlend, ships as a production integration on top of vLLM.

Serving engines are adding non-prefix cache reuse as a first-class feature.

The engineering teams working on inference optimization at scale are treating this as one of the highest-leverage problems to solve right now, because every percentage point of cache reuse translates directly into GPU cost saved.

Understanding how the KV cache works, why prefix caching fails for RAG, and what the three failure modes of naive cache reuse are is becoming essential knowledge for anyone building or operating a RAG system at any meaningful scale.

You can learn how to build such systems here →

Production RAG Deep Dive

👉 Over to you: Have you measured where your RAG latency actually sits?

ORM vs PRM: How LLMs verify their reasoning?

When a model solves a problem in steps, something has to score that work so the model can learn from it.

There are two ways to do it, and they differ based on what they assign credit to.

• An outcome reward model reads only the final answer and returns one number for the whole solution.

• A process reward model reads every step and returns a score after each one.

The process model wins as a judge because sometimes a model can land on the right answer through faulty reasoning.

• An outcome model cannot tell the difference because it sees the right answer and approves the whole solution, broken steps included.

• A process model catches the bad step, and a correct final answer does not save it.

OpenAI actually measured this difference.

The model’s weights stayed frozen while it generated many solutions per problem, and each scorer picked the one it rated highest.

The process scorer solved 78.2% of a math test set, the outcome scorer solved 72.4%, and taking the most common answer got 69.6%.

That is the ranking job, and step-level scoring done by process reward model is clearly better at it.

Using that signal to train the model is a different job.

The score stops being a verdict on finished work and becomes the objective the model is pushed toward.

• In ranking, a wrong rating picks one bad solution and the model never learns anything from it.

• In training, a wrong rating comes back as a weight update, and the model produces more of whatever earned it.

Which is why the two jobs need different scorers.

A process scorer has to be a model. It reads a step written in natural language and predicts whether that step was sound, so it carries the usual model gaps and misjudges some steps.

Ranking absorbs those gaps because nothing adapts to them. Training turns them into a target, since gradient descent will find whatever the scorer rates highly and push toward it, correct or not.

An outcome check on a math or code problem is not a model at all. It runs the test or compares the answer to ground truth.

There is no direction to push a rule like that. The answer is right or it is not.

So the choice comes down to what the scorer is made of, not how much it sees. A process scorer sees more and can be fooled by the model it is teaching. An outcome rule sees almost nothing and cannot be.

DeepSeek-R1 used exactly this. They tested process scoring during development and shipped a rule that checks the final answer and the output format.

As a practical takeaway:

• Use outcome scoring when the answer is checkable and the scorer sits inside the training loop.

• Use process scoring when the reasoning itself has to be correct, or when ranking finished candidates and guiding search across them.

To dive deeper, we covered all of this in depth across 13 parts of our RL series:

• Part 1 covers the foundations, the agent-environment loop, and the reward hypothesis →

• Part 2 covers MDPs and value functions →

• Part 3 covers Bellman equations and dynamic programming →

• Part 4 covers model-free learning, Q-learning, and SARSA →

• Part 5 covers function approximation and what breaks when states stop being enumerable →

• Part 6 covers deep RL and DQN →

• Part 7 covers policy gradients and actor-critic →

• Part 8 covers PPO and why it became the default for LLM post-training →

• Part 9 covers RLHF, preference data, and reward model training →

• Part 10 covers verifiable rewards and GRPO, which is the rule-based path R1 took →

• Part 11 covers the reward signal problem for agents →

• Part 12 covers environments, trajectories, and the training loop →

• Part 13 covers how AI teams use RL in production, including Cursor’s five-hour checkpoint loop →

Good day!

• ​Free Observability Engineering Masterclass with Liz Fong-Jones & Honeycomb.

• ​​Build a stock market research Agentic workflow.

• Double Descent vs. Bias-Variance Trade-off.

• [Hands-on] Data version control to build reproducible ML systems.

Free Observability Engineering Masterclass with Liz Fong-Jones & Honeycomb

Honeycomb is running a six-session live masterclass, turning theories from Observability Engineering (2nd Ed.) into practice. This masterclass series is ideal for individual contributors and senior leaders.

It’s taught by Liz Fong-Jones, who co-wrote the O’Reilly book Observability Engineering.

Each session covers one chapter and turns it into something you can use that same week in production:

• Aug 3 → Instrument with OpenTelemetry so you capture important bits.

• Aug 19 → Learn to wire observability into CI/CD and catch regressions.

• Sept 2 → Get your AI work funded when leadership keeps pushing it back

• Sept 16 → Set reliability targets that reflect what users actually feel

• Sept 30 → Cut your observability bill without losing the data

• Oct 14 → Learn Observability in every domain (mobile, data pipelines, ML systems, and IoT)

Live sessions include Q&A with Liz. Recordings and hands-on labs will also be made available after each one.

The first session is this Monday (3rd August).

Register here for free now so that you don’t miss out →

Learn AI observability for free.

Thanks to Honeycomb for partnering today!

​Build a stock market research Agentic workflow​

Recently, we talked about ​Sim, a lightweight, user-friendly framework to build AI agent workflows in minutes.

​Sim GitHub repo

Key features:

• Real-time workflow execution

• Connects with your favorite tools

• Works with local models via Ollama

• Intuitive drag-and-drop interface using ReactFlow

• Multiple deployment options (NPM, Docker, Dev Containers)

Based on our testing, Sim is a better alternative to n8n with:

• An intuitive interface

• A much better copilot for faster builds

• AI-native workflows for intelligent agents

We used it to build a stock market research agent & connected it to Telegram in minutes.

Tech stack:

• ​Sim​ to build the workflow

• Alpha-Vantage MCP to access stock market data

• Docker to locally host everything

The video at the top provides a step-by-step guide along with all the setup instructions.

You can also find all the details in Sim’s GitHub repository.

​GitHub repo →​ (don’t forget to star)

Double Descent vs. Bias-Variance Trade-off

It is well-known that as the number of model parameters increases, the model gradually overfits the data.

For instance, consider fitting a polynomial regression model trained on this dummy dataset below:

In case you don’t know, this is called a polynomial regression model:

Here, as we’ll increase the degree (m):

• The training loss will get closer to zero.

• The test (or validation) loss will first decrease and then increase.

This is also evident from the following loss plot:

But notice what happens when you continue increasing the degree (m):

Why does the test loss increase to a certain point but then decrease again?

What you see here is called the “double descent phenomenon,” and it is commonly observed in many deep learning models.

It depicts that increasing the model complexity beyond the point of interpolation can improve generalization performance.

And it’s hard to fathom since it challenges the traditional bias-variance trade-off:

To the best of our knowledge, this is still an open question, and it isn’t entirely clear why neural networks exhibit this behavior.

Some theories suggest that the model applies an implicit regularization that allows it to precisely focus on an apt number of parameters for generalization.

You can actually try it yourself:

• Create a small dummy dataset of size n.

• Train a polynomial regression of degree m, starting from 1 to a value greater than n.

• Plot the test loss and training loss for each m.

👉 Over to you: What do you think about the possible causes?

​Data Version Control​

Versioning GBs of datasets is practically impossible with GitHub because it imposes an upper limit on the file size we can push to its remote repositories.

That is why Git is best suited for versioning a codebase, which is primarily composed of lightweight files.

However, ML projects are not solely driven by code.

Instead, they also involve large data files, and across experiments, these datasets can vastly vary.

To ensure proper reproducibility and experiment traceability, it is also necessary to version datasets.

Data version control (DVC) solves this problem.

The core idea is to integrate another version controlling system with Git, specifically used for large files.

Here’s everything you need to know ​​(with implementation)​​ about building 100% reproducible ML projects →​

Good day!

​

Zep is an agent memory at enterprise scale.

Its Context Lake manages, governs, and serves everything your agents need to know about users, accounts, and your business, providing a prompt-ready Context Block in under 200ms for custom agents or any MCP client.

Learn more here →

Thanks to Zep for partnering today!

6 automatic optimization methods for LLM systems

Tuning an AI system no longer means training it.

Several methods now improve a system by editing its text, the prompt, the code, sometimes the training loop itself, and all of it runs automatically.

The visual below depicts several such methods:

They share one skeleton:

• An LLM proposes a change

• An evaluator scores it

• And the best changes are kept while the rest are dropped.

What separates them is what they edit and what feedback they get to read.

Let’s understand each of these techniques below.

1) OPRO, from Google DeepMind, treats the language model itself as the optimizer.

It hands the model a running leaderboard of past prompts and their scores, then asks it to write a stronger instruction each round, which gets scored and added back to the board.

This method typically surfaces instructions like asking a model to do deep research or think before it solves a problem.

It needs no gradients and labels beyond a score, which makes it the simplest of the six.

But it plateaus on hard tasks, reacts strongly to how the meta-prompt is phrased, and was never built to beat specialized solvers on large problems.

A meta-prompt is the prompt handed to the optimizer model, not the one being optimized. It holds the task description, the leaderboard of past instructions and their scores, and the request to write a better one.

2) MIPROv2, from DSPy, tunes two things at once:

• One is the wording of the instruction.

• The other is the set of a few-shot examples, the solved cases shown inside the prompt to demonstrate the task.

Those examples are not something a person hand-picks. MIPROv2 generates them by running the program over a batch of labeled data and keeping the runs that reached the correct answer, so the demonstrations come from what already works.

Alongside that, it writes candidate instructions informed by the data and the program, then runs a Bayesian search, which tries the promising combinations rather than every one, to find which instruction and example set score best together.

Pairing them in one search is a critical improvement, since an instruction written to fit the examples it ships with reads more cohesively than tuning either half alone.

It works well once a few hundred labeled examples are on hand. The catch is that it locks in its pool of candidate instructions and examples up front, so it cannot react to a specific failure the way a trace-reading method can, and running all the scoring trials gets expensive.

3) TextGrad, from Stanford, borrows backpropagation almost directly.

It turns a system into a graph where each node is a piece of text and each edge is an LLM call, then passes natural-language criticism backward through that graph so every component learns what it specifically got wrong.

That design lets it optimize whole multi-step pipelines, and even non-text artifacts like candidate drug molecules, through the same loop.

The interface mirrors PyTorch, so anyone who has written a training loop can pick it up. But the feedback grows unstable once the graph runs more than three or four nodes deep, and each step costs several LLM calls, which adds up fast.

4) GEPA, from Berkeley, reads the full execution trace of a run instead of collapsing it into a single score.

It diagnoses why the attempt failed, proposes a targeted fix, and keeps a Pareto set of candidates so a variant that is best at one slice of the task survives even when its overall average is worse.

Reading the trace, rather than a bare reward, lets it reach a working prompt in a fraction of the rollouts that reinforcement learning needs, with no weight updates at all.

Keeping the specialists on the frontier also stops a strong but narrow candidate from being averaged away too early. Its main dependency is that the traces have to be rich enough to diagnose, which not every task provides.

We covered GEPA in detail here if you want to learn more →

5) AlphaEvolve, from DeepMind, points the same loop at code.

Two Gemini models, one tuned for breadth and one for depth, propose diffs to a program, automated evaluators score each candidate, and the best survive into a database that seeds the next generation.

Surprisingly, this has produced results humans had not found, including a way to multiply 4x4 matrices with fewer multiplications than a method that stood for 56 years, and a data center scheduler now running in Google’s production fleet.

It even sped up a kernel used to train the Gemini models behind it. The limit is that every candidate has to be checked by an automatic evaluator, so it only fits problems where correctness can be measured by a machine (like it happened in the matrix multiplication case), and the evolutionary search burns a lot of compute.

6) AutoResearch, Andrej Karpathy’s method, runs the loop on a machine learning training script and lets it work autonomously.

A coding agent reads the code, edits it, runs a fixed five-minute experiment, commits the change if the metric improved, and reverts it with a git reset if it did not.

Because only improvements are ever committed, the codebase moves in one direction and never regresses, and the git history doubles as a readable log of what worked.

On his own training code, the agent found around 20 improvements in two days, including a bug in the attention implementation he had missed.

The tradeoff is that it cannot step backward to set up a bigger gain later, so it can get stuck in a local optimum, and it only works when experiments are deterministic and comparable.

The pattern underneath all six is similar. Turn a system into text, define a way to measure it, and let a language model improve it in a loop.

They differ mostly in what they edit and how much feedback they read, and no single one wins everywhere. The choice comes down to what is being optimized and what can be measured cleanly.

👉 Over to you: Have you used automatic optimization methods before?

Good day!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

AI enters a codebase as at least five distinct asset types.

External LLM APIs, MCP servers, agents, tool plugins, and datasets are each pulled in through a different mechanism.

A lockfile catches none of them. The API call lives in application code, the MCP server in a config, the dataset behind a loader script, so no single scan surface exists, and security teams end up tracking fragments.

Checkmarx AI Inventory (part of Checkmarx One) scans for all five in one pass and lists each with its provider, version, and exact file and line, exported as an AI-BOM for audits.

Gartner named Checkmarx a Leader in its inaugural Magic Quadrant for Software Supply Chain Security.

Learn more about Checkmarx AI Inventory here →

Thanks to Checkmarx for partnering today!

Why 86% of Claude Code bill has nothing to do with your prompts

You run a Claude Code session and ask it to build a simple Python utility. The session burns through millions of tokens. The Anthropic console shows the total, but it doesn’t show you where those tokens went.

You assume the cost is your prompts and the generated code. That assumption is wrong.

We traced a real Claude Code deployment across a 45-person engineering team over 30 days and broke down every token. Only 14% of input tokens were actual user prompts.

The rest was context that the developer never explicitly shared. Prior assistant context alone consumed 30-45% of the total input spend.

The scale of this problem is already visible in the industry right now.

Uber rolled Claude Code out to roughly 5,000 engineers and watched per-person bills climb to $500-$2,000 per month.

Another report of one developer on a $200 Max plan consumed $50,000 worth of tokens in a single month just by using the features as offered.

Claude Code has already crossed $1 billion in annualized revenue. The spend is real, but the visibility into what’s driving it is almost nonexistent for most teams.

The anatomy of a Claude Code session

Every Claude Code turn is a fresh API request.

The model doesn’t carry state between turns. Each message includes the full conversation history, the full tool schema, and every piece of context that’s been loaded into the session.

One thing to address upfront: Anthropic’s prompt caching does reduce the per-token price of replayed context to roughly 10% of the standard input rate. Claude Code is architecturally built around caching, and the cost figures throughout this article already account for it. Without caching, the picture would be dramatically worse. But caching reduces the price per token, not the volume of tokens being replayed. A bloated configuration still burns tokens on every turn, just at a discounted rate. The overhead is cheaper than it could be. It’s still overhead.

Here’s what that context stack looks like, broken down by where the tokens land.

Static overhead

Static overhead is the base system prompt, tool definitions, and behavioral instructions. It loads on every single turn, and you can’t configure it away.

In most individual setups without active cost management, it balloons significantly because the system prompt carries the weight of every tool definition, every MCP schema, and every behavioral instruction Claude Code needs to function.

These are fixed costs per turn that you pay regardless of what you’re working on.

Skills and CLAUDE.md

These are text files that give Claude Code domain-specific knowledge. They load into context based on glob patterns or always-on settings.

Most developers add them over time and rarely remove them. Skills that were useful for one project persist into the next, consuming tokens on every turn without contributing anything.

CLAUDE.md in particular is read at the start of every session, and every word in it costs tokens every time you send a message. Developers commonly accumulate CLAUDE.md files that grow to 5,000 or even 10,000 tokens.

There’s roughly a 10x cost difference between placing CLAUDE.md at the project root (where it gets injected into every tool call) versus in .claude/rules/ where it loads more selectively.

MCP server schemas

MCP server schemas are the tool definitions for every connected MCP server. Each server’s tools get serialized into a JSON blob describing every available tool, its purpose, and the shape of its inputs and outputs. That schema gets prepended to your context on every turn, not just once at session start.

A developer with 10 MCP servers and 50 tools between them can pay up to 16,000 tokens per turn just in schema definitions. Every MCP you connect stays connected until you remove it manually. Most developers never do.

Tool results and built-in tool calls

These sit on both sides of the token ledger. On the input side, tool results (outputs from file reads, bash commands, grep operations) account for 23% of input spend at enterprise scale. Every time Claude reads a file or runs a command, that output gets captured in the conversation history and replayed on subsequent turns.

On the output side, built-in tool calls (file writes, bash commands, code edits) consume 24% of output spend. These are the tokens Claude spends invoking tools rather than generating text you see.

Together, tool-related tokens make up the largest combined category in a typical session.

Prior assistant context

Simply put, this is the conversation history carried forward. Long sessions replay Claude’s own earlier thinking, text, and tool call results on every turn.

In a detailed breakdown of this category, 78% of prior assistant context cost came from replayed tool_use results, not the conversation text itself.

Previous tool outputs (file reads, grep results, bash commands) are the biggest replay tax, and they compound with every turn.

Thinking tokens

These are output tokens consumed by Claude’s reasoning process.

Until recently, Claude Code defaulted to “high” thinking effort on Opus. Developers on Reddit reported burning through Max limits 10x faster than on the previous model version.

Anthropic changed the default to “medium” after SWE-bench data showed 76% fewer output tokens at medium versus high with the same task completion rate.

At enterprise scale, Thinking can consume up to ~40% of total output spend.

Model selection

This is the silent multiplier behind all of this. Sonnet models cost roughly 0.6x what Opus costs per tier and performs comparably on most development work.

Running Opus by default for every task, including formatting, linting, and boilerplate generation, is one of the most expensive configuration choices a team can leave unexamined.

The visibility gap

Developers currently have a limited number of options for understanding their Claude Code costs, and each one stops short.

• The /cost command shows session-level totals. It tells you how much a session costs in aggregate, but it doesn’t tell you that 45% of your input spend was built into tool replays or that your MCP servers are adding thousands of tokens every turn.

• The Anthropic Console shows organization-level usage over time. It tells you the total tokens consumed across your workspace. If one team is burning tokens on MCPs they don’t need, the console shows a line going up. It doesn’t show why.

• Claude Code’s /context command shows a snapshot of your current context window usage. It breaks down how much of the window is consumed by the system prompt, conversation history, and tool definitions. It doesn’t track patterns over time, compare across developers, or tell you which configuration elements are costing the most money. Worth noting that /context also has known calculation bugs and doesn’t always return accurate token counts, so even at the individual level the numbers aren’t always reliable.

• Manual auditing works for one person. You can check your system prompt size or inspect .claude.json to count registered MCPs. But for an org with 50 or 100 developers, each with their own configurations, manual auditing breaks down. Configurations drift, MCPs accumulate and nobody removes anything because nobody knows what’s safe to remove.

The gap across all four is the same. None of them connect token attribution (which categories are eating tokens) to configuration policy (what should change) to implementation (applying changes across the team).

Those are three separate problems, and none of the built-in tools solves more than the first one partially.

What token-level observability looks like

Comet built Cost Intelligence inside Opik to close that gap.

They built a proxy plugin that intercepts API traffic between Claude Code and the Anthropic API. It hashes content for privacy (your prompts and code are never stored or sent to Comet’s servers), extracts structural metadata like category, character count, and model, and sends only the cost metrics to Opik.

No SDK integration and code changes needed. Data appears after the first traced session.

The entire setup is a single config block in your Claude Code settings file.

{
  “enabledPlugins”: {
    “opik-cipx@opik-enterprise”: true
  },
  “env”: {
    “OPIK_CIPX_API_KEY”: “<workspace-scoped-api-key>”,
    “OPIK_CIPX_BASE_URL”: “https://www.comet.com/opik/api”,
    “OPIK_CIPX_PROJECT”: “claude-code”
  }
}

Once connected, the Home dashboard shows a Sankey diagram that maps every token from its source category through the coding agent to its output category.

You can click into any category and see exactly what’s driving the cost. Clicking on Prior Assistant Context shows the cost split between replayed tool_use (64.5%), text (35.4%), and thinking (0.1%).

Click on MCP Servers, and you get a per-server ranked view showing total spend, call volume, and a “Waste” metric that flags servers with zero-call installs still loading their schemas every turn.

The User Leaderboard shows per-developer spend alongside their model choice, token consumption, skills count, MCPs count, and MCP call volume.

A “High spend” flag marks outlier users. This is the view that lets engineering leaders ask informed questions instead of enforcing blanket restrictions.

From visibility to fixes

The part that separates Cost Intelligence from a dashboard is the cost savings engine.

The Cost Savings page surfaces specific recommendations ranked by estimated savings, each with an explanation and a quality impact warning where relevant.

Here’s what those recommendations look like in practice.

• Switch default model to Sonnet: On development work, moving to Sonnet cuts the main-loop bill by roughly half. Sonnet costs 0.6x Opus per tier and handles the same tier-weighted work. Estimated savings can vary from 22% upto 48% of total spend.

• Compact context sooner: Long sessions re-read their whole context every turn at the cache-read rate. Lowering the auto-compaction threshold trims the prefix before each replay. Estimated savings can be 10-15% of spend.

• Lower thinking effort: Prior-assistant context replays Claude’s earlier thinking, so it compounds across turns. Disabling always-on thinking and capping thinking tokens shrinks both the current turn and every replay after it. Estimated saving of 6-7%.

• Block specific MCP servers. Cost Intelligence identifies servers with high inactive user counts, flags zero-call installs that still load schemas every session, and calculates recoverable spend per server.

• Cap output tokens. Bounds the maximum response length so long generations can’t run away on cost. This prevents those sessions where Claude produces thousands of lines of code you didn’t ask for, each of which gets replayed on subsequent turns.

• Disable auto memory. Auto memory reloads accumulated notes into context each session. Turning it off removes that recurring overhead while keeping project CLAUDE.md and rules unaffected.

• Drop git instructions: Git workflow instructions sit in the system prompt on every turn. Remove them where teams don’t rely on them.

Each recommendation comes with an “Apply” button that implements the configuration change directly.

For org-wide deployment, the settings export generates a managed-settings.json that enforces all enabled policies across the team.

Comet’s internal engineering team ran this same exercise on their own Claude Code usage before building it into a product. They traced cost patterns back to specific prompt structures, context buildup, and workflow behaviors using Opik.

After refactoring their configuration (centralizing rules, migrating always-on skills to on-demand, adopting short session loops), their median output cost dropped from $229 to $181 per million output tokens. That’s a 21% reduction with zero change in development velocity.

A 15-minute audit you can run today

You can audit your own setup right now. Here’s where to look.

• Audit your MCP servers: Open ~/.claude.json and your workspace .mcp.json. Disable any server you haven’t used in the past two weeks. The token savings from removing even two or three unused servers compound across every turn of every session.

• Shrink your CLAUDE.md: Keep it under 200 lines. Move domain knowledge, workflow guidance, and one-off fixes into on-demand skills that load only when relevant. The global file should contain only true invariants such as build commands, project architecture, and hard constraints.

• Set thinking effort to medium: Same task completion rate, 76% fewer output tokens. This is the single highest-leverage setting change available.

• Keep sessions short: Do one task, commit, compact or start a fresh session. Long sessions are where prior assistant context balloons and stale tool results keep getting replayed on every turn.

• Review your default model: If Opus is your org default, check whether each task type needs it. Switching to Sonnet for routine development work cuts costs by roughly 40% with comparable output quality for most coding tasks.

For individual developers, the Opik Claude Code plugin is free and open source.

Install it via a single command:

/plugin marketplace add comet-ml/opik-claude-code-plugin 

You’ll get session-level tracing with full span breakdowns.

For teams that need org-level attribution, configuration policies, and one-click optimization across every developer, Cost Intelligence turns what would take hours of manual auditing into a single dashboard and is the single best optimization you can opt for.

​Learn more about Opik Cost Intelligence here →​

Here’s the Comet ML Opik GitHub Repo →

(don’t forget to star 🌟)

👉 Over to you: Have you audited your Claude Code configuration for token waste, and what was the biggest surprise when you looked at where your tokens were going?

Good day!

• Your agents are doing web search wrong.

• 6 LLM deployment formats in production.

• ​Prevent data leakage in ML pipelines​.

Your agents are doing web search wrong

The agent isn't searching badly. It gets handed a table of contents and is left to go read the web itself.

A search call returns links and thirty-word snippets, so the agent fetches each page, strips the markup, and cleans the text before any reasoning starts, and that work repeats on every hop of the loop.

The same question runs above 28,000 tokens through a three-hop search loop and under 7,000 through an owned index that already holds the processed page.

Seltz is that index. It crawls and processes pages ahead of the query, so one call returns a finished document across its people, news, and wiki scopes.

Try Seltz here →

Get Seltz API key

Thanks to Seltz for partnering today!

6 LLM deployment formats in production

Google and Anthropic agree on one thing about LLM inference:

The faster a model runs, the fewer machines can run it.

This sounds counterintuitive, but a model file gets faster by locking in hardware-specific choices into the file.

And those choices only fit the hardware they were made for.

That is why they build the same model more than once, one version per chip or runtime it has to run on.

The visual below covers the six formats used to run LLMs in production:

1-2) Pickle and safetensors hold raw weights. Both run anywhere Python runs, so neither is faster than the framework around it.

A .pt file is a list of instructions that torch.load executes and loading a checkpoint can also run a payload (possibly malicious).

Safetensors holds plain numbers instead, laid out the way memory needs them, so there is nothing to rebuild when the file opens.

Both still need Python, the model code, and a tokenizer alongside.

3) GGUF is llama.cpp’s format, built so a model can run on a machine with no ML framework installed.

To achieve this, weights, tokenizer, and chat template ship in one file, and llama.cpp or Ollama runs it directly.

4) ONNX is an interchange format, built so a model trained in one framework can run somewhere that framework was never installed.

It decides the operations and the order they run in, but leaves the hardware open, so the runtime picks CPU, CUDA, or an NPU when the file loads.

5) MLX is Apple’s framework for its own silicon, where the CPU and GPU share a single pool of memory.

It decides where the weights live, so they are never copied between the two, and that advantage does not help on any other machine.

6) TensorRT is NVIDIA’s compiler, which turns a model into machine code for one specific GPU.

Everything is settled at build time, down to instructions tested on the exact card it compiles for, which is why the file will not load on a different GPU architecture.

The safe default is to stop as high on the chart as the latency budget allows, because every row down trades speed with the types of hardware it can run on.

To dive deeper into the full LLMOps lifecycle, we have covered every bit of it in the LLMOps course, starting from fundamentals to production:

• Read Part 1 on fundamentals of LLMOps here →

• Read Part 2 on understanding the core building blocks of LLMs →

• Read Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixture-of-experts, and the fundamentals of pretraining and fine-tuning →

• Read Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applications →

• Read Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and different prompting techniques →

• Read Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more →

• Read Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techniques for building high-signal inputs →

• Read Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic context injection, and common failure modes in agentic applications →

• Read Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong understanding of the fundamental concepts →

• Read Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for evaluation of LLM apps →

• Read Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming →

• Read Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO →

• Read Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model parallelism →

• Read Part 14 on the fundamentals of LLM serving, including API-based access, inference with vLLM, and practical decisions.

​Prevent data leakage in ML pipelines​

​Data leakage​ happens when your model accidentally gets access to information during training that it won’t have during inference.

And the scary part is that it can be incredibly subtle.

Let us walk you through the most common ways it sneaks in.

​To specifically explore data leakage in greater depth with code, we covered it in Part 6 of the MLOps course here →​

Why is data leakage dangerous?

A leaked feature can make a model seem extremely accurate during training/validation, because it’s indirectly using the answer!

But when deployed, that info isn’t available, so the model fails unexpectedly.

Leakage can be subtle and is often discovered only after deployment, when the model behaves too well on historical data but poorly on new data.

Common causes of data leakage

Train/test contamination

This is the most straightforward but pernicious case. Training data somehow bleeds into the test set. For example, random shuffling might break temporal integrity in time-series data, accidentally leaking future information into the model’s eyes.

Leaking through preprocessing

Do not scale or transform using the combined statistics of your dataset. This subtle form of leakage creeps in when the test data informs transformations like scaling.

For example, scaling features to 0-1 using the min and max of the entire dataset (including test) leaks knowledge of the test distribution into the train.

To prevent this type of leakage, always fit preprocessing only on the training set, then apply it to the validation/test sets.

Using target-derived features

This is a common mistake in feature engineering. For instance, say you’re predicting whether a user will churn next month, and you accidentally include a feature like “number of logins in next month,” which obviously references the outcome.

Sometimes this happens less obviously: e.g., you include a summary that was computed including the target period.

How to prevent this? Think carefully. Any feature that wouldn’t be available at prediction time (or that uses information from the future relative to prediction) is a leakage risk.

How to prevent leakage

Holdout validation

Always assess your models on a truly independent dataset. Steep drops in performance from training to evaluation signal potential leakage.

Feature importance analysis

Calculating feature importance reveals suspicious features masquerading as informative only due to leakage. If a particular feature is overwhelmingly important, inspect it; could it be leaking info?

Null model testing

In deliberately nonsensical setups (e.g., predicting 2019 with a 2020 model), leakage causes even ill-formed models to perform inexplicably well.

A leak-safe approach

1. Perform train/validation/test splits before peeking or processing.

2. Use stats from only the training data for things like scaling and encoding.

3. In time-dependent data, keep the chronological order for validation.

Understanding leaks and vigilantly safeguarding against them ensures your ML pipeline runs smoothly and reliably in the real world.

If you want to learn more about these real-world ML practices and start your journey with MLOps, we have already covered MLOps from an engineering perspective in our 18-part crash course.

It covers foundations, ML system lifecycle, reproducibility, versioning, data and pipeline engineering, model compression, deployment, Docker and Kubernetes, cloud fundamentals, virtualization, a deep dive into AWS EKS, monitoring, and CI/CD in production.

​Start with MLOps Part 1 here →​

​To specifically explore data leakage in greater depth with code, we covered it in Part 6 of the MLOps course here →

Good day!​

You have 80,000 agent trajectories from production. You need to find top 100 worth reviewing to improve your agent.

No LLM allowed to evaluate trajectories. How will you do this?

Let’s look at some approaches.

The simplest solution one could start with is random sampling. Pick 100 random trajectories and review.

But most production agents handle routine requests just fine, so you end up wasting a big chunk of your annotation budget.

Another approach can filter for longer conversations since 10+ user messages means more complexity.

But longer conversations skew heavily toward outright failures. You’ll surface obvious breakdowns but miss subtle issues hiding in conversations where the agent technically succeeded.

A ​recent paper from DigitalOcean takes a new approach.

It computes lightweight behavioral signals directly from the trajectory data using deterministic rules.

The signals fall into three groups:

1) Interaction signals:

• If a user rephrases the request or corrects the agent, that’s misalignment.

• Agent repeating itself is stagnation.

• User abandoning the agent is disengagement.

• User confirming something worked is satisfaction.

All are detected through normalized phrase matching and similarity checks.

2) Execution signals:

• A tool call that doesn’t advance the task is a failure signal.

• Repeated calls with identical or drifting inputs indicate a loop.

These are straightforward to extract from execution logs.

3) Environment signals, like rate limits, context overflow, and API errors.

• Useful to diagnose but not for training since they reflect system constraints, not agent decisions.

Each trajectory gets scored based on which signals fire, and you sample the highest-signal ones for review.

On τ-bench, they compared all three approaches on 100 trajectories:

• Random sampling hit a 54% informativeness rate.

• The length-based heuristic reached 74%.

• Signal-based sampling reached 82%.

This means roughly 4 out of every 5 trajectories are genuinely useful to improve the agent.

In fact, among conversations where the agent completed the task correctly, signal sampling still identified useful patterns in 66.7% of cases vs. 41.3% for random.

These are the subtle issues like policy violations, inefficient tool use, and unnecessary steps that don’t break the task but still matter for optimization.

The whole framework runs without any LLM overhead and can sit always-on in a production pipeline.

If you want to see this in practice, this signal-based approach is already integrated into ​Plano​, an open-source AI-native proxy that handles routing, orchestration, guardrails, and observability in one place.

​Here’s the Plano GitHub repo →​

​Here’s the paper on arxiv →​

👉 Over to you: What is your approach to solve this?

Serverless vs on-prem vs edge deployment

Serverless, on-prem, and edge deployment are three different answers to the same question of where your model actually runs, and each one costs you something different.

The visual below depicts them:

• Serverless means you hand your model to a provider that only turns on a machine when a request arrives. Between requests the container shuts down, so you pay nothing while idle. The weights get unloaded along with it, so the next call has to pull gigabytes back into memory before it can answer, which adds up to 90 seconds. A reranker sitting in the middle of a search cannot spend that long waking up, so you keep instances warm, and you are back to paying for idle GPUs.

• On-premise means you rent or own the GPU yourself and keep the server running inside your own network. Nothing leaves your walls, there is no cold start because the engine already holds the weights, and you pay a flat hourly rate instead of a per-token meter. The card runs whether traffic arrives or not.

• Edge means the model runs on the user’s own device, on its NPU or CPU, through a small local runtime. The model has to be shrunk down to fit, so it works offline with nothing ever leaving the hardware, but it handles one narrow task rather than a full pipeline.

On-premise is where serious teams land, and it only pays off if several models share one card.

In reality, this rarely happens because vLLM pre-allocates 90% of the GPU at startup and ignores every other vLLM instance on it, while HuggingFace TEI (Text Embeddings Inference) takes one model-id per process. Neither one knows the other is there.

So four small models end up on four separate cards. You switched to small models to spend less, and you are now renting four GPUs to run what one could hold.

The serving engine box in on-prem decides whether on-prem is cheap or just differently expensive.

The card is not the constraint. Instead, the missing piece is one server that holds all the models and shifts memory between them as traffic moves, instead of four servers each convinced they own the whole GPU.

The Superlinked Inference Engine (SIE) is the open-source project built for that slot. One server handles embeddings, reranking, extraction, and generation, loads a model the first time a request needs it, and drops the least recently used one when memory runs short.

GitHub repo: https://github.com/superlinked/sie

(don’t forget to star 🌟)

We also wrote a detailed breakdown on what SIE is and how to get started with it.

Read it here →

​CPU vs. GPU vs. TPU vs. NPU vs. LPU​

5 hardware architectures power AI today.

Each one makes a fundamentally different tradeoff between flexibility, parallelism, and memory access.

The visual below maps the internal architecture of all five side by side:

CPU

It is built for general-purpose computing. A few powerful cores handle complex logic, branching, and system-level tasks.

It has deep cache hierarchies and off-chip main memory (DRAM). It’s great for operating systems, databases, and decision-heavy code, but not that great for repetitive math like matrix multiplications.

GPU

Instead of a few powerful cores, GPUs spread work across thousands of smaller cores that all execute the same instruction on different data.

This is why GPUs dominate AI training. The parallelism maps directly to the kind of math neural networks need.

TPU

They go one step further with specialization.

The core compute unit is a grid of multiply-accumulate (MAC) units where data flows through in a wave pattern.

Weights enter from one side, activations from the other, and partial results propagate without going back to memory each time.

The entire execution is compiler-controlled, not hardware-scheduled. Google designed TPUs specifically for neural network workloads.

NPU

This is an edge-optimized variant.

The architecture is built around a Neural Compute Engine packed with MAC arrays and on-chip SRAM, but instead of high-bandwidth memory (HBM), NPUs use low-power system memory

The design goal is to run inference at single-digit watt power budgets, like smartphones, wearables, and IoT devices.

Apple Neural Engine and Intel’s NPU follow this pattern.

LPU (Language Processing Unit)

This is the newest entrant, by Groq.

The architecture removes off-chip memory from the critical path entirely. All weight storage lives in on-chip SRAM.

Execution is fully deterministic and compiler-scheduled, which means zero cache misses and zero runtime scheduling overhead.

The tradeoff is that it provides limited memory per chip, which means you need hundreds of chips linked together to serve a single large model. But the latency advantage is real.

AI compute has evolved from general-purpose flexibility (CPU) to extreme specialization (LPU). Each step trades some level of generality for efficiency.

The visual below maps the internal architecture of all five side by side, and it was inspired by ByteByteGo’s post on CPU vs GPU vs TPU. We expanded it to include two more architectures that are becoming central to AI inference today.

That said, if you want to get hands-on with actual GPU programming using CUDA, learn about how CUDA operates GPU’s threads, blocks, grids (with visuals), etc., we covered it here: ​​​​Implementing (Massively) Parallelized CUDA Programs From Scratch Using CUDA Programming.

👉 Over to you: Which of these 5 have you actually worked with or deployed on?

Good day!

• Agent memory and state are not the same thing!

• Graph engineering, clearly explained.

• ​The anatomy of diffusion LLMs.

Agent memory and state are not the same thing!

If an agent forgets something it has already learned, that’s a memory problem. If it forgets where it was in the middle of a task and starts over, that’s a state problem.

We once killed one of our agents mid-task to test something else, and it started over like the execution so far never happened.

That’s when it clicked that we’d been treating two different problems as one.

State is tied to the current run as to what task the agent is working on and what it’s already found.

None of that exists unless something writes it down.

The fix was to add a checkpoint after every completed step that records the agent’s progress, so if the process dies, it resumes from that exact point instead of starting from scratch.

Memory is a different thing entirely. It’s what survives across runs as facts, lessons, and findings that are worth retaining.

At first, we had one shared memory for all our agents and assumed that was enough. But it wasn’t until our agents started reading each other’s findings and treating them as their own.

That’s why giving each agent its own memory scope is important with memory = memory.scope(”/agent”).

Once we separated state from memory, everything became much easier to reason about.

Here’s what the final Agent Harness looks like:

• separate memory from state as two different problems

• scope memory per agent when findings shouldn’t be shared

• write a checkpoint after every completed task

• resume interrupted runs from the last checkpoint

• fork a checkpoint into a new branch without redoing previous work

This is the harness baseline to build any agent. It’s enough for general-purpose workflows, but coding agents and long-running systems need other layers along.

We wrote an article that builds that next layer from scratch. It walks through planning, the agent loop, subagents, sandboxing, memory, and checkpointing, one layer at a time.

The whole thing is built with CrewAI, a 100% open-source framework.

You can read the article here →

Graph engineering clearly explained

The moment you have several loops that need to work together, you have a coordination problem, and graphs are how engineers have always described coordination.

That’s the whole idea behind graph engineering that Peter Steinberger mentioned a few days back.

Today, let’s understand what exactly graph engineering is!

First, the graph itself

A graph is three things.

• Nodes are units of work, whether that’s an agent, a plain model call, a deterministic function, a tool, or a human approving something.

• Edges decide what runs next, either in sequence, in parallel, or conditionally based on what the last node produced.

• State is a shared object that flows along the edges. Every node reads from it and writes to it.

graph.add_node("research", research_agent)
graph.add_node("write", writer_agent)
graph.add_node("review", reviewer_agent)

graph.add_edge("research", "write")
graph.add_edge("write", "review")

graph.add_conditional_edge("review",
    lambda state: "done" if state.approved else "write")

This is the starter graph almost every example uses. A researcher gathers material, a writer drafts, and a reviewer judges. If the review passes, the run ends. If it fails, an edge sends the draft back to the writer.

There are three nodes and four edges, and one of those edges forms a loop.

But here’s what reframes everything. A single agent loop is just a one-node graph with an edge pointing back to itself.

Graphs don’t replace loops but rather connect and govern them.

The stack kept growing

The center of gravity in AI keeps drifting away from the model, and each shift picked up a name.

• Prompt engineering → The words you send.

• Context engineering → Everything the model sees, not just your instructions.

• Harness engineering → The code around the model that runs tools, tracks state, and handles errors.

• Loop engineering → The autonomous cycle that drives one agent toward a goal.

• Graph engineering → The coordination layer across many loops, covering what runs when, in what order, and who checks whom.

Each layer wraps the one before it:

A graph is made of loops, each loop needs a good harness, each harness call is a context problem, and every context contains prompts.

If you skip a lower layer, the graph just fails in a more elaborate way.

Of course, none of this is new technology.

LangGraph shipped this exact model involving nodes and edges over shared state back in January 2024.

Microsoft’s AutoGen has GraphFlow, and Google built ADK 2.0’s entire workflow runtime on the same idea.

So while the name is new, the practice isn’t. In fact, the discipline isn’t inventing graphs but rather knowing when to use one, and how to keep it from rotting.

That part has four hard problems.

Hard part 1: knowing when a node deserves to exist

The most common failure is turning “summarize this PDF” into a five-node graph with a fetcher, a chunker, a summarizer, a reviewer, and a formatter.

A node earns its place only if it represents a real specialty, meaning a different model, a different toolset, or a genuinely separate role like a read-only reviewer. Steps you could inline into an existing loop are not nodes.

A useful filter is that if you can’t draw the graph on a napkin, it’s too complex. And if collapsing two nodes into one loses nothing, they were never two nodes.

Hard part 2: keeping shared state clean

In a loop, the failure mode is context rot.

In a graph, the same problem moves into a shared state.

Every node writes to the state object, so an uninformed write in node two will become a confident input for node five.

Nobody notices until the output is wrong, and by then, the bad data has flowed through half the system.

The solutions are simple and boring, yet effective:

• Give the state a typed schema.

• Decide explicitly which nodes may write to which fields.

• Checkpoint state between nodes so you can replay a run and see exactly where it went bad.

One caution applies to replays though.

Nodes after a checkpoint execute again, so any node with external side effects, like sending an email or creating a record, must be safe to run twice.

Hard part 3: routing you can trust

An edge is a decision, and the question is who makes it.

If a model decides the route, you get flexibility and instability together.

The same state can take different paths on different runs, which makes debugging miserable.

Google’s design rule for ADK 2.0 is the cleanest position in the discourse. Deterministic code should control predictable routing, and models should only handle the steps that need actual judgment.

Route with code wherever the condition is checkable, and spend model calls only where interpretation is genuinely required.

Hard part 4: agents agreeing with each other

Loop engineering’s sharpest rule was to never let an agent grade its own homework.

Graphs raise these stakes.

If you have 20 agents built on the same base model and all are reading the same flawed context, then they will happily agree with each other, and models measurably prefer their own outputs.

You can fix this with a reviewer node that runs on a different model, give it fresh context instead of the full conversation, and anchor its verdict to evidence the graph can’t fabricate, like tests that actually ran or code that actually compiled.

Note: Cognition landed in the same place after a year of running Devin, their coding agent. Their working setup lets several agents read the work and weigh in, but only one agent is ever allowed to change anything.

Reading is safe to do in parallel, because a bad opinion costs you nothing until someone acts on it. Writing is where the damage happens, so you keep it in one place where you can see it.

Where the graph is an overkill

Most of the time.

Anthropic’s own numbers suggest this.

A single agent burns roughly 4x the tokens of a chat interaction, and multi-agent systems burn roughly 15x.

Every node you add multiplies that.

Anthropic’s multi-agent research system outperformed a single Opus agent by 90.2% on their internal research eval, because research fans out into independent searches naturally.

But their standing advice from Building Effective Agents hasn’t changed. Find the simplest solution possible, and only add complexity when the task demands it.

Even LangGraph’s own guidance says that if your agent is a straightforward loop with tools, LangGraph is overkill.

The decision rule is pretty straightforward: Reach for a graph when the work splits into genuine specialties, needs parallel fan-out and join, needs different models per step, or needs failure isolation and auditable routing. Otherwise, stay in the loop.

Where to start

You don’t need an org chart of agents on day one. Build up to it.

• Master a single loop first, with brakes, a real completion check, and a critic. A graph of weak loops is just distributed failure.

• Draw the graph on paper before writing code, and challenge every node to justify its existence.

• Define the state schema and write access up front. State drift is the main way graphs rot.

• Make the reviewer node a different model with fresh context, and anchor it to external evidence.

• Put budget caps on every node. A graph has many loops spending tokens in parallel, and a weak verifier now burns money concurrently.

Key takeaway

Graph engineering isn’t a new discipline replacing loop engineering. It’s the name that stuck for a decision every agent builder eventually faces. When one loop stops being enough, coordination becomes the engineering.

The word may not survive the year. The design question will.

Here’s a summary of key takeaways in graph engineering.

​The anatomy of diffusion LLMs​

We recently published a deep dive that covers one of the most important architectural shifts happening in language modeling right now: diffusion LLMs.

​Read the full Part 1 deep dive here →​

​Read the full Part 2 deep dive here →​

It builds a complete understanding from first principles:

• how autoregressive generation is structurally memory-bandwidth bound)

• why Gaussian noise can’t work on discrete tokens

• how masked diffusion solves this with an ELBO-derived training objective

• the math behind the forward and reverse processes

• unmasking strategies

• block diffusion for KV cache compatibility

• a detailed engineering comparison between the two paradigms.

• the training techniques that scaled dLLMs from 8B to 100B parameters (including converting pre-trained autoregressive models like LLaMA into diffusion models via attention mask annealing)

• the inference acceleration stack (block-wise KV caching with Fast-dLLM, confidence-aware parallel decoding, token editing with LLaDA 2.1),

• production serving with SGLang

• hands-on code for running Dream 7B and serving LLaDA 2.0,

• and a decision framework for when dLLMs actually make sense over autoregressive models.

​Read the full Part 1 deep dive here →​

​Read the full Part 2 deep dive here →​

Why care?

Every production LLM today, GPT-4, Claude, Gemini, LLaMA, generates text the same way: one token at a time, left to right.

Each token requires loading the full model weights through GPU memory, performing a tiny computation, and then loading all the weights again for the next token. On an A100, this means roughly 1 FLOP per byte of data moved, while the GPU is designed for 100+ FLOPs per byte.

​Diffusion LLMs​ take a completely different approach. They start with a fully masked sequence and iteratively unmask all tokens in parallel, using bidirectional attention at every step. This shifts inference from memory-bandwidth bound to compute-bound, which is exactly where modern GPUs are efficient.

The results are catching up fast. Block diffusion (BD3-LM) is within 0.5 perplexity points of autoregressive on LM1B. LLaDA at 8B parameters matches LLaMA 3 on MMLU and exceeds it on TruthfulQA and HumanEval. And models like Dream 7B are already being served in production with SGLang.

Understanding how it works at a mathematical level, from the forward masking process to the ELBO objective to block-level KV caching, is going to be increasingly valuable as these models scale.

​You can read Part 1 here →​

​You can read Part 2 here →​

👉 Over to you: Do you think the future of LLM generation is pure diffusion, pure autoregressive, or some hybrid of the two?

Good day!

Kimi K3 leans on a new mechanism called delta attention that does not keep a growing KV cache.

That is how it holds a million tokens of context without the memory blowing up.

Before we can understand delta attention, we need to understand attention itself.

It is a lookup. Every token stores a key, which works like an address, and a value, which is the content at that address.

To build its output, a token sends out a query, matches it against every key in the sequence, and pulls back a blend of the values whose keys matched.

Standard attention keeps every one of those key and value pairs as a list, one entry per token.

That list is the KV cache, and it grows with the sequence, so each new token has to scan the whole thing to build its output.

Delta attention keeps the lookup process but throws away the list.

The entire past collapses into one fixed-size matrix that still behaves like the lookup table.

When you hand it a key, it returns the value vector, blended from everything the past tokens wrote, and it’s weighted by how closely each stored key matches the one you handed in.

Writing is a constrained operation. A fixed matrix has no free slot to append to, so every write lands in a direction that earlier writes already occupy and merges the two associations together.

The delta rule handles this in two moves per token, depicted below:

→ It reads before it writes. It hands the matrix the new token’s key and sees what value the memory currently returns, its existing guess for that address.

→ It writes the difference, not the value. It compares that guess against the value it actually wants stored and writes only the gap. That gap is the delta, and it corrects the old association instead of stacking a new one on top.

The matrix also lets old entries fade over time, so a fixed size can keep absorbing a long sequence without filling up.

Standard attention remembers by keeping everything and pays a quadratic cost to re-scan it.

Delta attention remembers by rewriting one matrix and pays a linear cost.

Of course, a compressed matrix cannot store every token exactly, so recall of any single token becomes approximate. That is why production models interleave the two, i.e., a few full-attention layers for exact lookup and the rest running linear.

Read more in the official announcement blog here →

11 LLM evaluation methods

A model can answer a question correctly and still score close to zero on BLEU.

That happens whenever the model says the right thing in different words than the reference. BLEU compares n-gram overlap, so a clean paraphrase reads as a total miss.

This one failure is why LLM evaluation is fragmented into several methods, depicted below:

Each method encodes a different assumption about what a correct output looks like, and different metrics are useful in different use cases.

Let’s understand the 11 must-know LLM evaluation metrics below

1) BLEU

Splits output and reference into n-grams and measures how much of the output is supported by the reference, with a brevity penalty so short outputs cannot win by omission.

It works for translation and constrained generation where the reference wording is close to the only acceptable wording. Report corpus-level BLEU instead of averaging sentence scores, and supply multiple references when more than one phrasing is correct.

2) ROUGE

Flips the direction of BLUE and measures recall, meaning how much of the reference is covered by the output.

It is the default for summarization and extraction, where missing content hurts more than extra content. ROUGE-L uses longest common subsequence and tolerates reordering better than ROUGE-2. Recall alone rewards verbosity, so it needs a precision-side metric next to it.

3) BERTScore

Embeds both texts, matches each output token to its most similar reference token, and reports precision, recall, and F1 over those similarities.

It handles the paraphrase case that causes problems in BLEU and ROUGE. Absolute values lie within a narrow high band, so scores are only meaningful when compared across systems, not read as accuracy. Rescaling the baseline helps, and the embedding model has to match the domain and language.

4) G-Eval

It hands a task description and criteria to a judge model to generate chain-of-thought evaluation steps, then scores against those steps with the result weighted by token probabilities.

It fits subjective criteria with no reference answer, like tone, instruction following, or domain correctness. Criteria written as a concrete checklist score far more consistently than criteria written as adjectives.

Ideally, it is recommended to run the judge at a low temperature and calibrate against a small human-labeled set before trusting it.

5) LLM-as-Judge

The pairwise form feeds two or more outputs to a judge with a rubric, the judge picks the better one:

It is the practical way to compare two models, two prompts, or two retrieval configs when absolute scores drift between runs.

Make sure to randomize which output appears first, because judges might favor a position. You also need to control for length, since judges reliably prefer longer answers regardless of quality.

6) Human eval

Annotators score outputs across defined dimensions, and the aggregate becomes the reference that every automated metric gets calibrated against.

It belongs in two places, building the calibration set that validates a judge, and the final gate before release.

You must define the rubric before annotation starts and measure inter-annotator agreement. Low agreement means the rubric is ambiguous, not that the annotators are wrong.

7) LLM juries

Runs several judges independently over the same output and averages their scores into one verdict.

It exists because a single judge carries a stable bias, including a preference for outputs from its own model family.

Use different model families rather than the same model three times, since correlated judges average away noise but not bias. Several small models in a jury often beat one large judge at a lower cost.

8) DAG

Scores through a decision tree where each node asks one narrow question and the answer routes to the next node, with the final leaf carrying the score.

It fits rubrics with hard requirements and ordering, like format compliance, required sections, or a mandatory disclaimer.

Because branching is deterministic, the same output always gets the same score. Always put cheap deterministic checks near the root so failing outputs exit before any model call.

9) Trajectory accuracy

Captures the agent’s full sequence of thoughts, tool calls and observations, then scores that path against the expected one.

It matters because an agent can reach the right answer through the wrong path, burning tokens on redundant calls or touching a tool it should not have.

Score path and outcome separately, otherwise a lucky correct answer hides a broken execution path. This needs tracing in place first, since there is no trajectory to score without it.

10) Multi-turn eval

Treats the whole conversation as the unit and scores role adherence, knowledge retention across turns, and coherence.

Per-turn scoring misses the failures that only appear over time, like contradicting an earlier answer or dropping a constraint set five turns ago.

Run it on real conversation logs rather than synthetic two-turn exchanges, because rule retention failures show up at depth.

11) Safety eval

Runs bias, toxicity and PII classifiers in parallel over the output and flags violations instead of producing a quality score.

It belongs as a gate, not as a term inside an average. A single PII leak matters regardless of how good the mean quality score looks, which is how folding safety into an aggregate ends up shipping violations.

To use them in practice, most of these already ship as built-in metrics in Comet Opik, an open-source LLM evaluation and observability platform, including BLEU, ROUGE, BERTScore, G-Eval, LLM juries, trajectory accuracy, conversation-level metrics, and moderation, all running over traced production data.

Here’s the repo → https://github.com/comet-ml/opik

(don’t forget to star it ⭐️)

Also, to dive deeper in the full LLMOps lifecycle, we have covered every bit of it in the LLMOps course, starting from fundamentals to productions:

• Read Part 1 on fundamentals of LLMOps here →

• Read Part 2 on understanding the core building blocks of LLMs →

• Read Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixture-of-experts, and the fundamentals of pretraining and fine-tuning →

• Read Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applications →

• Read Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and different prompting techniques →

• Read Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more →

• Read Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techniques for building high-signal inputs →

• Read Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic context injection, and common failure modes in agentic applications →

• Read Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong understanding of the fundamental concepts →

• Read Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for evaluation of LLM apps →

• Read Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming →

• Read Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO →

• Read Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model parallelism →

• Read Part 14 on the fundamentals of LLM serving, including API-based access, inference with vLLM, and practical decisions.

👉 Over to you: What else would you add to the master tree?

Good day!

Long-running agent flows fail mid-run, and in most frameworks, that means restarting from zero and burning the tokens again.

​CrewAI just shipped checkpointing in v1.14. Every flow method becomes a recovery point, written automatically when an event you specify fires (e.g., method_execution_finished).

You can resume in one line and fork from any saved state into a new branch with full lineage tracking. An async TUI lets you browse checkpoints, inspect events, and resume or fork from the UI.

The pipelines become resumable, inspectable, and branchable processes, with zero extra infra.

​You can find the CrewAI repo here →​

5 techniques to optimize LLMs in production

A tricky LLM interview question:

You’re serving an LLM in FP16 on an A100.

Token generation is quite slow, so you upgrade to an H100, which has over 3x the FP16 compute.

But upgrading the GPU didn’t improve the token generation speed much.

Why did this happen?

LLM token generation is a memory bandwidth problem disguised as a compute problem.

An A100 can do about 300 trillion FP16 operations per second. But it can only move ~2 TB per second between HBM and its compute units.

During generation, every new token requires reading the model weights and the entire KV cache from memory. While the computation process finishes almost instantly, the chip still has to spend time waiting for data.

This is why the highest-impact serving optimizations all try to tackle the same two things:

• how much data moves

• and how often the GPU sits idle

The visual below covers five of them.

1) Flash attention:

Standard attention writes the full attention matrix in HBM, and it grows quadratically with sequence length.

Flash Attention splits Q, K, and V into tiles, computes attention inside on-chip SRAM, and writes only the final output back.

The output is mathematically identical, but the quadratic intermediate never exists in memory, so long sequences become relatively manageable.

2) Paged attention:

Naive serving reserves one contiguous memory chunk per request, sized for the maximum possible sequence length.

vLLM paper measured 60-80% of KV cache memory wasted this way.

Paged attention is inspired by virtual memory from OS concepts.

The cache stays in small fixed-size blocks mapped through a block table, allocated as generation proceeds and freed on completion, which reduces waste to under 4%.

3) Continuous batching:

Static batching waits for the slowest request in a batch before admitting anything new. So one long generation keeps every finished slot idle until it completes.

Continuous batching makes scheduling decisions after every decode step instead of after every batch.

When a request finishes, a queued request takes its slot at the next decode step, and the GPU stays saturated.

4) Speculative decoding:

A small draft model proposes K tokens cheaply, then the large target model verifies all K in a single forward pass, accepting matches and correcting the first mismatch.

The rejection sampling scheme guarantees an output distribution identical to running the large model alone, so the speedup costs zero quality.

5) Kernel fusion:

Every GPU operation runs as its own kernel by default, and each one writes its result to HBM for the next kernel to read back.

Kernel fusion merges LayerNorm, MatMul, and activations into one kernel that keeps intermediates in registers and SRAM.

Compilers like TensorRT-LLM and torch.compile do this automatically from the model graph.

That said, all five techniques above are built for one model owning the whole GPU.

Paged attention and continuous batching fit more concurrent requests when more memory is available for KV cache blocks.

So vLLM pre-allocates nearly all GPU memory at startup and turns it into one big block pool.

But production stacks rarely run just one model.

A typical pipeline also runs several small specialized models like embedders, rerankers, and extractors alongside the LLM.

Serving each one the standard way requires a separate vLLM or TEI process, and every process claims its memory upfront with no visibility into the others.

So a 600M-parameter embedder reserves a GPU it will never fully use, and models that could comfortably share one card end up spread across many.

This is also why moving to small yet better models does not lead to substantial cost savings alone.

Superlinked’s open-source inference engine (SIE) solves this by running all of those models behind one API with shared GPU memory and on-demand model loading, so the card is actually shared instead of claimed.

GitHub repo → https://github.com/superlinked/sie

(don’t forget to star it ⭐ )

We wrote a full breakdown of this small-model serving problem and how SIE solves it.

It covers why switching to small specialized models doesn’t reduce inference costs unless those models share GPUs, and why vLLM’s design makes that impossible.

You can read it here →

Good day!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

“You are still responsible for your software, just as before. You are not allowed to introduce vulnerabilities because of vibe coding. “

He said it while drawing the line between vibe coding and agentic engineering. Agents write more of the code now, but none of that takes the responsibility off you.

The assumption underneath that is that a careful enough reader catches the problem. But some failures don’t show up in anything there is to read.

For instance, a common fear with a RAG agent is that it could hallucinate when a question asks something outside its corpus.

But such cases are actually well handled by any competent model now. If nothing in the retrieved context looks relevant, there’s no material to build an answer on.

Instead, the majority of failures originate when the retrieved context has partial coverage.

The retrieval pipeline returns context that’s topically correct but doesn’t cover the full question, and the model completes the remainder from parametric knowledge.

There are no token-level labels in the output to tell what was generated using retrieved context and what came from weights.

Both are streamed the same way.

Detecting this for production-grade apps needs a metric written for it, one that’s also aligned with principles of agentic engineering.

And the solution is actually implemented in the eval skill that comes with Google’s Agents CLI.

We described the concern to Claude Code in plain English. It read the agent’s code, came back with a plan I approved.

It then reported that no built-in metric isolates the behaviour and wrote a custom rubric called corpus_abstention.

It assigned a single categorical verdict per case rather than aggregating everything into one score, since the built-in raters regenerate their rubrics each run and leave no stable number to trend.

• GROUNDED_ANSWER

• CORRECT_ABSTENTION

• UNGROUNDED_ANSWER (answered entirely from outside knowledge)

• MIXED_LEAKAGE (grounded, but slips in one unsupported claim)

• WRONG_ABSTENTION (refused something the docs actually covered)

After this, it automatically generated 33 scenarios partitioned by where the failure could occur, like:

• in-corpus

• off-domain

• out-of-corpus but plausibly answerable

• boundary cases where the topic is covered, but a specific detail isn’t.

The baseline score was 19 of 33.

• Off-domain passed 3 of 3, as expected.

• But 6 of 15 in-corpus cases retrieved the right document, cited it correctly, answered accurately, and added a claim the source never made.

The root cause was one line in the agent’s instruction: “If you already know the answer to a simple question and no document lookup is needed, you may respond directly without citations.”

The eval skill helped flag this, and then Claude removed it and forced retrieval on every question.

This took the suite to 30 of 33, and ungrounded answers went from 6 to 0. The full recording of our run is below, and we worked with the Google Cloud team on this.

Here’s the Agents CLI GitHub repo → https://github.com/google/agents-cli

(don’t forget to star it ⭐ )

Are we ready for an agent-native memory system?

First, what does that even mean?

It’s memory built for an agent that keeps running, not a store you bolt on at the end. It has to hold facts across sessions, update them when they change, drop the stale ones, and hand back the right piece when the agent asks. A real data system, not a pile of saved chunks.

A new study checked whether anything out there actually does this, and what decides the answer surprised me.

It takes 12 agent memory systems and splits each into four stages, covering how memory is represented, extracted, retrieved, and maintained. Then it checks which stage decides whether the agent stays correct.

Turns out it’s write time, not query time.

The systems that stay reliable under fact updates and cross-session reasoning are the ones that add structure as the memory is written. The graph-based methods lead right there.

The flat stores fail the same way every time. An LLM pulls out entities and relationships with no schema, and the graph fills up with generic nodes that nothing downstream can filter.

A few things follow from this.

• Structure has to be set at extraction. Once the graph is built from untyped nodes, no query can recover a distinction the extractor never made.

• Updates return stale facts. Append-only and freeform stores keep handing back old versions, which the paper calls hallucinations of the past.

• A bigger model does not save it. Swapping the backbone moves the scores but rarely changes which pipeline wins, so the memory’s behavior is set before generation runs.

The paper also weighs the cost. The heavily structured systems pay for it, sometimes orders of magnitude more time per query, and the win only holds when the upkeep stays local instead of rewriting the whole store on every change.

So the takeaway is less about picking a winner and more about where the structure lives. Decide it at write time, keep the maintenance cheap, and retrieval has something real to rank.

Retrieval only ranks what extraction chose to keep.

Link to the paper: https://arxiv.org/pdf/2606.24775

If you’re into agent memory systems, we wrote a full article on exactly this.

It walks through defining the schema as a Pydantic ontology in Zep, with custom entity and edge types, source and target constraints, and the extraction pipeline that turns raw conversations into typed, queryable memory.

This memory system is 100% open-source: https://github.com/getzep/graphiti

You can read it here →

Good day!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

Every dependency in a modern codebase shows up in a manifest, except the AI ones.

Open-source models, LLM APIs, MCP servers, and agents enter production through the same pull requests as ordinary packages, but no lockfile lists them and no dependency scanner sees them.

Checkmarx found that 70% of teams expect AI components in production, while 43% have zero governance over them.

Checkmarx AI Inventory (part of Checkmarx One) catalogs every model, SDK, and MCP server deterministically, traces each one to the exact file and line, and exports an AI-BOM for compliance audits.

Gartner named Checkmarx a Leader in its inaugural Magic Quadrant for Software Supply Chain Security.

Learn more about Checkmarx AI Inventory here →

Thanks to Checkmarx for partnering today!

5 LLM Quantization Techniques

A 70B parameter model in FP16 requires 140GB of memory for weights alone.

That exceeds any single GPU available today:

• An H100 carries 80GB

• An RTX 4090 carries 24GB

So serving the model in half precision means multi-GPU setups and tensor parallelism before the first token is generated.

Quantization reduces this footprint by storing weights in lower bits.

The standard scheme maps each weight from FP16 onto a small integer grid: compute the dynamic range of the tensor, divide it into 2^b levels (16 levels at 4-bit), round each weight to the nearest level, and store a scale factor to dequantize during inference.

At 4-bit, the weight memory drops 4x. The 140GB model fits in 35GB, within the range of a single 40GB or 48GB card.

That said, if rounding were lossless in practice, one method would suffice, and the diagram below would have had one row:

But it has five because weights and activations in large transformers are not uniformly important.

The LLM.int8() paper quantified this. Beyond roughly 6.7B parameters, every transformer layer develops a handful of hidden dimensions carrying values 20x to 100x larger than the rest.

These dimensions make up about 0.1% of the model’s features, but zeroing them out nearly destroys the model’s ability to predict text.

These outliers are also what break naive rounding. The quantization grid spans the range of the tensor, so a single activation of magnitude 60 among values near 1 inflates the scale factor and collapses the remaining values into a few levels, destroying their precision.

Each method in the visual shown earlier solves this saliency problem differently.

1) RTN (round to nearest) does not handle it at all. It applies the scale-and-round procedure directly with no calibration data, which makes it the cheapest baseline and the weakest at low bit widths, where accumulated rounding error is unrecoverable.

2) GPTQ corrects rounding damage as it happens. It quantizes the weights of a layer a few at a time, measures how much error the rounding just introduced, and nudges the not-yet-quantized weights to cancel out that error before moving on.

The adjustment is not uniform. GPTQ uses statistics from calibration data to work out which remaining weights can absorb the error with the least effect on the layer’s output, which is what separates it from blind rounding.

This runs fast enough to matter in practice. A 175B model quantizes to 4-bit in around four GPU hours. The known weakness is that fitting the calibration data this closely can tune the model to it, and the AWQ authors showed this hurts accuracy on inputs that look different from the calibration set.

3) AWQ protects the important weights before rounding instead of repairing damage afterward. It runs calibration samples through the model and watches which weights get multiplied by the largest incoming values, because a rounding error on those weights gets amplified the most in the layer’s output.

Only around 1% of weight channels turn out to matter this much. Rather than storing them at higher precision, AWQ multiplies them by a scaling factor before rounding, so they spread across more levels of the grid and lose less detail, and the math is rearranged so a matching inverse scale elsewhere keeps the layer’s output unchanged.

Every weight still ends up in plain INT4, with no special-case handling at inference time. AWQ also depends far less on which calibration samples were used than GPTQ does, which is a big part of why it became the standard choice in serving engines like vLLM.

4) LLM.int8() isolates the outliers instead of fighting them. When the model is loaded, it finds the dimensions carrying extreme values and splits every matrix multiplication into two:

• The outliers run in full FP16

• The other 99.9% of values run in INT8, and the two results are added back together.

Nothing needs to be computed ahead of time. That is why it is also available as the load_in_8bit flag in bitsandbytes.

The cost is speed. Splitting and merging every matrix multiplication is slower than running one optimized 4-bit kernel, so this path is common for local development and QLoRA fine-tuning, not for serving traffic at scale.

5) QAT modifies training rather than the quantization procedure. During a short fine-tuning run, every forward pass rounds the weights to INT4 before using them, so the model experiences the exact damage that quantization will cause, and its predictions are scored under that damage.

The weight updates still happen in full precision behind the scenes, since rounding itself cannot be trained through directly. Over the fine-tuning run, the weights drift toward values that keep the loss low even after rounding, so when the real quantization is applied at the end, there is almost nothing left to break.

Google produced its Gemma 3 QAT checkpoints with roughly 5,000 fine-tuning steps guided by the original model’s outputs, and the int4 versions hold close to full-precision quality at about 3x lower memory.

All five methods described above produce the same artifact, i.e., a model at a fraction of its trained precision.

They differ in where the outlier problem gets addressed, at rounding time, after rounding, before rounding, at inference, or during training itself.

To dive deeper, we have already written a full deep dive on Quantization, specifically, which covers several of these methods with their simplified mathematics.

Learn how Quantization optimizes LLMs to run them on tiny hardware here →

Good day!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

Part 13 of the Reinforcement Learning course is available now.

Here, we look at what actual AI labs and companies are actually doing with everything we learned across twelve chapters.

Read Part 13 of the RL course here →

RL Nanodegree Part 13

It covers four case studies, each showing a different role RL plays in the industry:

• Cursor’s real-time RL loop that ships improved checkpoints every five hours

• How frontier labs turned verifiable rewards into reasoning and agentic capabilities

• Scale AI’s enterprise results, where a 4B model beat GPT-5 on domain-specific tasks

• The emerging market where RL environments are built, priced, and sold as products

Read Part 13 of the RL course here →

RL Nanodegree Part 13

Why care?

Across all twelve chapters of this series, we built the full mechanics of RL piece by piece.

We covered bandits, value functions, Bellman equations, policy gradients, PPO, GRPO, judges, trajectories, and environments.

One real test of understanding RL is whether you can read a company’s blog post about their training pipeline and immediately identify the environment, the reward source, the trajectory structure, and the failure modes they are likely facing.

That is what this chapter exercises.

We cover stories of very different companies, each using RL in a way that maps directly onto concepts from this series.

The vocabulary you built is the same vocabulary these teams operate in, like reward hacking, credit assignment, environment design, and the tension between learned and verifiable rewards.

This is the final chapter, and this is what we have covered in the full course:

• Part 1: Foundations of Reinforcement Learning →

• Part 2: MDPs and Value Functions →

• Part 3: Bellman Equations and Dynamic Programming →

• Part 4: Model-Free Learning →

• Part 5: Function Approximation →

• Part 6: Deep RL and DQN →

• Part 7: Policy Gradients and Actor-Critic →

• Part 8: Proximal Policy Optimization →

• Part 9: RLHF →

• Part 10: Verifiable Rewards and GRPO →

• Part 11: Reward signal problem for Agents →

• Part 12: Environments, trajectories, and the training Loop →

• Part 13: How AI teams use RL in production →

👉 Over to you: What topics would you like us to cover in this RL series?

LLM Routing vs Mixture of Agents

When you’re building a system that needs to handle diverse queries, two architectural patterns are considered.

One picks the best model for each task, and the other gets multiple models to collaborate on the same task.

Both involve multiple LLMs, but what they do with those models is completely different.

The visual below explains how each one works:

In LLM routing, an intent classifier reads the prompt, and a cost ranker weighs latency, token price, and capability, so only one model gets the call while everything else stays idle.

• The prompt is classified by domain and action.

• A cost ranker picks the best model given latency and price constraints.

• One model responds, and the rest never see the query.

In Mixture of Agents, a set of consulting models each analyze the query independently and hands their output to a single aggregator, which uses every perspective to write the final answer and make the tool calls.