You’re fine-tuning a model for Python code generation. The data was generated using the strongest LLMs like Opus/GPT.

But the fine-tuned model performs better when you use a weaker teacher instead.

Why did this happen?

A stronger teacher model can produce worse fine-tuning results. This sounds counterintuitive, but it is a well-documented effect in knowledge distillation research.

Large models solve a basic problem using abstractions, type hints, and patterns.

A Qwen3-8B model does not have enough capacity to reproduce those patterns. So instead of learning clean solutions, it learns an approximation of something it cannot fully represent.

However, a weaker teacher solves the same problem correctly, but with simpler patterns that the student can actually replicate.

A recent paper from Fastino Labs also documented this.

The researchers used Pioneer, their fine-tuning agent that takes a task description, generates training data, selects a base model, runs experiments, and iterates until the model hits a performance target, all without human intervention.

During one of those runs, Pioneer fine-tuned Qwen3-8B on Python code generation.

The agent tried two different teacher models for synthetic data generation: one large frontier model and one smaller model.

• The frontier model’s data hurt performance.

• The smaller model’s data performed much better in fewer iterations.

And the fine-tuning Agent was smart enough to catch this behavior. It measured the results from both teachers, saw that the frontier model was making things worse, and dropped it.

A human engineer would likely have defaulted to a bigger model because it is the stronger model, and might not have questioned that choice.

The paper explains three reasons this happens:

→ Capacity mismatch: The student cannot learn the teacher’s internal representations when the gap is too large. Increasing teacher size first helps, then hurts beyond a certain point.

→ Forgetting pretrained knowledge: Qwen3-8B already knows how to write Python from pretraining. Fine-tuning on a complex coding style from a much larger model can overwrite that existing capability.

→ Over-complexity in training data: A large model will solve “reverse a linked list” with elegant abstractions and comprehensive error handling. That is correct code, but it is also unnecessary complexity for the task. A simpler teacher generates solutions that match the task’s actual complexity, and the student learns them cleanly.

As a takeaway, always match the teacher to the student’s capacity and the task’s complexity.

To fine-tune a 3B or 8B model on a well-defined task, a mid-tier teacher will often produce better training data than powerful one.

You can find the paper here →

How to beat GRPO without touching model weights

GRPO needs tens of thousands of rollouts to converge. Each rollout produces a 5,000-token trace full of reasoning steps, tool calls, and self-corrections, but GRPO reduces all of it to a single scalar reward.

So we end up backpropagating on one bit per trajectory while throwing away thousands of bits of structured signal.

GEPA takes a different approach.

Instead of computing policy gradients on that scalar, it hands the full rollout trace to a reflection LLM and asks “what went wrong, and how should the prompt change?”

The reflection model writes a new prompt, you test it, and if it improves, you keep it.

The paper came out in July 2025. It was accepted at ICLR 2026, DSPy made it a first-class optimizer, and Hugging Face and OpenAI both shipped cookbooks around it.

On compound AI systems (multi-module pipelines with separate prompts), GEPA matches or beats GRPO while spending 10-50x less compute and requiring no training infrastructure at all.

Let’s break down why it works, how it compares to GRPO, and how to use it in DSPy.

We started a course series on RL recently. Read Part 1 here →

This first chapter covers:

• what makes RL fundamentally different from supervised and unsupervised learning

• the agent-environment interaction loop

• the exploration-exploitation tradeoff

• multi-armed bandits as the simplest RL setting, four action-selection strategies (greedy, ε-greedy, optimistic initialization, UCB)

• and a complete hands-on implementation of the classic 10-armed testbed with results and analysis.

The signal compression problem in RL

Reinforcement learning on language models has a signal problem that most practitioners overlook. Every rollout an agent produces is a 5,000-token document, containing:

• Reasoning steps

• Tool calls

• Self-corrections

• Compiler errors

• Judge rationales

That trace is rich and structured, containing exactly the kind of diagnostic information you’d want to learn from.

While training the agent, GRPO takes all of that and reduces it to a single number.

And it throws away thousands of bits of structured info, which partly explains why it needs tens of thousands of rollouts to converge.

The signal isn’t sparse, but the final reward makes it sparse.

Letting the signal read itself

GEPA’s core idea is that the rollout is already a natural language artifact, so let an LLM read it. Don’t reduce the trace to a number.

Hand it to a reflection model along with the failure mode, and ask: “What went wrong here, and how should the prompt change?”

The reflection model writes a new prompt. You test it. And if it improves, you keep it.

That’s the full optimization loop. Everything else in the paper is engineering that makes it work at scale.

What GEPA actually optimizes

GEPA targets compound AI systems.

A pipeline of LLM modules with their own prompts, glued together by Python control flow. For instance, a multi-hop QA agent might have:

• A first-hop query writer

• A retriever

• A summarizer

• A second-hop query writer

• A final answerer

Each module has a prompt. GEPA evolves all of them.

The optimization target is simple: maximize expected metric on your task, subject to a rollout budget. The novelty is in how you spend that budget.

The feedback function

GEPA replaces your scalar metric with a feedback function μ_f.

It includes the same score that GRPO gives plus a natural language description of what happened.

• For multi-hop QA, it returns which gold docs you retrieved and which you still need.

• For instruction-following, it returns per-constraint pass/fail descriptions.

• For code generation, it returns the actual compiler errors and profiler traces.

• For privacy-preserving rewriting, it splits the score into quality and PII-leakage with breakdowns.

The 6-step algorithm

Each iteration of the main loop does this:

1. Pick a candidate prompt set from the population (Pareto sampling, more on this below)

2. Pick a module to mutate (round-robin across modules)

3. Sample 3 examples from the training set

4. Run rollouts and collect full traces plus feedback from μ_f

5. Reflect: feed traces and feedback to a reflection LLM, get a new prompt

6. Accept or reject: rerun on the same 3 examples. If better, keep it. If not, discard.

Repeat until the budget runs out and return the best candidate. The entire loop runs without gradients, PPO, or KL penalties.

GEPA vs GRPO, head to head

A quick look at GRPO first:

Both GRPO and GEPA take feedback and improve the system. That’s where the similarity ends.

GRPO updates model weights with policy gradients on scalar rewards. GEPA updates prompts with natural language reflection on full traces.

Here’s the side-by-side comparison:

One important caveat. GRPO can change what your model knows. GEPA can only change how you ask it.

If your base model can’t do the task at all, no prompt evolution will save you. Fine-tune when you need new capabilities. Use GEPA when you need to extract more from what’s already there.

A real example

Let’s make this concrete with a real example from the paper.

The task: HotpotQA is a multi-hop question answering benchmark. You get a question that needs information from two different Wikipedia articles to answer. You can’t find the answer in just one place.

Example question: “What is the population of the region containing the parish of São Vicente?”

To answer this, your agent has to:

1. 1st hop → Search “São Vicente parish”, retrieve a doc, learn it’s in Madeira.

2. 2nd hop → Search “Madeira population”, retrieve that doc, get the number.

3. Answer: Combine both

The agent has separate modules for each hop. We’re going to look at the prompt for the second-hop query writer, the module that decides what to search for after the first retrieval.

Inputs the module receives:

• question: the original user question

• summary_1: a summary of what the first hop retrieved

Output it produces:

• query: the search query for the second hop

Here’s the seed prompt that DSPy gives you by default:

“Given the fields question, summary_1, produce the fields query.”

This is generic since it just describes the schema and it scores around 38% on validation.

GEPA runs this on a few examples and watches what happens. For instance, the query writer might keep doing the same thing wrong, like it paraphrases the original question and retrieves the same documents it already had.

For our São Vicente example, given a summary about the parish, it would search “São Vicente parish population” again, retrieve nothing new and fail.

The reflection LLM sees this failure pattern across multiple examples in the trace. It writes a new prompt:

“Generate a search query optimized for the second hop of multi-hop retrieval. The first-hop query was the original question, so first-hop docs already cover the entities mentioned directly. Your goal: retrieve documents NOT found in the first hop but necessary to answer completely. Avoid paraphrasing the original question. Target connected or higher-level entities mentioned in summary_1 but not explicitly in the question. Example: if summary_1 describes a parish but the question asks about the wider region’s total population, your query should target the region, not the parish. So for a question about São Vicente’s region, query ‘Madeira archipelago population’ rather than ‘São Vicente population’.”

That rewritten prompt scores 69%, up from 38% on the seed.

The model and task stayed identical. The only thing that changed was the prompt for one module out of several.

The reflection LLM didn’t just rephrase the seed but it also absorbed an actual strategy from observed failures:

• Don’t paraphrase the question

• The first hop already covered the directly-mentioned entities

• Target the broader entity that connects them

• Here’s a worked example of the pattern

That’s the kind of information you cannot encode in a policy gradient. RL would just tell you “this trajectory was 0.3 below the group mean” and let backprop figure it out across thousands of tokens. GEPA writes the lesson down in plain English and ships it as the new prompt.

Pareto selection

The idea of evolving/optimizing the prompt is not new.

But most approaches often mutate from the best candidate so far, which sounds reasonable, but it collapses to local optima fast.

GEPA uses something smarter, borrowed from quality-diversity optimization.

Imagine three candidate prompts and four tasks:

A greedy approach will pick C every time since it has the best average.

But A is the only one who handles Task 1 well, and B does Task 2 well. If you only mutate C, you lose those strategies forever.

Pareto selection keeps anyone who’s best at at least one task. Then it samples parents weighted by how many tasks they win. So C is most likely to be picked, but A and B stay in the pool. Their distinctive strengths can later be combined with C’s.

This single design choice is what separates GEPA from earlier evolutionary prompt methods.

Where GEPA fits in the landscape

Quick map of who does what:

• APE, OPRO: Both use an LLM to propose prompt candidates scored by a scalar metric. APE generates candidates from input-output demos and picks the best. OPRO feeds prior prompts and their scores into a meta-prompt so the LLM can propose informed improvements. Single prompt, no reflection on traces.

• EvoPrompt, Promptbreeder: Evolutionary operators (crossover, mutation) applied to prompt populations via LLM calls. Promptbreeder adds a self-referential layer: it also evolves the mutation-prompts themselves. Both use scalar fitness for selection and target single prompts, not multi-module pipelines.

• Reflexion: Agents reflect on task feedback after each trial and store reflections in an episodic memory buffer for the next attempt. Improves per-instance behavior across retries, not population-level prompt evolution across a training set.

• TextGrad: PyTorch-style backpropagation but with natural language critiques instead of numerical gradients. An LLM propagates textual feedback through a computation graph to produce per-variable improvement suggestions. Single candidate per iteration, no population.

• MIPROv2: DSPy’s prior flagship. Bootstraps few-shot examples from training data, proposes instruction candidates grounded in data summaries and traces, then uses Bayesian Optimization (Optuna TPE) to search the joint instruction-demo space across all modules. Generates all candidates upfront rather than evolving them through reflection.

• GRPO: Actual RL with weight updates. Samples a group of rollouts per prompt, uses the group mean as baseline to compute per-trajectory advantage, then updates weights via policy gradients with a KL penalty. The only method here that changes what the model knows, not just how you prompt it.

GEPA borrows ideas from several of these methods. It uses verbal reflection like Reflexion, but applies it across a population of candidates instead of a single agent’s memory.

It evolves that population using selection pressure like EvoPrompt, but with natural language feedback driving mutations instead of scalar fitness.

It targets compound multi-module pipelines like MIPROv2, but evolves prompts iteratively through reflection instead of generating all candidates upfront.

The piece that’s new to GEPA is Pareto selection, which preserves candidates that are best at even one task rather than always mutating from the highest-average performer.

Using it in DSPy

The API is one line different from MIPROv2:

optimizer = dspy.GEPA(
    metric=metric_with_feedback,
    auto="medium",
    reflection_minibatch_size=3,
    candidate_selection_strategy="pareto",
    reflection_lm=dspy.LM("gpt-5", temperature=1.0, max_tokens=32000),
    use_merge=True,
    track_stats=True,
)

optimized = optimizer.compile(program, trainset=train, valset=val)

The catch: your metric function needs the right signature.

def metric(gold, pred, trace=None, pred_name=None, pred_trace=None):
    # return dspy.Prediction(score=float, feedback=str)

Return a prediction with both a score and a feedback string. The feedback is what gets fed to the reflection LLM, so make it diagnostic and specific.

If your feedback string is just “wrong answer”, you’re back to scalar territory, and GEPA degrades to a slower MIPROv2.

If your feedback says “missed entity X, retrieved doc Y when gold was Z, format violation in step 3”, then GEPA works well with that level of detail.

A 2026 reality check

GEPA beats GRPO specifically, not every RL method.

The field has stopped framing this as GEPA vs RL and started framing it as GEPA and RL. The paper itself points to hybrid recipes as the natural next step.

Reflection is far more sample-efficient than RL on compound systems. The two are increasingly combined, not pitted against each other.

One more nuance worth knowing is that Decagon’s March 2026 production ablations found that more data isn’t always better with GEPA. 20 to 100 examples often beats 500. The reflection loop overfits when you feed it too much.

GEPA learns from patterns in failures. With 50 well-chosen examples, the reflector sees a clean signal. With 500, it starts chasing noise.

Use small, high-quality training sets and don’t assume scale helps.

When to reach for what

If you’re building a compound AI system today, here’s the decision tree.

• Use GEPA when: You have a small training set, expensive rollouts, no access to weights, and a metric you can describe in words.

• Use GRPO when: You have abundant cheap rollouts, open weights, and a verifiable terminal reward.

• Use MIPROv2 when: You specifically need bootstrapped few-shot exemplars in your prompts.

• Use TextGrad when: Your computation graph is deep and you want explicit per-variable critique propagation.

For most practical compound-system work in 2026, GEPA is the default to try first.

RL still has its place, but it’s no longer the obvious default when reading a rollout costs less than running ten thousand more.

To dive deeper into RL, we started a course series on RL recently. Read Part 1 here →

The first chapter covers:

• what makes RL fundamentally different from supervised and unsupervised learning

• the agent-environment interaction loop

• the exploration-exploitation tradeoff

• multi-armed bandits as the simplest RL setting, four action-selection strategies (greedy, ε-greedy, optimistic initialization, UCB)

• and a complete hands-on implementation of the classic 10-armed testbed with results and analysis.

Thanks for reading!

Claude Code has two context gaps that no amount of CLAUDE.md optimization will fix.

The first is web scraping.

• web_fetch doesn’t return raw page content. It runs the page through a smaller model and returns a summary with a 125-character quote limit, so you can’t use it to extract full tutorials, product specs, or thread content.

• curl returns raw HTML but gets blocked by sites with anti-bot protection (Amazon, LinkedIn, most e-commerce), can’t render JavaScript SPAs, and fails at scale due to rate limiting. Both truncate at around 100KB.

The second is backend integration.

• When Claude Code talks to a backend like Supabase through MCP, it discovers state through multiple separate calls (list_tables, execute_sql, list_extensions), each returning a partial view.

• Auth provider config isn’t queryable at all. And when something fails, error messages don’t distinguish between platform-level and code-level rejections, so the agent enters retry loops that burn tokens with every attempt. In our recent test, a single RAG app built on Supabase consumed 10.4M tokens and needed 10 manual fixes.

Bright Data solves the first problem. InsForge solves the second.

Today, let’s look at how to set up both as skills in Claude Code, with an interesting way we tend to use them.

Bright Data setup

Bright Data skill (open-source) adds scraping infrastructure that handles everything web_fetch and curl can’t.

The agent gets a four-tier fallback that escalates based on what the target site requires, like native fetch, curl, browser automation, and a proxy network with residential IPs and automatic CAPTCHA solving.

For agent workflows, the more useful capability is structured data extraction.

Instead of raw HTML that the agent has to parse, Bright Data provides pre-built extractors for 40+ platforms (Amazon, LinkedIn, Instagram, TikTok, YouTube, Reddit) that return clean JSON with specific fields like product prices, review scores, profile data, and post content.

npx skills add brightdata/skills

This installs several skills covering scraping, search, structured data feeds, MCP orchestration, SDK best practices, and the bdata CLI.

InsForge setup

We covered the backend problem in depth in a recent issue. The same RAG app that consumed 10.4M tokens on Supabase consumed 3.7M on InsForge with zero errors.

InsForge (open-source, Apache 2.0) acts as the backend context engineering layer for agents using Skills and CLI.

• Install all four Skills (primary documentation and diagnostic layer):

npx skills add insforge/insforge-skills

This installs insforge (SDK patterns), insforge-cli (infrastructure commands), insforge-debug (failure diagnostics), and insforge-integrations (third-party auth providers). Total metadata cost: ~714 tokens at session start.

• Link the CLI to your project (primary execution layer):

npx @insforge/cli link --project-id <project-id>

Building a Google Doc clone

This 10 hr video teaches how to build a Google Doc clone:

Ten hours of video means hundreds of implementation details like real-time collaboration, document state syncing, editor toolbar structure, and permissions.

That’s a lot of context that Claude Code will likely struggle to scrape. Even if it manages to scrape, compression will remove many of those details.

With both skills installed, here’s what you can do:

I want to build what's shown here:
https://www.youtube. com/watch?v=gq2bbDmSokU
Use Bright Data skills to scrape it and then
InsForge as the backend to implement.
Add Google OAuth and build a clean Google-doc
like interface. On every doc, add an "Ask AI"
button that chats with GPT-4o about the content.
Use InsForge's model gateway for the LLM capabilities.

The video below depicts the whole build with the final app, built in one shot:

• Bright Data scraped the full video content (transcript, metadata, structured descriptions), and Claude Code used it as the build spec.

• InsForge handled the backend in one shot: Google OAuth, database schema with RLS, storage, edge functions, and the model gateway for GPT-4o chat through InsForge’s built-in functionalities.

Finally, it gave a working Google Docs clone with real-time editing, Google OAuth, and AI-powered document chat, built from a single prompt with zero errors.

The YouTube example is the simple case for the scraping side and this isn't limited to tutorials or building from scratch.

The same workflow applies to any technical content on the web, like:

• A Reddit thread where someone describes how they optimized their real-time sync.

• A Hacker News discussion walking through an auth architecture.

• A competitor's product page with a feature worth replicating.

Give the link to Claude Code with the Bright Data skill, and the agent scrapes the content, understands what was described, and implements it in the existing app.

So any technical content on the web can become a build spec.

For basic sources, native scraping tools work. Bright Data is essential with sources that actively resist scraping, like Reddit threads with aggressive rate limiting, Amazon product pages behind anti-bot detection, LinkedIn profiles that fingerprint your browser, and JavaScript SPAs that need full browser rendering to even load content.

• You can find the Bright Data Skills repo here →

• You can find Insforge GitHub repo here →

👉 Over to you: what skills live in your default Claude Code setup?

Naive RAG vs Blockify

There’s a new RAG approach that:

• cuts corpus size by 40x.

• reduces tokens per query by 3x.

• improves vector search relevance by 2.3x.

And it delivered 260% accuracy improvement on the medical RAG benchmark over standard RAG.

The diagram compares it with naive RAG:

Chunks in a standard RAG pipeline typically carry no info about version, clearance level, or source authority.

The embedding model encodes it the same way regardless of whether the chunk is an outdated draft or the latest approved version.

During retrieval, if an outdated chunk and a latest chunk get retrieved as context, the LLM has no signal to prefer one over the other.

So it combines both and hallucinates.

The issue is not retrieval but rather the representation. The unit itself is wrong, and the fix has to happen before retrieval, at the data layer.

Blockify is an open-source data preprocessing engine that solves this at the data layer.

The engine sits between the document parser and the vector store.

Here’s how it works:

• First, a context-aware splitter finds natural breaks (paragraph boundaries, section breaks, topic shifts).

• Instead of embedding raw segments directly, a purpose-built LLM processes each one and extracts structured knowledge units called IdeaBlocks (typically 2-3 sentences). Each unit isolates a different fact or concept.

• Each unit is paired with a contextualized question and answer. This mirrors how users query the system and ensures the query embedding sits closer to real queries in the vector space (HyDE does something similar).

• Each block also carries metadata info like entity name, entity type, version, and clearance level. This helps rank retrieval by recency and authority, not just similarity.

The pipeline runs in two stages.

• The Ingest model converts raw text into IdeaBlocks as described above.

• The Distill model then clusters semantically similar blocks across the full set and merges duplicates into one canonical unit before indexing.

The retrieved units now answer a specific question instead of returning a paragraph that might contain the answer somewhere in the middle.

On the published benchmarks:

• The pipeline reduces a corpus to roughly 2.5% of its original size while preserving 99% factual integrity.

• Token consumption per query drops by 3x, from 1.5k tokens (naive top-5 chunks) to 500 tokens (top-5 IdeaBlocks).

• Vector search relevance improves 2.3x, measured by cosine distance.

In medical evaluation, the same pipeline delivered up to 650% accuracy improvement on clinical-grade RAG with a quantized Llama 3.2 3B model running on-device.

Blockify composes with LangChain and LlamaIndex. You can swap out the chunking stage (NodeParser/TextSplitter) and produce IdeaBlock nodes that the rest of the pipeline consumes normally.

For storage, you can integrate it directly with most vector DBs like Milvus, Elastic, etc.

There is also a Claude Code skill in the repo that runs the full Ingest and Distill pipeline while referencing the project documentation.

For production workloads on Intel Xeon, an optimized build is available through OpenVINO.

Here’s the GitHub repo →

Thanks for reading!

Vector RAG scores chunks independently against a query. That works for single-hop fact lookups, but breaks when a query requires connecting info across multiple chunks because there’s no structural link between what gets retrieved.

FalkorDB’s GraphRAG SDK builds a knowledge graph from your source data (PDFs, CSVs, HTML, URLs), auto-detects an ontology using an LLM, and converts natural language into Cypher graph queries at query time.

Retrieval follows entity relationships instead of embedding distance, so the LLM gets a structurally connected context rather than isolated fragments.

On GraphRAG-Bench (ICLR’26), the solution ranked #1 overall across all four task types (fact retrieval, complex reasoning, contextual summarization, creative generation) against 8 systems, including Microsoft GraphRAG.

It’s LLM-agnostic, supports multi-agent orchestration with domain-specific KG agents, and runs on a single machine.

You can find the repo here →

[Hands-on] Build OpenClaw’s core in a single visual workflow

OpenClaw runs a seven-stage agentic loop with multi-channel routing, persistent memory, and tool execution, all inside a local gateway process.

The orchestration logic is powerful but opaque. It lives inside the runtime, and what you interact with is a JSON config file and CLI commands.

You can still somewhat control what the agent does, but not how the routing and decision-making are wired together.

In the demo below, we rebuilt OpenClaw’s core in a single visual workflow using Sim (open-source with 27k stars) and made that wiring explicit:

Sim is an open-source workflow builder where every routing decision, tool call, and memory read is a visible node on a canvas.

We rebuilt the full OpenClaw pipeline in it using:

• 25 blocks

• 29 connections

• short-term and long-term memory

• and multi-channel output across Telegram and Slack.

So essentially, you have the same capabilities, but the entire orchestration graph is inspectable and editable.

To build this, we also used the Sim Copilot to generate the workflow from natural language. A single prompt produced the entire 25-block workflow, all connections intact.

Moreover, the full stack is open-source, self-hostable, and runs local models through Ollama.

Watch the full walkthrough above in the video.

You can find the Sim GitHub repo here → (don’t forget to star it)

Thanks for reading!

There’s a new method that could replace LLM-as-a-judge for production agents.

Most teams rely on a giant LLM as a judge to evaluate and guard their agent. But it has two major drawbacks:

• It’s slow and expensive at inference time

• It often misses domain-specific failures

Vibe training flips this.

Researchers at Plurai distilled a small language model that’s specialized for your agent’s exact use case. The SLM becomes your evaluator and your runtime guardrail, both in one.

The training data isn’t hand-curated either.

They spin up a swarm of adversarial agents that debate and stress-test every use case your agent is supposed to handle. That synthetic interaction data trains the specialized SLM.

So the judge actually understands what wrong looks like in your specific domain.

The reported gains vs. standard LLM-as-a-judge setups:

• ~8x faster inference

• ~50% fewer evaluation errors

Smaller, faster, and more accurate because it’s specialized for the job. The SLM-for-agents thesis is playing out in a very concrete way.

If LLM-as-a-judge is your current evaluation layer, this is worth benchmarking against.

You can find the paper here →

And you can try it live here →

Thanks to Plurai for partnering today!

Who actually builds AI Image models

The AI image generation ecosystem runs only on about a dozen companies that train their own models from scratch.

Some build models and ship consumer products around them. Others build and distribute models as open weights or APIs without a consumer product. A growing group started as product companies and now trains their own models for control and differentiation. And a final layer routes requests across all of them.

That number has stayed roughly constant since early 2024, even as the total number of AI image products has exploded.

Training a frontier image model requires 800M+ image-text pairs, thousands of GPU-hours, and a research team iterating on diffusion or autoregressive architectures for months.

Most companies invoke an API instead. But which API you pick, and which layer of this stack your provider sits on, determines everything downstream, like your per-image cost, your latency budget, whether you can fine-tune for your use case, and how much you depend on someone else's roadmap.

This dynamic has produced a four-layer stack defined by a core distinction: whether a company owns the foundational model or builds on top of it.

This stack includes:

• Model-first companies (frontier model builders that have integrated their models into mass-market apps)

• Model-only companies that build and distribute foundational models via open source or APIs with no consumer product

• Product-first builders, companies that evolved from using third-party tools to building their own models for control and differentiation

• And finally, Orchestrators (platforms that provide convenient access and routing to models built by others).

Let’s walk through each layer, who the real players are, and where the boundaries are collapsing.

Four types of companies in AI image generation

1. Model‑first companies / frontier builders

These are companies that started with a foundational model (trained from scratch) and later integrated it into their own mass‑market user product.

Note: Some of these companies offer API-based access to their models, but most users access them through their own mass-market products.

2. Model‑only companies / foundational contributors

These companies build foundational models from scratch but do not build a mass‑market product around them.

Their priority is making the model accessible to others via open weights, API, or commercial licensing.

Some of these companies (Tencent, Alibaba) also have their own products. Still, their open‑source/API strategy is so significant that they belong here — their models live separately from their products.

3. Product-first builders

Companies that started as product solutions (design, e‑commerce, photo editing) using third‑party models, then realized they needed their own model to control quality, differentiation, and costs.

Today, they are full model builders.

4. Orchestrators/Inference platforms

These companies do not train their own foundational models. They provide access to others’ models, unified APIs, interfaces, or hosting for fine-tuning. They create value through convenient access but do not control the model layer.

Hybrid case: Adobe Firefly

Adobe doesn’t fit into one category. It is simultaneously:

• a model builder (Firefly Image Model 4 and Ultra, 18B+ assets, customization via Foundry),

• An orchestrator because Firefly Boards let users pick Google Imagen 3, OpenAI GPT Image, and others alongside Adobe’s own models.

Adobe has strengthened its position by becoming both a model builder and a major orchestrator of AI models for the creative industry.

This is a unique hybrid showing where the market is heading.

Where does the value go long‑term?

If you own your own foundational model, trained from scratch (pretraining), you control:

• your cost structure (no API bills),

• your latency and deployment,

• your differentiation (no one else has the same model),

• your independence (no one can cut you off).

If you don’t own a model, you compete on UX, curation, or price, but you are always a tenant on someone else’s land.

Only about a dozen companies worldwide train their own models from scratch and produce mass-market products. The rest, hundreds of apps and platforms, are built on top of those.

That distinction, owning your model or not, remains the most important line on the map.

👉 Over to you: Do you think the product‑born model builders (Recraft, Canva) will eventually overtake the lab‑born ones (OpenAI, Midjourney)? Or does being a model‑first company still carry an unassailable advantage?

Thanks for reading!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

The interaction above works in discrete steps. At each step, three things happen in order:

• The agent observes the current state of the environment (S). A state is a description of the situation the agent is in, enough to decide what to do next. For instance, in chess, the state is the board position, and in a dialogue model, the state is the conversation history so far.

• The agent picks an action (A) based on what it sees. The action is the agent’s output, the only way it can influence the environment. For instance, in chess, an action is a legal move. For an LLM, an action is the generated response.

• The environment then does two things: it transitions to a new state (S’), and it emits a reward (R), a scalar number that evaluates the action. The next step begins, and the loop continues.

Stringing these steps together gives a trajectory:

Reading left to right, this is the entire history of the agent’s interaction with the environment. Each (S, A, R, S’) quartet is one transition, and much of RL is about learning from these transitions.

Applying RL to LLMs

When RL was first applied to LLMs, the environment was human preference.

OpenAI’s InstructGPT (2022) introduced RLHF (Reinforcement Learning from Human Feedback), where:

• humans ranked model outputs

• those rankings trained a reward model

• and PPO (Proximal Policy Optimization) used that reward model to fine-tune the LLM.

ChatGPT was built on this exact pipeline.

But humans can’t sit in the training loop rating every output in real time. If the model generates 16 responses per prompt across thousands of training steps, that’s hundreds of thousands of evaluations.

OpenAI solved this by splitting the process into two phases.

• First, the offline phase. Here, humans ranked a relatively small set of model outputs and generated pairwise comparisons. This was the expensive human labor part, but it was a one-time cost.

• Second, they trained a reward model on those rankings, which was a separate LLM that learned to predict what humans would prefer. Now you had a neural network that could score any output instantly, without waiting for a human. The reward model was a compressed approximation of human judgment, fast enough to sit inside the training loop.

With the reward model in place, PPO could run the actual RL training at GPU speed. The model generated responses, the reward model scored them, and PPO updated the weights, without extensive need for humans.

The cost, however, was that PPO required four full-size models in memory simultaneously.

• The policy (the LLM being trained).

• The reference policy (a frozen copy of the original, used to prevent training from drifting too far via a KL divergence penalty).

• The reward model (the human-preference approximator discussed above to score every output).

• And the critic, also called the value model (more about it below).

The critic exists to answer one question:

Was this reward good or bad relative to what we’d normally expect for this prompt?

We need this because a raw reward of 0.7 means nothing in isolation. For instance, on a simple factual question where most responses score 0.9, a 0.7 is below average.

But on a complex open-ended question where most responses score 0.4, a 0.7 is excellent.

The critic learns this baseline by observing thousands of (prompt, reward) pairs during training.

PPO’s actual training signal is the advantage, which is estimated as the reward minus the critic’s predicted baseline.

This makes the signal stable across prompts of different difficulty. But the cost involved here is that the critic is a full-size LLM itself, adding another model’s worth of memory.

For a 7B parameter LLM, that meant roughly 28B parameters in memory at once.

DeepSeek R1 breakthrough using verifiable rewards

In January 2025, DeepSeek released R1 with a fundamentally different approach to the reward signal.

Instead of training a reward model from human preferences (Phases 1 and 2 of the RLHF pipeline), they used RLVR (Reinforcement Learning with Verifiable Rewards).

It’s a simple, rule-based verification where the environment itself provides the signal.

For instance:

• For math problems, the verifier checked if the model’s answer matched the known solution.

• For code, a compiler ran the output and returned pass or fail. Binary rewards: 1 for correct, 0 for wrong.

There are no human rankings or explicit reward models required since the ground truth was available (or inferable) to be used as the reward.

The RL optimizer was GRPO (Group Relative Policy Optimization), which stripped away most of PPO’s infrastructure.

It removed the critic model entirely.

Instead of training a separate model to predict expected reward per prompt, GRPO generated multiple responses to the same prompt (typically 16) and normalized rewards within each group.

If 4 out of 16 responses got the math problem right, those 4 received a positive advantage, and the other 12 received a negative advantage.

This step cut an entire full-size model from memory.

GRPO also removed the need for the learned reward model, since RLVR’s verifier handled scoring directly.

So the four-model PPO setup (policy + reference + critic + reward model) collapsed to just two, i.e., the policy being trained and a reference copy for KL regularization.

In fact, in practice, some implementations fold the reference into the policy checkpoint, bringing it close to a single-model setup.

With this setup. DeepSeek R1-Zero, trained with just GRPO and verifiable rewards (no supervised fine-tuning at all), went from 15.6% to 77.9% on AIME 2024 math problems.

With majority voting, it hit 86.7%, matching OpenAI’s o1.

The model developed self-verification, reflection, and chain-of-thought reasoning on its own, purely from the binary correct/incorrect signal, and nobody taught it to reason step by step.

The RL training loop discovered that reasoning improved the reward, so the model learned to reason.

RLVR with GRPO became the dominant approach for training reasoning models through 2025.

Every major lab released a reasoning variant following this recipe.

The problem

GRPO itself is general-purpose.

It doesn’t care whether the reward comes from a math verifier, a code compiler, a human, or a Python script.

It just needs a number for each response, and it normalizes within each group to produce the training signal.

But a clear bottleneck here is where these reward comes from.

For math and code, this is fine since the environment provides a deterministic signal.

But agents that interact with real-world tools and data don’t produce outputs you can string-match against a gold answer.

A RAG agent retrieves context and generates a response. There’s no single correct answer to compare against. A customer support agent drafts a reply. There’s no compiler to run it through. A summarization agent condenses a 20-page document. There are many valid summaries, and no string-matching verifier can distinguish a good one from a mediocre one.

In these cases, the environment doesn’t hand you a reward signal the way a math problem does.

Of course, some agent tasks do have verifiable outcomes, and for these, RLVR works just fine, even with multi-step tool use. The verifiability depends on the task’s outcome, not on whether the model is acting as an agent.

But for the majority of agent workflows, the outcome is subjective or multi-dimensional.

Intuitively, GRPO is still the right fit here because Agents that take multiple steps, call tools, and compose responses would benefit from learning through exploration, trying different approaches, and getting reinforced for what works.

So, while the RL framework is the right fit, the missing piece is the scoring function.

One solution is to write custom reward functions where Python code scores each output based on hand-defined criteria.

• A RAG reward function might check whether the response used the retrieved context (faithfulness), penalize content that wasn’t in the context (hallucination), reward completeness, and handle cases where the context itself is ambiguous.

• A tool-use reward function might score partial progress through a multi-step task, penalize unnecessary API calls, and measure whether the agent reached the correct final state.

Each criterion returns a partial score, and these get summed or weighted into a final reward.

This works, but it introduces its own set of problems.

Writing a good reward function takes days of iteration. Researchers need to anticipate edge cases, calibrate the weights between different criteria, and test that the function actually rewards the behavior you want.

A reward function that over-weights format compliance and under-weights faithfulness will train an agent that produces beautifully formatted hallucinations.

Reward functions are also brittle. If you change the retrieval pipeline, add a new tool, or modify the system prompt, the reward function needs to be rewritten.

Debugging is problematic too.

When the agent learns bad behavior during training, the cause could be the reward function, the training hyperparameters, the data, or something else entirely.

But because the reward function is custom code, you often can’t tell whether the function is measuring what you think it’s measuring until you’ve already trained a model on it and evaluated the outputs.

This is the primary reason RL has been widely adopted for verifiable tasks (math, code, logic) but not for agent workflows (RAG, customer support, tool use, summarization).

RLVR gave reasoning models a general-purpose, automatic reward signal where they could check the answer and return 0 or 1. No such equivalent exists for most agentic workflows.

The distinction isn’t about the model. The same Qwen 2.5 14B can serve both roles.

The distinction is about the task. Can we verify if an Agent is producing an output that can be automatically checked?

How are AI labs approaching this?

This isn’t a gap that only open-source practitioners are noticing.

The major AI labs have been converging on the same problem from different directions.

Anthropic demonstrated that you don’t need humans in the RL loop at all.

Their Constitutional AI work showed that if you write down a set of principles (a “constitution”), an AI can evaluate outputs against those principles and generate preference data for RL training.

The AI judged its own outputs against the written principles and used those judgments as the RL signal. This was a significant conceptual shift that a document of rules replaced an army of human evaluators.

OpenAI has been working on something similar internally. They are developing “Universal Verifiers,” a technique to extend RL beyond math and code into domains like biology, medicine, and general knowledge, where answers can’t be checked with a simple string match.

The details aren’t public, but the direction is clear that we need general-purpose reward signals that work across any domain, not just the ones with deterministic verifiers.

RULER

If you want to see this in practice, RULER, built into OpenPipe’s ART framework (open-source with 9k+ stars) is a general-purpose reward function that replaces all of that custom scoring code with a single function call.

It uses an LLM-as-judge to rank multiple trajectories, and it works by exploiting the same property that makes GRPO powerful, i.e., only relative rankings matter.

Here’s how it works step-by-step:

• For each training step, you generate N trajectories for the same scenario (typically 4 to 8).

• RULER sends all N to a judge LLM (like o3, o4-mini, or even a local Qwen3 32B).

• The judge reads the agent’s system prompt to understand what the agent was supposed to do, then scores each trajectory from 0 to 1 relative to the others.

Two properties make this work:

1) Relative scoring is easier than absolute scoring.

LLMs struggle with absolute scoring because there’s no shared calibration.

But asking “which of these 4 responses best follows the system prompt’s instructions” is a comparison task, and LLMs do those consistently well.

RULER leans into this by presenting all trajectories together and asking the judge to rank them against each other.

2) GRPO normalizes within each group anyway.

Whether the best trajectory scored 0.9 or 0.3 in absolute terms doesn’t matter.

GRPO takes the scores within a group, computes the mean and standard deviation, and normalizes.

The training signal comes from the relative ordering by understanding which trajectories were above average and which were below. RULER’s relative rankings map directly onto what GRPO expects.

A rough walkthrough

Before jumping into code, let’s trace what happens conceptually. Say you’re training a RAG agent. At each training step, GRPO generates multiple responses for the same query:

Scenario: "What is the refund policy?"
Retrieved context: "Refunds within 30 days. Digital products non-refundable..."

(Faithful)
Response A: "Refunds within 30 days. Email support@example.com."     

(hallucinated)
Response B: "Refunds within 30 days. Also store credit for 90 days."

(ignored context)
Response C: "Not sure, check the website."

(verbose but accurate)
Response D: "The policy states that refunds are available within..."

In a traditional setup, you’d write a reward function to score each of these:

def reward_function(response, context):
    score = 0.0
    if uses_context(response, context):
        score += 0.4
    if not has_hallucination(response, context):
        score += 0.3
    if is_complete(response, context):
        score += 0.2
    if is_concise(response):
        score += 0.1
    return score

Each of those helper functions (uses_context, has_hallucination, is_complete, is_concise) is its own engineering project.

You need to define what “uses context” means precisely, decide thresholds, handle edge cases, and test everything.

With RULER, you replace all of that with:

scored_group = await ruler_score_group(group, "openai/o3")

The judge LLM reads the system prompt (”Answer using ONLY the retrieved context. Do not add information not in the context.”), reads all four responses and scores them.

The system prompt already defines faithfulness, hallucination, and completeness implicitly. The judge applies those criteria without implementing them in Python.

Trajectories and Groups

ART represents each agent response as a Trajectory, and it is a sequence of messages (system, user, assistant) packaged with metadata that GRPO needs for training.

Multiple trajectories for the same scenario form a TrajectoryGroup. This is the unit RULER scores and GRPO trains on.

# A single trajectory: one complete agent interaction
traj = art.Trajectory(
    messages_and_choices=[
        {"role": "system", "content": "You are a RAG support agent..."},
        {"role": "user", "content": "What is the refund policy?\n\n[Context]: ..."},
        Choice(finish_reason="stop", index=0,
               message=ChatCompletionMessage(role="assistant", content="...")),
    ],
    reward=0.0,  # RULER fills this in
)

# A group: multiple trajectories for the same scenario
group = art.TrajectoryGroup([traj_a, traj_b, traj_c, traj_d])

# Score the entire group relatively
judged_group = await ruler_score_group(group, "openai/o3")

The reward=0.0 initialization is a placeholder.

• After ruler_score_group returns, each trajectory’s reward field is updated with the judge’s score.

• The Choice and ChatCompletionMessage objects are OpenAI’s standard types, so if you’re already using the OpenAI SDK for inference, these are the same objects you’re already working with.

Two concrete examples

RULER has two levels of API.

1) The lower-level ruler function works with plain message dictionaries, so no ART-specific objects are needed.

This is the fastest way to see what RULER actually does.

Let’s look at a minimal example with three trajectories for the same RAG query.

Below, we have defined three message lists: one is faithful, one hallucinates, and one ignores the context entirely.

message_lists = [
    # Good: uses retrieved context accurately
    [
        {"role": "system", "content": "You are a RAG-based support agent. Use the retrieved context to answer user queries accurately."},
        {"role": "user", "content": "What is our refund policy?\n\n[Retrieved context]: Refunds are available within 30 days of purchase. Digital products are non-refundable after download. Contact support@example.com to initiate."},
        {"role": "assistant", "content": "You can get a refund within 30 days of purchase. However, digital products are non-refundable once downloaded. To start the process, email support@example.com."},
    ],
    # Mediocre: partially uses context, adds hallucinated details
    [
        {"role": "system", "content": "You are a RAG-based support agent. Use the retrieved context to answer user queries accurately."},
        {"role": "user", "content": "What is our refund policy?\n\n[Retrieved context]: Refunds are available within 30 days of purchase. Digital products are non-refundable after download. Contact support@example.com to initiate."},
        {"role": "assistant", "content": "Our refund policy allows returns within 30 days. You can also get store credit for up to 90 days. Just call our hotline at 1-800-REFUND."},
    ],
    # Bad: ignores context entirely
    [
        {"role": "system", "content": "You are a RAG-based support agent. Use the retrieved context to answer user queries accurately."},
        {"role": "user", "content": "What is our refund policy?\n\n[Retrieved context]: Refunds are available within 30 days of purchase. Digital products are non-refundable after download. Contact support@example.com to initiate."},
        {"role": "assistant", "content": "I'm not sure about the refund policy. You should check the website or contact someone from the billing team."},
    ],
]

Next, we run the scoring:

from art.rewards import ruler

scores = await ruler(message_lists, "openai/o3")

for label, score in zip(["Faithful", "Hallucinated", "Ignored context"], scores):
    print(label)
    print("→", score.score)
    print("→", score.explanation)

This produces the following output:

Faithful:
→ 0.97 
→ Accurately reflects the retrieved policy
  details, complete and concise.

Hallucinated:
→ 0.45
→ Gives correct 30-day refund info but adds
  unsupported details (90-day credit, hotline),
  reducing accuracy.

Ignored context:
→ 0.05
→ Provides no useful information and ignores available context.

Notice that we never wrote a faithfulness checker or coded a hallucination detector.

The system prompt mentioned “Use the retrieved context to answer user queries accurately,” and the judge applied that as the evaluation criteria.

The hallucinated response scored 0.45 (not zero) because it partially used the context. The 30-day refund part was correct.

The judge gave partial credit for what it got right and penalized what it invented.

That’s a nuanced distinction that would take significant engineering to encode in a rule-based reward function.

Moreover, the scores are spread across the 0-1 range: 0.97, 0.45, 0.05, unlike binary pass/fail.

RULER produces a gradient that reflects relative quality. GRPO can use this gradient to apply proportional updates to strongly reinforce the faithful behavior, mildly suppress the hallucination pattern (since it was partially correct), and strongly suppress the context-ignoring behavior.

2) The ruler function above works for understanding and experimentation, but ART’s training loop operates on Trajectory and TrajectoryGroup objects.

These carry the reward field that GRPO reads, debug logs for inspection, and the structure that model.train() expects.

After this, the higher-level ruler_score_group function handles the conversion.

Below, let’s look at the same RAG scenario structured the way you’d use it in a real training pipeline, now with 4 trajectories instead of 3.

# The system prompt defines the agent's goal
# RULER uses this as the implicit reward function
system_msg = {
    "role": "system",
    "content": (
        "You are a RAG-based support agent. "
        "Answer user queries using ONLY the retrieved context. "
        "Do not add information that is not in the context."
    ),
}

user_msg = {
    "role": "user",
    "content": (
        "What is the refund policy?\n\n"
        "[Retrieved context]: Refunds are available within 30 days "
        "of purchase. Digital products are non-refundable after "
        "download. Contact support@example.com to initiate."
    ),
}

responses = [
    "You can get a refund within 30 days of purchase. Digital products "
    "are non-refundable once downloaded. Email support@example.com to start.",

    "Refunds are available within 30 days. You can also get store credit "
    "for up to 90 days, and our hotline is 1-800-REFUND.",

    "I'm not sure about the refund policy. Please check the website or "
    "contact the billing team for more details.",

    "Based on the information I have, the refund policy states that "
    "refunds are available within 30 days of purchase. It is important "
    "to note that digital products cannot be refunded after they have "
    "been downloaded. If you wish to initiate a refund, you should "
    "reach out to support@example.com.",
]

Now we have 4 trajectories instead of 3. The fourth is a verbose but accurate response that uses only the retrieved context but wraps it in unnecessary filler words/sentences.

Moving on, we define our Trajectories and Groups as we discussed earlier:

import art
from openai.types.chat.chat_completion import Choice
from openai.types.chat import ChatCompletionMessage

trajectories = []
for resp in responses:
    traj = art.Trajectory(
        messages_and_choices=[
            system_msg, user_msg,
            Choice(
                finish_reason="stop", index=0,
                message=ChatCompletionMessage(role="assistant", content=resp),
            ),
        ],
        reward=0.0,
    )
    trajectories.append(traj)

group = art.TrajectoryGroup(trajectories)

Finally, we run the scoring:

from art.rewards import ruler_score_group

judged_group = await ruler_score_group(group, "openai/o3", debug=True)

With debug=True, RULER prints the judge’s raw reasoning with the actual scores.

This is the raw reasoning:

{
    "scores": [
        {
            "trajectory_id": "1",
            "explanation": "Accurately answers the question using only the retrieved context, concisely and completely.",
            "score": 0.98
        },
        {
            "trajectory_id": "2",
            "explanation": "Includes unsupported details about store credit and a hotline that are not in the retrieved context, so it violates the instruction to use only the context.",
            "score": 0.2
        },
        {
            "trajectory_id": "3",
            "explanation": "Does not answer the question despite the needed information being present in the retrieved context.",
            "score": 0.05
        },
        {
            "trajectory_id": "4",
            "explanation": "Accurately and completely answers the question using only the retrieved context, though slightly more verbose than necessary.",
            "score": 0.96
        }
    ]
}

And these are the scores (ranked):

Rank 1 | Score: 0.980 — Concise, faithful response
Rank 2 | Score: 0.960 — Verbose but accurate response
Rank 3 | Score: 0.200 — Hallucinated store credit and hotline
Rank 4 | Score: 0.050 — Ignored the retrieved context entirely

If you notice closely...

• The concise faithful response (0.98) scored just above the verbose accurate one (0.96). Both used only the retrieved context, both were correct, but the system prompt said “Answer using ONLY the retrieved context,” and the concise version did that more directly. The judge recognized verbosity as a minor quality issue, not a correctness issue. That’s a nuanced distinction that would be hard to encode in a scoring function because how do you write a rule that says “technically correct but unnecessarily wordy, penalize by 0.02”?

• The hallucinated response dropped from 0.45 in the first experiment to 0.20 here. The difference is the system prompt. The first experiment said “Use the retrieved context to answer accurately.” This one says “Do not add information that is not in the context.” The stricter instruction produced stricter scoring. The judge adapted automatically. If you tighten your system prompt, RULER tightens its evaluation to match, without you changing any scoring code.

• The context-ignoring response scored 0.05 in both experiments. When the answer is right there in the retrieved context, and the agent says “I’m not sure,” there’s no ambiguity regardless of how the system prompt is worded.

These scored trajectories are exactly what model.train() expects, so let’s look at that ahead.

The full training loop

To actually train with these scores, you replace the hardcoded responses with real model inference.

ART’s gather_trajectory_groups handles the orchestration.

Essentially, for each scenario, it generates a group of trajectories using the model’s current weights, scores them with RULER, and collects the results for GRPO:

for step in range(num_steps):
    groups = await art.gather_trajectory_groups(
        (
            art.TrajectoryGroup(
                rollout(model, scenario) for _ in range(4)
            )
            for scenario in scenarios
        ),
        after_each=lambda g: ruler_score_group(
            g, "openai/o3"),
    )
    await model.train(groups)  # GRPO updates LoRA weights

In every step, the model generates 4 responses per scenario using its current weights, RULER ranks them relatively, and GRPO reinforces the high-scoring behavior while suppressing the low-scoring behavior.

The agent gets better at following the system prompt’s instructions with every iteration.

Over multiple steps, the model learns the patterns that score well (faithfulness, conciseness, grounding in context) and unlearns the patterns that score poorly (hallucination, ignoring context, verbosity).

And notice that no reward function was defined anywhere in this code.

Custom rubrics

For most tasks, the system prompt provides enough signal for RULER to score effectively. But when you need more specific evaluation criteria, RULER supports custom rubrics:

custom_rubric = """
- Prioritize responses that are concise and clear
- Penalize responses that include emojis or informal language
- Reward responses that cite sources
"""

await ruler_score_group(group, "openai/o3", rubric=custom_rubric)

The rubric is natural language, not Python, so iterating on it is fast.

You just change a sentence, rerun, and check the scores.

Compare this to editing a reward function where a misplaced weight or a buggy condition can silently teach the agent bad behavior that you won’t notice until after training.

Application to non-verifiable tasks

RULER is general-purpose. It works on any task, not just freeform ones where custom rewards are painful.

The practical question is when RULER adds value over simpler alternatives.

For purely deterministic tasks (did the SQL query return the right rows?), a binary verifier is cheaper and gives a cleaner signal.

For purely subjective tasks (was the summary good?), RULER is the only automatic option. For tasks that sit in between (did the agent find the right answer AND explain it well?), you can combine both:

judged_group = await ruler_score_group(group, "openai/o3")

for traj in judged_group.trajectories:
    independent_reward = verify_correctness(traj)  # binary 0/1
    traj.reward += independent_reward

RULER preserves any rewards you assign during rollout under a separate metric, so you can layer LLM-judge scoring on top of deterministic verification without losing either signal.

Practical details

Here are some practical insights we have gathered based on using RULER:

• You don’t need the most expensive model as the judge. Cheaper models like Qwen3 32B often work well. You can also use Claude, local models through Ollama, or any model supported by LiteLLM. The choice is a cost-quality tradeoff, not a hard requirement.

• 4 to 8 trajectories per group is the recommended range. Fewer than 4 gives the judge too little to compare against. More than 8 can confuse the judge and increase the cost without proportional benefit.

• When all trajectories in a group share the same system prompt and user message (which they usually do), RULER deduplicates the common prefix automatically. The judge only sees the shared context once, followed by the different responses. This cuts token usage significantly for long system prompts or multi-turn conversations.

• RULER caches judge responses to disk. If you rerun the same trajectories, it won’t hit the API again. This matters during debugging when you’re iterating on the system prompt or rubric.

The bottleneck in applying RL to agents was never the optimization algorithm.

GRPO handles that well.

It was always the reward signal.

RLVR solved this for verifiable tasks by letting the environment score outputs directly.

RULER solves it for every task (verifiable or non-verifiable) by letting an LLM judge score outputs relatively.

The full implementation is in the ART repository, along with Colab notebooks that walk you through the training loop end-to-end.

Repo: https://github.com/OpenPipe/ART (don’t forget to star it ⭐️)

Thanks for reading!

Read Part 1 of the Reinforcement Learning course here →

Reinforcement learning nanodegree part 1

This first chapter covers:

• what makes RL fundamentally different from supervised and unsupervised learning

• the agent-environment interaction loop

• the exploration-exploitation tradeoff

• multi-armed bandits as the simplest RL setting, four action-selection strategies (greedy, ε-greedy, optimistic initialization, UCB)

• and a complete hands-on implementation of the classic 10-armed testbed with results and analysis.

Read Part 1 of the Reinforcement Learning course here →

Why care?

Look at what has happened in the past two years.

• DeepSeek-R1 used GRPO for reasoning.

• ChatGPT was shaped by RLHF.

• Claude uses constitutional AI with RL.

Every frontier LLM released recently has some form of reinforcement learning in its post-training pipeline.

RL is no longer a niche subfield for robotics and game-playing. It is a core component of how the most capable AI systems are built today.

Google Trends reflects this.

Search interest for “reinforcement learning” was nearly flat from 2004 to 2024. In the past year, it has gone vertical, hitting an all-time high.

The demand for RL expertise has followed.

If you look at ML engineering roles at labs like OpenAI, Anthropic, DeepMind, or any team working on post-training, alignment, or agentic systems, RL fluency shows up as a requirement consistently.

Understanding how reward signals shape model behavior, how policy optimization works, and how exploration interacts with credit assignment is becoming as fundamental as understanding backpropagation was five years ago.

This series is structured the same way as our MLOps/LLMOps course: concept by concept, with clear explanations, diagrams, math where it matters, and hands-on implementations you can run.

And no prior RL background is needed.

You can start reading Part 1 here →

Reinforcement learning nanodegree part 1

Over to you: What topics would you like us to cover in this RL series?

Thanks for reading!

DeepSeek just released V4-Pro, a 1.6T total parameter MoE model with only 49B active parameters, open-sourced under MIT.

It’s the strongest open-weight model for agentic coding right now, hitting 80.6% on SWE-Bench Verified.

DeepSeek’s own engineers reportedly use it as their internal coding agent, rating it above Claude Sonnet 4.5 and close to Opus 4.6 in non-thinking mode.

The model also supports a 1M token context window natively, and V4’s new hybrid attention architecture brings KV cache usage down to 10% of V3.2, which is what makes that context length practical for long-running agent loops.

If you’re looking to try it out, Lightning AI has a hosted endpoint ready to go →

Thanks to Lightning AI for partnering today!

8 prompting techniques to generate better LLM outputs

Zero-shot prompting (just sending a query with no additional structure) is the default for most people using LLMs.

It’s also where most output quality complaints come from: inconsistent formatting, shallow reasoning, missing constraints, and lack of diversity.

Each of these failure modes maps to a specific prompting technique that fixes it.

Some have been around for a few years (few-shot, CoT), others are from 2025 research (ARQ hit 90.2% instruction adherence vs. 81.5% for direct prompting; Verbalized Sampling improved output diversity by 1.6-2.1x).

Let’s walk through all eight, how they work, and when each one is the right tool.

Few-shot prompting

Instead of describing what you want, you show the LLM a few input-output examples directly in the prompt. The model picks up the pattern and applies it to your new input.

Three to five examples are typically enough. Use this when the task has a specific format: natural language to SQL, text classification into custom categories, or any conversion where showing is easier than telling.

Chain-of-Thought (CoT) prompting

CoT asks the model to reason step by step before producing a final answer. Adding “Let’s think step by step” to prompts improved accuracy on GSM8K math benchmarks from 17.7% to 78.7% with PaLM 540B.

It works because it breaks complex reasoning into smaller, verifiable steps rather than forcing a single-pass answer. Most useful for math, logic, code debugging, or any task that depends on a chain of intermediate conclusions.

​We covered it in detail here, along with the Self-consistency technique and the Tree of Thought technique →

Prompt hierarchy

LLM APIs expose multiple levels of instruction: system prompts, developer prompts, and user prompts. System prompts set behavioral constraints, developer prompts define task-specific logic, and user prompts carry the actual query.

When instructions conflict across levels, the model prioritizes higher levels. This separation of concerns lets you put immutable rules in the system prompt so user inputs can’t override your core constraints.

Role-specific prompting

Assigning a persona (”You are a financial advisor” vs. “You are a security researcher”) shifts the model’s responses toward a specific expertise profile. The model conditions on different subsets of its training data, producing different vocabulary, framing, and decision criteria.

You can also run the same query through multiple personas and compare outputs for diverse perspectives on the same problem.

Negative prompting

Instead of only telling the LLM what to do, you specify what to avoid: “do not use marketing language,” “avoid bullet points,” “do not mention pricing unless asked.” The model treats these as hard constraints during generation.

Most useful for content generation where you need to prevent specific failure modes like jargon, hallucinated references, or unnecessary caveats.

JSON prompting

You define a JSON schema in the prompt and instruct the LLM to return its output matching that structure.

Include something like {"task": "summarize", "input": "...", "output_format": {"sentiment": "", "summary": ""}} and the model responds with values filled in.

Unlike API-level structured outputs (OpenAI’s response_format, Anthropic’s tool-use), JSON prompting works with any model and any interface because the constraint lives in the prompt itself.

You get ~90%+ schema compliance on capable models without any API-specific setup.

​We covered it in detail here →

Attentive reasoning queries (ARQ)

ARQ replaces free-form CoT with targeted, domain-specific questions organized in a predefined JSON schema. Instead of “think step by step,” you give the model a structured checklist it must answer before generating its response.

In testing across 87 scenarios within the Parlant framework, ARQ hit 90.2% success rate vs. 86.1% for CoT and 81.5% for direct prompting.

CoT lets the model freely ignore instructions. ARQ reinstates critical constraints at the exact point where reasoning happens, using the recency effect to keep rules in an active context.

​We covered it in detail here →

Verbalized sampling

Post-training alignment (RLHF, DPO) causes LLMs to collapse toward a narrow set of “safe” outputs.

Verbalized Sampling fixes this with a single prompt change. Instead of asking for one response, you ask the model to “generate 5 responses with their corresponding probabilities.”

This forces the LLM to verbalize its internal distribution rather than collapsing to the mode.

In experiments mentioned in its research paper, diversity improved by 1.6-2.1x over direct prompting, while human evaluation scores went up by 25.7%. The technique is orthogonal to temperature, so you can stack them.

​We covered it in detail here →​

These techniques aren’t mutually exclusive. Few-shot + CoT is a common combination. JSON prompting + negative prompting gives you structured outputs with explicit constraints. ARQ is a structured version of CoT designed for multi-turn agent conversations.

The choice depends on the failure mode: inconsistent format (JSON prompting), shallow reasoning (CoT or ARQ), lack of diversity (Verbalized Sampling), unwanted content (Negative prompting), or missing domain expertise (Role-specific prompting).

👉 Over to you: Which of these techniques do you use most in your day-to-day workflows, and have you tried combining any of them?

Thanks for reading!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

The idea above sounds counterintuitive, but it is actually a direct result of how agent memory is built today.

Agent memory inherits the cognitive shape of its store.

• A vector DB gives it associative memory to recognize familiar patterns.

• A graph gives it relational memory to understand how things connect.

Most agents run on the first and skip the second.

Here’s an example that explains the failure it leads to:

Say a study assistant stores three facts about a student in a vector DB:

• Mark is in grade 10.

• Grade 10 has final exams in March.

• The library closes 2 weeks before final exams.

Mark asks: “Will the library be open next week?”

The vector DB likely returns the first and third facts, because the query mentions Mark and the library.

But it skips the middle fact, which links Mark’s grade to the exam time, because that fact mentions neither Mark nor the library.

It sits in embedding space too far from the query to make it to the retrieved context.

So the Agent answers with partial info, or it fills the gap with a plausible guess that sounds right but might be off by weeks.

This is not a corner case, but it’s actually what real queries look like. Any question that spans two or more hops exceeds what a similarity search can do.

Increasing context windows and retrieving more context is one solution.

But accuracy drops over 30% when the relevant fact sits in the middle of a long context, which is the well-known “lost in the middle” problem.

A bigger window is not the same as better memory. It just gives the model more room to miss things.

To actually solve this problem, you need to stop treating memory as a single store and start treating it as three complementary layers, each doing a job the others cannot.

• Relational: It stores where a fact came from, when it was stored, and who has access. This is the provenance layer.

• Vector: It stores what a fact means and what it is semantically similar to. This is the retrieval layer.

• Graph: It stores how facts connect, what depends on what, and who relates to whom. This is the reasoning layer.

All three are important and complementary:

• A vector DB alone gives similarity without relationships.

• A graph alone gives relationships without semantic search.

• A relational store alone tracks where data came from but cannot reason over it.

If you want to see this in practice, Cognee (open-source) implements this approach.

It runs an ECL pipeline (Extract, Cognify, Load) that writes into all three stores in a single pass and keeps them synchronized as new data arrives.

So the vectors and graph edges are built together during indexing, not glued together later.

On top of this, there are two things Cognee does differently from most memory tools:

1) Smarter entity resolution:

You can give Cognee a domain vocabulary file, and it uses it to merge duplicate mentions automatically.

So “car manufacturer,” “automobile maker,” and “vehicle producer” collapse into one canonical node instead of being available as three separate entries.

2) Local-first defaults:

The default stack runs on a single pip install and stays fully local. You can switch to Postgres and Neo4j for production without changing the API.

We wrote a first-principles walkthrough of agent memory that takes the same problem and works through every layer of the stack, ending in a real working agent built on Cognee.

You can find it here →

And you can find the Cognee GitHub repo here →

Thanks for reading!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

Moonshot released Kimi K2.6 yesterday, and for the first time, an open-weight model holds its ground against Claude Opus 4.6 on the benchmarks that matter for agentic work.

It also costs a fraction of the price.

Here’s a graphical comparison:

Pricing:

• Kimi K2.6 → $0.95/million input tokens and $4/million output tokens.

• Claude Opus 4.6 → $5/million input tokens and $25/million output tokens.

With cache hits, the gap widens.

• K2.6 drops to $0.16 per million on cached inputs.

• Opus 4.6 drops to $0.50.

That’s roughly 5-6x cheaper across the board, before and after caching.

Benchmarks:

K2.6 leads Opus 4.6 on four of the six head-to-head comparisons Moonshot published:

• SWE-bench Pro: 58.6 vs 53.4 (agentic coding)

• HLE with tools: 54.0 vs 53.0 (agentic reasoning)

• DeepSearchQA: 92.5 vs 91.3 (deep research)

• LiveCodeBench: 89.6 vs 88.8

Opus 4.6 still wins on SWE-bench Multilingual and BrowseComp, but the gap is under a point in both.

The part that actually matters

Benchmarks are the easy story. The harder and more interesting story is long-horizon execution.

K2.6 ran a single autonomous task for over 12 hours, making 4,000+ tool calls, to port and optimize inference for a small LLM in Zig, a language most models barely touch.

It ended up running around 20% faster than LM Studio on the same hardware.

Separately, it refactored an 8-year-old financial matching engine across 13 hours, delivering a 133% peak throughput gain.

This is the capability gap that usually separates frontier closed models from open ones. K2.6 closes a meaningful chunk of it.

You get weights you can actually deploy, a Modified MIT license, 5-6x lower inference cost, and performance that no longer forces you to compromise on agentic workloads.

Read more in the official blog here →

We’ll publish a thorough evaluation and a model deep dive on Kimi soon.

Claude Opus 4.7 isn’t a drop-in replacement for 4.6

Opus 4.7 thinks differently, follows instructions more literally, spawns fewer subagents, and reasons more aggressively after every user turn.

The patterns that worked before now cost you tokens without proportional quality gains. The fix is not complicated, but it requires understanding what changed and adjusting your workflow accordingly.

Let’s walk through it today!

The delegation mindset

The single biggest shift with Opus 4.7 is how you frame your role. Treat Claude like a capable engineer you delegate to, not a pair programmer you guide line by line.

Every user turn in an interactive session triggers reasoning overhead. With 4.6, you could spread instructions across multiple turns without much penalty.

With 4.7, that pattern inflates token usage because the model reasons deeply after each message you send. You pay for reasoning on every turn, whether the turn deserves it or not.

Three concrete changes will fix this.

• Specify the task upfront in the first turn. Include intent, constraints, acceptance criteria, and relevant file paths. Ambiguous prompts spread across many turns reduce both token efficiency and output quality.

• Batch your questions. Every user message adds reasoning overhead, so give the model enough context to keep moving without checking in.

• Use auto mode for trusted tasks. For long-running work where you’ve already provided full context, auto mode (Shift+Tab) cuts cycle time by removing unnecessary check-ins.

The 5 effort levels

Opus 4.7 introduces xhigh, a new effort level between high and max. It’s now the default for Claude Code.

If you’re an existing user who never manually set your effort level, you’ve been auto-upgraded to xhigh.

Here’s how the five tiers break down.

• low is for latency-sensitive, tightly scoped work. The model won’t go above and beyond, but it still outperforms Opus 4.6 at the same effort level.

• medium fits cost-sensitive tasks where you’re willing to trade intelligence for speed.

• high balances intelligence and cost. It’s a good pick for concurrent sessions or budget-conscious work without a large quality drop.

• xhigh is the default and the sweet spot for coding and agentic tasks. You get strong autonomy and intelligence without the runaway token usage that max can produce.

• max squeezes out extra performance on genuinely hard problems, but with diminishing returns. It’s more prone to overthinking, so use it deliberately for eval ceiling testing or extremely intelligence-sensitive work.

One practical tip: you can toggle effort mid-task. Start at xhigh for the complex design phase, drop to high for straightforward implementation, and bump to max for a tricky debugging session. This gives you fine-grained control over token spend.

Opus 4.7 respects effort levels more strictly than 4.6, so if a task at low or medium feels underthought, raise the effort instead of prompting around it.

Adaptive thinking replaces fixed budgets

If you were using Extended Thinking with budget_tokens on Opus 4.6, that’s gone. Opus 4.7 uses adaptive thinking instead.

With fixed budgets, you allocated a set number of thinking tokens upfront, and the model used them whether it needed to or not.

With adaptive thinking, the model decides when and how much to think at each step.

Simple queries get fast responses, complex reasoning steps get deep thought, and steps that don’t benefit from thinking skip it entirely.

Over a long agentic run, this adds up to faster responses and lower token usage compared to a blanket thinking budget.

The migration is straightforward:

# Before (Opus 4.6 with extended thinking)
client.messages.create(
    model="claude-opus-4-6",
    max_tokens=64000,
    thinking={"type": "enabled", "budget_tokens": 32000},
    messages=[{"role": "user", "content": "..."}],
)

# After (Opus 4.7 with adaptive thinking)
client.messages.create(
    model="claude-opus-4-7",
    max_tokens=64000,
    thinking={"type": "adaptive"},
    output_config={"effort": "xhigh"},
    messages=[{"role": "user", "content": "..."}],
)

You can still steer the thinking rate with prompts. To get more thinking, try something like “Think carefully and step-by-step before responding; this problem is harder than it looks.”

To get less thinking, use “Prioritize responding quickly rather than thinking deeply. When in doubt, respond directly.”

If you’re running at max or xhigh effort, set a large max output token budget (start at 64k), so the model has room to think and act across subagents and tool calls.

Behavior changes to note

Opus 4.7 has several default behavior changes that will catch you off guard if you’ve tuned your prompts or harnesses for 4.6. Let’s go through them.

Response length

Opus 4.7 calibrates response length to task complexity. Simple lookups get short answers, open-ended analysis gets long ones.

If your use case depends on a specific length or style, state it explicitly. Positive examples of the voice you want work better than negative “don’t do this” instructions.

Fewer tool calls

The model calls tools less often and reasons more. This produces better results in many cases.

But if you need more aggressive tool use for search or file reading, provide explicit guidance about when and why tools should be used. Raising effort to high or xhigh also increases tool usage.

Fewer subagents

Opus 4.7 is more judicious about delegating to subagents. If your workflow benefits from parallel fan-out, spell it out clearly:

“Do not spawn a subagent for work you can complete directly in a single response. Spawn multiple subagents in the same turn when fanning out across items or reading multiple files.”

More literal instruction following

This is the change most likely to break existing setups. Opus 4.7 interprets prompts more literally, especially at lower effort levels.

It won’t silently generalize an instruction from one item to another, and it won’t infer requests you didn’t make. The upside is precision and less trash.

The downside is that if you need an instruction applied broadly, you must state the scope explicitly. For example: “Apply this formatting to every section, not just the first one.”

Tone shift

Opus 4.7 is more direct and opinionated, with less validation-forward phrasing and fewer emoji than 4.6’s warmer style.

If your product relies on a specific voice, re-evaluate your style prompts against the new baseline.

Code review

Opus 4.7 is meaningfully better at finding bugs. It shows 11 percentage points better recall on Anthropic’s hardest bug-finding eval based on real PRs, with higher precision too.

But here’s the catch. If your review harness says “only report high-severity issues” or “be conservative,” Opus 4.7 follows that instruction more faithfully than 4.6 did.

It may investigate code just as thoroughly, identify bugs, and then not report findings it judges below your stated bar. Precision rises, but measured recall drops.

The fix is to separate finding from filtering:

Report every issue you find, including ones you are uncertain about or consider low-severity. Do not filter for importance or confidence at this stage. Your goal here is coverage: it is better to surface a finding that later gets filtered out than to silently drop a real bug. For each finding, include your confidence level and an estimated severity so a downstream filter can rank them.

If you need single-pass filtering, be concrete about where the bar is rather than using qualitative terms. For example: “Report any bugs that could cause incorrect behavior, a test failure, or a misleading result; only omit pure style or naming preferences.”

Session management with 1M context

Claude Code now has a 1 million token context window. That’s enough to build a full-stack app from scratch in a single session.

But more context doesn’t always mean better results. Context rot is real.

As the context grows, attention spreads across more tokens. Older, irrelevant content starts to distract from the current task, and the model gets less intelligent as its context window fills up.

You have five options at every turn.

• Continue means sending another message. Use this when everything in the window is still relevant.

• Rewind (double-Esc) jumps you back to a previous message to re-prompt from there. Failed attempts get dropped from context, which is often better than typing “that didn’t work, try X instead,” because you keep the useful file reads while dropping the failed approach.

• /compact with a hint summarizes the session and keeps going. It’s lossy but low effort, and you can steer it with instructions like /compact focus on the auth refactor, drop the test debugging.

• /clear starts a fresh session. You write down what matters yourself, giving you zero rot and full control.

• Subagents delegate work that generates lots of intermediate output. The subagent gets its own fresh context window, and only the final result comes back.

The mental test here: will you need the tool output again, or just the conclusion? If just the conclusion, use a subagent.

What causes bad auto-compaction

Auto-compaction fires when you’re nearing the context limit. The problem is that this is exactly when the model is at its least intelligent point due to context rot.

A common failure looks like this: autocompact fires after a long debugging session and summarizes the investigation. Your next message references something that was dropped from the summary.

With one million context, you have more time to compact proactively with a description of what matters. Don’t wait for auto-compaction to kick in.

Important prompting techniques

Several foundational prompting techniques remain effective with Opus 4.7, and you shouldn’t abandon them just because the model is smarter.

• Use XML tags to structure complex prompts. Wrap instructions, context, examples, and inputs in their own tags (<instructions>, <context>, <input>) to reduce misinterpretation.

• Give Claude a role in the system prompt. Even a single sentence focuses on behavior and tone.

• Use 3-5 examples wrapped in <example> tags, covering edge cases and varying enough that the model doesn’t pick up unintended patterns.

• Put longform data at the top of your prompt, above your query. Queries at the end can improve response quality by up to 30% on complex, multi-document inputs.

• Ground responses in quotes. For long document tasks, ask Claude to quote relevant parts before carrying out the task.

Controlling tool use

Opus 4.7 calls tools less frequently by default. To increase tool use, raise effort or add explicit guidance: “Use [tool] when it would enhance your understanding of the problem.”

Parallel tool calling is a strength worth leaning into. The model will run multiple searches, read several files at once, and execute bash commands in parallel.

You can boost parallel execution to near 100% with explicit guidance in XML tags.

Overthinking mitigation

At higher effort levels, Opus 4.7 can think extensively, inflating thinking tokens. Add this prompt to keep it focused:

“When you’re deciding how to approach a problem, choose an approach and commit to it. Avoid revisiting decisions unless you encounter new information that directly contradicts your reasoning.”

Migration checklist

If you’re moving from Opus 4.6 to 4.7, here’s what to update.

1. Switch to adaptive thinking by replacing thinking: {type: “enabled”, budget_tokens: N} with thinking: {type: “adaptive”}.

2. Set effort to xhigh, which is the new default for coding work, and set max output tokens to 64k at xhigh or max effort.

3. Update code review prompts to separate finding from filtering so you preserve recall.

4. Reduce user turns by front-loading context into the first message, and add explicit subagent guidance so the model knows when to fan out.

5. Specify design preferences concretely rather than relying on generic instructions to override the house style.

6. Migrate away from prefilled responses. Starting with Claude 4.6 models, prefilled responses on the last assistant turn are deprecated, and on Mythos Preview, they return a 400 error.

Computer use

Computer use works across resolutions up to a new maximum of 2576px or 3.75MP. Sending images at 1080p provides the best balance of performance and cost.

For cost-sensitive workloads, 720p or 1366x768 are strong lower-cost options.

Opus 4.7 rewards upfront specification and punishes incremental, multi-turn prompting. The model is more capable, more literal, and more autonomous than 4.6.

Give it a well-specified task, set effort to xhigh, and let it run. The developers who will get the most out of this model are the ones who stop guiding it step by step and start delegating like they would to a senior engineer.

Try this today: take your next coding task, write a single detailed prompt with intent, constraints, and acceptance criteria, and send it in one turn at xhigh. Compare the result and token usage to your old multi-turn pattern.

Thanks for reading!

Read the full Part 2 deep dive here →

Diffusion LLMs Part 2

It covers:

• the training techniques that scaled dLLMs from 8B to 100B parameters (including converting pre-trained autoregressive models like LLaMA into diffusion models via attention mask annealing)

• the inference acceleration stack (block-wise KV caching with Fast-dLLM, confidence-aware parallel decoding, token editing with LLaDA 2.1),

• production serving with SGLang

• hands-on code for running Dream 7B and serving LLaDA 2.0,

• and a decision framework for when dLLMs actually make sense over autoregressive models.

Read the full Part 2 deep dive here →

Why care?

Every production LLM today, GPT-4, Claude, Gemini, LLaMA, generates text the same way: one token at a time, left to right.

Each token requires loading the full model weights through GPU memory, performing a tiny computation, and then loading all the weights again for the next token. On an A100, this means roughly 1 FLOP per byte of data moved, while the GPU is designed for 100+ FLOPs per byte.

Diffusion LLMs take a completely different approach. They start with a fully masked sequence and iteratively unmask all tokens in parallel, using bidirectional attention at every step. This shifts inference from memory-bandwidth bound to compute-bound, which is exactly where modern GPUs are efficient.

The results are catching up fast.

Block diffusion (BD3-LM) is within 0.5 perplexity points of autoregressive on LM1B. LLaDA at 8B parameters matches LLaMA 3 on MMLU and exceeds it on TruthfulQA and HumanEval. And models like Dream 7B are already being served in production with SGLang.

But knowing that dLLMs exist and knowing how to train them, serve them, and decide when they are the right choice over autoregressive models are very different things.

Part 2 covers the full practical stack, like how teams are converting existing AR checkpoints into dLLMs at a fraction of the training cost, what makes inference fast in practice, and how to run Dream 7B and serve LLaDA 2.0 with code you can execute.

If you haven’t read Part 1, start there first. It covers the theory, math, and generation mechanics that this article builds on.

And you can read Part 2 for practical implementations here →

👉 Over to you: Do you think the future of LLM generation is pure diffusion, pure autoregressive, or some hybrid of the two?

Thanks for reading!

When Claude moved from Sonnet 4.5 to Sonnet 4.6, backend token usage through Supabase’s MCP server went up, from 11.6M to 17.9M tokens across 21 database tasks.

The model got smarter, but the backend token usage actually increased.

The reason is subtle, and it has nothing to do with the model.

Instead, it has to do with how the backend exposes info to the agent. When context is incomplete, a more capable model doesn’t just skip the gap.

It spends more tokens reasoning about the gap, runs more discovery queries, and retries more frequently. So the missing context doesn’t disappear with a better model. It gets more expensive.

Let’s look at why backends are a token sink for agents, what an alternative architecture looks like, and what the cost difference is on a real project.

Why Supabase’s MCP server wastes tokens

Supabase is a great backend. But it wasn’t designed to be operated by AI agents, and the MCP server that was added later inherits that limitation.

Three specific mechanisms cause the token bloat.

1) Documentation retrieval returns everything

When CC needs to set up Google OAuth through Supabase, it invokes the search_docs MCP tool.

Supabase’s implementation returns full GraphQL schema metadata on every call, which has 5-10x more tokens than the agent actually needs.

If the agent asked for OAuth setup instructions, it got the entire authentication docs, including sections on email/password, magic links, phone auth, SAML, and SSO.

This happens on every search_docs call, like database queries, storage configuration, and edge function deployment.

Each call dumps the full metadata for that entire domain. Across a session where the agent sets up auth, database, storage, and functions, the docs overhead alone can account for thousands of wasted tokens.

2) No visibility into backend state

When you use Supabase as a human dev, you open the dashboard and see everything at a glance, like active auth providers, tables, RLS policies, configure storage buckets, deployed edge functions, etc.

An agent can’t see the dashboard.

Supabase’s MCP server does expose some state through individual tools like list_tables and execute_sql, but there’s no way to ask “what does my entire backend look like right now?” and get one structured response.

So the agent pieces it together through multiple calls, each call returns a partial view, and some info (like which auth providers are configured) isn’t available through MCP at all.

This fragmented discovery process costs tokens, and the agent often needs several attempts because the information comes back incomplete or in a format that requires further queries to interpret.

3) No structured error context

When something goes wrong (and it will, because the agent is guessing), Supabase returns raw error messages. It could be a 403 from an RLS denial, a 500 from a misconfigured edge function, etc.

A human dev would look at it, check the Supabase dashboard, cross-reference with the logs, and fix the issue.

The agent doesn’t have that path. It gets the error message, reasons about what might have caused it, and tries a fix.

If the fix is wrong, it retries. Each retry re-sends the entire conversation history and compounds the token cost.

These three mechanisms (doc overhead, state discovery, error retry loops) compound fast.

A model that reasons more extensively, like Sonnet 4.6, makes each exploration step more thorough and more expensive.

That’s why the token gap widened from Sonnet 4.5 to 4.6, and it’ll likely widen further with each new model release.

What “backend context engineering” should look like

The fix isn’t switching to a worse model.

It’s giving the agent a structured backend context so it doesn’t have to explore and guess.

This is what Karpathy means by context engineering: “the delicate art and science of filling the context window with just the right information for the next step.”

He explicitly includes tools and state as part of that context. Most people apply the idea to prompts and RAG retrieval.

But the backend is part of the context window too, and right now, it’s the part almost nobody is optimizing.

To see what this looks like in practice, InsForge (open source, Apache 2.0) implements exactly this approach.

It provides the same primitives as Supabase (Postgres with pgvector, auth, storage, edge functions, and realtime) but structures the information layer so agents can consume it efficiently.

The key architectural difference is how it delivers context to Claude Code.

Three layers work together:

• Skills for static knowledge.

• CLI for direct backend operations.

• MCP for live state inspection

Each layer solves a different problem and reduces tokens for a different reason.

1) Skills: static knowledge with zero round-trips

InsForge’s primary approach for knowledge is Skills. They load directly into the agent’s context at session start, so the SDK patterns, code examples, and edge cases for every backend operation are available without any tool calls.

Skills also use progressive disclosure, wherein only the metadata (name, description, ~70-150 tokens per skill) loads initially.

The full skill content loads only when the agent determines it matches the current task. This means you can have 100+ skills installed without context bloat, which isn’t possible with MCP’s all-or-nothing schema loading.

Four skills cover the full stack, each scoped to a specific domain:

• insforge for frontend code that talks to the backend.

• insforge-cli for backend infrastructure management

• insforge-debug for structured error diagnosis across common failures like auth errors, slow queries, edge function failures, RLS denials, deployment issues, and performance degradation)

• insforge-integrations for third-party auth providers (Clerk, Auth0, WorkOS, Kinde, Stytch).

Install all four with one command:

npx skills add insforge/insforge-skills

2) CLI for direct execution

For actually executing backend operations (creating tables, running SQL, deploying functions, managing secrets), the InsForge CLI is the primary interface.

Every command supports --json for structured output, -y to skip confirmation prompts, and returns semantic exit codes so agents can detect auth failures, missing projects, or permission errors programmatically.

This is helpful because Claude Code can pipe CLI output through jq, grep, and awk in ways that would require multiple sequential MCP tool calls.

These are some example operations the agent actually runs:

# Inspect backend state (run first to discover what's configured)
npx @insforge/cli metadata --json

# Database operations
npx @insforge/cli db query "CREATE TABLE posts (...)" --json
npx @insforge/cli db policies  # inspect existing RLS policies

# Edge functions
npx @insforge/cli functions deploy my-handler
npx @insforge/cli functions invoke my-handler --data '{"action":"test"}' --json

# Storage
npx @insforge/cli storage create-bucket documents --json
npx @insforge/cli storage upload ./file.pdf --bucket documents

# Frontend deployment
npx @insforge/cli deployments env set VITE_INSFORGE_URL https://...
npx @insforge/cli deployments deploy ./dist --json

# Diagnostics
npx @insforge/cli diagnose db --check connections,locks,slow-queries

The agent parses the JSON and handles errors based on exit codes.

3) MCP tools for live backend state

MCP is still useful, but for a narrower purpose, like inspecting the current state of your backend when that state is changing.

InsForge’s MCP server exposes a lightweight get_backend_metadata tool that returns a structured JSON with the full backend topology in a single call:

{
  "auth": {
    "providers": ["google", "github"],
    "jwt_secret": "configured"
  },
  "tables": [
    {"name": "users", "columns": ["id", "email", "created_at"], "rls": "enabled"},
    {"name": "posts", "columns": ["id", "title", "body", "author_id"], "rls": "enabled"}
  ],
  "storage": { "buckets": ["avatars", "documents"] },
  "ai": { "models": [{"id": "gpt-4o", "capabilities": ["chat", "vision"]}] },
  "hints": ["Use RPC for batch operations", "Storage accepts files up to 50MB"]
}

In one call and ~500 tokens, the agent knows the full backend topology. The hints field provides agent-specific guidance that reduces incorrect API usage.

The key design choice here is that MCP is used for state inspection (which changes as the agent works), not for documentation retrieval (which doesn’t).

This inverts the typical usage pattern and is the main reason InsForge consumes far fewer tokens than Supabase on equivalent tasks.

Supabase vs Insforge: Build DocuRAG with Claude Code

To make this concrete, we built the same DocuRAG app using Claude Code.

Users sign in via Google OAuth, upload PDFs, the system chunks and embeds the text (text-embedding-3-small, 1536 dimensions), stores the vectors in pgvector, and users ask natural-language questions answered via GPT-4o.

This touches nearly every backend primitive at once: user auth, file storage, a documents table, vector embeddings, embedding generation, chat completion, a retrieval edge function, and RLS to isolate each user’s documents.

Here's the setup for each.

Supabase

• Create a Supabase account and create a new project.

• Connect the MCP server to Claude Code and authenticate:

claude mcp add --scope project --transport http supabase \
  "https://mcp.supabase.com/mcp?project_ref=<your-project-ref>"

claude /mcp

• Install Supabase's Agent Skills (marked as “Optional” in Supabase's official setup):

npx skills add supabase/agent-skills

This installs two skills:

• supabase: broad catch-all skill covering Database, Auth, Edge Functions, Realtime, Storage, Vectors, Cron, Queues, client libraries (supabase-js, @supabase/ssr), SSR integrations (Next.js, React, SvelteKit, Astro, Remix), CLI, MCP, schema changes, migrations, and Postgres extensions

• supabase-postgres-best-practices: Postgres performance optimization across 8 categories

Supabase ships one broad skill that triggers on "any task involving Supabase," plus a specialized Postgres optimization skill. When the supabase skill activates, all its content loads because the trigger conditions cover almost the entire product surface.

Insforge

• Create an Insforge account and create a new project (you can also self-host and run it fully locally using Docker Compose).

• Install all four Skills (primary documentation and diagnostic layer):

npx skills add insforge/insforge-skills

This installs insforge (SDK patterns), insforge-cli (infrastructure commands), insforge-debug (failure diagnostics), and insforge-integrations (third-party auth providers). Total metadata cost: ~714 tokens at session start.

• Link the CLI to your project (primary execution layer):

npx @insforge/cli link --project-id <project-id>

InsForge ships four narrowly scoped skills, each covering a specific domain.

• When you're writing frontend code, only insforge activates.

• When you're creating tables, only insforge-cli activates.

• When something breaks, only insforge-debug activates.

The prompt is nearly identical for both sessions, with one key difference.

• Supabase:

Build a chat with document app called DocuRAG.
It will be a typical RAG setup where a user
can upload a document. It will be chunked, embedded,
and stored in a vector DB. Once done, a user can ask
questions about the document. The engine will retrieve
the relevant chunks after embedding the query. Finally,
it will generate a coherent response using GPT-4o based
on the query and the retrieved context. Add Google OAuth.
Use Supabase as the backend and LLMs/embedding models via
the OpenAI API. Build frontend in next.js.

• InsForge:

Build a chat with document app called DocuRAG.
It will be a typical RAG setup where a user
can upload a document. It will be chunked,
embedded, and stored in a vector DB. Once done,
A user can ask questions about the document.
The engine will retrieve the relevant chunks
after embedding the query. Finally, it will
generate a coherent response using GPT-4o based on
the query and the retrieved context. Add Google OAuth.
Use Insforge as the backend and also for the model
gateway. Build the front-end in Next.js.

The Supabase prompt says "LLMs/embedding models via the OpenAI API" (two systems to wire). The InsForge prompt says "also for the model gateway" (one system).

We ran both sessions side by side and recorded the full build. Here’s the side-by-side video showing what happened from prompt to working app.

It also showcases the final output from both sessions, built on two different backends.

One important thing: Supabase required manual Google OAuth setup outside of Claude Code. We had to navigate to Google Cloud Console, create an OAuth 2.0 client ID, configure the consent screen, add our email as a test user, copy the Client ID and Client Secret, then paste it into Supabase’s dashboard. This was not required in Insforge.

Before diving into the session-specific details, here’s what the numbers looked like at the end:

• Supabase: 10.4M tokens; $9.21 Cost with 12 user messages (10 error reports)

• InsForge: 3.7M tokens; $2.81 Cost, with 1 user messages (0 error reports)

Now let’s look at what actually happened in each session.

To analyze both sessions objectively, we exported the full Claude Code session history from both runs (as JSONL files) and fed them to a separate Claude instance. The analysis below, including tool call counts, error sequences, and token breakdowns, comes from parsing those session logs.

Supabase (consumed 10.4M tokens with $9.21 cost)

The initial build went smoothly.

The agent loaded the supabase skill, discovered the backend state via MCP tools (list_tables, list_extensions, execute_sql), scaffolded the Next.js project, created the database schema, wrote two edge functions (ingest-document and query-document), and deployed everything. The build passed.

First problem: login didn’t work

When we tried to sign in with Google OAuth, the app threw an error. The agent had wired the authentication using the wrong Supabase client library for Next.js.

In Next.js, the OAuth callback runs on the server, but the agent used a client-side library that stores login state in the browser. The browser state isn’t available on the server, so the login flow broke.

The agent fixed this by switching to a different library (@supabase/ssr), rewriting how the app handles login sessions, and rebuilding.

Document upload failed (took 8 turns to fix)

After the login was fixed, we tried uploading a document. The edge function returned an error, we reported it, it tried a fix, failed, then we tried again, and it returned the same error. This cycle repeated 8 times:

• The agent tried adding auth headers manually → Same error.

• Redeployed with extra logging to see what was happening → Same error.

• Tried showing the real error message instead of the generic one → Different error (now a network/CORS issue).

• Fixed the CORS issue → Back to the original error.

• Tried a different way of reading the user’s login token → Same error.

• Tried yet another authentication approach → Same error.

In plain terms, Supabase has a security layer that checks login tokens before the edge function code even starts. The new auth library the agent installed (to fix the first problem) was sending a token format that this security layer didn’t recognize.

So every request was getting rejected at the door before the function code had a chance to run. That’s why none of the code-level fixes worked.

The agent spent 8 rounds fixing code-level issues when the problem was upstream of the code entirely.

The solution was simple: turn off the platform’s automatic token checking and handle authentication inside the function code instead.

It took 8 attempts because every time, it saw a 401 (unauthorized) error, but nothing told it where the rejection was coming from. Without that signal, it kept attempting to fix the code.

But during this debugging process, the edge function was redeployed 8 times (on top of 2 initial deploys during the build). Each redeployment, log check, and retry re-sent the entire growing conversation history, compounding the token cost.

Final session stats involved:

• 12 user messages (10 were error reports)

• 135 tool calls

• 30+ MCP tool calls.

• 10.4M tokens

• $9.21 Cost

Insforge (consumed 3.7M tokens with $2.81 cost)

The InsForge session completed without any errors.

The agent started by inspecting the backend state.

Its first action was npx @insforge/cli metadata --json, which returned a structured overview of the project, including the configured auth providers, existing tables, storage buckets, available AI models, and real-time channels.

This gave the agent a complete picture of what it was working with before it wrote any code.

In the Supabase session, the agent needed multiple MCP calls (list_tables, list_extensions, execute_sql) to piece together a similar understanding, and even then, it missed critical details like the verify_jwt behavior.

The schema setup ran through 6 CLI commands, all of which succeeded.

The agent enabled pgvector, created the documents and chunks tables (with a vector(1536) column), enabled Row Level Security on both, created the access policies, and set up the match_chunks similarity search function.

Each command returned structured output confirming what happened, so the agent could verify each step before moving to the next.

The auth and edge function problems from the Supabase session didn't occur here.

The insforge skill included the correct client library patterns for Next.js, so the agent wired authentication correctly on the first attempt.

And the two edge functions (embed-chunks and query-rag) both deployed and ran without errors because the model gateway for embeddings and chat completion was part of the same backend.

The agent didn't need to integrate OpenAI separately, manage a second API key, or deal with cross-service authentication.

The metadata response already listed text-embedding-3-small and gpt-4o as available models, so the agent called them directly through the InsForge SDK.

Final session stats involved:

• 1 user message

• 77 tool calls

• 0 MCP tool calls.

• 3.7M tokens

• $2.81 Cost

We asked Claude to generate a side-by-side summary and here’s what it produced:

The Supabase session’s token cost was driven by the error retry loop.

Each of the 8 edge functions redeploys re-sent the entire conversation history (which grew with each attempt).

The agent checked logs 6 times, redeployed functions 8 times, and tried 6 different authentication strategies before finding the root cause.

None of this was the agent’s fault. The Supabase platform’s verify_jwt gate was rejecting the token before the function code ran, and the logs didn’t distinguish between platform-level and code-level rejections.

The Insforge session avoided these problems because the skills loaded the correct auth patterns from the start, and the CLI gave structured feedback on every operation.

This comparison highlights a problem that goes beyond Supabase specifically.

Most backends were designed for human developers who can read dashboards, interpret ambiguous errors, and mentally track state across multiple services.

When an agent takes over that workflow, the assumptions break. The agent can’t see the dashboard. It can’t tell where an error came from if the logs don’t say. And every time it guesses wrong, the token cost compounds.

InsForge is built around a different set of assumptions.

• The backend exposes its state through structured metadata and the CLI gives the agent programmatic control with clear success/failure signals.

• The skills encode the correct patterns so the agent doesn’t have to discover them through trial and error.

• And the model gateway keeps LLM operations inside the same backend, which removes the cross-service integration issues that caused most of the Supabase session's debugging.

InsForge is fully open source under Apache 2.0 and you can self-host it via Docker. The code, the skills, and the CLI are all on its GitHub repo: https://github.com/InsForge/InsForge.

Thanks for reading!

If you’re using GPT or Claude, you’re using the same model as everyone else, with the same capabilities, the same cost, and no competitive edge.

But if you take a small open-source model and fine-tune it on your specific task, it can outperform a model 100x its size, at a fraction of the cost and latency.

Devs typically associate fine-tuning with a painful setup, like curating datasets, labeling outputs, and hand-crafting reward functions.

In 2026, that’s no longer the case.

Modern fine-tuning using GRPO and RULER is redefining fine-tuning.

You can now train agents that genuinely improve through experience, without writing a single reward function or collecting a single labeled example.

Today, let’s walk through exactly how!

SFT vs. Reinforcement Fine-Tuning

In supervised fine-tuning (SFT), you collect input-output pairs and the model learns to imitate them.

The problem is that SFT teaches the model what to say, not how to succeed.

For agents that search, call APIs, and reason across multiple steps, imitation isn’t enough. You want improvement through trial and error.

Think of it this way:

• SFT = studying a textbook (memorizing answers to known questions)

• RL = on-the-job training (learning from trial, error, and feedback)

This is Reinforcement Fine-Tuning (RFT). You give the model a reward signal and let it discover the best strategies on its own.

How GRPO Works

GRPO (Group Relative Policy Optimization) is the most popular RFT algorithm today. It’s the same algorithm that powered DeepSeek-R1’s reasoning capabilities.

Essentially, instead of training a separate model to score responses, GRPO generates multiple completions and grades them relative to each other.

Here’s how it works for each prompt:

1. Sample a group: Generate N completions from the current model

2. Score each one: A reward function evaluates each attempt

3. Normalize within the group: Calculate relative advantage vs. the group average

4. Update the model: Reinforce above-average behaviors, suppress below-average ones

GRPO only needs relative rankings, not absolute scores. Whether completions score 0.3, 0.5, and 0.7 or 30, 50, and 70 doesn’t matter. Only the ordering drives learning.

ART: Agent Reinforcement Trainer

GRPO is powerful, but how do you actually apply it to a real-world agent?

ART (Agent Reinforcement Trainer) is a 100% open-source framework that brings GRPO to any Python application.

Most RL frameworks are built for simple chatbot interactions, involving one input, one output, and the job is done.

Real agents are fundamentally different. They search documents, invoke APIs, and reason across multiple steps before producing an answer.

ART is built for exactly this. It provides:

• Native support for tool calls and multi-turn conversations

• Integrations with LangGraph, CrewAI, and ADK

• Efficient GPU utilization during training

Architecture

ART splits into two parts: a Client and a Backend.

The Client is where your agent code lives. It sends inference requests to the backend and records every action into a Trajectory, the complete history of one agent run.

The Backend is where the heavy lifting happens. It runs vLLM for fast inference and Unsloth-powered GRPO for training. After each training step, a new LoRA checkpoint loads automatically into the inference server.

The full training loop

1. Client sends an inference request

2. Backend generates model outputs

3. Agent takes actions in the environment (tool calls, searches, etc.)

4. Environment returns a reward

5. Trainer updates the model via GRPO

6. A new LoRA checkpoint loads into the inference server

7. Repeat, with each cycle, the model gets a little better than before

RULER: RL without manual reward functions

Defining a good reward function has always been the hardest part of RL.

Training an email agent requires labeled correct answers. Training a code agent requires test suites. Each one is its own unique engineering project.

RULER (Relative Universal LLM-Elicited Rewards) eliminates this bottleneck entirely. It uses an LLM-as-judge to compare multiple agent trajectories and rank them, with no labeled data required.

It works because of two key insights:

• Asking an LLM “rate this 0-10” produces inconsistent results

• Asking “which of these 4 attempts best achieved the goal?” is far more reliable.

And since GRPO only needs relative scores, the absolute values don’t matter anyway.

The process is three steps:

1. Generate N trajectories for a scenario

2. Pass them to an LLM judge, which scores each from 0 to 1

3. Use those scores directly as rewards in GRPO

A practical example

We put together a fully working notebook that trains a 3B model to master how to use any MCP server through reinforcement learning using ART.

Simply provide an MCP server URL, and the notebook will:

1. Query the server’s tools

2. Generate a set of input tasks that use those tools

3. Train the model on those tasks using automatic RULER evaluation

You can find more examples to adapt and get started in the ART GitHub repo.

Here’s the GitHub Repo →

12 must-use features in Claude Code​

• CLAUDE .md is your project’s memory. It stores your stack details, conventions, and rules so Claude loads them at every session start.

• Permissions let you whitelist or block tools like Bash per session. If you’re working on anything production-facing, this is non-negotiable.

• Plan Mode makes Claude draft a step-by-step plan before touching any code. You get to approve or reject before anything runs.

• Rules let you set project-wide behavioral guardrails with specific dos and don’ts beyond what CLAUDE(.)md covers.

• Skills are reusable instruction files you store in .claude/skills/. Write them once and Claude follows them automatically every time.

• Hooks fire shell scripts on events like PreToolUse and PostToolUse, which makes them perfect for auto-linting or triggering tests.

• MCP connects Claude to databases, APIs, and services. This is how you give it real-world access beyond your codebase.

• Plugins let you add Docker, pytest, and VS Code extensions without writing any integration code.

• Slash Commands store workflow shortcuts in .claude/commands/ so you can trigger complex flows with a single keystroke.

• Subagents spawn parallel Claude instances that divide and conquer multi-step workflows simultaneously.

• Voice Mode lets you talk to Claude hands-free, which is great for quick queries while your hands are on the keyboard.

• Rewind lets you step back to any checkpoint in your session and restart cleanly from that point.

We covered the anatomy of the .claude folder in a recent issue.

​Read it here →​

👉 Over to you: Which features do you use the most in CC?

Thanks for reading!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

Most RAG cost optimization happens at the model layer, like smaller models, fewer calls, and batching.

The retrieval payload itself rarely gets measured.

A typical setup retrieves 5 chunks per query, each around 300 tokens. That’s 1,500 input tokens before the LLM writes a single word, and at scale, that compounds.

But the bigger problem is accuracy. Enterprise documents repeat the same facts across multiple file versions.

When retrieved chunks say slightly different versions of the same thing, the LLM blends them. The answer sounds confident and is wrong in ways that are hard to catch.

Blockify (GitHub repo) sits between your raw docs and vector store.

Instead of splitting text into raw chunks, it uses a fine-tuned LLM to generate small, structured knowledge units called IdeaBlocks, where each one is built around one question and one validated answer. Average size: 98 tokens.

It runs on Intel Xeon CPUs, so no GPU server is needed to get started.

On a published benchmark, the IdeaBlock index outperformed raw chunked indexing by 13.55% on vector accuracy, using the same source documents and embedding model.

The token count dropped 3.09X as a direct result of the smaller unit size.

The cost drops because the quality improved, not separately from it.

You can find the Blockify GitHub repo here →

72 techniques to optimize LLMs in production

On an H100 running Llama 70B, a single inference request hits 92% GPU compute utilization during prefill, then drops to 28% during decode on the same hardware a moment later. The workload changed, not the GPU.

For context:

• Prefill processes the entire prompt in parallel and saturates tensor cores.

• Decode generates one token at a time and reads the full KV cache from HBM at every step, which makes it memory-bandwidth bound.

This asymmetry is why a single optimization never gets you very far, and why LLM inference prices have still fallen roughly 10x per year, with GPT-4-level performance going from $20 per million tokens in late 2022 to around $0.40 today.

Most of that drop came from the serving stack, and we put together this visual, which lists the techniques that go into optimizing LLMs in production.

Every technique in the grid above is a response to one of three bottlenecks: prefill compute, decode memory bandwidth, or the cost of everything that wraps the model.

Stacking enough of these techniques closes the 5-8x cost-efficiency gap between optimized vLLM or TensorRT-LLM deployments and naive FP16 inference.

Today, let’s walk through the nine layers, what each one actually solves, and how they stack up in a real production deployment.

We covered a lot more in the LLMOps course with implementations and engineering logic.

You can start reading it here →

1. Model compression

Model weights live in GPU memory all the time.

A 70B model in FP16 is 140GB before you load a single token of context. Compression attacks this usage directly.

• INT8 halves the memory vs FP16.

• INT4 cuts it 4x.

• FP8 gives you native tensor core support on Hopper and Blackwell, which means compression plus speedup.

GPTQ, AWQ, and SmoothQuant are the three main algorithms here.

• GPTQ uses Hessian-based second-order information

• AWQ preserves salient weights based on activation magnitudes,

• SmoothQuant handles both weights and activations at W8A8.

Distillation and pruning attack the parameter count itself rather than the bits per parameter.

Multi-LoRA serving is the escape hatch for multi-tenant deployments, where you keep one base model in memory and hot-swap small adapter weights per request.

We covered this specific pillar in

• Part 9 of MLOps course →

• Part 10 of MLOps course →

• Part 12 of LLOps course →

2. Attention and architecture

Standard attention is O(N²). At 128K context, this will have 16 billion computations, which is why naive attention is infeasible at long context even on H100-class hardware.

FlashAttention reorders the attention math to be IO-aware, avoiding materializing the full N×N matrix.

PagedAttention applies OS-style virtual memory to the KV cache, eliminating fragmentation.

MQA, GQA, and MLA attack the number of KV heads.

MQA shares one KV head across all queries, GQA groups them, MLA compresses keys and values into a low-rank latent. DeepSeek-V2 reported a 93.3% KV cache reduction from MLA alone.

Sliding window attention restricts each token to a local window. MoE activates only a subset of experts per token. These are architectural choices driven entirely by serving economics.

We covered this specific pillar in:

• Part 3 of LLMOps course →

• Part 13 of LLMOps course →

3. Decoding

Decode is memory-bound because every new token requires a full pass over the weights and KV cache.

• Speculative decoding sidesteps this by generating a draft with a cheap model, then verifying in parallel with the main model.

• Medusa attaches extra prediction heads to the model itself, so the same model can draft its own candidate tokens without needing a separate smaller model.

• EAGLE improves on this by predicting at the hidden-state level rather than the token level, which gives higher draft accuracy and better speedups.

• Lookahead decoding skips the draft model entirely. It generates and verifies multiple tokens in parallel from the main model alone.

• Prompt lookup decoding copies spans directly from the input prompt, which is surprisingly effective for tasks with heavy prompt-output overlap like summarization or code edits.

• Constrained decoding enforces grammars at the token level, which is how providers guarantee valid JSON.

• Multi-token prediction trains the model to emit several tokens per forward pass.

We covered this specific pillar in:

• Part 4 of LLMOps course →

• Part 13 of LLMOps course →

4. KV cache

The KV cache grows linearly with context length, and for long conversations, it dominates memory (learn KV caching here)

A 70B model with 4K context per request already consumes several gigabytes of KV just for a modest batch size.

• Prefix caching reuses KV across requests sharing the same prefix, which is why system prompts and few-shot examples are effectively free after the first request.

• KV offload tiers cold cache entries to CPU RAM or NVMe.

• KV cache quantization compresses the cache itself, separate from the weights.

• Token eviction methods like H2O and SnapKV drop low-attention tokens from the cache. SnapKV reports 92% KV compression at a 1024-token budget with a 3.6x decode speedup.

• Attention sinks, from the StreamingLLM paper, keep the first few tokens permanently in the cache to prevent long-context generation from going incoherent past the cache limit.

• Chunked prefill splits long prompts into smaller pieces so decode steps can interleave with prefill work.

We covered this specific pillar in:

• Part 13 of LLMOps course →

5. Batching and scheduling

LLM inference is memory-bandwidth bound during decode, which means the GPU is usually starved. Batching more requests together amortizes memory reads across more useful work.

• Continuous batching does this at the iteration level. As soon as one request finishes generating, a new one takes its slot mid-flight.

• Dynamic batching waits for a short window to group arriving requests. Batching 32 requests together cuts per-token cost roughly 85% with minor latency impact.

• Prefill-decode disaggregation splits the two phases onto separate GPU pools. Perplexity, Meta, and Mistral run this in production because co-locating prefill and decode on the same GPU means decode requests freeze every time a new prefill enters the batch.

• SLO-aware scheduling prioritizes interactive traffic over background jobs.

• Spot GPU scheduling runs preemptible workloads on cheap capacity.

• Request deduplication merges identical in-flight queries.

We covered this specific pillar in:

• Part 13 of LLMOps course →

• Part 14 of LLMOps course →

• Part 15 of MLOps course →

6. Parallelism and kernels

• Tensor parallelism splits weight matrices across GPUs.

• Pipeline parallelism splits layers.

• Expert parallelism shards MoE experts across devices.

• Sequence parallelism splits along the token dimension.

• CUDA graphs reduce kernel launch overhead, which matters because decode launches thousands of tiny kernels per second.

• Kernel fusion combines multiple operations into one launch.

• Torch compile produces fused kernels automatically via graph-level compilation.

We covered this specific pillar in:

• Part 13 of LLMOps course →

7. Application caching

The cheapest inference is the one you skip.

• Prompt caching reuses the KV state of static prefixes across calls. Anthropic reports up to 90% cost reduction and 85% latency reduction for long cached prompts.

• Semantic caching matches queries by embedding similarity rather than exact string match, which handles paraphrases.

• Exact-match caching is the hash-based baseline.

• Response caching stores completed outputs.

• Embedding deflection routes simple queries to a vector search without ever calling the LLM.

• Batch API endpoints run async jobs at roughly half the per-token price for non-realtime workloads

We covered this specific pillar in:

• Part 13 of LLMOps course →

• Part 14 of LLMOps course →

8. Input/output shaping

Output tokens cost 3-10x more than input tokens across every major provider.

Claude Sonnet 4 is $3 per million input versus $15 per million output, so trimming either side of the call translates directly into margin.

• Prompt compression with tools like LLMLingua achieves up to 20x compression with minimal quality loss.

• Context pruning drops irrelevant retrieved chunks before they reach the model.

• System prompt optimization trims static prefixes that bloat every request.

• Response length caps, structured output modes, and few-shot pruning all attack output volume.

• Context distillation summarizes long histories into a shorter state.

• RAG over long context is often cheaper than stuffing everything into the window. Retrieval keeps the prefill bill bounded.

We covered this specific pillar in:

• Part 5 of LLMOps course →

• Part 6 of LLMOps course →

• Part 7 of LLMOps course →

• Part 8 of LLMOps course →

9. Routing and cost

Not every query needs a frontier model.

• Model routing picks a smaller model when a smaller model suffices.

• Model cascading runs a cheap model first and escalates to a larger one only when confidence is low. Advisor strategy is somewhat similar to this:

• Classifier routing learns which queries go where.

• Multi-provider failover routes across APIs for reliability and cost.

• QoS tiers separate fast-and-cheap traffic from slow-and-high-quality.

• Task-specific fine-tuning lets a 7B model match a 70B model on a narrow domain.

• Function calling offloads deterministic logic to tools so the model doesn’t spend tokens computing what code could.

We covered this specific pillar in:

• Part 12 of LLMOps course →

• Part 14 of LLMOps course →

Putting it together

A serious production stack touches most of these.

A reasonable setup for a general-purpose API might run FP8 weights, GQA-based attention with FlashAttention kernels, PagedAttention for KV, continuous batching with prefill-decode disaggregation, prefix caching for system prompts, semantic caching at the application layer, prompt compression for long retrieved contexts, and model routing to send trivial queries to a small model.

The gap between this stack and a naive FP16 deployment with static batching is 5-8x on cost-per-token, and each technique alone moves the number only a small amount, which is exactly why the compounding across all nine layers is what defines a serious production setup.

We covered a lot more in the LLMOps course with implementations and engineering logic.

You can start reading it here →

Thanks for reading!

AI agents write code without knowing your dependency graph, quality profiles, or security rules. So when something goes wrong, CI catches it minutes later.

SonarQube Agentic Analysis moves that verification into the agent's inner loop.

During a regular CI run, SonarQube stores full project context, like dependencies, compiled artifacts, type information, and build configuration.

When the agent writes a file, it invokes SonarQube Agentic Analysis mid-workflow.

The engine restores that cached context, applies your team's quality profiles and security rules, and runs the same analysis your pipeline uses. Same precision as a full CI scan, in seconds.

The agent generates, verifies, fixes, re-verifies, and commits. PRs that pass quality gates the first time, without the back and forth.

Compatible with Claude Code, Cursor, Codex, Gemini CLI, and VS Code with Copilot. Direct API available for automated pipelines.

Core analysis is free during the beta period with a SonarQube Cloud Teams or Enterprise plan.

Get started with SonarQube Agentic Analysis here →

Thanks to Sonar for partnering today!

Evolution of Agent Landscape From 2022-26

The biggest shift in AI agents hasn’t been about making models smarter.

They do have their part but it has been more about making the environment around them smarter.

Here’s how agent engineering evolved in just 4 years, across three distinct phases:

Phase 1: weights (2022)

Everything was about the model itself. Bigger models, more data, better training. Scaling laws suggested that progress will come from more parameters.

RLHF and fine-tuning shaped behavior in this phase.

If you wanted a better agent, you trained a better model. This worked great for single-turn tasks.

But it hit a wall fast. Updating one fact meant retraining. Auditing behavior was nearly impossible. And personalization across millions of users from one frozen set of weights didn’t happen.

Phase 2: context (2023-2024)

A key realization that happened in this phase was that you don’t always need to change the model.

You can change what the model sees.

Prompt engineering, few-shot examples, chain-of-thought, and RAG led the way here.

Suddenly, the same frozen model could behave completely differently based on what you put in front of it.

Developers stopped fine-tuning and started iterating on prompts and retrieval pipelines instead. It was cheaper, faster, and surprisingly effective.

But context windows are finite. Long prompts get noisy. Models attend unevenly (the “lost in the middle” problem is real).

And every new session starts fresh with zero memory of what happened before.

Context made agents flexible. It didn’t make them reliable.

Phase 3: Harness engineering (2025-2026)

This is where we are now.

The question changed from “what should we tell the model?” to “what environment should the model operate in?”

The model is no longer the sole location of intelligence. It sits inside a harness that includes persistent memory, reusable skills, standardized protocols (like MCP and A2A), execution sandboxes, approval gates, and observability layers.

The model stays the same. What changes is the task it’s being asked to solve.

An example could be a coding agent asked to implement a feature, run tests, and open a PR.

Without a harness, the model must keep repo structure, project conventions, workflow state, and tool interactions all inside a fragile prompt.

With a harness, persistent memory supplies context, skill files encode conventions, protocolized interfaces enforce correct schemas, and the runtime sequences steps and handles failures.

So you have the same model but completely different reliability.

The pattern across all three phases is simple:

• weights encoded knowledge in parameters (fast but rigid)

• context staged knowledge in prompts (flexible but ephemeral)

• harnesses externalized knowledge into persistent infrastructure (reliable and governable)

Each phase didn’t replace the previous one but rather built on top of what existed.

Weights still matter and so does context engineering. But the center of gravity has moved outward.

The most consequential improvements in agent reliability today rarely come from changing the base model.

They come from better memory retrieval, sharper skill loading, tighter execution governance, and smarter context budget management.

Building better agents increasingly means building better environments for models to operate in.

There’s a great paper on this titled Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering.

You can read it here →

We also published this deep dive (article) on agent harness engineering, covering the orchestration loop, tools, memory, context management, and everything else that transforms a stateless LLM into a capable agent.

You can read the Agent Harness article here →

Thanks for reading!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

To understand why it matters, you need to see where it sits.

Level 1: Prompt → Response

Each call is stateless. The model can use tools/APIs within a single request but nothing persists. Most production LLM apps are sophisticated Level 1 wrappers.

Level 2: Interactive assistant

The platform handles persistence for you with memory, tools, files, connectors. ChatGPT and Claude live here. These are capable, but entirely reactive.

Level 3: Delegated execution

You define the goal and the agent owns the execution. Claude Code, Codex, and deep research operate here. Your task keeps running when you walk away, but the agent won’t start new work on its own.

Level 4: Autonomous scheduled operation

The agent runs on its own clock using cron, webhooks, or event triggers with persistent state across runs. OpenClaw with heartbeat, n8n with AI nodes, or the custom stacks devs stitch together.

Level 5: Self-building systems

Tools like Lovable and Bolt already go from prompt to deployed app. But the output is a web app that sits there until someone interacts with it.

Level 5 is different.

You can say “monitor my competitors’ blogs, store new posts in a table, and Slack me when they launch a product” and then the agent creates the database schema, connects the integrations, sets the schedule, and deploys a workflow that runs every morning on its own. No one needs to be present.

The workflow it just created runs on a schedule, maintains persistent state, and acts without human initiation.

Those are the exact characteristics of Level 4. So the output of a Level 5 agent is itself a Level 4 agent.

If you want to see this in practice, Sim (GitHub repo) shipped Mothership as an early implementation of this.

You can describe what you need, and it creates tables, wires workflow blocks, configures integrations, and sets the schedule.

It’s fully open-source (27k+ GitHub stars) so you can easily self-host it and see the full implementation on GitHub.

Here’s the GitHub repo →

Google solved an old RNN problem

A new paper from Google Research introduces “Memory Caching,” and the idea is quite simple.

Here’s the problem it solves:

Modern RNNs compress the entire input into a single fixed-size memory state. As sequences get longer, old information gets overwritten. That’s why they still struggle with recall-heavy tasks compared to Transformers.

Memory Caching addresses this by splitting the sequence into segments and saving the RNN’s memory state at the end of each segment.

When generating output, each token looks back at all these saved checkpoints, not just the current memory.

The complexity trade-off is elegant:

• Standard RNNs: O(L)

• Transformers: O(L²)

• Memory Caching: O(NL), where N = number of segments

You control the trade-off by choosing how many segments to cache. The model smoothly interpolates between RNN-like efficiency and Transformer-like recall.

The paper proposes four ways to use these cached memories:

1. Residual Memory: just sum all cached states (simplest)

2. Gated Residual Memory (GRM): input-dependent gates that weigh each segment’s relevance to the current token

3. Memory Soup: interpolates the actual parameters of cached memories into a custom per-token network

4. Sparse Selective Caching (SSC): MoE-style routing that picks only the most relevant segments

Gated Residual Memory (GRM) consistently performs best across tasks.

Under simplifying assumptions, hybrid architectures that interleave RNN and attention layers can be viewed as a special case of Memory Caching. This gives clean intuition for why hybrid models work. They’re implicitly caching memory states.

On recall-heavy tasks, Memory Caching significantly closes the gap between RNNs and Transformers. When applied to already strong models like Titans, it pushes them even further ahead on language understanding benchmarks.

Transformers still lead on the hardest retrieval tasks like UUID lookup at long contexts. But the direction is clear that you don’t need to choose between fixed memory and quadratic attention. There’s a useful middle ground now.

All experiments are at an academic scale (up to 1.3B params). Whether these gains hold at the frontier scale remains open.

This comes from the same team behind Titans and MIRAS, so it’s part of a larger research program on memory-augmented sequence models.

You can read the paper here →

Thanks for reading!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

You have 80,000 agent trajectories from production. You need to find top 100 worth reviewing to improve your agent.

No LLM allowed to evaluate trajectories. How will you do this?

Let’s look at some approaches.

The simplest solution one could start with is random sampling. Pick 100 random trajectories and review.

But most production agents handle routine requests just fine, so you end up wasting a big chunk of your annotation budget.

Another approach can filter for longer conversations since 10+ user messages means more complexity.

But longer conversations skew heavily toward outright failures. You’ll surface obvious breakdowns but miss subtle issues hiding in conversations where the agent technically succeeded.

A recent paper from DigitalOcean takes a new approach.

It computes lightweight behavioral signals directly from the trajectory data using deterministic rules.

The signals fall into three groups:

1) Interaction signals:

• If a user rephrases the request or corrects the agent, that’s misalignment.

• Agent repeating itself is stagnation.

• User abandoning the agent is disengagement.

• User confirming something worked is satisfaction.

All are detected through normalized phrase matching and similarity checks.

2) Execution signals:

• A tool call that doesn’t advance the task is a failure signal.

• Repeated calls with identical or drifting inputs indicate a loop.

These are straightforward to extract from execution logs.

3) Environment signals, like rate limits, context overflow, and API errors.

• Useful to diagnose but not for training since they reflect system constraints, not agent decisions.

Each trajectory gets scored based on which signals fire, and you sample the highest-signal ones for review.

On τ-bench, they compared all three approaches on 100 trajectories:

• Random sampling hit a 54% informativeness rate.

• The length-based heuristic reached 74%.

• Signal-based sampling reached 82%.

This means roughly 4 out of every 5 trajectories are genuinely useful to improve the agent.

In fact, among conversations where the agent completed the task correctly, signal sampling still identified useful patterns in 66.7% of cases vs. 41.3% for random.

These are the subtle issues like policy violations, inefficient tool use, and unnecessary steps that don’t break the task but still matter for optimization.

The whole framework runs without any LLM overhead and can sit always-on in a production pipeline.

If you want to see this in practice, this signal-based approach is already integrated into Plano, an open-source AI-native proxy that handles routing, orchestration, guardrails, and observability in one place.

Here’s the Plano GitHub repo →

Here’s the paper on arxiv →

👉 Over to you: What is your approach to solve this?

10 Must-use Slash Commands in Claude Code

Setting up shell aliases is such a natural part of working in a terminal that most developers do it almost reflexively. If you run a command often enough, you alias it.

With Claude Code prompts, though, devs typically skip this step entirely and keep retyping the same 10-15 line instructions from memory, like their code review checklist, test gen constraints, pre-commit scan...and all this session after session.

The real cost isn’t just the repetition you do as a dev, but the prompt drift.

Every time you retype a prompt from memory, the wording shifts slightly. For instance, you might forget a constraint or phrase the expected output format differently.

With shell commands, this doesn’t matter because they’re deterministic, but with an LLM, slightly different phrasing may produce noticeably different output.

Claude Code’s custom commands fix both problems.

You can save a markdown file in .claude/commands/, and it becomes a slash command you can invoke with identical instructions every time.

The prompts are version-controlled through Git, so your whole team runs the same commands, and when someone improves a prompt, everyone gets the update on their next pull.

This is the same pattern Boris Cherny described in his thread on Claude Code workflows, where his every repeated workflow becomes a command, checked into Git, and shared with the team:

Let’s walk through how to set them up, then the 10 commands that have been most useful in my workflow. I’ll demo each one on a real ML inference service (FastAPI, scikit-learn, Alembic) so you can see the actual output, with full prompt templates you can drop into your own project.

How custom commands work

A custom command is a Markdown file inside a .claude/commands/ directory. The filename becomes the command name.

# Project-scoped (shared via Git, shows as "(project)" in autocomplete):
your-repo/.claude/commands/preflight.md  →  /preflight

# User-scoped (personal, works in all projects):
~/.claude/commands/orient.md  →  /orient

# Subdirectories create prefixed commands:
.claude/commands/db/migrate.md  →  /db:migrate

The file content is the prompt that gets sent to Claude when you run the command. You can use $ARGUMENTS as a placeholder for anything typed after the command name.

For instance, running “/dissect src/auth/session.ts” substitutes $ARGUMENTS with “src/auth/session.ts“.

You can also inject dynamic context using shell commands with the !command syntax:

## Current state
- Branch: !`git branch --show-current`

- Staged changes: !`git diff --cached --stat`

- Last 3 commits: !`git log --oneline -3`

Claude runs those shell commands before processing the prompt, so the context is always fresh.

Lastly, an optional YAML frontmatter at the top of the file lets you pre-approve tools (so Claude doesn’t ask for permission on every git call), set a model override, or add a description:

---
description: Pre-commit check for debug artifacts and code smells
allowed-tools: Bash(git *), Bash(grep *), Read, Glob
---

That’s the entire system, which includes a markdown file, an optional YAML header, and $ARGUMENTS for dynamic input.

Below are the 10 commands we’ve found most useful in practice:

The newsletter ahead is a bit too long to share over email due to size constraints.

We have shared the full setup guide, with usage videos and prompts here →

Thanks for reading!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

And the “memory” you feel when chatting with ChatGPT is an illusion created by re-sending the entire conversation history with every request.

That trick works for casual chat. It falls apart the moment you try to build a real agent.

Here are 7 failure modes show up the instant you skip memory:

1. Context amnesia: the agent asks for information you already gave it

2. Zero personalization: every interaction feels generic

3. Multi-step task failure: intermediate state silently drops mid-task

4. Repeated mistakes: no episodic recall means the same errors, forever

5. No knowledge accumulation: every session starts from scratch

6. Hallucination from gaps: when context overflows, the model invents

7. Identity collapse: no continuity, no trust

The obvious response is “throw more context at it.” That’s why 128K and 200K token windows feel like they should solve everything.

They don’t.

Accuracy drops over 30% when relevant information sits in the middle of a long context. This is the well-documented effect.

Context is a shared budget. Details like the system prompts, retrieved docs, conversation history, and output…all fight for the same tokens.

Even at 100K tokens, the absence of persistence, prioritization, and salience makes raw context length insufficient.

Memory isn’t about cramming more text into the prompt. It’s about structuring what the agent remembers so it can find what matters.

The cognitive science frame that actually helps

Lilian Weng’s 2023 formulation has become the default framework here.

Agent = LLM + Memory + Planning + Tool Use.

The four co-equal pillars.

Her taxonomy borrows from cognitive science, where human memory splits into three systems:

• Sensory memory captures raw perceptual input and holds it for a fraction of a second. Only the portions you pay attention to get passed forward.

• Working memory is where active thinking happens. It holds roughly 7±2 items at a time (Miller’s 1956 finding). Lose focus, and the contents disappear.

• Long-term memory is durable storage with no practical capacity limit. Retrieval is the bottleneck: you can store millions of things and still fail to recall the one you need.

Each maps directly to a component in modern agent architectures:

Long-term memory itself splits further:

• Episodic: specific past events (”on Tuesday, the PostgreSQL cluster went down”)

• Semantic: facts and concepts (”PostgreSQL is a relational database”)

• Procedural: skills and workflows (”when a user asks for a refund, first check the purchase date”)

The bridge between episodic and semantic is memory consolidation: repeated specific events distilling into general knowledge.

An agent that notices “users consistently prefer executive summaries” across dozens of interactions should turn that into a reusable rule. Without consolidation, your agent replays individual events rather than learning from them.

The minimal agent, and what breaks first

If you strip away the frameworks, an agent is a loop which goes like: perceive, think, and act.

If you tell it “I have 4 apples,” then ask “I ate one, how many left?” and it has no idea what apples you’re talking about. Each call exists in isolation.

Layer 1: The Python list

The first fix everyone reaches for is maintaining the interaction in a messages list:

Multi-turn works now. The apples question gets answered correctly because the full conversation re-ships with every call.

Two problems show up fast:

• The list grows unbounded. Around turn 200, you hit the context ceiling and the oldest messages silently drop. The user’s name from turn 1 disappears long before yesterday’s throwaway joke. No prioritization, just strict chronological order.

• Everything lives in RAM. The moment the Python process ends, your agent has no idea who you are.

Layer 2: Markdown files for persistence

The next move is writing memory to disk.

Markdown is a natural fit since they are human-readable, Git-friendly, and the agent can read it back as plain text. Claude Code uses exactly this pattern with CLAUDE.md and MEMORY.md files:

Persistence is solved because if you restart the script, and the conversation is still on disk. You could also maintain a separate facts file that the agent extracts over time:

You can open the file in any editor, see exactly what the agent knows, and fix it by hand. Genuinely useful for prototyping.

With 4 facts, this works perfectly. Load the entire file into context and the LLM handles any question about Sarah, her company, or her industry.

Now fast-forward three months. Your agent has 2,000 extracted facts and 200 conversation logs. That’s 500K+ tokens of markdown on disk, and your context window is 128K.

You can no longer load everything. You need to selectively retrieve only the facts relevant to the current query. With flat files, your only option is keyword search:

At small scale, markdown files work. At real scale, they force keyword retrieval, and keywords can’t handle synonyms, paraphrases, or connections across facts.

The information is on disk. But you can’t load all of it, and keyword search is too brittle to find the right pieces.

OpenClaw, for instance, stores memory as markdown checkpoint files, and over weeks of daily use, earlier facts quietly slip away as context accumulates and gets compacted. The storage is there but the retrieval isn’t.

Layer 3: Vector search

Next step is to chunk the markdown, embed them, and search by cosine similarity, which solves the synonym problem.

But then you face a new problem. Consider these three facts in your vector DB:

User asks: “Was Alice’s project affected by Tuesday’s outage?”

The query mentions Alice and Tuesday’s outage, so vector search ranks the first and third facts high.

But the critical bridge, “Project Atlas uses PostgreSQL,” mentions neither Alice nor Tuesday. It’s the connecting piece, and it’s the one that won’t surface.

Each fact is an isolated point in embedding space. The connective tissue linking them is invisible to vectors.

This isn’t an edge case but rather the normal shape of real-world questions.

Business knowledge is inherently relational and any question that crosses two or more hops exceeds what flat vector retrieval can answer.

The capability matrix

Each layer fixes the previous pain but reveals a deeper one:

You need persistence, semantic understanding, and relational reasoning in a single memory layer.

Building this yourself means gluing together a vector database, a graph database, a relational store, an entity extractor, a deduplication pipeline, and an edge-weighting system.

That’s weeks of infrastructure work before you write a single line of agent logic.

Cognee as the memory layer

Cognee is an open-source knowledge engine built for agent memory. It combines vector search with knowledge graphs and a relational provenance layer into a single system.

The entire API surface is four async calls:

Under the hood, these four calls encapsulate a three-store architecture.

Each store captures a dimension of knowledge the others can’t:

• Relational store → provenance: where data came from, when it was ingested, who has access

• Vector store → semantics: what content means, what it’s similar to

• Graph store → relationships: how entities connect, what causes what, who reports to whom

If you flatten any of these, you’ll lose information that matters for retrieval accuracy.

What cognify actually does?

cognee.cognify() runs a multi-stage pipeline that converts raw text into structured, interconnected knowledge:

1. Document classification by type and domain

2. Permission checking for multi-tenant access control

3. Chunk extraction that respects paragraph structure (not fixed-size cuts)

4. Entity and relationship extraction via LLM, with automatic deduplication through content hashing

5. Summary generation for efficient retrieval

6. Dual indexing into the vector store (embeddings) and graph store (edges)

The deduplication step matters more than it sounds. If the same entity shows up across 50 documents, Cognee merges it into a single graph node with 50 inbound edges.

Your agent no longer sees “Alice” as 50 different strangers. And the pipeline is incremental by default so only new or updated files get reprocessed.

Every graph node has a corresponding embedding. This dual representation is the core trick since it allows you to enter through vectors (find semantically similar content) and exit through the graph (follow relationships to connected entities), or the reverse.

That’s what makes multi-hop queries work without sacrificing semantic search.

Memify: memory that learns

memify() is another interesting practical detail, which runs an RL-inspired optimization pass over the graph:

• Strengthening useful paths that led to good retrieval

• Pruning stale nodes that haven’t been touched

• Auto-tuning edge weights based on real usage

• Adding derived facts by identifying implicit relationships

A customer support agent’s graph naturally strengthens paths through product docs and refund policies while letting rarely-queried HR edges decay. The graph develops its own sense of relevance over time.

Fourteen retrieval modes

Cognee ships 14 search modes but these are the most useful ones:

Building a real agent with Cognee memory

Here’s the complete pattern wiring Cognee into the perceive-think-act loop:

The memory cycle follows: ingest, extract, store, retrieve, respond, store again.

Each turn enriches the knowledge graph, and incremental processing means you only pay to index new content.

Session memory handles pronoun resolution automatically:

Multi-tenancy is built in at the graph level with per-dataset permissions (read, write, delete, share).

Takeaway

If you’re building an agent today, the real starting question is: “what does my agent need to remember, and what kind of questions will it answer?”

If your queries only need similarity search (”find conversations like this one”), vector-only memory works.

The moment queries cross entity boundaries (”Was Alice’s project affected by Tuesday’s outage?”), you need graph traversal.

You can wire together separate vector, graph, and relational stores yourself. Teams that go this route typically burn weeks on infrastructure for a memory layer that still doesn’t learn from its own usage.

Cognee collapses that into four API calls. Embedded defaults get you running in minutes. Swappable backends (Postgres, Qdrant, Neo4j) take you to production without changing your agent code.

Intelligence requires structure, not just storage. The three storage paradigms (relational, vector, graph) aren’t competing options. They’re complementary layers of the same memory system.

Check out Cognee on GitHub →

Thanks for reading!

Read the full Part 1 deep dive here →

Diffusion LLMs Part 1

It builds a complete understanding from first principles:

• how autoregressive generation is structurally memory-bandwidth bound)

• why Gaussian noise can’t work on discrete tokens

• how masked diffusion solves this with an ELBO-derived training objective

• the math behind the forward and reverse processes

• unmasking strategies

• block diffusion for KV cache compatibility

• and a detailed engineering comparison between the two paradigms.

Read the full Part 1 deep dive here →

Why care?

Every production LLM today, GPT-4, Claude, Gemini, LLaMA, generates text the same way: one token at a time, left to right.

Each token requires loading the full model weights through GPU memory, performing a tiny computation, and then loading all the weights again for the next token. On an A100, this means roughly 1 FLOP per byte of data moved, while the GPU is designed for 100+ FLOPs per byte.

Diffusion LLMs take a completely different approach. They start with a fully masked sequence and iteratively unmask all tokens in parallel, using bidirectional attention at every step. This shifts inference from memory-bandwidth bound to compute-bound, which is exactly where modern GPUs are efficient.

The results are catching up fast. Block diffusion (BD3-LM) is within 0.5 perplexity points of autoregressive on LM1B. LLaDA at 8B parameters matches LLaMA 3 on MMLU and exceeds it on TruthfulQA and HumanEval. And models like Dream 7B are already being served in production with SGLang.

Understanding how it works at a mathematical level, from the forward masking process to the ELBO objective to block-level KV caching, is going to be increasingly valuable as these models scale.

You can read the Part 1 here →

👉 Over to you: Do you think the future of LLM generation is pure diffusion, pure autoregressive, or some hybrid of the two?

Thanks for reading!

Unsloth Studio is a local, browser-based GUI for fine-tuning LLMs without writing any code.

It wraps the training pipeline in a clean interface that handles model loading, dataset formatting, hyperparameter configuration, and live training monitoring.

The process to fine-tune the latest Gemma 4 is simple:

1. Open the Unsloth Colab notebook (available here).

2. Pick your model and dataset

3. Hit start training

You can find the notebook here →

Advisor strategy in LLMs to optimize token costs

Yesterday, Anthropic shipped an “advisor tool” in the Claude API that lets Sonnet or Haiku consult Opus mid-task, only when the executor needs help.

The benefit is that you get near Opus-level intelligence on the hard decisions while paying Sonnet or Haiku rates for everything else. So frontier reasoning only kicks in when it’s actually needed, not on every token.

Back in February, UC Berkeley published a paper called “Advisor Models” that trains a small 7B model with RL to generate per-instance advice for a frozen black-box model.

The paper’s approach was to take Qwen2.5 7B, train it with GRPO to generate natural language advice, and inject that advice into the prompt of a black-box model.

The black-box model never changes, and the advisor learns what to say to make it perform better.

To test it, they found that GPT-5 scored 31.2% on a tax-filing benchmark. But adding the trained advisor took that to 53.6%.

Moreover, on SWE agent tasks, a trained advisor cuts Gemini 3 Pro’s steps from 31.7 to 26.3 while keeping the same resolve rate.

Anthropic’s advisor tool takes a different path to the same idea. Sonnet runs as the executor to handle tools and iteration.

When it hits something it can’t resolve, it consults Opus, gets a plan or correction, and continues.

Sonnet with Opus as advisor gained 2.7 points on SWE-bench Multilingual over Sonnet alone, while costing 11.9% less per task.

Haiku with Opus scored 41.2% on BrowseComp. Haiku alone scored 19.7%.

Implementation-wise, it’s a one-line API change. The advisor tokens bill at Opus rates, and the advisor typically generates only 400-700 tokens per call.

response = client.messages.create(
    model="claude-sonnet-4-6",  # executor
    tools=[
        {
            "type": "advisor_20260301",
            "name": "advisor",
            "model": "claude-opus-4-6",
            "max_uses": 3,
        },
        # ... your other tools
    ],
    messages=[...]
)

So the combined cost stays well below running Opus end-to-end.

Both approaches point to the same thing that you don’t need the most powerful model on every token.

You need it at the right moments, for the right inputs.

Here’s the paper by UC Berkeley →

Thanks for reading!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

A key limitation of coding agents today is that they can’t fetch live web data on their own.

And platforms like LinkedIn, X, and Reddit (where a ton of developer discussions happen) are notoriously hard to scrape due to bot detection, CAPTCHAs, and JavaScript rendering.

Bright Data open-sourced a skills plugin (brightdata/skills), which provides live web access directly into Claude Code, Cursor, Windsurf, and 40+ other coding agents, with automatic handling of all those access barriers.

You can use it to scrape any webpage as clean markdown, run Google searches that return structured JSON, and extract structured data from 40+ platforms, including Amazon, LinkedIn, YouTube, TikTok, and Reddit.

Here’s the GitHub repo →

Thanks to Bright Data for partering today!

How to build Agents that don’t fail in production

Here are some damages caused by AI in production:

• Replit’s Agent wiped out a production DB.

• Zillow lost $304M due to its home-buying AI.

• iTutor paid $365k when AI auto-rejected old applicants.

Today, let’s learn how to build Agents that don’t fail in production (with code).

One primary challenge with customer-facing AI agents is that they either escalate the issue too quickly to a human agent or confidently mislead the user.

Also, it’s not just the frequency of mistakes, but their severity, especially when even the worst 0.001% could matter.

Talking specifically about user-facing use cases, placing control guidelines and embedding business logic into instruction-following Agents is helpful.

Let’s use Parlant (open-source with 18k stars) to build a compliant conversational finance agent that processes and approves loans.

Parlant is a framework to build customer-facing agents that behave exactly as instructed.

1️⃣ Create Agent

We start by defining a loan-approval Agent.

Next, we declare some domain-specific terms the Agent may need to know to answer confidently.

2️⃣ Journey

Parlant introduces the idea of Journeys. They inform the Agent about the multi-step conversational flow that helps it guide the user through the conversation as intended.

Check this loan approval journey:

This provides the following Journey:

• Determine the type of loan the user wants

• Collect loan and income-related details

• Call the tool to check eligibility

• End if not eligible, else ask them to upload docs

• Call the tool to process the uploaded docs

• End if invalid docs, else loan approved

3️⃣ Guidelines

Guidelines in Parlant tell the agent how to approach specific situations through condition-action pairs.

This provides more control over the agent’s behaviour.

This reduces misalignments by ensuring the Agent’s behavior aligns with the business needs.

We can do more things to embed more control, but this simple setup itself gives a powerful instruction-following Agent.

In the video below, no matter what we try to get a loan approved, the Agent refuses to do so.

You can also inspect the exact reasoning.

This was just the primer on what you actually do with Parlant. We are still exploring and will cover more on building extensive, production-grade, and user-facing AI systems that don’t fail.

Building Agents is about engineering “behavior” at scale. So you cannot vibe-prompt an Agent and expect it to work.

Parlant gives the structure to build Agents that behave exactly as instructed.

In the meantime, here’s the Parlant GitHub repo →

​The code for today’s issue is available in this GitHub repo →​

Thanks for reading!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.