Onyx is a self-hostable AI chat platform that works with any LLM, like Claude, GPT, Gemini, Llama, or any open-weight model you want.

Onyx GitHub Repo

Here’s what it ships with:

• Agents that chain multiple tools in sequence

• RAG with full indexing across 40+ connectors (Slack, Drive, Confluence, Jira, GitHub, email, call transcripts)

• Deep research ranked #1 on DeepResearchBench, above every proprietary alternative

• MCP support for connecting to external systems

• Code interpreter for data analysis and file generation

• Self-host on your own infrastructure via Docker in a few mins

Unlike Claude’s MCP-based connectors that query your tools at runtime, Onyx actually indexes all your data. That means faster, more reliable search across everything your team has ever written.

The entire code is open-source under the MIT license, so you can see the full implementation on GitHub and try it yourself.

Find the GitHub repo here → (don’t forget to star it ⭐️)

Agent memory is only as good as its schema

When you give an agent a knowledge graph for memory, the default behavior is that the LLM handling extraction decides the structure on its own.

It picks the entity types, the relationship labels, and the attributes.

The results are generic.

Everything becomes a Topic or Object and every connection is labeled RELATES_TO, resulting in a graph that can’t be queried meaningfully.

Zep’s Graphiti (open-source with 26k+ GitHub stars) fixes this with a prescribed ontology.

You can define entity types and edge types as Pydantic models, and the extraction model classifies against these definitions instead of guessing.

Let’s look at why this matters, how the extraction pipeline works, with code!

Why flat retrieval breaks on multi-hop reasoning

Vector-based memory stores facts as text chunks and retrieves them by semantic similarity. That works until a query requires connecting facts that don’t appear in the same chunk.

Consider three facts stored about a project.

• Alice manages Project Atlas

• Project Atlas runs on PostgreSQL

• The PostgreSQL cluster went down Tuesday

A query like “was Alice’s project affected by Tuesday’s outage” needs all three.

Vector search will retrieve just facts 1 and 3 because both mention relevant terms. Fact 2 is the bridge connecting Alice to PostgreSQL through Project Atlas, but it mentions neither Alice nor Tuesday. Similarity search misses it.

A knowledge graph stores entities as nodes and relationships as edges. Instead of matching text, it traverses connections.

That chain (Alice → manages → Project Atlas → runs on → PostgreSQL) is what makes multi-hop reasoning work, and it is invisible to flat vector retrieval.

Where extraction fits in the pipeline

Every graph-based memory system follows the same pipeline. LLM accepts the raw data and extracts entities and relationships.

These get stored as nodes and edges, and at query time, the system searches the graph and injects relevant facts into the agent’s prompt.

The extraction step determines everything downstream. It decides what the graph contains, how it’s structured, and what can be queried.

When extraction is unstructured, the LLM picks entity types and relationship labels on its own.

A developer conversation about building a web app called Nexus with Python, TypeScript, React, and Docker produces nodes labeled Topic and Object with RELATES_TO edges across the board.

Two things break without proper structure:

• Retrieval collapses to semantic similarity because it can’t filter by type when everything shares the same label.

• Domain constraints disappear because there are no structured attributes like status or category to distinguish “active vs. completed” or “frontend framework vs. database.”

Defining the schema with Pydantic

The fix is the same pattern used everywhere in the AI stack.

• FastAPI endpoints get Pydantic response models.

• Function calling tools get Pydantic schemas.

• Agent memory works the same way in Zep.

Define custom entity types using EntityModel (a subclass of Pydantic’s BaseModel) with EntityText fields and descriptions that guide the extraction model.

from zep_cloud.external_clients.ontology import EntityModel, EntityText
from pydantic import Field

class Project(EntityModel):
    """
    Represents a specific software project, application, 
    or codebase that the user is building or contributing to.
    """

    project_status: EntityText = Field(
        description="Current status: active, completed, paused, or archived.",
    )
    project_type: EntityText = Field(
        description="Type of project: web app, mobile app, API, CLI tool, etc.",
    )

The docstrings and field descriptions are important here because good descriptions with concrete examples give the extractor enough signal to classify accurately.

The Pydantic descriptions above aren’t just classification instructions. They teach the extractor vocabulary it doesn’t know.

A Technology entity follows the same pattern.

class Technology(EntityModel):
    """
    Represents a programming language, framework, library, 
    database, or tool that the user works with.
    """

    tech_category: EntityText = Field(
        description="Category: programming language, framework, database, etc.",
    )

Edge types use EdgeModel and carry their own attributes.

from zep_cloud.external_clients.ontology import EdgeModel

class WorksOn(EdgeModel):
    """The user is currently working on, building, or contributing to a project."""
    role: EntityText = Field(
        description="User's role: lead developer, contributor, maintainer, etc.",
    )

class UsesTechnology(EdgeModel):
    """The user actively uses or works with a specific technology."""
    proficiency: EntityText = Field(
        description="Proficiency level: beginner, intermediate, advanced, or expert.",
    )

Finally, wire these into the graph with source/target constraints using EntityEdgeSourceTarget, which defines which entity types can connect through which edge types:

from zep_cloud import EntityEdgeSourceTarget

client.graph.set_ontology(
    entities={"Project": Project, "Technology": Technology},
    edges={
        "WORKS_ON": (
            WorksOn,
            [EntityEdgeSourceTarget(source="User", target="Project")],
        ),
        "USES_TECHNOLOGY": (
            UsesTechnology,
            [EntityEdgeSourceTarget(source="User", target="Technology")],
        ),
    },
)

The code enforces that

• WORKS_ON can only connect a User to a Project

• USES_TECHNOLOGY can only connect a User to a Technology.

• Any relationship that doesn't match these constraints won't produce a typed edge.

Under the hood

When a conversation is ingested with a schema, Zep’s extraction pipeline runs through five steps:

1. Entity extraction identifies named entities in the text and classifies them against the defined types

2. Entity resolution merges duplicates (”Nexus” and “the Nexus project” become one node)

3. Fact extraction identifies relationships and outputs them as typed edges using the source/target constraints

4. Fact resolution detects contradictions and invalidates outdated facts, preserving history through Graphiti’s bi-temporal model, which tracks both event time and ingestion time

5. Temporal extraction parses time references and maps them to validity windows on each edge

The Pydantic schema defined above guides steps 1 and 3.

Entity types tell the extractor what to look for. Edge types with their constraints tell it what relationships to classify. Resolution and temporal processing happen automatically.

Every edge also carries explicit validity intervals (t_valid, t_invalid). When information changes (”Alex moved from Project Atlas to Project Nexus”), the old fact is invalidated, not deleted.

This way, you can query what’s true now as well as reconstruct what the state would have been at any point in time.

The output in practice

We ingest a conversation where a developer named Alex discusses their work (an active web app called Nexus, their tech stack, proficiency levels):

Querying for Project nodes returns Nexus with populated project_status and project_type attributes.

The node isn’t a generic “Topic” or “Object.” It’s a Project with structured fields as defined in the schema.

The edges are typed too.

• WORKS_ON carries role: lead developer

• USES_TECHNOLOGY carries proficiency: advanced for Python and Docker, proficiency: intermediate for TypeScript.

This can now filter projects by status, technologies by category, and query “which active projects use PostgreSQL” with a precise answer.

Context templates

The final piece is context templates, which assemble typed facts into a prompt-ready block.

You can define which edge types and entity types to include, and Zep formats them with temporal annotations into a single string injected into the agent’s prompt.

client.context.create_context_template(
    template_id="dev-context",
    template="""# PROJECTS
%{edges types=[WORKS_ON] limit=5}

# TECH STACK
%{edges types=[USES_TECHNOLOGY] limit=10}

# PROJECT DETAILS
%{entities types=[Project] limit=5}

# TECHNOLOGIES
%{entities types=[Technology] limit=10}""",
)

It looks like this:

Every entry in the resulting context block is typed, temporally annotated, and carries the attributes defined. Save the template once, reference it by ID in agent calls.

The 10/10/10 constraint and schema as a reasoning boundary

Zep enforces a hard limit of 10 custom entity types, 10 custom edge types, and 10 fields per type.

That’s intentional to force a dev to think about what matters in a domain rather than modeling everything.

The source/target constraints also act as guardrails on what an agent is allowed to remember. If a schema doesn’t include an edge type connecting Project to Competitor, the extraction model won’t create that relationship, even if a conversation mentions both.

The schema defines the space of valid memories.

This is the same principle behind typed function calling, where we constrain the LLM’s output space so that it can’t produce invalid arguments. Memory schemas apply that same constraint to what the agent stores.

Start with 3-4 entity types and 3-4 edge types that capture 80% of your domain logic, and add complexity incrementally.

Agent memory without schema discipline is a graph that behaves like a vector store.

In a way, you pay the cost of graph construction without getting the benefit of structured retrieval.

The schema is how you get that benefit back, and the fact that it’s Pydantic means there’s nothing new to learn.

You can find Zep’s GitHub repo here → (don’t forget to star 🌟)

👉 Over to you: have you defined a custom ontology for your agent’s memory, or are you letting the LLM decide the structure on its own?

Thanks for reading!

In 2023, AI products were thin wrappers. The LLM call was the entire workload, so cheaper models meant cheaper bills.

That’s not what AI queries look like today.

A single request can trigger a vector DB, web search, tools, and multiple LLM inferences before streaming a response back. The model got cheaper, but the infrastructure serving each query got heavier.

And when all of that runs inside a single deployable unit, auto-scaling makes it worse.

A spike in inference forces the entire app to scale, even when every other component is idle. The bill lands at 10x for a single-component problem.

AWS’ managed services decompose this by running each layer independently. Inference on Bedrock, vector search on Aurora pgvector, orchestration on EKS, each scaling on its own load curve.

Innovaccer runs this pattern for a healthcare AI platform serving 54M patients. After decomposing, they achieved 33% lower cloud costs and 65% less management overhead.

You can explore AWS’s resources for building AI-native SaaS here →

Thanks to AWS for partnering today!

Karpathy’s prediction about RL is coming true now!

Karpathy called reward functions unreliable and argued that a single reward number is too low-dimensional to teach an agent what “good” means for complex tasks.

To solve this, Agents need a knowledge-guided review as a higher-dimensional feedback channel.

Every major AI lab trains models with RL today (OpenAI, Anthropic, DeepSeek).

And their key bottleneck has always been the reward functions.

GRPO by DeepSeek worked well for math and code because the environment gave a binary signal.

But for real agent tasks, someone still has to hand-code the scoring function. That takes days and breaks every time the pipeline changes.

RULER (implemented in OpenPipe ART, 10k stars) addresses the exact problem Karpathy identified.

The reward criteria are defined in plain English, and an LLM evaluates each trajectory against that description to provide feedback for training.

We trained a Qwen3 1.4B agent that plays 2048 using GRPO with this exact workflow.

In this case, the agent saw the board, picked a direction, and RULER evaluated the outcome, all from this natural language definition.

You can see the full implementation on GitHub and try it yourself.

Here’s the ART Repo → (don’t forget to star it ⭐ )

ART GitHub Repo

Just like RLHF replaced manual rankings and GRPO replaced the critic model, natural language rewards are replacing hand-coded scoring functions.

RL reward engineering is now prompt engineering.

We also wrote a full walkthrough covering RL for LLM agents, from RLHF to GRPO to RULER. You can read it here →

Thanks for reading!

Knowledge graphs are insanely good at giving agents human-like memory!

Recently, we built an MCP-powered memory layer that can be shared across all your AI apps, like Cursor, Claude Desktop, etc.

Here’s a complete walk-through:

It’s built using a real-time knowledge graph.

Tech stack:

• ​Graphiti as a memory layer (open-source)

• Neo4j to store the knowledge graph

• Docker to self-host the MCP server

Everything is 100% open-source and self-hosted.

The anatomy of ~/.hermes folder

One folder controls everything your Hermes agent knows, remembers, and can do. Understanding its layout is the difference between treating Hermes as a black box and actually customizing it.

Here’s what lives inside and why each piece matters.

Configuration

config.yaml is the source of truth for everything non-secret, like model choice, terminal backend, tool enablement, and MCP servers.

The env holds your API keys and bot tokens and auth.json stores OAuth credentials.

Then there’s SOUL.md. It occupies slot #1 in the system prompt, before anything else loads.

It defines who the agent is, like its personality, tone, communication style, and hard limits. Everything the agent writes, creates, and remembers passes through this identity layer.

Knowledge

memories/ contains two tiny files. MEMORY.md (2,200 chars) holds project conventions, tool quirks, and lessons learned. USER.md (1,375 chars) holds your profile.

Both get injected into the system prompt as frozen snapshots at session start. When they fill up, the agent consolidates by merging entries, dropping redundancy, keeping only what’s dense and useful.

Capabilities

skills/ is where the learning loop lives. Each skill is a self-contained ability with a SKILL.md (the procedure), a references/ folder (docs the agent reads), and scripts/ (executable helpers).

Skills come from three sources. Either they are bundled with Hermes, downloaded from the hub via hub/, or created by the agent itself during your sessions.

Hermes ships with 687 skills across 18 categories, and you can add any GitHub repo as a custom tap.

Runtime state

sessions/ stores per-platform session metadata. state.db is the SQLite database with FTS5 indexing that backs tier 2 memory.

Automation

cron/ holds scheduled jobs in jobs.json and their outputs in output/. The gateway daemon ticks every 60 seconds and runs due jobs in isolated sessions. You describe schedules in plain English, Hermes converts them.

Extension + observability

plugins/, hooks/, and skins/ are the surface area for user customization. logs/ gives you agent.log, gateway.log, and errors.log for debugging.

You won’t manually edit most of these files. But knowing this layout means you understand exactly where identity, memory, skills, automation, and state live, and how they connect.

We wrote a full deep dive covering Hermes agent’s architecture, memory system, self-evolving skills, GEPA optimization, and setting up multiple specialized agents.

You can find the full deep dive here →

Thanks for reading!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

​Sim is a lightweight, user-friendly platform for building AI agent workflows in minutes.

​Sim GitHub repo

Key features:

• Real-time workflow execution

• Connects with your favorite tools

• Works with local models via Ollama

• Intuitive drag-and-drop interface using ReactFlow

• Multiple deployment options (NPM, Docker, Dev Containers)

Based on our testing, Sim is a better alternative to n8n with:

• An intuitive interface

• A much better copilot for faster builds

• AI-native workflows for intelligent agents

​GitHub repo → (don’t forget to star)

The top Hermes integrations to give your agent superpowers

The top Hermes integrations to give your agent superpowers:

1. Obsidian

The Karpathy-style second brain, but one that talks back.

Every note, page, and backlink in the vault becomes live context. The agent doesn’t just store knowledge, it reasons over it across everything that’s been written and saved.

2. Reddit

Unfiltered opinions from real users on any product, niche, or problem.

No SEO fluff, no corporate blogs. Just raw signal from people who actually use the thing. One of the best research integrations for market validation.

3. InsForge​

A full agentic backend behind one semantic layer.

Auth, database, storage, edge functions, all accessible without wiring five services together. The agent reasons about backend primitives directly instead of calling disconnected APIs.

Closest analogy: a PaaS built for agents.

​GitHub →

(don’t forget to star 🌟)

4. GitHub

Code, issues, PRs. Turns Hermes into an engineering teammate that can actually read the repo.

Essential for anyone shipping software.

5. Firecrawl

Web search designed specifically for agents.

Returns clean structured data instead of raw HTML, which means faster responses and fewer tokens burned per query. Worth keeping on by default.

6. YouTube transcripts

Converts any video into searchable text. Hour-long podcasts, tutorials, conference talks, all become indexed notes in seconds.

Easily the most underrated research integration in the stack.

7. Google Workspace

Gmail, Calendar, Drive, Docs, and Sheets through one connector.

An agent that can’t check the inbox, read the calendar, or write to shared docs is basically decorative. This should probably be the first integration anyone enables.

8. Discord

Ideal for channel-based automation.

Hermes can be plugged into specific channels with dedicated workflows in each. Support tickets from email can be scanned, categorized, and dropped into an organized channel every morning without anyone lifting a finger.

9. Stripe

Revenue, refunds, subscription changes, failed charges, all surfaced through a single question instead of clicking through dashboards.

“How many trials converted last week” or “which customers downgraded this month” gets a direct answer. Turns Stripe from a payment processor into a queryable business intelligence layer.

10. Bland (or Twilio)

Gives Hermes a voice for real phone calls. Booking reservations, confirming appointments, following up on invoices.

The call recordings are worth listening to just for entertainment.

11. Graphiti (by Zep)

Real-time knowledge graphs that build structured relationships from conversations and documents.

Instead of flat vector similarity, the agent traverses typed connections between entities. The difference between “find similar text” and “understand how things actually relate.”

​GitHub →

(don’t forget to star 🌟)

12. FireFlies

Every meeting transcript, fully searchable. “What did that client say about pricing last month” gets answered instantly instead of scrubbing through a 45-minute recording.

That said, if you’re looking to set up Hermes, I wrote a full deep dive covering the Hermes agent’s architecture, memory system, self-evolving skills, GEPA optimization, and how to set up multiple specialized agents.

​Get started here →

Thanks for reading!

Doc Holiday solves one of engineering’s most persistent problems: knowledge degeneration.

Every time code ships, the gap between what the product does and what the rest of the company knows widens. Documentation is usually the first thing to fall behind. The support team spends the next sprint answering questions that the last release already answered.

Drop it into your CI/CD pipeline and connect any upstream source, like Notion, Jira, Slack, Zendesk, Confluence, or Google Docs.

When a PR merges, it reads the commit history, linked tickets, and connected specs, then generates the changelog, release notes, and documentation updates automatically.

It keeps documentation in sync with the codebase without adding a step to the release process.

Get started with Doc Holiday here→

Thanks to Doc Holiday for partnering today!

Implement knowledge distillation from scratch.

Model performance is rarely the only factor in determining which model will be deployed.

This is because we also consider several operational metrics, such as:

• Inference Latency: Time taken by the model to return a prediction.

• Model size: The memory occupied by the model.

• Ease of scalability, etc.

Knowledge distillation is a common technique to compress ML models before deployment.

Let’s learn how to implement it!

We implemented 5 more techniques to compress ML models and reduce costs here: ​Model Compression: A Critical Step Towards Efficient Machine Learning​.

What is knowledge distillation?

In a gist, the idea is to train a smaller/simpler model (called the “student” model) that mimics the behavior of a larger/complex model (called the “teacher” model).

This involves two steps:

• Train the teacher model as we typically would.

• Train a student model that matches the output of the teacher model (there are some other types of knowledge distillation techniques as well).

DistillBERT, for instance, is a student model of BERT.

• DistilBERT is approximately 40% smaller than BERT.

• But it retains approximately 97% of the BERT’s capabilities.

Next, let’s look at the implementation.

Knowledge distillation implementation

Imagine we have already trained a CNN model on the MNIST dataset:

Its epoch-by-epoch training loss and validation accuracy are depicted below:

Let’s define a simpler model without any convolutional layers:

Since it’s a classification model, it will output a probability distribution over the <N> classes.

Thus, we can train the student to match the probability distribution of the teacher on all samples.

KL divergence (​also used to train tSNE​) can be used as a loss function.

It measures how much information is lost when we use distribution Q to approximate the distribution P.

Thus, in our case:

• P → probability distribution from teacher.

• Q → probability distribution from student.

The loss function is implemented below:

Finally, we train the student model:

The following image compares the training loss and validation accuracy of the two models:

The performance of the student model is not as good as the teacher model, which is expected.

However, it is still promising since it’s a simple feed-forward network.

Also, the student model is 35% faster than the teacher model, which is a significant increase in the inference run-time of the model for a 1-2% drop in the performance.

That said, one downside of knowledge distillation is that one must still train a larger teacher model first to train the student model.

This may not be feasible in a resource-constrained environment.

Today, we only covered knowledge distillation.

We implemented 5 more techniques to compress ML models and reduce costs: ​Model Compression: A Critical Step Towards Efficient Machine Learning.

👉 Over to you: What are some other ways to build cost-effective models?

Thanks for reading!

Reinforcement Learning Nanodegree part 4

The previous chapters covered MDPs, value functions, Bellman equations, and DP. All of that assumes you have full access to the environment's transition probabilities.

This chapter removes that assumption and covers how agents learn purely from interacting with the environment.

It covers:

• It covers Monte Carlo prediction and control

• The structural limits of MC that motivated temporal-difference learning

• The intersection of MC and DP and the bias-variance tradeoff between the two,

• SARSA, Q-learning, and maximization bias

• And a complete SARSA vs. Q-learning experiment on the Cliff Walking gridworld with code and analysis.

Everything is covered from scratch, so no RL background is required.

You can read Part 4 of the course here →

Why care?

Supervised learning handles prediction. There are inputs and labels, and the model learns the mapping.

But a growing share of the most important problems in AI right now are not prediction problems.

They are sequential decision problems.

• An LLM deciding which token to generate next based on a reward signal is making sequential decisions.

• An agent that searches documents, calls APIs, and composes a multi-step answer is making sequential decisions.

• A post-training pipeline that uses GRPO to teach a model to reason by generating multiple attempts and reinforcing the better ones is orchestrating sequential decisions.

RL is the ML paradigm designed for this, and it gives us the mathematical framework for learning what to do, not just what to predict, in settings where actions have consequences that unfold over time.

That is why it keeps showing up at the center of every major AI development. It is the right abstraction for the problems the field is increasingly trying to solve.

This series builds that understanding from the ground up, concept by concept, with math where it matters and hands-on code you can run.

• Part 1: Foundations of Reinforcement Learning →

• Part 2: Markov Decision Processes and Value Functions →

• Part 3: Bellman Equations and Dynamic Programming →

• Part 4: Model-Free Learning in RL →

👉 Over to you: What topics would you like us to cover in this RL series?

Thanks for reading!

Agent quality doesn’t live in the model. It lives in the harness. The same model with a better harness will be faster, smarter, and more efficient.

But maintaining that harness is still manual work in most teams. Someone is tuning the context window, classifying tool errors, tracking degradations, and manually repairing them.

Opik (open-source with 18k stars) implements patterns to solve this problem:

Test suites grow organically from real production failures.

You can write assertions in plain English (something like “the response must not hallucinate facts absent from the retrieved context”) and Opik evaluates traces against them.

Also, the whole regression suite builds itself over time from actual bugs.

Opik also ships with a debugging agent called Ollie. It reads the full span tree of a failing trace, reads the source code through opik connect, proposes a fix as a diff, reruns the agent with the same inputs, and saves the fix as a test case.

Agent configurations like prompts, tools, models, and params are versioned together as one unit with one-click rollback. The Agent Playground runs the full instrumented graph before deploying anything.

You can find the official documentation here →

And here’s the GitHub repo →

We are working on a hands-on demo on this. Stay tuned!

​Building custom tools for AI Agents​

Agentic frameworks provide native support for several tools.

For instance, here are some powerful tools provided by CrewAI:

However, you may need to build custom tools at times.

Today, let us show how it’s done!

We’ll build a real-time currency conversion tool. Instead of making an Agent guess exchange rates, we will integrate a custom tool to help the Agent get live exchange rates.

Let’s begin!

First, get an ​exchange rate API key​ (it’s free) and specify it in the .env file:

Next, we have some standard import statements:

To build a custom tool in CrewAI, you need to implement two classes:

#1) Class for input schema

Here, we declare the input fields the tool expects using Pydantic. Descriptions are important since they help the agent prepare a tool call.

#2) Class for the actual tool

Next, we define the CurrencyConverterTool by inheriting from BaseTool and declare attributes like name, description, arguments schema (which is the same as the input schema we defined above):

Every tool class has a _run method that implements the tool logic.

In the above code, we fetch live exchange rates using an API request. We also handle errors if the request fails or the currency code is invalid.

Done!

Now we build the Agent as usual and specify the tool we created above:

We define its Tasks:

We define the Crew and execute it:

Printing the response, we get this:

Works as expected!

Here, note that we directly passed a structured input to the Crew.

But real-world queries could look like this:

How much is 100 dollars in euros today?

We can introduce a Query Parser Agent (or even better…a regular LLM call) and produce structured outputs first. We can then pass the structured output like we did above.

Consider that an assignment from today’s newsletter issue, and feel free to reach out with your solution.

That said, we published a course to help you implement reliable Agentic systems, understand the underlying challenges, and develop expertise in building Agentic apps on LLMs, which every industry cares about now.

• ​​​​In Part 1​​​​, we covered the fundamentals of Agentic systems, understanding how AI agents act autonomously to perform tasks.

• ​​​​In Part 2​​​​, we extended Agent capabilities by integrating custom tools, using structured outputs, and we also built modular Crews.

• ​​​​In Part 3​​​​, we focused on Flows, learning about state management, flow control, and integrating a Crew into a Flow.

• ​​​​In Part 4​​​​, we extended these concepts into real-world multi-agent, multi-crew Flow projects.

• In ​​​​Part 5​​​​ and ​​​​Part 6​​​​, we moved into advanced techniques that make AI agents more robust, dynamic, and adaptable, like Guardrails, Async execution, Callbacks, Human-in-the-loop, Multimodal Agents, and more.

• In ​​​​Part 7​​​​, we covered Knowledge of agentic Systems.

• In ​​​​Part 8​​​​ and ​​​​Part 9​​​​, we primarily focused on 5 types of Memory for AI agents, which help agents “remember” and utilize past information.

• In ​​​​Part 10​​​​, we implemented the ReAct pattern from scratch.

• In ​​​​Part 11​​​​, we implemented the Planning pattern from scratch.

• In ​​​​Part 12​​​​, we implemented the Multi-agent pattern from scratch.

• In ​Part 13​ and ​Part 14​, we covered 10 practical steps to improve Agentic systems.

Of course, if you have never worked with LLMs, that’s okay. We cover everything in a practical and beginner-friendly way.

Thanks for reading!

The standard path from “I have a thing to run” to “it’s running and accessible” involves provisioning a VPS, configuring SSH keys, opening firewall ports, setting up a reverse proxy, and wiring HTTPS manually.

For a side project or internal tool, that overhead often exceeds the work itself.

exe.dev bundles compute, networking, and persistence into a single step.

One command gives you a persistent virtual machine, already exposed to the internet over HTTPS, with no firewall configuration or reverse proxy required.

The pricing model matches how developers actually work: shared CPU and RAM across machines, billed for underlying resources rather than per VM. Spin up several for different projects without paying for idle capacity.

It sits practically between a VPS and serverless, built for developers who want infrastructure control without the setup tax.

Get started with exe.dev here →

Thanks to the team for partnering today!

​Claude Code’s /goal command​

Claude Code completes a turn, then stops and waits for your next prompt.

For a 200-file migration or a full test suite fix, that means you’re typing “keep going” every few minutes for an hour. The bottleneck in long-running agentic coding sessions isn’t the model, it’s the human pressing enter.

/goal removes that bottleneck.

You define a completion condition once, and Claude keeps working turn after turn until a separate evaluator model (Haiku by default) confirms the condition is met by reading the conversation transcript.

The evaluator doesn’t run commands or read files on its own. Instead, it only judges what Claude has already surfaced.

Let’s look at how the evaluator loop works, what makes a good condition, and where it breaks down.

How the evaluator loop works

/goal runs one evaluation cycle per turn.

Claude finishes a turn (reads files, runs commands, edits code), and the evaluator model receives the full conversation transcript plus your condition.

The evaluator doesn’t call tools, run commands, or read files independently. It can only judge what Claude has already surfaced in the conversation.

This constraint shapes how you write conditions.

The condition describes the desired end state, not the current state.

For instance:

• /goal all tests in test/auth pass and the lint step is clean means “keep working until every test passes and lint is clean.” Claude will edit code, run the test suite, read failures, fix them, and re-run, turn after turn, until the evaluator sees passing output in the transcript. The condition works because the test output is visible in the conversation.

• But a condition like /goal the app is production-ready fails because “production-ready” doesn’t produce verifiable output. There’s no command whose result proves that claim, so the evaluator has nothing concrete to check against.

The evaluator returns a short reason with every judgment. That reason appears in the status view and in the transcript, so you can track what Claude is working toward at any point.

Run /goal with no argument to see the turns elapsed, tokens spent, and the last evaluator reason.

Writing effective conditions

Conditions can be up to 4,000 characters. The official docs recommend four components, including a measurable end state, a stated check (how Claude should prove it), constraints on what shouldn’t change, and optionally a turn or time cap.

Conditions that work well:

• /goal all tests in test/auth pass and the lint step is clean

• /goal CHANGELOG.md has an entry for every PR merged this week

• /goal every call site of the old API has been migrated and the build succeeds, stop after 20 turns

Each one describes an observable end state that Claude can demonstrate through its own output. The evaluator has concrete criteria to check against.

Vague conditions lead to two failure modes and both waste compute:

• Claude loops burning tokens without making progress

• or the evaluator hallucinates success because there’s nothing concrete to verify.

Interactive, -p, and remote control modes

/goal works across three Claude Code execution modes.

• Interactive mode is the default. You type /goal <condition> in your terminal session. Setting a goal starts a turn immediately with the condition as the directive, so you don’t need a separate prompt.

• Non-interactive mode with the -p flag runs the loop to completion in a single invocation: claude -p "/goal CHANGELOG.md has an entry for every PR merged this week"

• Remote Control mode lets multiple developers share goal-tracking agents in collaborative environments.

In all three modes, a ◎ /goal active indicator shows live elapsed time, turn count, and token usage as an overlay panel.

/goal vs. /loop vs. Stop Hooks vs. Auto Mode

Claude Code now has four autonomous workflow mechanisms. Each solves a different problem.

• /goal keeps a session alive across turns until a model-verified condition is met. Best for tasks with a defined end state: migrations, refactors, test suite fixes.

• /loop re-runs a prompt on a time schedule (every 5 minutes, every hour). Best for polling or periodic tasks: checking a deploy, summarizing new GitHub comments. It’s not the right tool for “finish this refactor.”

• Stop hooks live in your settings file, apply to every session in their scope, and can run custom scripts for deterministic checks or prompts for model-evaluated ones. More flexible than /goal, but requires configuration.

• Auto mode removes per-tool confirmation prompts within a turn but doesn’t start new turns on its own. The practical combination is auto mode + /goal: each turn runs unattended, and the loop runs until done.

Where /goal doesn’t work

There’s no built-in token budget. Claude keeps going until the condition is met or you interrupt with Ctrl+C (or /goal clear). Including a turn cap in your condition (stop after 20 turns) is the simplest mitigation.

The evaluator only sees the transcript. If Claude doesn’t print a test result or file diff, the evaluator can’t verify it. Conditions must be phrased around observable output.

Compound objectives overwhelm it. “Redesign auth, add OAuth, write tests, update docs” is too much for one goal. Break complex work into sequential goals, each with its own verifiable end state.

/goal runs only in workspaces where you’ve accepted the trust dialog. It’s unavailable when disableAllHooks is set at any settings level or when allowManagedHooksOnly is set in managed settings.

Setting up the project

Three things make /goal runs more reliable.

1. CLAUDE.md at your project root. Claude reads this automatically during every turn. Define architecture decisions, coding conventions, and acceptance criteria here so Claude has consistent context across a 30-turn run.

2. Hooks for auto-validation. Set up PostToolUse hooks to auto-run lint or type-checking after every file edit. Claude catches issues mid-run instead of discovering them at the end.

3. Auto mode enabled. Without it, a long /goal run stalls waiting for you to approve every file write. Auto mode + /goal is the combination that makes unattended runs practical.

If you’re not sure how to phrase your goal, let Claude write it for you. Paste something like this into Claude Code before running /goal:

Write me a /goal prompt. Ask me what I’m trying to do first, then keep asking follow-up questions until you can describe ‘done’ in specific, measurable terms.

The feature doesn’t solve every problem. But for any task where you can write a clear finish line, it removes the biggest friction in AI-assisted coding, which is you, sitting there, pressing enter over and over.

​As further reading, if you want to dive into the anatomy of the .claude/ folder: we covered it here in full detail →​

👉 Over to you: Have you tried /goal for a real codebase task yet? What condition did you set, and how did it go?

Thanks for reading!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

Hermes Agent takes a fundamentally different approach which makes it much more practically useful over OpenClaw. It ships with a learning loop that:

• Remembers across sessions

• Writes its own reusable skills

• Prunes them in the background

• And validates them offline through an evolutionary engine called GEPA

No other open-source agent combines all of them. Not even OpenClaw.

This guide covers how this learning loop works, what each memory layer does, and how to configure everything from scratch.

By the end, you’ll have three fully isolated agents running on your machine: a programmer (who uses your Claude code), a deep researcher, and a designer, each with its own personality, memory, skills, and Telegram bot.

Check this out:

The whole setup takes minutes, and everything here is reproducible on your own hardware.

What Hermes is, and what makes it architecturally different

The one-line pitch: an agent that gets better the longer you use it.

What makes that real is that three usually separate capabilities sit in one framework: runtime skill learning, persistent multi-layer memory, and an optional weight-training pipeline. No other open-source agent ships all three.

The closest comparison in the open ecosystem is OpenClaw. Both are persistent, messaging-friendly, but they make opposite architectural choices.

A clean framing from the Kilo blog captures it: “Hermes packages a gateway around a learning agent. OpenClaw packages an agent around a messaging gateway.”

How it’s built

Before the learning loop makes sense, you need a basic picture of how Hermes is structured.

Everything flows through a single AIAgent class in a run_agent.py script. CLI, messaging gateway, batch runner, IDE integration: they’re all entry points into the same core agent.

This is what makes the platform-agnostic story actually work.

The core loop is ReAct-style and synchronous. Build the system prompt, check if compression is needed, make an interruptible API call, execute any tool calls, and loop again.

A few key details:

• The agent can run commands in six different places. Local terminal, Docker, SSH, Modal, Daytona, or Singularity. Same code, just a config change. Move execution from your laptop to a cloud GPU server without touching anything else.

• It works with almost any model. A translation layer routes any provider through one of three API formats. That’s why you can swap from Claude to GPT to Gemini to local Ollama with one command and nothing breaks.

• The agent has a hard cap of 90 turns per task. Without it, an agent stuck in a loop (retrying a failing API, re-reading the same file) would silently burn through your credits. Subagents share the same budget, so a runaway delegation chain can’t sneak past either.

Before memory

Before we get to memory and self-evolving skills, there’s a layer that sits above both: identity.

Memory is what the agent knows. Skills are how it does things. But neither tells you who it is when it shows up. Without an identity layer, every agent feels like the same agent wearing different hats.

Hermes solves this with a single file: SOUL.md.

It lives at ~/.hermes/SOUL.md and occupies slot #1 in the system prompt, before anything else loads. It defines the agent’s personality, tone, communication style, and hard limits.

# SOUL.md

You are a pragmatic senior engineer with strong taste.
You optimize for truth, clarity, and usefulness
over politeness theater.

SOUL.md is hand-authored and static. You write it once, tweak it over time, and it stays consistent across every project and every session. If the file is missing, Hermes falls back to a built-in default identity.

Why does this matter for the self-improving story? Because everything that follows (the memory the agent writes, the skills it creates, the way it consolidates knowledge) happens through the lens of this identity.

SOUL.md is the fixed frame. Memory and skills are the moving parts inside it.

The memory system

Hermes doesn’t have a single “memory.” It has three layers, each designed for a different purpose.

Tier 1: Two tiny Markdown files.

At the core are two files stored on disk:

• MEMORY.md (2,200 chars max) holds the agent’s notes about your environment, project conventions, tool quirks, and lessons learned.

• USER.md (1,375 chars max) holds your profile: name, communication preferences, skill level, and things to avoid.

Both are injected into the system prompt as a frozen snapshot when a session starts. If the agent writes a new memory entry mid-session, that change persists to disk immediately but won’t appear in the system prompt until the next session.

When memory fills up (~80% capacity, shown as a percentage in the system prompt header), the agent has to consolidate.

It merges related entries into denser, more information-packed versions, so that only useful information survives.

Tier 2: Full-text session search.

Every conversation (CLI and messaging) is stored in SQLite with full-text search. The agent can search weeks of past conversations from this.

The tradeoff is clear: Tier 1 is always in context but tiny. Tier 2 has unlimited capacity but requires an active search plus LLM summarization.

Critical facts live in memory. Everything else is searchable on demand.

Tier 3: External memory providers (8 plugins).

For deeper persistent memory, Hermes ships with 8 pluggable providers that run alongside built-in memory (never replacing it). Only one can be active at a time.

When any external provider is active, Hermes automatically prefetches relevant memories before each turn, syncs conversation turns after each response, and extracts memories on session end.

Self-evolving skills

Memory handles facts. Skills handle procedures.

Skills are Markdown files with YAML frontmatter, and function as the agent’s procedural memory.

Here’s the anatomy of a skill:

---
name: k8s-pod-debug
description: >
  Activate for crashing pods, CrashLoopBackOff,
  "why is my pod restarting", container failures.
version: 1.2.0
author: agent
platforms: [linux, macos]
---

## Procedure
1. Get pod status → check events → pull logs
2. Look for OOMKilled, ImagePullBackOff, config errors

## Pitfalls
- Forgetting --previous flag on restarted containers

## Verification
- Pod stays Running with 0 restarts for 5+ minutes

To keep token costs low, skills use progressive disclosure:

• Level 0 → The agent sees names + descriptions only (~3k tokens for the full catalog)

• Level 1 → It loads the full skill content when it actually needs one

• Level 2 → It can drill into specific reference files within a skill

The self-improvement loop

This is the core differentiator. The agent creates its own skills autonomously using the skill_manage tool. Skill creation triggers when:

• The agent completes a complex task (5+ tool calls)

• It hits errors or dead ends and finds the working path

• The user corrects its approach

• It discovers a non-trivial workflow

So the loop works like as follows: the agent encounters a problem → solves it through trial and error → saves the successful approach as a SKILL.md file → next time it encounters a similar problem, it loads the skill and follows the proven procedure instead of rediscovering the approach from scratch.

The tool supports six actions: create, patch (targeted fix, preferred because it’s token-efficient), edit (full rewrite), delete, write_file, and remove_file.

Garbage collection for skills

Without maintenance, agent-created skills pile up. You end up with dozens of narrow, overlapping playbooks that waste tokens and pollute the catalog.

The Curator is a background maintenance system that handles this.

It runs on an inactivity check (not a cron daemon). Of 7 days have passed since the last run and the agent has been idle for 2+ hours, a background fork of the agent spins up with its own prompt cache, never touching the active conversation.

It operates in two phases:

1. Automatic transitions (deterministic, no LLM): Skills unused for 30 days become stale. Skills unused for 90 days get archived.

2. LLM review (up to 8 iterations): A forked agent surveys all agent-created skills and decides per-skill whether to keep, patch, consolidate, or archive.

Two important constraints:

• The Curator never touches bundled or hub-installed skills. Only agent-authored ones.

• It never auto-deletes. The worst outcome is archival to ~/.hermes/skills/.archive/, which is recoverable with one command.

Before every Curator pass, Hermes takes a tar.gz snapshot of the entire skills directory. Rollback is one command, and rollbacks are themselves reversible.

You can also pin critical skills with hermes curator pin <skill> to protect them from archival and deletion. Patches and edits still go through, so the agent can improve a pinned skill without requiring you to unpin it first.

GEPA: Evolving skills offline with execution traces

Here’s where it gets interesting.

The in-agent learning loop (skill creation + Curator) has a known weakness:

• The agent tends toward self-congratulation. It almost always thinks it performed well, even when it didn’t. Community feedback has confirmed this.

• The same system that auto-generates skills can also overwrite manual customizations with worse versions.

GEPA (Genetic-Pareto Prompt Evolution) solve this.

It is not built into the Hermes runtime. It lives in a companion repository (NousResearch/hermes-agent-self-evolution) and operates as an offline optimization pipeline.

The core idea is that instead of asking the agent “did you do well?”, GEPA reads execution traces to understand why things failed, then proposes targeted improvements through evolutionary search.

The pipeline:

1. Read the current skill from the Hermes repo

2. Generate an evaluation dataset (synthetic test cases via Claude Opus, real session history from SQLite, or hand-curated golden sets)

3. Run the GEPA optimizer: read execution traces → understand failure points → generate candidate variants

4. Evaluate candidates using LLM-as-judge scoring with rubrics (not binary pass/fail)

5. Apply constraint gates: full test suite must pass 100%, skills stay under 15KB, caching compatibility is preserved, semantic purpose doesn’t drift

6. Best variant goes out as a PR against the Hermes repo. Never a direct commit.

No GPU required. Everything runs through API calls. Cost: roughly $2-10 per optimization run.

This is something that can be skipped initially, but is highly effective when you hit a wall and don’t want to spend time and money on finetuning (RL/GRPO)

More details in this repo →

We recently wrote an article on GEPA →

It’s a great alternative to try before moving to full fine-tuning or RL-based fine-tuning.

To summarise:

SOUL.md sets the identity. The runtime loop captures experience. The Curator keeps the library clean. GEPA makes sure what’s in the library actually works.

That’s the full theory. Now let’s get it running on your machine.

Getting up and running

Linux, macOS, or WSL2. Python 3.11+ comes with the installer. 8GB RAM is fine for API-based usage.

curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash
source ~/.bashrc   # or ~/.zshrc

Run the setup wizard. It walks through provider, API key, model, and tools:

hermes setup

Then start chatting in the terminal:

hermes

If you want to talk to your agent from your phone instead of the terminal, point it at a Telegram bot.

Get a bot token from @BotFather (run /newbot), then get your Telegram user ID from @userinfobot.

This gives you a working agent:

What’s inside ~/.hermes/

Right after installation, your home directory gets a new folder.

It’s worth understanding the layout because everything you do with Hermes touches one of these paths.

~/.hermes/
├── config.yaml           # Main configuration
├── .env                  # API keys and secrets
├── auth.json             # OAuth provider credentials
├── SOUL.md               # Agent identity (slot #1 in system prompt)
│
├── memories/
│   ├── MEMORY.md         # Persistent agent facts
│   └── USER.md           # User model
│
├── skills/               # All skills (bundled, hub, agent-created)
│   ├── mlops/
│   │   ├── axolotl/
│   │   │   ├── SKILL.md
│   │   │   ├── references/
│   │   │   └── scripts/
│   │   └── vllm/
│   ├── devops/
│   └── .hub/             # Skills Hub state
│
├── sessions/             # Per-platform session metadata
├── state.db              # SQLite session store with FTS5
├── cron/
│   ├── jobs.json         # Scheduled jobs
│   └── output/           # Cron run outputs
│
├── plugins/              # Custom plugins
├── hooks/                # Lifecycle hooks
├── skins/                # CLI themes
└── logs/                 # agent.log, gateway.log, errors.log

A few files deserve a closer look.

• config.yaml is the source of truth for everything non-secret. Model choice, terminal backend, tool enablement, MCP servers all live here. Edit with hermes config edit or set values one at a time with hermes config set <key> <value>.

• .env holds your secrets. API keys, bot tokens, passwords. Hermes routes secret-looking values here automatically.

• SOUL.md is slot #1 in the system prompt, before everything else. Identity layer, covered earlier.

• skills/ is where the entire learning loop lives. Every skill the agent creates, plus everything you install, lands here.

• state.db is the SQLite database backing session search.

You won’t manually edit most of this. But knowing the layout makes everything else click.

Adding new Skills

Hermes maintains its own official Skills Hub with 687 skills across 18 categories. The breakdown:

• 87 built-in skills that ship with the agent

• 79 optional skills you can enable on demand

• 16 from Anthropic (frontend-design, pdf, pptx, docx, mcp-builder, etc.)

• 505 from LobeHub (broader community contributions)

You can also add any GitHub repo as a custom tap:

hermes skills tap add yourname/your-skills-repo
hermes skills install yourname/your-skills-repo/<skill-name>

This is how you’d share skills across a team or maintain your own private collection.

Going from 1 to 10 agents

One agent is fine. Multiple specialized agents is where Hermes gets interesting.

Hermes has a first-class feature for this called profiles. Each profile is a fully isolated Hermes instance with its own config, memory, skills, sessions, and SOUL.md. They share nothing by default.

We’ll set up three: a designer, a programmer, and a researcher.

hermes profile create designer --clone
hermes profile create programmer --clone
hermes profile create researcher --clone
hermes profile list

--clone copies your default profile’s config and .env as a starting point.

To give each one its own Telegram bot, each profile needs its own bot from BotFather. Telegram only allows one connection per token, so sharing breaks things.

Run /newbot three times with BotFather and save the three tokens. Then run the gateway wizard once per profile:

hermes -p designer gateway setup
hermes -p programmer gateway setup
hermes -p researcher gateway setup

The setup is exactly the same as a regular agent, where you can again create new bots in Bot Father and connect them to their respective agents.

Give each one a personality via SOUL.md

This is where the agents become genuinely different from each other. Edit each profile’s SOUL.md.

Designer at ~/.hermes/profiles/designer/SOUL.md:

# Soul

You are an expert at creating hand-drawn illustrations that explain
AI, machine learning, and software engineering concepts. Think
whiteboard sketches, not polished marketing art.

Every illustration should make a technical idea click. You lead with
the concept, then choose the metaphor, then commit to the sketch.
You prefer simple line work and clear labels over visual flourish.

Be opinionated about what to draw and what to leave out. Say when an
illustration would hurt more than help.

Check out these examples:

Programmer at ~/.hermes/profiles/programmer/SOUL.md:

# Soul

You are my staff engineer. Terse, direct, pragmatic.

You read code before you write code. You write the smallest change
that solves the problem. You prefer standard library over dependencies,
boring tech over shiny tech, and explicit over clever.

Always check: does this already exist in the codebase? Are there
tests? What breaks if this fails? Run the tests before saying "done."

Researcher at ~/.hermes/profiles/researcher/SOUL.md:

# Soul

You are my deep researcher for the AI and machine learning space.
Your main job is a daily Telegram digest of what's new and what
matters.

Cover four streams: trending GitHub repos, big tech and lab
announcements, fresh research papers, and the social pulse on X,
Reddit, and Hacker News. Lead with what changed since yesterday.
Cite every claim with a URL. Flag when signal is thin.

Use delegate_task aggressively to parallelize across streams. Never
state a contested claim as settled. Never fabricate a citation.

Route programmer’s execution through Claude Code

The programmer is more interesting if it doesn’t just write code itself, but delegates execution to the Claude Code CLI while Hermes orchestrates.

Claude Code does the file edits, runs commands, and manages git. Hermes reads the result and decides what’s next.

This is also how we run our on top of our Claude Max subscription without needing any separate API key. Claude Code uses Max credentials automatically.

Start a session and send this single activation prompt:

We already have a Claude Max subscription. You are my staff engineer who
helps me with my day-to-day coding tasks, and under the hood you use
Claude Code for all the executions. Set yourself up accordingly.

The programmer will install the autonomous-ai-agents/claude-code skill on its own, verify Claude is on PATH, and start using it for code execution. From the next message onward, anything coding-related (read files, write code, run tests, commit, push) routes through Claude Code under the hood.

Two things worth knowing:

• Make sure claude is on your PATH before activating. which claude command should print a real binary path.

• Claude Code has both a print mode (one-shot, fast, no TUI) and an interactive mode (full tmux session). The programmer picks based on the task. You don’t need to think about it.

Customizing the designer to teach it your visual style

The designer becomes genuinely useful when they can generate images in your style, not generic AI output. The pattern is to feed it reference designs, let it study them, and ask it to create a skill that generates new images in the same style.

This is the self-improving loop being used as a setup mechanism. Instead of writing a skill by hand, you’re showing the agent good examples and asking it to encode the pattern itself.

Start a session with the designer and paste your reference images (drag-and-drop in CLI, or attach in Telegram). Then send this prompt:

Carefully study these reference illustrations. Note the color palette,
line weight, level of detail, composition, and overall aesthetic.

I want you to create a new skill called "my-design-style" that captures
this visual style. The skill should:

1. Document the style fingerprint in plain language (palette, line
   weights, composition rules, recurring motifs)
2. Include a Python script that takes a text description of a new
   illustration and generates the image using the Nano Banana model
   (google/gemini-2.5-flash-image) via the OpenRouter API in this style
3. Read OPENROUTER_API_KEY from the environment

Use skill_manage to create it. Test the generated script on a sample
prompt before saying it's done.

The designer will study the references, write the SKILL.md, generate the Python script, save it under ~/.hermes/profiles/designer/skills/my-design-style/, and verify the script runs.

If you already ran hermes setup and picked OpenRouter as your provider, the key is already in the designer profile’s .env thanks to --clone. If not, add it once:

hermes -p designer config set OPENROUTER_API_KEY <your-key>

From then on, asking the designer for a new illustration triggers the skill. It writes a prompt informed by your style fingerprint, calls Nano Banana through OpenRouter, and saves the output.

The same pattern works for any style-specific output. Feed reference content, ask the agent to build a skill that reproduces the pattern. Newsletter intros, X threads, code review comments, anything where consistency matters.

Scheduling work with cron jobs

The researcher’s SOUL.md says it’s responsible for a daily Telegram digest. That implies a job running on its own schedule, without you remembering to ask. That’s what Hermes cron is for.

Hermes ships with a built-in scheduler. The gateway daemon ticks every 60 seconds, runs any due jobs in isolated agent sessions, and delivers output to whichever messaging platform you specify.

Jobs survive restarts. They live in ~/.hermes/cron/jobs.json and output goes to ~/.hermes/cron/output/.

You don’t need to write cron expressions. You describe what you want in English and Hermes converts it.

To wire up the researcher’s daily digest, open a session with the researcher and send this prompt:

Every weekday at 8am India time, prepare a deep digest of what's new
in the AI and machine learning space over the last 24 hours. Cover
four streams in this order:

1. Trending GitHub repos (especially new AI/ML tooling)
2. Big tech and lab announcements (Anthropic, OpenAI, Google, Meta,
   xAI, Nous, etc.)
3. Fresh research papers worth reading
4. Social pulse from X, Reddit, and Hacker News

Lead with what changed since yesterday. Cite every claim with a URL.
Keep it under 800 words. Deliver to Telegram.

Set this up as a recurring cron job.

The researcher creates the job using its cronjob tool, delivery target defaults to the current chat (Telegram in this case), and the scheduler takes over from there. Verify it was created:

hermes -p researcher cron list

You should see the job with its next scheduled run time.

Other useful patterns

The cron syntax is flexible. A few variations worth knowing:

• One-shot delays → /cron add 30m “Remind me to check the build” runs once in 30 minutes.

• Recurring intervals → /cron add “every 2h” “Check server status” runs every two hours.

• Standard cron expressions → /cron add “0 9 * * 1-5” “...” for precise control. Weekdays at 9am, in this case.

• Skill attachment → /cron add “every 1h” “Summarize new feed items” --skill blogwatcher loads a skill before running the prompt.

You can also chain jobs. One cron’s output becomes the next cron’s input via a context_from flag. Useful for multi-stage automations where you want a research step to feed a writing step.

We’ll continue to write a lot more about Hermes since we have been using it extensively.

Also, if you learn better from video, we’ll drop a full Hermes Agent walkthrough in a couple of days!

👉 Over to you: Have you used Hermes Agent?

Thanks for reading!

We built a Hugging Face fine-tuning studio that lets you fine-tune any LLM directly from Claude:

The app connects to the HF Hub for model and dataset search. It handles chat template formatting for the training data, and lets you configure LoRA rank, quantization, batch size, and learning rate directly from Claude.

Training runs on HF’s GPU infra via AutoTrain.

Once training finishes, you can also chat with your fine-tuned model (or any other LLM on HF) directly from Claude

The studio’s built with the mcp-use SDK, an open-source full-stack framework to build MCP Apps for Agents.

In mcp-use, any MCP tool can be associated with a UI.

You define a tool handler, create a React component, and the mcp-use framework handles the tool registration, prop mapping between server and widget, bundling, and hot reload during development.

The widgets follow the MCP Apps standard, inspired by OpenAI’s Apps SDK.

You can find the mcp-use GitHub repo here →

And you can find the code for this fine-tuning studio here →

Speculative decoding in LLMs

Google uses speculative decoding in AI Overviews to serve over a billion Search users.

It’s also how Anthropic, Meta, and most major inference providers reduce latency at scale and get 2-3x more tokens per second, with mathematically identical outputs.

We covered the mechanics of LLM inference (prefill and decode phases, KV caching, speculative decoding, batching, and optimization techniques that improve latency and throughput here in the LLMOps course →

The loop has three steps and the diagram below depicts how speculative decoding differs from standard decoding:

• A small model generates K candidate tokens autoregressively. Because it’s 10-100x smaller than the target, this costs roughly 1-2% of a target forward pass.

• Then a large model processes all K draft tokens in a single forward pass, computing its distribution p(x) at each position. This parallel verification is structurally identical to prefill, so it saturates GPU compute instead of being memory-bandwidth bottlenecked like single-token decoding.

• Finally, each draft token is accepted or rejected by comparing the probability distributions of the small and large models. The algorithm walks through tokens sequentially. If token 3 is the first rejection, tokens 1-2 are accepted as-is (the small model got them right), the large model replaces token 3 with its own prediction (which it already computed during the verification pass), and tokens 4-5 are discarded without evaluation.

Note: Traditionally, both models must share the same tokenizer since verification compares probability distributions token-by-token. Universal Assisted Generation (UAG, Transformers 4.46+) relaxes this via text-level re-encoding, but same-tokenizer pairs remain faster because they skip that overhead.

In the best case, you get K+1 tokens from just one large model call.

And in the worst case, where every draft token is rejected, you still get 1 token from the large model, which is the same as standard decoding.

Implementation-wise, Hugging Face Transformers already exposes this as assistant_model in the generate() call:

For production serving, vLLM also supports this.

Here are some production tradeoffs:

• The same tokenizer gives the best speedups. This is because when draft and target share a tokenizer, verification happens directly at the token ID level with zero overhead. Cross-tokenizer speculation also works, but it’s slow.

  • Cross-tokenizer pairs → 1.5-1.9x speedup

  • Same-tokenizer pairs → 1.5-3x speedup

• A larger draft model will have a higher acceptance rate, but the drafting overheads themselves get expensive and eat the gains. For instance:

  • Llama 3.2 1B as the drafter achieved 2.31x.

  • The larger Llama 3.1 8B only hit 2.08x despite higher acceptance.

Eliminating the second model

The classic setup has three practical pain points.

• You need a matched draft model from the same family.

• The model consumes extra GPU memory.

Several variants have emerged to eliminate one or more of these.

• EAGLE removes the separate-model problem entirely. Instead of a standalone drafter, it trains a lightweight head (< 1B params) directly on the larger model’s hidden states.

• Medusa solves the same problem differently. It adds multiple prediction heads to the larger model, each predicting tokens at different future positions simultaneously.

• Self-speculative decoding (LayerSkip, SWIFT) eliminates both the extra model and the training requirement. It uses the target model’s own early layers as the drafter, then the full model for verification.

The direction is that speculative decoding is converging toward single-model solutions where the draft capability is built into the target model itself, either through trained heads or layer skipping.

For most production setups today, though, the two-model approach with a same-family drafter remains the simplest path to 2-3x speedups.

As further reading:

We covered the mechanics of LLM inference (prefill and decode phases, KV caching, speculative decoding, batching, and optimization techniques that improve latency and throughput here in the LLMOps course →

And we covered 72 techniques to optimize LLMs in production here →

Here’s the visual again for your reference:

Thanks for reading!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

RAG can’t keep up with real-time data.

Graphiti builds live, bi-temporal knowledge graphs so your AI agents always reason on the freshest facts. Supports semantic, keyword, and graph-based search.

100% open-source with 26,000 stars.

GitHub repo (don’t forget to star the repo) →

Graphiti on GitHub

Claude Code’s architecture, explained visually!

Claude Code is a lot more than a CLI that invokes the Claude models.

The actual system has six layers, and the model is just one node inside the loop. The diagram breaks down every component:

𝗜𝗻𝗽𝘂𝘁 𝗟𝗮𝘆𝗲𝗿 handles session management, permission gating, and YAML-based trust tiers before anything reaches the model.

𝗞𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 𝗟𝗮𝘆𝗲𝗿 holds the skill registry, context compressor, task graph, and cross-session memory store. This is where harness intelligence lives outside the weights.

The context compressor is a 5-layer cascade that kicks in when the context window hits roughly 95% capacity. It doesn’t summarize your conversation the way ChatGPT does. Instead, it runs structured extraction on file paths, code snippets, and error histories while pruning redundant tool outputs. The goal is to keep the context usable, not just smaller.

𝗘𝘅𝗲𝗰𝘂𝘁𝗶𝗼𝗻 𝗟𝗮𝘆𝗲𝗿 runs tool dispatch through a typed registry with one handler per tool, like bash, read, write, grep, glob, and revert.

The streaming runtime handles parallel execution, and the prompt cache reuses stable prefixes at roughly 10% of the original cost.

𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻 𝗟𝗮𝘆𝗲𝗿 connects the MCP runtime to external servers (filesystem, git, custom). Tools register inward, and memory writes outward to a markdown file (agent_memory.md) that persists across sessions.

𝗠𝘂𝗹𝘁𝗶-𝗔𝗴𝗲𝗻𝘁 𝗟𝗮𝘆𝗲𝗿 is the most underappreciated piece, and it works very differently from what most people assume.

Claude Code supports two levels of parallelism: subagents and agent teams (covered here in detail).

• Subagents are lightweight workers that run inside your session. They get their own context window, do a focused task (search the codebase, explore a file tree), and return results to the parent. They can’t talk to each other, and they can’t spawn their own subagents. It’s a strict parent-child hierarchy.

• Agent teams go further. One session acts as a team lead, and it spawns independent teammates, each running as a full Claude Code instance with its own context window. The team lead breaks a task into subtasks, assigns them, and monitors progress.

The coordination happens through two mechanisms → a shared task list (JSON files on disk) and a mailbox system for peer-to-peer messaging.

Each teammate gets git worktree isolation. It’s a separate working directory with its own branch, sharing the same repository history.

This means agents can write to overlapping parts of the codebase without file conflicts. When they finish, worktrees with no changes are cleaned up automatically. Worktrees with changes persist for human review before merging.

𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗟𝗮𝘆𝗲𝗿 wraps everything. An event bus with lifecycle hooks logs all tool calls and messages, creating a complete audit trail of the agent’s actions and decisions.

Background executors run daemon threads non-blocking, so observability never stalls the main loop.

The master agent loop sits at the center of all six layers, and it’s deliberately simple. It assembles context, calls the model, receives a tool request, executes it, feeds the result back in, and repeats. Every iteration is one turn.

Within a turn, the model might request a tool call. That request flows through the permission system, gets executed, and the output feeds back into the loop as the next input.

The loop itself is single-threaded on purpose. All the intelligence lives in the layers around it, not in the loop logic. Anthropic calls it a “dumb loop” because the model reasons, and the harness mediates.

This is the architecture behind Claude Code.

We recently wrote this article, which is a deep-dive on how Anthropic, OpenAI, LangChain, and others build this pattern from the ground up →

That’s a wrap!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

Part 3 is now available, and you can read it here →

Reinforcement Learning Nanodegree Part 3

Part 2 covered the MDP framework and value functions. This one teaches you the equations and algorithms that actually compute them.

It covers:

• The Bellman expectation equations

• the Bellman optimality equations

• and the dynamic programming methods built on top of them, like iterative policy evaluation, policy improvement, policy iteration, and value iteration, with hands-on implementations.

Everything is covered from scratch, so no RL background is required.

You can read Part 3 of the course here →

Why care?

Most ML practitioners today have deep intuition for supervised learning.

But RL operates on a fundamentally different set of ideas. There is no labeled dataset. The agent generates its own training data through interaction. Actions have delayed consequences. Exploration is not optional but rather a core part of the learning process.

This is also where the biggest breakthroughs in AI are compounding right now. The field is having its moment, and the demand for engineers who understand it deeply is growing fast.

• Every technique that made recent LLMs dramatically better (RLHF, GRPO, DPO, constitutional AI) is a direct application of RL.

• Every agentic system that takes multi-step actions, calls tools, and operates over long horizons is an RL problem.

When you read about these in a paper or blog post, they only land if you understand what a policy is, what a value function measures, why reward shaping is hard, and how exploration works.

This series builds that understanding from the ground up, concept by concept, with math where it matters and hands-on code you can run.

This series is structured the same way as our MLOps/LLMOps course: concept by concept, with clear explanations, diagrams, math where it matters, and hands-on implementations you can run.

And no prior RL background is needed.

• If you haven’t read Part 1, start there first. It covers the agent-environment loop, exploration vs. exploitation, and multi-armed bandits →

• And part 2 covers Markov Decision Processes and Value Functions (the formal language that every RL algorithm is built on) →

Over to you: What topics would you like us to cover in this RL series?

Thanks for reading!

The above line sounds counterintuitive, but MCPMark V2 benchmarks confirmed this across 21 backend tasks.

And it’s not a minor 3-5% difference.

But 54% higher token usage.

The reason has nothing to do with the model itself.

Instead, it has to do with what the agent needs to know before it can start building.

When you’re building a full-stack app, CC must understand the entire backend, like:

• what tables already exist

• what RLS policies are active

• what storage buckets are available

• which auth providers are configured

• and what edge functions are deployed

Most backends don’t hand over this info cleanly.

For instance, with Supabase, asking for OAuth setup via MCP returns the entire auth docs, including sections on email/password, magic links, phone auth, SAML, and SSO.

That’s 5-10x more tokens than the agent actually needed. And this happens on every MCP call across every domain.

The agent then discovers the state through separate calls to list_tables, execute_sql, and list_extensions, each returning a partial view.

Some info, like which auth providers are configured, isn’t queryable through MCP at all.

And when something breaks, Supabase returns the same error code whether the rejection came from the platform layer or from the function code.

The agent has no way to infer accurately, so it cycles through code-level fixes for a problem that might not be in the code at all.

A better model does not have a magical way to skip these gaps.

In fact, it tries even harder to fill them, which means more discovery queries, more reasoning, and more retries. That’s why the token cost went up with a better Claude model.

A smarter approach is actually implemented in InsForge, an open-source backend (self-hostable via Docker) that offers the same primitives as Supabase but structures everything around the assumption that an agent is operating the backend, not a human on a dashboard.

Before writing any code, a single CLI call returns the full backend topology in ~500 tokens.

The agent sees every table, auth provider, storage bucket, and available AI models in one structured response.

Instead of one broad skill like Supabase that triggers on everything, it has four narrowly scoped skills.

• Creating tables only activates the CLI skill.

• Debug skill only activates when code breaks.

• Building frontend only activates the SDK skill.

• Wiring third-party auth only activates the integrations skill.

This keeps the agent’s cognitive load lean since it only loads what matches the current task.

The CLI returns structured JSON with semantic exit codes on every operation, so the agent always knows whether something succeeded or failed and why. There are no ambiguous 401s that may indicate three different things.

We tested both backends on the same full-stack RAG app and recorded the full sessions.

Supabase:

• consumed 10.4M tokens

• needed 10 manual interventions

InsForge:

• consumed 3.7M tokens

• completed the entire build without any errors

This isn’t a Supabase-specific problem. Most backends were designed for humans who can see dashboards and interpret raw errors.

When an agent operates the backend instead, every missing piece of context needs a discovery call, and every ambiguous error enters a retry loop.

Fixing this requires giving agents structured backend context before they start writing code.

InsForge is an open-source implementation of exactly this, and you can self-host it via Docker.

GitHub repo (9k+ stars): https://github.com/InsForge/InsForge

(don’t forget to star it ⭐ )

You can read our walkthrough on building the full-stack RAG with Supabase and Insforge in this newsletter issue →

Checkpoint any method automatically using CrewAI

Long-running agent flows fail mid-run, and in most frameworks, that means restarting from zero and burning the tokens again.

CrewAI just shipped checkpointing in v1.14. Every flow method becomes a recovery point, written automatically when an event you specify fires (e.g., method_execution_finished).

You can resume in one line and fork from any saved state into a new branch with full lineage tracking. An async TUI lets you browse checkpoints, inspect events, and resume or fork from the UI.

The pipelines become resumable, inspectable, and branchable processes, with zero extra infra.

You can find the CrewAI repo here →

Thanks for reading!

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

Your RAG system scores 90% retrieval accuracy on 5k company docs.

But scaling to 500k docs drops the accuracy to just 50%, with the same embedding model and retriever.

Why did this happen?

The simplest answer is that more documents mean more competition for the top-k retrieval slots. That is true, but it doesn’t explain why accuracy drops this dramatically.

The answer comes down to how enterprise docs are distributed in the embedding space.

Today, a single product decision in a company generates meeting transcripts, Slack threads, Confluence docs, Jira tickets, and email threads.

They are related to the same event, so they all land in a similar region of the embedding space.

As the company operates over months, this pattern repeats for every project/customer/roadmap, and the embedding space fills up with clusters of closely related documents.

But all related docs don’t contain the same facts.

• Slack thread covers the decision made

• Jira has the implementation deadline

• Confluence has the technical spec

• Email thread has the customer request

When a query is about a specific fact (like a deadline), the answer lives in one of those docs.

At a 5K corpus size, there might be 3-5 docs touching that topic, and the correct one easily lands in the top-k results.

But at a 500K corpus size, there could be 40-60 total docs, and the one containing the actual answer can easily get pushed out of the top-k by other topically relevant docs, degrading retrieval.

A recent research paper from Onyx documented this.

The researchers used their newly open-sourced EnterpriseRAG-Bench dataset.

EnterpriseRAG-Bench Data Repo

It has 500k+ synthetic enterprise documents spread across Slack, Gmail, Jira, GitHub, Confluence, Google Drive, HubSpot, Fireflies, and Linear, with realistic noise like misfiled documents, near-duplicates, and conflicting versions.

They ran the same retrievers at five corpus sizes from 5K to 500K.

• Vector search accuracy dropped from 90.7% at 5K documents to 50.6% at 500K docs.

• BM25 degraded more gracefully, from 85.8% to 68.4%.

• At every scale, higher neighborhood density in the embedding space monotonically correlated with lower recall.

The practical implication here is that retrieval accuracy on a 5k test set tells you almost nothing about production-scale performance.

Always test at a realistic volume to measure the neighborhood density in your embedding space to estimate how much headroom the retriever actually has.

The entire EnterpriseRAG-Bench dataset (500K docs with questions, and the whole evaluation harness) is open-source.

Run your retriever against it at 5K, then at 500K, and see where your own accuracy curve breaks.

You can find the GitHub repo here →

EnterpriseRAG-Bench Data Repo

Build your own 100% local AI second brain

Karpathy’s LLM Wiki compiles raw sources into a persistent Markdown wiki with backlinks and cross-references.

The LLM reads papers, extracts concepts, writes encyclopedia-style articles, and maintains an index. The knowledge is compiled once and kept current, so the LLM never re-derives context from scratch at query time.

This works because research is mostly about concepts and their relationships, which are relatively stable.

But this pattern breaks when you apply it to actual work, where context evolves across conversations constantly.

A compiled wiki would have a page about a project, but it wouldn’t track that a deadline agreed in one email thread and moved to a later date in another thread, while the team still assumed the original date.

A wiki doesn’t track ground truth effectively.

We wrote about this recently, and Karpathy liked it:

Tracking this requires a different data structure altogether. Not a wiki of summaries, but a knowledge graph of typed entities where people, decisions, commitments, and deadlines are separate nodes linked across conversations.

Rowboat (GitHub Repo) is an open-source implementation of exactly this, built on the same Markdown-and-Obsidian foundation that Karpathy uses, but extended into a work context.

The way it works is that it ingests conversations from Gmail, Granola, and Fireflies, and instead of writing a summary page per topic, it extracts each decision, commitment, and deadline as its own Markdown file with backlinks to the people and projects involved.

That’s structurally different from a wiki because:

• A wiki page about “Project X” gives you a summary of what was discussed.

• But a knowledge graph gives you every decision made, who made it, what was promised, when it was promised, and whether anything has shifted since.

Next, let’s set up Rowboat from scratch, walk through what the knowledge graph looks like on disk, and see what happens once the graph is live.

Setup

Rowboat is a local desktop app (Mac, Windows, Linux) that runs entirely on your machine and lets you bring your own model from Ollama, LM Studio, or any hosted API.

It stores everything in ~/.rowboat/ as plain Markdown files in an Obsidian-compatible vault. If you already use Obsidian for notes, you can point it at the same vault and browse Rowboat’s knowledge graph alongside your own files.

If you don’t use Obsidian, the Markdown files are still readable in any editor.

To start, download the app here: rowboatlabs.com/downloads

Next, open ~/.rowboat/config/models.json and point it at whatever you’re running.

For a hosted provider, you can specify:

{
  "provider": {
    "flavor": "openai",
    "apiKey": "sk-..."
  },
  "model": "gpt-4o"
}

Anthropic, Google, and OpenRouter use the same structure, so you can just swap the flavor property above. If you want inference fully on your machine, Ollama works too:

{
  "provider": {
    "flavor": "ollama",
    "baseURL": "http://localhost:11434"
  },
  "model": "llama3.2"
}

Lastly, you can also set the preferred LLM providers using their respective API Keys from the UI itself. Visit Settings → Models:

Once a model is connected, Rowboat wraps it with system prompts that make the LLM aware of your knowledge graph structure.

The model knows about the knowledge/ directory, understands the entity types (People, Projects, Organizations, Topics), can traverse backlinks between notes, and can read Today.md for current context before responding.

It’s effectively a work-oriented system prompt layer on top of whatever model you chose, so the LLM operates as a context-aware copilot rather than a blank chat session.

Moving on, you’ll need your own Google Cloud project with OAuth credentials so that you can invoke Gmail, Calendar, and Drive directly from your machine using those credentials.

The full walkthrough is in google-setup MD file in the Rowboat GitHub repo.

The core flow is this:

• Create a project at console. cloud. google. com

• Enable Gmail, Calendar, and Drive APIs

• Set up an OAuth consent screen (Testing mode works)

• Add your email as a test user

• Create an OAuth Client ID (type: Web application)

• Set redirect URI to http://localhost:8080/oauth/callback

• Paste your Client ID and Secret into Rowboat when prompted

Once connected, the first sync kicks off, and the graph starts building.

If you use Fireflies or Granola, you can add your API keys here:

// ~/.rowboat/config/fireflies.json  # Fireflies meeting transcripts

{
  "apiKey": "<your-fireflies-api-key>"
}


// ~/.rowboat/config/granola.json  # Granola meeting notes

{
  "apiKey": "<your-granola-api-key>"
}

The transcripts get pulled into the Meetings/ folder automatically, with decisions and action items extracted into the knowledge graph.

Additionally, you can add other integrations if needed, based on what fits your workflow.

~/.rowboat/config/deepgram.json      # voice input and notes
~/.rowboat/config/elevenlabs.json    # voice output
~/.rowboat/config/exa-search.json    # web research via Exa
~/.rowboat/config/composio.json      # external tools
~/.rowboat/config/mcp.json           # other MCP tools

Once the first sync completes, the graph is ready to query.

The vault structure on disk

After the first sync, your ~/.rowboat/ directory is live.

• The config/ folder has all your API keys and model config

• The gmail_sync/ and calendar_sync/ folder has synced data before it gets processed into the graph

• The events/ folder contains the background agent activity, split into done/ and pending/ folders.

• Anything in the sites/ folder gets served at http://localhost:3210/sites/<slug>/ and can be embedded as a live iframe in any note.

• The logs/ and runs/ folders contain the operation history and agent run records.

The more interesting part of the structure is inside the knowledge/ folder:

This is what it holds:

• Agent Notes/ is Rowboat’s memory layer, written from your behavior and writing patterns over time.

• Meetings/ contains processed transcripts from Fireflies or Granola, with decisions and action items extracted.

• Notes/ is for your own Markdown files.

• Today.md is the file Rowboat reads first before answering anything. It aggregates recent emails, meeting notes, and drafts into a single entry point for every query.

In addition to this, there are also People/, Organizations/, Projects/, and Topics/ folders.

They all start empty and populate as the signal accumulates. The Rowboat Agent only creates an entity file once there’s enough evidence across emails, meetings, and decisions to justify it.

This is a deliberate design choice because premature entity creation from noisy data (spam senders, marketing emails) would pollute the graph.

Rowboat doesn’t load the full vault on every query. It reads Today.md first, then pulls only the entity files relevant to your request. The graph can grow to hundreds of notes while the query cost stays flat.

Querying the graph

Once sync completes, here’s a useful first query you can ask:

Prep me for my 2pm meeting with Sarah Chen

This depicts the result:

Under the hood, the Rowboat Agent traversed Sarah’s entity node and its backlinks.

• Her role and firm came from email signatures and meeting attendees.

• Your interaction history was reconstructed from calendar events and threads.

• Open items were extracted from past meeting transcripts.

• And talking points come from related entities in the graph.

Any [[entity]] reference in the brief is a live backlink that you can click through:

This will show the full entity node, so additional context for any account, person, or project becomes visible in just one hop:

Here’s another query:

Create a voice summary of my day ahead

This depicts the result:

In this case, the Agent pulled today’s meetings from your calendar, gathered context on attendees from the graph, generated a script grounded in your priorities, and handed the audio back as an MP3.

This requires the ElevenLabs or Deepgram integration discussed above for the audio part specifically.

For ongoing tracking, tagging @ rowboat on any note turns it into a live note. Rowboat keeps it updated automatically as new relevant information comes in from connected sources.

For instance, if you’re tracking a deal, a project, or an account across multiple conversations, the note stays current without manual upkeep.

Finally, you can also spin up agents that run on a schedule independently of your queries. You control what runs, when it runs, and what gets written back to the vault.

Knowledge graph evolution over time

As expected, the knowledge graph gets denser with every email and meeting. New signals attach to existing entity nodes rather than creating isolated files.

Imagine that Sarah was in the Series B intro call 3 weeks back. That link already exists from day one.

But when she emails term sheet feedback now, that update attaches to the same node as a new line in her activity log.

The graph doesn’t create a second “Sarah Chen” file. It extends the existing one.

At month 2, when you query “what has Sarah committed to this quarter,” the Agent traverses connected context across every interaction, not individual threads sitting in separate inboxes.

If you set a live note set on a deal in week one, it stays current because the Agent keeps pulling new relevant context into it automatically.

So you set it once, and the graph maintains it.

This is the structural difference from Karpathy’s wiki pattern. A wiki compiles concepts into pages. A knowledge graph tracks state across conversations, and that state compounds as new interactions link back to existing entities

Rowboat takes the compounding knowledge base Karpathy’s pattern describes and makes it work for the context that actually changes day to day.

It stores everything as plain Markdown in ~/.rowboat/ so nothing is locked into a proprietary format.

Model calls go to wherever you pointed models. json.

• With Ollama, the entire pipeline runs on your hardware.

• With a hosted API key, your prompts go directly to that provider under your own account’s terms.

Rowboat doesn’t proxy or intercept model calls.

OAuth credentials for Gmail, Calendar, and Drive live in ~/.rowboat/config/ and call Google’s API directly from your machine. Every integration follows the same architecture, i.e., they are invoked locally and outputs are written to the local vault.

The repo is Apache-2.0 licensed.

GitHub repo here (13k+ stars): github.com/rowboatlabs/rowboat (don’t forget to star it 🌟)

Thanks for reading!

Most agent workflows die the moment the process does.

Serverless cold starts interrupt long-running tasks. Cloud VMs require networking setup, HTTPS configuration, and infra scaffolding before anything actually runs.

exe.dev takes a different approach. It gives developers persistent virtual machines, accessible over HTTPS immediately, with no dashboard or configuration required.

Launch a machine via SSH, and it stays alive, disk included. For agent workflows specifically, the fit is direct: agents need a stable execution environment, an addressable endpoint, and persistence across runs. exe.dev bundles all three.

You can also share a running service as easily as sharing a Google Doc, with optional built-in authentication.

Positioned between VPS and serverless, it removes the infrastructure overhead without abstracting away control.

Get started with exe.dev here→

Thanks to the team for partnering today!

Build a deep researcher that beats OpenAI, Gemini, and Perplexity

ChatGPT Deep Research, Claude, and Perplexity are all closed-source SaaS running in someone else’s cloud.

Every query and every connected internal document sits on their servers, not yours.

That’s fine for some use cases, but not all.

Today, let’s look at a fully open-source deep research stack that runs on your own infrastructure using three:

• Onyx for retrieval

• CrewAI for orchestration

• Voxtral for voice.

Here’s the full system running end-to-end, from voice query to narrated research report:

The rest of this article breaks down how it works and walks you through building the same stack yourself. Before any of that, though, it’s worth being clear about why this is worth building at all.

Why self-hosting actually matters

Every major AI research tool is a closed cloud service. That has real consequences:

• Your queries go to their servers. The questions you ask reveal what you’re working on.

• Your connected data gets indexed on their infrastructure. Integration is convenient, but the index lives on their side.

• Retention, logging, and audit are their call, not yours. Enterprise tiers soften this, but don’t eliminate it.

• Quotas and pricing change on their timeline. The tool you depend on today can reprice or rate-limit tomorrow.

For regulated industries, teams with IP-sensitive work, or anyone working under data residency rules, that list isn’t theoretical. It’s the reason AI-assisted research still feels out of reach for a lot of serious work.

Unless you can run the whole stack yourself, with no compromise on quality.

Why existing research tools break

Most research tools do one pass. They search, collect whatever comes back, and hand it to the LLM to write something up.

That works for shallow queries. It breaks the moment you ask something that requires synthesis across sources, contradiction detection, or reasoning across multiple hops.

Here’s what that failure looks like in practice:

• The agent finds a source and a contradicting source. It picks one and moves on. The contradiction never surfaces.

• Two sources say the same thing in different words. The report cites both as independent evidence.

• A critical connecting fact lives in a document that wasn’t retrieved, because keyword matching doesn’t understand that “cloud migration” and “moving our PostgreSQL cluster to AWS” are the same thing.

And they all share the same root cause: research isn’t one task.

What good deep research actually requires

Five things, regardless of tools:

1. Separation of stages → Gathering, analysis, and writing should be isolated. Each stage receives only the clean output of the previous one.

2. Retrieval that reasons → Keyword search is brittle. Vector similarity breaks on multi-hop questions. The retrieval layer needs parallel query variants, intelligent recombination, and an LLM filtering step before synthesis. Without that filtering, hallucinations enter.

3. Reflection in the loop → Static plans break when findings contradict expectations. The system should pivot when something unexpected surfaces, while still tracking coverage of the original plan.

4. Unified search across public and internal sources → The research layer should query the open web and internal knowledge bases in one pipeline, with permissions enforced per-document. Whether indexing runs on your infrastructure or a vendor’s determines who owns the data.

5. A voice layer → Voice input for queries is faster than typing, especially for complex or exploratory questions. Audio output for reports makes long-form results consumable without reading through pages of text.

Onyx as the open-source retrieval layer

Onyx is an open-source AI platform built around these principles. It gives any model RAG, web search, code execution, deep research, and custom agents out of the box.

Fully self-hostable, so your data never leaves your infrastructure.

Moreover, Onyx is ranked #1 (ahead of OpenAI Deep Research, Gemini 2.5 Pro, and Perplexity Deep Research) on DeepResearch Bench, an independent academic benchmark covering 100 PhD-level research tasks across 22 fields, evaluated on report quality and citation accuracy.

Here’s how the Onyx deep research translates to architecture.

Instead of running a single retrieval loop, it separates the workflow into three distinct phases:

• Phase 1: Clarification → For short or ambiguous queries, the system generates up to 5 targeted follow-up questions to narrow scope. Detailed queries skip this step automatically.

• Phase 2: Planning → The query is decomposed into up to 6 exploration directions. The planner has no tool access, so it produces a research plan, not answers. This is a deliberate constraint.

• Phase 3: Iterative execution → An orchestrator and research agents alternate for up to 8 cycles, each dispatching up to 3 agents in parallel.

Design-wise:

• The orchestrator never searches directly.

• Research agents never see the full query or plan.

This forces self-contained task briefs per agent and prevents context from leaking across stages.

Adaptive strategy

Onyx explores alternative paths from the original plan based on what it finds. Between every dispatch, a mandatory reflection step produces structured output:

• What’s covered

• What gaps remain

• What new directions emerged

• Whether more cycles will yield new info

This runs every time, so the result behaves like a researcher, not a retrieval engine.

The 6-stage retrieval pipeline

Each agent runs through this full pipeline before the LLM synthesizes anything:

1. Query generation → The original query is expanded into parallel variants: semantic rephrasings, keyword alternatives, and broader searches. Multi-part questions are split automatically.

2. Search and recombination → Results are retrieved from a hybrid index (vector + BM25), combined via Reciprocal Rank Fusion, and adjacent chunks are merged.

3. LLM selection → The LLM reviews all retrieved chunks and keeps only the relevant ones. Skipping this step is where hallucinations typically enter the pipeline.

4. Context expansion → For each selected document, the LLM reads surrounding chunks to determine how much additional context is needed. This runs in parallel across documents.

5. Prompt building → Selected sections are assembled along with citations and chat history into a single prompt.

6. Answer synthesis → The LLM generates a grounded answer with inline citations linking back to source documents.

Onyx connects to 40+ enterprise data sources like Slack, Confluence, Jira, GitHub, Salesforce, Google Drive, SharePoint, Notion, Zendesk, HubSpot, Gong, and more.

The difference here from proprietary tools isn’t whether it can connect. It’s where the indexing happens. Onyx pre-indexes everything continuously on your own infrastructure, syncing content, metadata, and permissions in near real time.

This provides:

• One query spans the open web and every internal source at once.

• Users only see results from documents they’re authorised to view.

• Permissions sync automatically from each source.

• No internal data leaves your network to be indexed or stored by a vendor.

Finally, regarding citations:

• Agents cite inline as they write intermediate reports.

• Citations from parallel agents get merged and renumbered into one unified set.

• Every final claim traces back to a specific source document.

CrewAI: the orchestration layer

Onyx handles retrieval. CrewAI handles coordination.

The default pattern most developers reach for is one agent with three sequential tasks sharing a growing context window:

• The writer starts before the analyst finishes.

• Raw search noise bleeds into the final report.

• Source material gets reinterpreted twice before output.

CrewAI solves this with three primitives:

• Flows wire independent Crews together, each receiving only clean output from the stage before. No accumulated context.

• Skills inject domain-specific instructions into an agent’s prompt at runtime via SKILL.md. Instruction at the point of action.

• MCP Integration attaches MCP servers directly to an agent via the mcps field. No adapter, no context manager.

Onyx connects in one declaration:

from crewai import Agent

researcher_agent = Agent(
    role="Senior Research Analyst",
    goal="Gather information on research query with source URLs",
    backstory="You are a disciplined analyst. Record every source URL.",
    mcps=[
        f"{ONYX_MCP_URL}?token={ONYX_TOKEN}" # also runs locally without a token
    ]
) 

The Researcher agent gets three tools instantly without any tool wiring:

• Search the knowledge base

• Search the web

• Fetch full page content from any URL

Voxtral: the voice layer

Research workflows typically bottleneck at the keyboard, especially for exploratory queries that are easier to articulate verbally than to type out.

Voxtral is Mistral’s native audio model family, designed for both speech understanding and generation within the same architecture.

Transcription handles accents, background noise, and domain-specific vocabulary well. Generation produces natural-sounding speech rather than robotic TTS output.

This adds two things to the research pipeline:

• Voice input → A spoken question is transcribed and fed directly into the research pipeline, removing the typing step entirely.

• Report narration → The full Markdown report is read back as expressive audio. For long reports, listening is more practical than reading through pages on screen.

Putting it together

The full flow works like this:

• A research query comes in as typed text, spoken audio, or an uploaded PDF.

• The Researcher Agent searches the web and internal documents via Onyx MCP.

• The Analyst Agent deduplicates findings, flags contradictions, and groups them into themes.

• The Report Writer Agent produces a structured Markdown report with citations.

• The report can be narrated back via Voxtral TTS.

The natural first design is a single Crew with three sequential tasks. This doesn’t work well in practice.

Shared context across stages degrades ground truth. Onyx refers to this as “deep frying” where facts get reinterpreted, contradictions get smoothed over, and source material becomes unrecognizable by the time the Writer sees it.

This system uses a CrewAI Flow instead, with three separate Crews where each one receives only the clean output from the previous stage.

• Researcher Agent → Connects to Onyx via CrewAI’s MCP integration. It runs web searches, reads full URLs, and searches uploaded PDFs. Every finding carries a citation back to its source.

• Analyst Agent → Takes the raw findings and deduplicates overlapping facts, merges sources that say the same thing, flags explicit contradictions, and groups everything into coherent themes. The output is a structured summary, not a pile of search results.

• Report Writer Agent → Turns the analyst’s summary into a citation-backed Markdown report. It’s equipped with a CrewAI Skill (SKILL.md) that gets injected at generation time to enforce consistent structure.

deep-research-report/
├── SKILL.md       # Formatting rules, evidence standards, structure
├── scripts/       # Optional
└── references/    # Optional

SKILL.md uses YAML front matter and a Markdown body:

---
name: deep-research-report
description: >
  Guidelines for writing high-quality, publication-ready deep research reports.
  Covers structure, tone, evidence standards, and formatting rules.
metadata:
  author: deep-research-agent
  version: "1.0"
---

Instructions for the agent go here.
This markdown is injected into the agent's prompt when the skill is activated.

Here's a successful execution of the full flow:

You can find all the code for the project in this Lightning Studio →

Why build?

The point here isn’t that open-source has caught up to closed-source research tools.

It’s that ​Onyx runs deep research on infrastructure you can inspect, self-host, and modify. Combined with ​CrewAI’s​ enforced stage separation and ​Voxtral’s​ native speech layer, the resulting stack covers three things that closed-source alternatives fundamentally cannot offer together:

• Research quality that’s competitive with commercial tools, with full citation integrity maintained across stages.

• Complete data sovereignty, where every query and internal document stays on your infrastructure.

• Full transparency into the codebase, which you can read, audit, and extend.

If data sovereignty has been the reason your team hasn’t adopted AI-powered research, that constraint no longer has to exist.

Thanks for reading!

Until now, Anthropic’s Generative UI capabilities only existed inside its own products.

Open Generative UI by CopilotKit is an open-source implementation of that same pattern that works in any app.

The agent generates HTML/SVG at runtime, and CopilotKit streams it token-by-token into a sandboxed iframe inside the app’s chat.

So the user can watch the UI assemble itself in real time, not after the full response is ready.

The sandbox is fully isolated with no access to the parent app, the DOM, or user data. So if the agent hallucinates broken markup or unexpected JavaScript, nothing leaks outside the iframe.

Under the hood, the agent does not select from pre-built components. Instead, it generates arbitrary visuals from scratch every time.

The output is unconstrained by default, but you can shape it by defining prompt-based skills that teach the agent specific visual formats or guidelines.

For instance, a skill prompt can guide the agent toward producing a Chart.js dashboard with proper axis labels and responsive sizing, or an interactive 3D model with rotation controls.

The graphic below depicts this, and the output quality you see actually comes from the skills layer.

Open Generative UI runs on AG-UI, so it works out of the box with LangGraph, CrewAI, Mastra, Google ADK, AWS Strands, and more.

It also ships with a standalone MCP server that plugs into Claude Code, Cursor, or any MCP-compatible client.

And the entire stack is built on top of CopilotKit, the open-source frontend framework for agents and generative UI. 30k+ GitHub stars, with SDKs for React, Next.js, Angular, and Vue.

Here’s a live demo gallery if you want to test this yourself →

CopilotKit GitHub repo (30k+ stars) →

Train classical ML models on large datasets

The list of sklearn implementations that support a batch API is quite small:

This is concerning since, in the enterprise space, the data is primarily tabular.

Classical ML algorithms, such as tree-based ensemble methods, are frequently used for modeling.

However, typical implementations of these models are not “big-data-friendly” because they require the entire dataset to be in memory.

There are two ways to approach this:

• The first way is to use big-data frameworks like Spark MLlib to train them. ​​​​We covered this in detail →​

• There’s one more way: Random Patches. Let’s learn below.

Random Patches

Note: This approach will only work in an ensemble setting. So, you would have to train multiple models.

The idea is to sample random data patches (rows and columns) and train a tree model on each patch.

Repeat this step multiple times by randomly generating different data patches to obtain the entire random forest model.

These are the results mentioned in the thesis (check pages 174 and 178) on 13 datasets:

• In most cases, the random patches approach performs better than the traditional random forest.

• In other cases, there is a marginal difference in performance.

And this is how we can train a random forest model on large datasets that do not fit into memory.

Why does it work?

The idea is similar to what we discussed when we covered Bagging, which eventually allowed us to build our own variant of the Bagging algorithm: ​​​​Why Bagging is so ridiculously effective at variance reduction?​​​​

In a gist, building trees that are as different as possible guarantees a greater reduction in variance.

In this case, the dataset overlap between two trees will be less than that in a typical random forest. This aids in the Bagging objective and leads to a more robust model.

To understand this mathematically, read this: ​​​​Why Bagging is so ridiculously effective at variance reduction?

Thanks for reading!​

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) that align with such topics.

Develop "Industry ML" Skills

Here are some of them:

• Learn everything about MCPs in this ​crash course with 9 parts →​

• Learn how to build Agentic systems in a crash course with 14 parts.

• Learn how to build real-world RAG apps and evaluate and scale them in this crash course.

• Learn sophisticated graph architectures and how to train them on graph data.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.

• Learn how to run large models on small devices using ​Quantization techniques.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using ​Conformal Predictions.

• Learn how to identify causal relationships and answer business questions using causal inference in this crash course.

• Learn how to scale and implement ML model training in this practical guide.

• Learn techniques to reliably test new models in production.

• Learn how to build privacy-first ML systems using ​Federated Learning.

• Learn 6 techniques with implementation to compress ML models.

All these resources will help you cultivate key skills that businesses and companies care about the most.

​InsForge​ (open-source) solves the most frustrating bottleneck in AI-assisted development: backend configuration.

InsForge GitHub Repo

Agents can build a beautiful frontend in minutes, set up API routes, and lay out the component architecture. But the moment it needs to enable auth or configure a database, it completely falls apart.

The reason is that every backend platform today (Firebase, Supabase, AWS) was designed for humans clicking through dashboards. When agents try to interact with these platforms through MCP servers, they get fragmented context like table names without schema details or auth endpoints without security configs. So agents end up guessing, hallucinating, and generating broken code.

​​InsForge​​ fixes this at the infrastructure level rather than the tooling level. It introduces a semantic layer where every backend primitive (auth, database, storage, AI features) is exposed as structured, machine-readable capabilities with metadata, constraints, and documentation baked in.

Primitives are also aware of each other, so auth knows about database permissions and storage understands access policies.

Because agents get a complete, structured context instead of inferring what’s missing, InsForge delivers:

• Roughly 2x more accuracy than Supabase MCP

• 1.6x faster task completion

• 30% better token efficiency

To test this out, we built a full ChatGPT clone with auth, database, storage, and AI integration, built entirely with Claude Code using InsForge as the backend. No manual configuration was needed, not because of any magic, but because the agent could reason about the entire backend as one coherent system.

InsForge works with any AI coding agent, including Cursor, Claude Code, Windsurf, and Codex. You can use all the primitives together or just pick what you need, like database only or auth only.

It’s fully open-source under Apache 2.0.

​​Find GitHub repo here →​​ (don’t forget to star it ⭐️)

Practical defenses for prompt injection

OWASP ranks prompt injection as the #1 threat to LLM applications.

Most agent deployments still rely on a single defense, i.e., the system prompt instructions telling the model to ignore suspicious input.

That’s a behavioral constraint on the model, not a structural one, and an attacker only needs to bypass it once.

There are five composable defenses that move the security boundary to actual architectural enforcement. Let’s walk through each.

​To learn these defenses from full LLMOps principles with code, start here →​

We published the above LLMOps course, which covers the fundamentals of AI engineering & LLMs, Building blocks of LLMs like tokenization, embeddings, attention, architectural designs and training, decoding, generation parameters, the LLM Application Lifecycle, context engineering, prompt management, defense, control, memory, temporal context, evaluation, tool use, red teaming, Adaptive LLMs, and Serving.

Label before use

The simplest defense is to wrap untrusted text in explicit tags before it enters the prompt.

A developer prompt, for instance, would sandwich email body content inside <Untrusted> and </Untrusted> delimiters.

This gives the model a structural signal that the enclosed content is data, not instructions.

Google’s Spotlight research formalized this idea and showed that even simple delimiters reduce injection success rates significantly.

Encoding untrusted input in Base64 before passing it to the model is a more aggressive variant of the same principle.

Instruction hierarchy

Not all instructions should carry equal weight.

The instruction hierarchy defense assigns explicit trust ranks, like developer/system prompts rank highest, user messages rank medium, and third-party context (tool outputs, retrieved documents) ranks lowest.

When instructions conflict, the highest-ranked source wins.

OpenAI published a paper on this approach, training models to respect a tiered instruction priority.

The core insight is that an instruction embedded in a fetched webpage should never override a constraint set by the developer.

Principle of least privilege

If your agent only needs to read emails, don’t give them tools to send, draft, or delete them.

This is borrowed directly from OS-level security, and it’s the easiest defense to implement in practice.

Most agent frameworks let you define which tools are available per session or per task.

Restricting the tool set to the minimum required capability means that even a successful injection can’t escalate beyond the agent’s permissions.

An attacker who tricks the model into calling send_email gets nothing if that tool was never mounted.

Human in the loop

For sensitive actions (sending messages, modifying data, making purchases), require explicit user approval before execution.

The agent generates a plan, the user reviews it, and only then does the system run it.

This is a runtime defense, not a model-level one. It doesn’t prevent injection from happening, but it prevents injected instructions from causing real-world harm.

The tradeoff is latency and user friction, so most implementations apply it selectively to high-risk actions rather than every tool call.

Planner-executor separation

This is the most architecturally robust defense. Instead of one LLM that both reasons over untrusted data and calls tools, you split the work across two models.

The Planner (reasoning layer) sees untrusted data but has no tool access.

It produces a structured plan of actions. The Executor (action layer) has tools but never directly consumes untrusted input. It only executes the structured plan the Planner produced.

Google DeepMind’s CaMeL framework formalized this pattern, drawing from traditional security concepts like control flow integrity and information flow control.

CaMeL practically solved the AgentDojo security benchmark by ensuring that untrusted data can never influence the program’s control flow.

The Dual LLM pattern (in 2023) proposed the same core idea, where a privileged LLM plans and a quarantined LLM handles untrusted content, with strict isolation between them.

Combining defenses

No single defense is sufficient on its own. The strongest setups layer all five to label untrusted content, enforce instruction hierarchy, restrict tools to what’s needed, require approval for sensitive actions, and separate planning from execution when the stakes justify the complexity.

​To learn these defenses from full LLMOps principles with code, start here →​

We published the above LLMOps course, which covers the fundamentals of AI engineering & LLMs, Building blocks of LLMs like tokenization, embeddings, attention, architectural designs and training, decoding, generation parameters, the LLM Application Lifecycle, context engineering, prompt management, defense, control, memory, temporal context, evaluation, tool use, red teaming, Adaptive LLMs, and Serving.

👉 Over to you: Which of these five defenses are you using in your agent pipelines today, and which ones have you been putting off?

The main visual of this newsletter was inspired by ByteByteGo’s post on a similar topic.

Thanks for reading!

Part 2 is now available, and you can read it here →

Reinforcement Learning Nanodegree Part 2

Part 1 gave you the RL interaction loop and the exploration-exploitation tradeoff through bandits, and this one gives you the formal language that every RL algorithm is built on.

It covers:

• the Markov property and why it makes RL tractable

• the MDP as a 5-tuple (states, actions, transitions, rewards, discount factor)

• episodic vs. continuing tasks

• returns and discounting with concrete numerical examples

• the reward hypothesis and its limits (including reward hacking),

• deterministic and stochastic policies, state-value functions

• and a complete hands-on implementation of Monte Carlo policy evaluation on a 4×4 gridworld.

Everything is covered from scratch, so no RL background is required.

You can read Part 2 of the course here →

Why care?

Every frontier LLM released in the past two years uses RL in its post-training pipeline.

• ChatGPT was shaped by RLHF.

• DeepSeek-R1 used GRPO to develop reasoning capabilities.

• Claude uses constitutional AI with RL.

The pattern is consistent: pre-training gives the model knowledge, but RL is what gives it behavior.

And it is not just LLMs.

Agentic AI systems that take actions, interact with tools, and operate in multi-step environments are fundamentally RL problems. Robotics, recommendation systems, game-playing, autonomous driving, drug discovery: RL is the common thread.

Google Trends for “reinforcement learning” was nearly flat from 2004 to 2023. In the past year, it hit an all-time high.

The field is having its moment, and the demand for engineers who understand it deeply is growing fast.

This series builds that understanding from the ground up, concept by concept, with math where it matters and hands-on code you can run. No prior RL background needed.

This series is structured the same way as our MLOps/LLMOps course: concept by concept, with clear explanations, diagrams, math where it matters, and hands-on implementations you can run.

And no prior RL background is needed.

If you haven’t read Part 1, start there first. It covers the agent-environment loop, exploration vs. exploitation, and multi-armed bandits.

Reinforcement learning nanodegree part 1

Over to you: What topics would you like us to cover in this RL series?

Thanks for reading!

You’re fine-tuning a model for Python code generation. The data was generated using the strongest LLMs like Opus/GPT.

But the fine-tuned model performs better when you use a weaker teacher instead.

Why did this happen?

A stronger teacher model can produce worse fine-tuning results. This sounds counterintuitive, but it is a well-documented effect in knowledge distillation research.

Large models solve a basic problem using abstractions, type hints, and patterns.

A Qwen3-8B model does not have enough capacity to reproduce those patterns. So instead of learning clean solutions, it learns an approximation of something it cannot fully represent.

However, a weaker teacher solves the same problem correctly, but with simpler patterns that the student can actually replicate.

A recent paper from Fastino Labs also documented this.

The researchers used Pioneer, their fine-tuning agent that takes a task description, generates training data, selects a base model, runs experiments, and iterates until the model hits a performance target, all without human intervention.

During one of those runs, Pioneer fine-tuned Qwen3-8B on Python code generation.

The agent tried two different teacher models for synthetic data generation: one large frontier model and one smaller model.

• The frontier model’s data hurt performance.

• The smaller model’s data performed much better in fewer iterations.

And the fine-tuning Agent was smart enough to catch this behavior. It measured the results from both teachers, saw that the frontier model was making things worse, and dropped it.

A human engineer would likely have defaulted to a bigger model because it is the stronger model, and might not have questioned that choice.

The paper explains three reasons this happens:

→ Capacity mismatch: The student cannot learn the teacher’s internal representations when the gap is too large. Increasing teacher size first helps, then hurts beyond a certain point.

→ Forgetting pretrained knowledge: Qwen3-8B already knows how to write Python from pretraining. Fine-tuning on a complex coding style from a much larger model can overwrite that existing capability.

→ Over-complexity in training data: A large model will solve “reverse a linked list” with elegant abstractions and comprehensive error handling. That is correct code, but it is also unnecessary complexity for the task. A simpler teacher generates solutions that match the task’s actual complexity, and the student learns them cleanly.

As a takeaway, always match the teacher to the student’s capacity and the task’s complexity.

To fine-tune a 3B or 8B model on a well-defined task, a mid-tier teacher will often produce better training data than powerful one.

You can find the paper here →

How to beat GRPO without touching model weights

GRPO needs tens of thousands of rollouts to converge. Each rollout produces a 5,000-token trace full of reasoning steps, tool calls, and self-corrections, but GRPO reduces all of it to a single scalar reward.

So we end up backpropagating on one bit per trajectory while throwing away thousands of bits of structured signal.

GEPA takes a different approach.

Instead of computing policy gradients on that scalar, it hands the full rollout trace to a reflection LLM and asks “what went wrong, and how should the prompt change?”

The reflection model writes a new prompt, you test it, and if it improves, you keep it.

The paper came out in July 2025. It was accepted at ICLR 2026, DSPy made it a first-class optimizer, and Hugging Face and OpenAI both shipped cookbooks around it.

On compound AI systems (multi-module pipelines with separate prompts), GEPA matches or beats GRPO while spending 10-50x less compute and requiring no training infrastructure at all.

Let’s break down why it works, how it compares to GRPO, and how to use it in DSPy.

We started a course series on RL recently. Read Part 1 here →

This first chapter covers:

• what makes RL fundamentally different from supervised and unsupervised learning

• the agent-environment interaction loop

• the exploration-exploitation tradeoff

• multi-armed bandits as the simplest RL setting, four action-selection strategies (greedy, ε-greedy, optimistic initialization, UCB)

• and a complete hands-on implementation of the classic 10-armed testbed with results and analysis.

The signal compression problem in RL

Reinforcement learning on language models has a signal problem that most practitioners overlook. Every rollout an agent produces is a 5,000-token document, containing:

• Reasoning steps

• Tool calls

• Self-corrections

• Compiler errors

• Judge rationales

That trace is rich and structured, containing exactly the kind of diagnostic information you’d want to learn from.

While training the agent, GRPO takes all of that and reduces it to a single number.

And it throws away thousands of bits of structured info, which partly explains why it needs tens of thousands of rollouts to converge.

The signal isn’t sparse, but the final reward makes it sparse.

Letting the signal read itself

GEPA’s core idea is that the rollout is already a natural language artifact, so let an LLM read it. Don’t reduce the trace to a number.

Hand it to a reflection model along with the failure mode, and ask: “What went wrong here, and how should the prompt change?”

The reflection model writes a new prompt. You test it. And if it improves, you keep it.

That’s the full optimization loop. Everything else in the paper is engineering that makes it work at scale.

What GEPA actually optimizes

GEPA targets compound AI systems.

A pipeline of LLM modules with their own prompts, glued together by Python control flow. For instance, a multi-hop QA agent might have:

• A first-hop query writer

• A retriever

• A summarizer

• A second-hop query writer

• A final answerer

Each module has a prompt. GEPA evolves all of them.

The optimization target is simple: maximize expected metric on your task, subject to a rollout budget. The novelty is in how you spend that budget.

The feedback function

GEPA replaces your scalar metric with a feedback function μ_f.

It includes the same score that GRPO gives plus a natural language description of what happened.

• For multi-hop QA, it returns which gold docs you retrieved and which you still need.

• For instruction-following, it returns per-constraint pass/fail descriptions.

• For code generation, it returns the actual compiler errors and profiler traces.

• For privacy-preserving rewriting, it splits the score into quality and PII-leakage with breakdowns.

The 6-step algorithm

Each iteration of the main loop does this:

1. Pick a candidate prompt set from the population (Pareto sampling, more on this below)

2. Pick a module to mutate (round-robin across modules)

3. Sample 3 examples from the training set

4. Run rollouts and collect full traces plus feedback from μ_f

5. Reflect: feed traces and feedback to a reflection LLM, get a new prompt

6. Accept or reject: rerun on the same 3 examples. If better, keep it. If not, discard.

Repeat until the budget runs out and return the best candidate. The entire loop runs without gradients, PPO, or KL penalties.

GEPA vs GRPO, head to head

A quick look at GRPO first:

Both GRPO and GEPA take feedback and improve the system. That’s where the similarity ends.

GRPO updates model weights with policy gradients on scalar rewards. GEPA updates prompts with natural language reflection on full traces.

Here’s the side-by-side comparison:

One important caveat. GRPO can change what your model knows. GEPA can only change how you ask it.

If your base model can’t do the task at all, no prompt evolution will save you. Fine-tune when you need new capabilities. Use GEPA when you need to extract more from what’s already there.

A real example

Let’s make this concrete with a real example from the paper.

The task: HotpotQA is a multi-hop question answering benchmark. You get a question that needs information from two different Wikipedia articles to answer. You can’t find the answer in just one place.

Example question: “What is the population of the region containing the parish of São Vicente?”

To answer this, your agent has to:

1. 1st hop → Search “São Vicente parish”, retrieve a doc, learn it’s in Madeira.

2. 2nd hop → Search “Madeira population”, retrieve that doc, get the number.

3. Answer: Combine both

The agent has separate modules for each hop. We’re going to look at the prompt for the second-hop query writer, the module that decides what to search for after the first retrieval.

Inputs the module receives:

• question: the original user question

• summary_1: a summary of what the first hop retrieved

Output it produces:

• query: the search query for the second hop

Here’s the seed prompt that DSPy gives you by default:

“Given the fields question, summary_1, produce the fields query.”

This is generic since it just describes the schema and it scores around 38% on validation.

GEPA runs this on a few examples and watches what happens. For instance, the query writer might keep doing the same thing wrong, like it paraphrases the original question and retrieves the same documents it already had.

For our São Vicente example, given a summary about the parish, it would search “São Vicente parish population” again, retrieve nothing new and fail.

The reflection LLM sees this failure pattern across multiple examples in the trace. It writes a new prompt:

“Generate a search query optimized for the second hop of multi-hop retrieval. The first-hop query was the original question, so first-hop docs already cover the entities mentioned directly. Your goal: retrieve documents NOT found in the first hop but necessary to answer completely. Avoid paraphrasing the original question. Target connected or higher-level entities mentioned in summary_1 but not explicitly in the question. Example: if summary_1 describes a parish but the question asks about the wider region’s total population, your query should target the region, not the parish. So for a question about São Vicente’s region, query ‘Madeira archipelago population’ rather than ‘São Vicente population’.”

That rewritten prompt scores 69%, up from 38% on the seed.

The model and task stayed identical. The only thing that changed was the prompt for one module out of several.

The reflection LLM didn’t just rephrase the seed but it also absorbed an actual strategy from observed failures:

• Don’t paraphrase the question

• The first hop already covered the directly-mentioned entities

• Target the broader entity that connects them

• Here’s a worked example of the pattern

That’s the kind of information you cannot encode in a policy gradient. RL would just tell you “this trajectory was 0.3 below the group mean” and let backprop figure it out across thousands of tokens. GEPA writes the lesson down in plain English and ships it as the new prompt.

Pareto selection

The idea of evolving/optimizing the prompt is not new.

But most approaches often mutate from the best candidate so far, which sounds reasonable, but it collapses to local optima fast.

GEPA uses something smarter, borrowed from quality-diversity optimization.

Imagine three candidate prompts and four tasks:

A greedy approach will pick C every time since it has the best average.

But A is the only one who handles Task 1 well, and B does Task 2 well. If you only mutate C, you lose those strategies forever.

Pareto selection keeps anyone who’s best at at least one task. Then it samples parents weighted by how many tasks they win. So C is most likely to be picked, but A and B stay in the pool. Their distinctive strengths can later be combined with C’s.

This single design choice is what separates GEPA from earlier evolutionary prompt methods.

Where GEPA fits in the landscape

Quick map of who does what:

• APE, OPRO: Both use an LLM to propose prompt candidates scored by a scalar metric. APE generates candidates from input-output demos and picks the best. OPRO feeds prior prompts and their scores into a meta-prompt so the LLM can propose informed improvements. Single prompt, no reflection on traces.

• EvoPrompt, Promptbreeder: Evolutionary operators (crossover, mutation) applied to prompt populations via LLM calls. Promptbreeder adds a self-referential layer: it also evolves the mutation-prompts themselves. Both use scalar fitness for selection and target single prompts, not multi-module pipelines.

• Reflexion: Agents reflect on task feedback after each trial and store reflections in an episodic memory buffer for the next attempt. Improves per-instance behavior across retries, not population-level prompt evolution across a training set.

• TextGrad: PyTorch-style backpropagation but with natural language critiques instead of numerical gradients. An LLM propagates textual feedback through a computation graph to produce per-variable improvement suggestions. Single candidate per iteration, no population.

• MIPROv2: DSPy’s prior flagship. Bootstraps few-shot examples from training data, proposes instruction candidates grounded in data summaries and traces, then uses Bayesian Optimization (Optuna TPE) to search the joint instruction-demo space across all modules. Generates all candidates upfront rather than evolving them through reflection.

• GRPO: Actual RL with weight updates. Samples a group of rollouts per prompt, uses the group mean as baseline to compute per-trajectory advantage, then updates weights via policy gradients with a KL penalty. The only method here that changes what the model knows, not just how you prompt it.

GEPA borrows ideas from several of these methods. It uses verbal reflection like Reflexion, but applies it across a population of candidates instead of a single agent’s memory.

It evolves that population using selection pressure like EvoPrompt, but with natural language feedback driving mutations instead of scalar fitness.

It targets compound multi-module pipelines like MIPROv2, but evolves prompts iteratively through reflection instead of generating all candidates upfront.

The piece that’s new to GEPA is Pareto selection, which preserves candidates that are best at even one task rather than always mutating from the highest-average performer.

Using it in DSPy

The API is one line different from MIPROv2:

optimizer = dspy.GEPA(
    metric=metric_with_feedback,
    auto="medium",
    reflection_minibatch_size=3,
    candidate_selection_strategy="pareto",
    reflection_lm=dspy.LM("gpt-5", temperature=1.0, max_tokens=32000),
    use_merge=True,
    track_stats=True,
)

optimized = optimizer.compile(program, trainset=train, valset=val)

The catch: your metric function needs the right signature.

def metric(gold, pred, trace=None, pred_name=None, pred_trace=None):
    # return dspy.Prediction(score=float, feedback=str)

Return a prediction with both a score and a feedback string. The feedback is what gets fed to the reflection LLM, so make it diagnostic and specific.

If your feedback string is just “wrong answer”, you’re back to scalar territory, and GEPA degrades to a slower MIPROv2.

If your feedback says “missed entity X, retrieved doc Y when gold was Z, format violation in step 3”, then GEPA works well with that level of detail.

A 2026 reality check

GEPA beats GRPO specifically, not every RL method.

The field has stopped framing this as GEPA vs RL and started framing it as GEPA and RL. The paper itself points to hybrid recipes as the natural next step.

Reflection is far more sample-efficient than RL on compound systems. The two are increasingly combined, not pitted against each other.

One more nuance worth knowing is that Decagon’s March 2026 production ablations found that more data isn’t always better with GEPA. 20 to 100 examples often beats 500. The reflection loop overfits when you feed it too much.

GEPA learns from patterns in failures. With 50 well-chosen examples, the reflector sees a clean signal. With 500, it starts chasing noise.

Use small, high-quality training sets and don’t assume scale helps.

When to reach for what

If you’re building a compound AI system today, here’s the decision tree.

• Use GEPA when: You have a small training set, expensive rollouts, no access to weights, and a metric you can describe in words.

• Use GRPO when: You have abundant cheap rollouts, open weights, and a verifiable terminal reward.

• Use MIPROv2 when: You specifically need bootstrapped few-shot exemplars in your prompts.

• Use TextGrad when: Your computation graph is deep and you want explicit per-variable critique propagation.

For most practical compound-system work in 2026, GEPA is the default to try first.

RL still has its place, but it’s no longer the obvious default when reading a rollout costs less than running ten thousand more.

To dive deeper into RL, we started a course series on RL recently. Read Part 1 here →

The first chapter covers:

• what makes RL fundamentally different from supervised and unsupervised learning

• the agent-environment interaction loop

• the exploration-exploitation tradeoff

• multi-armed bandits as the simplest RL setting, four action-selection strategies (greedy, ε-greedy, optimistic initialization, UCB)

• and a complete hands-on implementation of the classic 10-armed testbed with results and analysis.

Thanks for reading!

Claude Code has two context gaps that no amount of CLAUDE.md optimization will fix.

The first is web scraping.

• web_fetch doesn’t return raw page content. It runs the page through a smaller model and returns a summary with a 125-character quote limit, so you can’t use it to extract full tutorials, product specs, or thread content.

• curl returns raw HTML but gets blocked by sites with anti-bot protection (Amazon, LinkedIn, most e-commerce), can’t render JavaScript SPAs, and fails at scale due to rate limiting. Both truncate at around 100KB.

The second is backend integration.

• When Claude Code talks to a backend like Supabase through MCP, it discovers state through multiple separate calls (list_tables, execute_sql, list_extensions), each returning a partial view.

• Auth provider config isn’t queryable at all. And when something fails, error messages don’t distinguish between platform-level and code-level rejections, so the agent enters retry loops that burn tokens with every attempt. In our recent test, a single RAG app built on Supabase consumed 10.4M tokens and needed 10 manual fixes.

Bright Data solves the first problem. InsForge solves the second.

Today, let’s look at how to set up both as skills in Claude Code, with an interesting way we tend to use them.

Bright Data setup

Bright Data skill (open-source) adds scraping infrastructure that handles everything web_fetch and curl can’t.

The agent gets a four-tier fallback that escalates based on what the target site requires, like native fetch, curl, browser automation, and a proxy network with residential IPs and automatic CAPTCHA solving.

For agent workflows, the more useful capability is structured data extraction.

Instead of raw HTML that the agent has to parse, Bright Data provides pre-built extractors for 40+ platforms (Amazon, LinkedIn, Instagram, TikTok, YouTube, Reddit) that return clean JSON with specific fields like product prices, review scores, profile data, and post content.

npx skills add brightdata/skills

This installs several skills covering scraping, search, structured data feeds, MCP orchestration, SDK best practices, and the bdata CLI.

InsForge setup

We covered the backend problem in depth in a recent issue. The same RAG app that consumed 10.4M tokens on Supabase consumed 3.7M on InsForge with zero errors.

InsForge (open-source, Apache 2.0) acts as the backend context engineering layer for agents using Skills and CLI.

• Install all four Skills (primary documentation and diagnostic layer):

npx skills add insforge/insforge-skills

This installs insforge (SDK patterns), insforge-cli (infrastructure commands), insforge-debug (failure diagnostics), and insforge-integrations (third-party auth providers). Total metadata cost: ~714 tokens at session start.

• Link the CLI to your project (primary execution layer):

npx @insforge/cli link --project-id <project-id>

Building a Google Doc clone

This 10 hr video teaches how to build a Google Doc clone:

Ten hours of video means hundreds of implementation details like real-time collaboration, document state syncing, editor toolbar structure, and permissions.

That’s a lot of context that Claude Code will likely struggle to scrape. Even if it manages to scrape, compression will remove many of those details.

With both skills installed, here’s what you can do:

I want to build what's shown here:
https://www.youtube. com/watch?v=gq2bbDmSokU
Use Bright Data skills to scrape it and then
InsForge as the backend to implement.
Add Google OAuth and build a clean Google-doc
like interface. On every doc, add an "Ask AI"
button that chats with GPT-4o about the content.
Use InsForge's model gateway for the LLM capabilities.

The video below depicts the whole build with the final app, built in one shot:

• Bright Data scraped the full video content (transcript, metadata, structured descriptions), and Claude Code used it as the build spec.

• InsForge handled the backend in one shot: Google OAuth, database schema with RLS, storage, edge functions, and the model gateway for GPT-4o chat through InsForge’s built-in functionalities.

Finally, it gave a working Google Docs clone with real-time editing, Google OAuth, and AI-powered document chat, built from a single prompt with zero errors.

The YouTube example is the simple case for the scraping side and this isn't limited to tutorials or building from scratch.

The same workflow applies to any technical content on the web, like:

• A Reddit thread where someone describes how they optimized their real-time sync.

• A Hacker News discussion walking through an auth architecture.

• A competitor's product page with a feature worth replicating.

Give the link to Claude Code with the Bright Data skill, and the agent scrapes the content, understands what was described, and implements it in the existing app.

So any technical content on the web can become a build spec.

For basic sources, native scraping tools work. Bright Data is essential with sources that actively resist scraping, like Reddit threads with aggressive rate limiting, Amazon product pages behind anti-bot detection, LinkedIn profiles that fingerprint your browser, and JavaScript SPAs that need full browser rendering to even load content.

• You can find the Bright Data Skills repo here →

• You can find Insforge GitHub repo here →

👉 Over to you: what skills live in your default Claude Code setup?

Naive RAG vs Blockify

There’s a new RAG approach that:

• cuts corpus size by 40x.

• reduces tokens per query by 3x.

• improves vector search relevance by 2.3x.

And it delivered 260% accuracy improvement on the medical RAG benchmark over standard RAG.

The diagram compares it with naive RAG:

Chunks in a standard RAG pipeline typically carry no info about version, clearance level, or source authority.

The embedding model encodes it the same way regardless of whether the chunk is an outdated draft or the latest approved version.

During retrieval, if an outdated chunk and a latest chunk get retrieved as context, the LLM has no signal to prefer one over the other.

So it combines both and hallucinates.

The issue is not retrieval but rather the representation. The unit itself is wrong, and the fix has to happen before retrieval, at the data layer.

Blockify is an open-source data preprocessing engine that solves this at the data layer.

The engine sits between the document parser and the vector store.

Here’s how it works:

• First, a context-aware splitter finds natural breaks (paragraph boundaries, section breaks, topic shifts).

• Instead of embedding raw segments directly, a purpose-built LLM processes each one and extracts structured knowledge units called IdeaBlocks (typically 2-3 sentences). Each unit isolates a different fact or concept.

• Each unit is paired with a contextualized question and answer. This mirrors how users query the system and ensures the query embedding sits closer to real queries in the vector space (HyDE does something similar).

• Each block also carries metadata info like entity name, entity type, version, and clearance level. This helps rank retrieval by recency and authority, not just similarity.

The pipeline runs in two stages.

• The Ingest model converts raw text into IdeaBlocks as described above.

• The Distill model then clusters semantically similar blocks across the full set and merges duplicates into one canonical unit before indexing.

The retrieved units now answer a specific question instead of returning a paragraph that might contain the answer somewhere in the middle.

On the published benchmarks:

• The pipeline reduces a corpus to roughly 2.5% of its original size while preserving 99% factual integrity.

• Token consumption per query drops by 3x, from 1.5k tokens (naive top-5 chunks) to 500 tokens (top-5 IdeaBlocks).

• Vector search relevance improves 2.3x, measured by cosine distance.

In medical evaluation, the same pipeline delivered up to 650% accuracy improvement on clinical-grade RAG with a quantized Llama 3.2 3B model running on-device.

Blockify composes with LangChain and LlamaIndex. You can swap out the chunking stage (NodeParser/TextSplitter) and produce IdeaBlock nodes that the rest of the pipeline consumes normally.

For storage, you can integrate it directly with most vector DBs like Milvus, Elastic, etc.

There is also a Claude Code skill in the repo that runs the full Ingest and Distill pipeline while referencing the project documentation.

For production workloads on Intel Xeon, an optimized build is available through OpenVINO.

Here’s the GitHub repo →

Thanks for reading!