Graph Engineering Clearly Explained!
Covered with best practices from the industry.
In today’s newsletter:
Agent memory and state are not the same thing!
Graph engineering, clearly explained.
The anatomy of diffusion LLMs.
Agent memory and state are not the same thing!
If an agent forgets something it has already learned, that’s a memory problem. If it forgets where it was in the middle of a task and starts over, that’s a state problem.
We once killed one of our agents mid-task to test something else, and it started over like the execution so far never happened.
That’s when it clicked that we’d been treating two different problems as one.
State is tied to the current run as to what task the agent is working on and what it’s already found.
None of that exists unless something writes it down.
The fix was to add a checkpoint after every completed step that records the agent’s progress, so if the process dies, it resumes from that exact point instead of starting from scratch.
Memory is a different thing entirely. It’s what survives across runs as facts, lessons, and findings that are worth retaining.
At first, we had one shared memory for all our agents and assumed that was enough. But it wasn’t until our agents started reading each other’s findings and treating them as their own.
That’s why giving each agent its own memory scope is important with memory = memory.scope(”/agent”).
Once we separated state from memory, everything became much easier to reason about.
Here’s what the final Agent Harness looks like:
separate memory from state as two different problems
scope memory per agent when findings shouldn’t be shared
write a checkpoint after every completed task
resume interrupted runs from the last checkpoint
fork a checkpoint into a new branch without redoing previous work
This is the harness baseline to build any agent. It’s enough for general-purpose workflows, but coding agents and long-running systems need other layers along.
We wrote an article that builds that next layer from scratch. It walks through planning, the agent loop, subagents, sandboxing, memory, and checkpointing, one layer at a time.
The whole thing is built with CrewAI, a 100% open-source framework.
You can read the article here →
Graph engineering clearly explained
The moment you have several loops that need to work together, you have a coordination problem, and graphs are how engineers have always described coordination.
That’s the whole idea behind graph engineering that Peter Steinberger mentioned a few days back.
Today, let’s understand what exactly graph engineering is!
First, the graph itself
A graph is three things.
Nodes are units of work, whether that’s an agent, a plain model call, a deterministic function, a tool, or a human approving something.
Edges decide what runs next, either in sequence, in parallel, or conditionally based on what the last node produced.
State is a shared object that flows along the edges. Every node reads from it and writes to it.
graph.add_node("research", research_agent)
graph.add_node("write", writer_agent)
graph.add_node("review", reviewer_agent)
graph.add_edge("research", "write")
graph.add_edge("write", "review")
graph.add_conditional_edge("review",
lambda state: "done" if state.approved else "write")This is the starter graph almost every example uses. A researcher gathers material, a writer drafts, and a reviewer judges. If the review passes, the run ends. If it fails, an edge sends the draft back to the writer.
There are three nodes and four edges, and one of those edges forms a loop.
But here’s what reframes everything. A single agent loop is just a one-node graph with an edge pointing back to itself.
Graphs don’t replace loops but rather connect and govern them.
The stack kept growing
The center of gravity in AI keeps drifting away from the model, and each shift picked up a name.
Prompt engineering → The words you send.
Context engineering → Everything the model sees, not just your instructions.
Harness engineering → The code around the model that runs tools, tracks state, and handles errors.
Loop engineering → The autonomous cycle that drives one agent toward a goal.
Graph engineering → The coordination layer across many loops, covering what runs when, in what order, and who checks whom.
Each layer wraps the one before it:
A graph is made of loops, each loop needs a good harness, each harness call is a context problem, and every context contains prompts.
If you skip a lower layer, the graph just fails in a more elaborate way.
Of course, none of this is new technology.
LangGraph shipped this exact model involving nodes and edges over shared state back in January 2024.
Microsoft’s AutoGen has GraphFlow, and Google built ADK 2.0’s entire workflow runtime on the same idea.
So while the name is new, the practice isn’t. In fact, the discipline isn’t inventing graphs but rather knowing when to use one, and how to keep it from rotting.
That part has four hard problems.
Hard part 1: knowing when a node deserves to exist
The most common failure is turning “summarize this PDF” into a five-node graph with a fetcher, a chunker, a summarizer, a reviewer, and a formatter.
A node earns its place only if it represents a real specialty, meaning a different model, a different toolset, or a genuinely separate role like a read-only reviewer. Steps you could inline into an existing loop are not nodes.
A useful filter is that if you can’t draw the graph on a napkin, it’s too complex. And if collapsing two nodes into one loses nothing, they were never two nodes.
Hard part 2: keeping shared state clean
In a loop, the failure mode is context rot.
In a graph, the same problem moves into a shared state.
Every node writes to the state object, so an uninformed write in node two will become a confident input for node five.
Nobody notices until the output is wrong, and by then, the bad data has flowed through half the system.
The solutions are simple and boring, yet effective:
Give the state a typed schema.
Decide explicitly which nodes may write to which fields.
Checkpoint state between nodes so you can replay a run and see exactly where it went bad.
One caution applies to replays though.
Nodes after a checkpoint execute again, so any node with external side effects, like sending an email or creating a record, must be safe to run twice.
Hard part 3: routing you can trust
An edge is a decision, and the question is who makes it.
If a model decides the route, you get flexibility and instability together.
The same state can take different paths on different runs, which makes debugging miserable.
Google’s design rule for ADK 2.0 is the cleanest position in the discourse. Deterministic code should control predictable routing, and models should only handle the steps that need actual judgment.
Route with code wherever the condition is checkable, and spend model calls only where interpretation is genuinely required.
Hard part 4: agents agreeing with each other
Loop engineering’s sharpest rule was to never let an agent grade its own homework.
Graphs raise these stakes.
If you have 20 agents built on the same base model and all are reading the same flawed context, then they will happily agree with each other, and models measurably prefer their own outputs.
You can fix this with a reviewer node that runs on a different model, give it fresh context instead of the full conversation, and anchor its verdict to evidence the graph can’t fabricate, like tests that actually ran or code that actually compiled.
Note: Cognition landed in the same place after a year of running Devin, their coding agent. Their working setup lets several agents read the work and weigh in, but only one agent is ever allowed to change anything.
Reading is safe to do in parallel, because a bad opinion costs you nothing until someone acts on it. Writing is where the damage happens, so you keep it in one place where you can see it.
Where the graph is an overkill
Most of the time.
Anthropic’s own numbers suggest this.
A single agent burns roughly 4x the tokens of a chat interaction, and multi-agent systems burn roughly 15x.
Every node you add multiplies that.
Anthropic’s multi-agent research system outperformed a single Opus agent by 90.2% on their internal research eval, because research fans out into independent searches naturally.
But their standing advice from Building Effective Agents hasn’t changed. Find the simplest solution possible, and only add complexity when the task demands it.
Even LangGraph’s own guidance says that if your agent is a straightforward loop with tools, LangGraph is overkill.
The decision rule is pretty straightforward: Reach for a graph when the work splits into genuine specialties, needs parallel fan-out and join, needs different models per step, or needs failure isolation and auditable routing. Otherwise, stay in the loop.
Where to start
You don’t need an org chart of agents on day one. Build up to it.
Master a single loop first, with brakes, a real completion check, and a critic. A graph of weak loops is just distributed failure.
Draw the graph on paper before writing code, and challenge every node to justify its existence.
Define the state schema and write access up front. State drift is the main way graphs rot.
Make the reviewer node a different model with fresh context, and anchor it to external evidence.
Put budget caps on every node. A graph has many loops spending tokens in parallel, and a weak verifier now burns money concurrently.
Key takeaway
Graph engineering isn’t a new discipline replacing loop engineering. It’s the name that stuck for a decision every agent builder eventually faces. When one loop stops being enough, coordination becomes the engineering.
The word may not survive the year. The design question will.
Here’s a summary of key takeaways in graph engineering.
The anatomy of diffusion LLMs
We recently published a deep dive that covers one of the most important architectural shifts happening in language modeling right now: diffusion LLMs.
Read the full Part 1 deep dive here →
Read the full Part 2 deep dive here →
It builds a complete understanding from first principles:
how autoregressive generation is structurally memory-bandwidth bound)
why Gaussian noise can’t work on discrete tokens
how masked diffusion solves this with an ELBO-derived training objective
the math behind the forward and reverse processes
unmasking strategies
block diffusion for KV cache compatibility
a detailed engineering comparison between the two paradigms.
the training techniques that scaled dLLMs from 8B to 100B parameters (including converting pre-trained autoregressive models like LLaMA into diffusion models via attention mask annealing)
the inference acceleration stack (block-wise KV caching with Fast-dLLM, confidence-aware parallel decoding, token editing with LLaDA 2.1),
production serving with SGLang
hands-on code for running Dream 7B and serving LLaDA 2.0,
and a decision framework for when dLLMs actually make sense over autoregressive models.
Read the full Part 1 deep dive here →
Read the full Part 2 deep dive here →
Why care?
Every production LLM today, GPT-4, Claude, Gemini, LLaMA, generates text the same way: one token at a time, left to right.
Each token requires loading the full model weights through GPU memory, performing a tiny computation, and then loading all the weights again for the next token. On an A100, this means roughly 1 FLOP per byte of data moved, while the GPU is designed for 100+ FLOPs per byte.
Diffusion LLMs take a completely different approach. They start with a fully masked sequence and iteratively unmask all tokens in parallel, using bidirectional attention at every step. This shifts inference from memory-bandwidth bound to compute-bound, which is exactly where modern GPUs are efficient.
The results are catching up fast. Block diffusion (BD3-LM) is within 0.5 perplexity points of autoregressive on LM1B. LLaDA at 8B parameters matches LLaMA 3 on MMLU and exceeds it on TruthfulQA and HumanEval. And models like Dream 7B are already being served in production with SGLang.
Understanding how it works at a mathematical level, from the forward masking process to the ELBO objective to block-level KV caching, is going to be increasingly valuable as these models scale.
👉 Over to you: Do you think the future of LLM generation is pure diffusion, pure autoregressive, or some hybrid of the two?
Good day!
Well Written Avi. Now, I need to write the code to understand step by step. Thanks for putting this together.