In today’s newsletter:
LLM routing can cost more than not routing at all.
Agent framework vs. agent harness, clearly explained.
A new technique just made attention 23x faster.
LLM routing can cost more than not routing at all
The basic idea behind LLM routing sounds simple, i.e., send easy requests to cheaper models and save frontier models for hard tasks.
Production routing is rarely that simple.
Every classification adds latency. General-purpose models can misread intent. Routing rules go stale.
And switching models during an agent session can wipe out your prefix-cache savings, raise costs, and make the agent behave inconsistently.
In this video, we cover:
Why DIY routing often breaks in production
How prefix caching changes the economics
Why agents need session pinning and model affinity
How Plano and Arch-Router handle intent resolution
How to build, test, and evaluate a router with DigitalOcean
Routing works, but only when the infrastructure accounts for the full session, not just the latest request.
So the question worth asking is how much of your AI traffic has been overpaying this whole time.
Get started with DigitalOcean Inference Router →
Thanks to DigitalOcean for partnering today!
Agent Framework vs. Agent Harness, clearly explained!
Giving an agent tools doesn’t solve the problem of keeping it productive across a long task. It still needs to track progress, recover from failed actions, and preserve useful context when its conversation grows.
Those responsibilities help explain the distinction between an agent framework and an agent harness.
A framework gives you components for building agent behavior. A harness packages an agent loop with an environment and operating conventions. The engineering decision is how much of that surrounding system you want to assemble and maintain.
A framework gives you control over the design
An agent framework provides abstractions for models, tools, state, and orchestration. You use those components to define how your application behaves.
Consider an invoice-processing system. It might extract fields, validate them against purchase orders, route discrepancies for review, and submit approved records.
Some steps need model judgment. Others should follow deterministic rules, especially when money moves or records change.
Explicit orchestration lets you enforce those boundaries. A model can suggest an action, while application code decides whether the required checks have passed.
However, the left side of the visual needs one qualification: frameworks aren’t limited to workflows you can draw upfront.
LangGraph, for example, supports both deterministic steps and model-driven decisions, along with persistence and human intervention. A graph can contain an agent loop whose next tool call depends on what it discovers.
The useful distinction is how much agent behavior comes assembled, and how much you define yourself.
A harness supplies an operating environment
An agent harness makes choices about how the model works inside a task.
Depending on the implementation, that can include file access, command execution, context management, progress tracking, and delegation to subagents.
Deep Agents is a concrete example. It builds on LangChain and LangGraph, adding filesystem tools, context offloading, and subagent support around the tool-calling loop. The framework and harness occupy different layers of the same system.
For a coding task, this changes your starting point.
Instead of implementing file inspection, editing, and execution behavior individually, you configure an existing environment with your repository instructions, tools, and access boundaries.
The model can inspect a failing test, read the relevant implementation, make a change, and run another check. Its intermediate actions depend on evidence gathered during execution.
You still own the definition of success. A working tool loop doesn’t establish that the requested feature works.
Long tasks
Consider a task spans multiple sessions or context windows.
A conversation summary might preserve the broad objective while losing a detail such as which migration ran or which test still fails. Persisting the conversation alone doesn’t guarantee that the next session can resume correctly.
Anthropic documented this problem in its work on long-running coding agents. Agents sometimes attempted too much at once, left incomplete work, or declared completion prematurely.
Its approach combined an initial setup phase with incremental coding sessions. Progress files, version history, and explicit feature checks helped subsequent sessions reconstruct the state of work.
The practical lesson is to make progress inspectable outside the model’s conversation.
A repository commit records what changed. A test result records what passed. A task file records what remains.
These artifacts answer different questions, so compressing everything into one narrative summary can remove information the next session needs.
Choosing what to build yourself
Start with the task’s control requirements.
For invoice processing, the required sequence and approval rules may be clear before execution. Explicit orchestration makes those rules visible and testable.
For investigating a failing build, the next useful action may only become clear after reading logs or running a test. A configurable harness can reduce the amount of execution machinery you need to build.
Many applications need both.
An outer workflow can enforce authorization and record final outcomes. Inside one step, an agent can investigate an issue using a harness, then return evidence for the workflow to validate.
That boundary also improves debugging. You can distinguish a failed business rule from an agent choosing an unhelpful tool or losing context.
Evaluate the behavior around the model
When comparing options, inspect what happens after a tool fails, what stays a restart, and how the system decides that work is complete.
Also check which operations require approval and whether those restrictions apply to delegated agents.
Test those behaviors with a deliberately interrupted task. Resume it and inspect whether the agent repeats completed work, misses an unresolved failure, or claims success without verification.
Frameworks give you building blocks. Harnesses package decisions about execution. Your application still needs evidence that those decisions produce the behavior you intended.
If you want to build both sides of this yourself, our AI Agents course has a full LangChain and LangGraph track. It starts with a single production-ready agent and works up to harness middleware, planning and verification loops, multi-agent orchestration, and durable agents:
Start the AI Agents with LangGraph course →
Part 1: First Steps Towards a Production-Ready Agent with LangChain →
Part 2: Middleware-Driven Context Engineering in LangChain →
Part 4: Planning and Verification Loops with LangGraph and Jev →
A new technique just made attention 23x faster
Full attention compares every query against every earlier key, so prefill costs O(T²). At 10M tokens, that cost dominates everything else.
Sparse attention reads only k keys per query. But something still has to pick those k keys.
That picker is now the bottleneck. DeepSeek’s DSA uses a lightning indexer that scores every key. GLM-5.3-Flash pools 4 keys into 1 before scoring. NSA, MoBA, and Quest score blocks of keys. In each case the indexer stays O(T²), and at 10M tokens it takes 98% of the layer’s prefill time.
Blocks also group keys by position, so one relevant key sitting among unrelated ones can get averaged away.
Latent Sparse Attention (LSA) groups keys by meaning instead:
Keys are projected into a 64-dim index space, and every 4 are pooled into 1.
Each 512-key block is clustered into buckets of about 8 keys, each with a centroid. Finished blocks are never rebuilt.
A query scores only the centroids, keeps the top 1,024 buckets, and expands them to 4,096 keys, plus 4 sink tokens and the 512 most recent keys.
Exact FlashAttention then runs over those keys with an unchanged kernel.
If you’ve worked with vector search, this will look familiar. It’s cluster-based search running inside attention, and the paper’s kernel design draws on FAISS.
The results at 10M tokens:
One layer’s prefill runs 72.4x faster than full attention on Qwen-3.8-27B, and 22.7x faster than GLM-5.3-Flash’s sparse attention.
End-to-end time to first token is 9.2x faster on Qwen and 2.2x faster on GLM.
Below roughly 512k tokens, the end-to-end gain is about break-even.
So the 23x is a per-layer prefill number. The end-to-end gain is smaller and shows up only at very long contexts.
One thing LSA doesn’t change is storage. It makes finding the right keys cheaper, but the KV cache still grows linearly with context.
We covered the storage side in detail in our KV cache engineering deep dive:
👉 Over to you: Which part of your agent system has required more engineering, the workflow itself or keeping the agent productive during execution?
Good day!