In today’s newsletter:
Benchmark the harness, not just the model.
Find production agent failures before users do.
Benchmark the harness, not just the model
We benchmark models all the time. In production, that model runs inside a harness, and the harness can change the result, the cost, and the runtime.
Devs just open-sourced Harness Arena, inside the HarnessRouter project. It runs one task across several harness and model combinations, side by side.
Every config gets the same prompt, files, tools, skills, and evaluation checks. Arena then returns:
Total cost
Execution time
Whether the task passed
The files and artifacts it produced
HarnessRouter is a plug-and-play layer that lets you run any harness under one interface, including Codex, Hermes, Claude Code, DeepSeek Harness, System One (powered by Jev), and 25 more.
The cheapest run does not automatically win. A config has to complete the task correctly first. Cost and execution time are compared only among the configurations that pass.
In one Harness Arena test:
Hermes + GPT-5.2: 5/5 successful runs using 0.47 credits
Claude Code + Opus 4.8: 3/5 successful runs using 223 credits
That is around a 474x difference in credits, and the more expensive configuration also failed two runs.
You can keep the model fixed to measure how much the harness changes the result, or vary both to find the best pairing for that task.
Once you pick a config, later tasks run through the same Unified Harness Protocol (UHP) interface for sessions, streaming, files, artifacts, cancellation, and failures.
So you are no longer choosing an agent stack from a model leaderboard alone. You can benchmark the complete runtime on the task your product actually needs to solve.
Explore the HarnessRouter repo →
(don’t forget to star it ⭐)
Find production agent failures before users do
There’s a new system that can test and evaluate your agents against the real world, then write the fixes, all before you deploy them.
AI agents look great in demos, until real users start breaking them. Your agent can tell a customer “refund done” on a $1,080 invoice without ever checking who she is, and the logs will still look clean.
Normally, you only find a failure like that after a real customer runs into it.
PlurAI Simulation lets your agent fail in a simulated world first, so you can fix the problem before any user sees it.
Here’s the whole system, step by step, from the first simulated user to a re-tested fix:
It rebuilds your actual agent, its tools, and its environment from your code
Simulated users push the agent into the edge cases that matter for your use case
Evals built for your agent check the actions it took, like whether it verified the customer before issuing that refund
Every failure comes with the run that exposed it and a fix you can verify
Any production failure can become a simulation you replay against every new version
It ships as a single plugin for coding agents like Claude Code and Codex, and it works with whatever agent you’re building. I’ve started running it on my own agents too.
It’s free for every AI engineer, and you can find it here →
👉 Over to you: Would you rather find the next agent failure in simulation, or in a customer ticket?
Thanks to the Plurai team for working with us on this.
Good day!