Karpathy Said Something You’ll Regret Ignoring
vibe coding vs agentic engineering!
Karpathy said something you’ll regret ignoring
“You are still responsible for your software, just as before. You are not allowed to introduce vulnerabilities because of vibe coding. “
He said it while drawing the line between vibe coding and agentic engineering. Agents write more of the code now, but none of that takes the responsibility off you.
The assumption underneath that is that a careful enough reader catches the problem. But some failures don’t show up in anything there is to read.
For instance, a common fear with a RAG agent is that it could hallucinate when a question asks something outside its corpus.
But such cases are actually well handled by any competent model now. If nothing in the retrieved context looks relevant, there’s no material to build an answer on.
Instead, the majority of failures originate when the retrieved context has partial coverage.
The retrieval pipeline returns context that’s topically correct but doesn’t cover the full question, and the model completes the remainder from parametric knowledge.
There are no token-level labels in the output to tell what was generated using retrieved context and what came from weights.
Both are streamed the same way.
Detecting this for production-grade apps needs a metric written for it, one that’s also aligned with principles of agentic engineering.
And the solution is actually implemented in the eval skill that comes with Google’s Agents CLI.
We described the concern to Claude Code in plain English. It read the agent’s code, came back with a plan I approved.
It then reported that no built-in metric isolates the behaviour and wrote a custom rubric called corpus_abstention.
It assigned a single categorical verdict per case rather than aggregating everything into one score, since the built-in raters regenerate their rubrics each run and leave no stable number to trend.
GROUNDED_ANSWER
CORRECT_ABSTENTION
UNGROUNDED_ANSWER (answered entirely from outside knowledge)
MIXED_LEAKAGE (grounded, but slips in one unsupported claim)
WRONG_ABSTENTION (refused something the docs actually covered)
After this, it automatically generated 33 scenarios partitioned by where the failure could occur, like:
in-corpus
off-domain
out-of-corpus but plausibly answerable
boundary cases where the topic is covered, but a specific detail isn’t.
The baseline score was 19 of 33.
Off-domain passed 3 of 3, as expected.
But 6 of 15 in-corpus cases retrieved the right document, cited it correctly, answered accurately, and added a claim the source never made.
The root cause was one line in the agent’s instruction: “If you already know the answer to a simple question and no document lookup is needed, you may respond directly without citations.”
The eval skill helped flag this, and then Claude removed it and forced retrieval on every question.
This took the suite to 30 of 33, and ungrounded answers went from 6 to 0. The full recording of our run is below, and we worked with the Google Cloud team on this.
Here’s the Agents CLI GitHub repo → https://github.com/google/agents-cli
(don’t forget to star it ⭐ )
Are we ready for an agent-native memory system?
First, what does that even mean?
It’s memory built for an agent that keeps running, not a store you bolt on at the end. It has to hold facts across sessions, update them when they change, drop the stale ones, and hand back the right piece when the agent asks. A real data system, not a pile of saved chunks.
A new study checked whether anything out there actually does this, and what decides the answer surprised me.
It takes 12 agent memory systems and splits each into four stages, covering how memory is represented, extracted, retrieved, and maintained. Then it checks which stage decides whether the agent stays correct.
Turns out it’s write time, not query time.
The systems that stay reliable under fact updates and cross-session reasoning are the ones that add structure as the memory is written. The graph-based methods lead right there.
The flat stores fail the same way every time. An LLM pulls out entities and relationships with no schema, and the graph fills up with generic nodes that nothing downstream can filter.
A few things follow from this.
Structure has to be set at extraction. Once the graph is built from untyped nodes, no query can recover a distinction the extractor never made.
Updates return stale facts. Append-only and freeform stores keep handing back old versions, which the paper calls hallucinations of the past.
A bigger model does not save it. Swapping the backbone moves the scores but rarely changes which pipeline wins, so the memory’s behavior is set before generation runs.
The paper also weighs the cost. The heavily structured systems pay for it, sometimes orders of magnitude more time per query, and the win only holds when the upkeep stays local instead of rewriting the whole store on every change.
So the takeaway is less about picking a winner and more about where the structure lives. Decide it at write time, keep the maintenance cheap, and retrieval has something real to rank.
Retrieval only ranks what extraction chose to keep.
Link to the paper: https://arxiv.org/pdf/2606.24775
If you’re into agent memory systems, we wrote a full article on exactly this.
It walks through defining the schema as a Pydantic ontology in Zep, with custom entity and edge types, source and target constraints, and the extraction pipeline that turns raw conversations into typed, queryable memory.
This memory system is 100% open-source: https://github.com/getzep/graphiti
Good day!
P.S. For those wanting to develop “Industry ML” expertise:
At the end of the day, all businesses care about impact. That’s it!
Can you reduce costs?
Drive revenue?
Can you scale ML models?
Predict trends before they happen?
We have discussed several other topics (with implementations) that align with such topics.
Here are some of them:
Learn everything about MCPs in this crash course with 9 parts →
Learn how to build Agentic systems in a crash course with 14 parts.
Learn how to build real-world RAG apps and evaluate and scale them in this crash course.
Learn sophisticated graph architectures and how to train them on graph data.
So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.
Learn how to run large models on small devices using Quantization techniques.
Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using Conformal Predictions.
Learn how to identify causal relationships and answer business questions using causal inference in this crash course.
Learn how to scale and implement ML model training in this practical guide.
Learn techniques to reliably test new models in production.
Learn how to build privacy-first ML systems using Federated Learning.
Learn 6 techniques with implementation to compress ML models.
All these resources will help you cultivate key skills that businesses and companies care about the most.