The Hands-on AI Engineer Playbook to Build RAG Apps for Production
Why RAG latency is a prefill problem, not a retrieval problem.
The Hands-on AI Engineer Playbook to Build RAG Apps for Production
We have published a new deep dive in the RAG Systems course.
In a typical RAG request, embedding the query takes milliseconds, vector search takes tens of milliseconds, and reranking takes maybe a hundred milliseconds.
Then the model reads the retrieved chunks, and that step alone can take seconds. This deep dive is about that step.
Read the full deep dive here →
The full 50 min deep dive covers:
Why prefill dominates RAG latency, not retrieval
How the KV cache works and what it costs in memory
Why prefix caching approaches zero hit rate for RAG workloads
Three independent failures when you try to reuse cached chunks
Six published approaches to fix them
Hands-on implementation that covers every problem and fixes
Best practices for production to assess what applies to your system
Read the full deep dive here →
Why care?
Processing 16,000 input tokens with a 14B parameter model on an NVIDIA L20 takes ~5.5 seconds to get the first token.
And in a RAG system, the vast majority of those tokens are retrieved chunks, not the user’s question.
Prefill is the root cause of this, and it scales quadratically with input length because every token must attend to every other token.
For production RAG systems serving thousands of requests per hour, this is the dominant line item on the inference bill.
The techniques covered in this deep dive (KV cache reuse, selective recomputation, prompt reordering, and more) directly solve that cost.
CacheBlend, for instance, reuses precomputed chunk caches and recomputes only 10 to 15 percent of tokens, cutting time to first token by 2 to 3x while keeping answer quality within 0.01 to 0.03 F1 of a full recompute.
TurboRAG reports up to 9.4x TTFT reduction by moving the entire prefill offline.
This is also an area the industry is actively investing in.
LMCache, the open-source package that implements CacheBlend, ships as a production integration on top of vLLM.
Serving engines are adding non-prefix cache reuse as a first-class feature.
The engineering teams working on inference optimization at scale are treating this as one of the highest-leverage problems to solve right now, because every percentage point of cache reuse translates directly into GPU cost saved.
Understanding how the KV cache works, why prefix caching fails for RAG, and what the three failure modes of naive cache reuse are is becoming essential knowledge for anyone building or operating a RAG system at any meaningful scale.
You can learn how to build such systems here →
👉 Over to you: Have you measured where your RAG latency actually sits?
ORM vs PRM: How LLMs verify their reasoning?
When a model solves a problem in steps, something has to score that work so the model can learn from it.
There are two ways to do it, and they differ based on what they assign credit to.
An outcome reward model reads only the final answer and returns one number for the whole solution.
A process reward model reads every step and returns a score after each one.
The process model wins as a judge because sometimes a model can land on the right answer through faulty reasoning.
An outcome model cannot tell the difference because it sees the right answer and approves the whole solution, broken steps included.
A process model catches the bad step, and a correct final answer does not save it.
OpenAI actually measured this difference.
The model’s weights stayed frozen while it generated many solutions per problem, and each scorer picked the one it rated highest.
The process scorer solved 78.2% of a math test set, the outcome scorer solved 72.4%, and taking the most common answer got 69.6%.
That is the ranking job, and step-level scoring done by process reward model is clearly better at it.
Using that signal to train the model is a different job.
The score stops being a verdict on finished work and becomes the objective the model is pushed toward.
In ranking, a wrong rating picks one bad solution and the model never learns anything from it.
In training, a wrong rating comes back as a weight update, and the model produces more of whatever earned it.
Which is why the two jobs need different scorers.
A process scorer has to be a model. It reads a step written in natural language and predicts whether that step was sound, so it carries the usual model gaps and misjudges some steps.
Ranking absorbs those gaps because nothing adapts to them. Training turns them into a target, since gradient descent will find whatever the scorer rates highly and push toward it, correct or not.
An outcome check on a math or code problem is not a model at all. It runs the test or compares the answer to ground truth.
There is no direction to push a rule like that. The answer is right or it is not.
So the choice comes down to what the scorer is made of, not how much it sees. A process scorer sees more and can be fooled by the model it is teaching. An outcome rule sees almost nothing and cannot be.
DeepSeek-R1 used exactly this. They tested process scoring during development and shipped a rule that checks the final answer and the output format.
As a practical takeaway:
Use outcome scoring when the answer is checkable and the scorer sits inside the training loop.
Use process scoring when the reasoning itself has to be correct, or when ranking finished candidates and guiding search across them.
To dive deeper, we covered all of this in depth across 13 parts of our RL series:
Part 1 covers the foundations, the agent-environment loop, and the reward hypothesis →
Part 5 covers function approximation and what breaks when states stop being enumerable →
Part 8 covers PPO and why it became the default for LLM post-training →
Part 9 covers RLHF, preference data, and reward model training →
Part 10 covers verifiable rewards and GRPO, which is the rule-based path R1 took →
Part 12 covers environments, trajectories, and the training loop →
Part 13 covers how AI teams use RL in production, including Cursor’s five-hour checkpoint loop →
Good day!