A 10-week Roadmap to Run LLMs in Production
...covered with hands-on resources.
In today’s newsletter:
ReAct vs Plan-and-Act pattern in agents.
A 10-week roadmap to run LLMs in production.
Phases of ML modeling.
ReAct vs Plan-and-Act Pattern in Agents
Every wrong turn a ReAct agent takes stays in its context for the rest of the run.
For more context, the ReAct pattern runs one model in a single loop.
The model writes a thought about what to do next, takes one action, reads the observation that comes back, appends all three to the same prompt, and repeats until it decides the task is done.
Nothing ever leaves the prompt.
So a failed search from step three is still retained and accessible at subsequent steps, competing with the original objective for the model’s attention.
Plan-and-Act splits the overall loop into two jobs.
The authors of the Plan-and-Act paper experimented on web navigation, so the observation coming back after every action is the raw HTML of the page the agent is currently on.
A planner reads the user query and the initial page and writes high-level steps.
An executor reads the plan, the task, its own past actions, and the current HTML, then emits one grounded action.
After each action, the executor strips the HTML it no longer needs before taking the next one, so the execution context does not grow the way a ReAct trace does.
Whether this is helpful is determined by plan granularity.
A good step covers one unit of work, like searching for the product in the search box.
An individual click is too small to be a step, and “analyze the search results” is not a step at all, because it pushes the reasoning back onto the executor.
A step must also name the actual values it needs.
The paper’s planner instructions ask for “input New York as the arrival city” instead of “input the arrival city”, because the second version leaves the executor to guess which city goes in the box.
The executor’s job is picking the right element and typing into it, not filling in the blanks the planner left open.
They also found that a badly trained planner makes things worse than no planner at all.
On WebArena-Lite, a ReAct-style executor with no planner scored 36.97%.
But the same executor with a naively finetuned planner scored just 20.60%.
The planner had never seen those sites, so it wrote steps that read fine but matched nothing on the page, and the executor followed them anyway.
A properly trained planner reached 43.63%.
A plan written once and never revised has its own problem. For instance, if the search for “library at CMU” gives no results, the executor will still hold a step that cannot work anymore, and it will keep trying it anyway.
Replanning after every action is necessary to recover from that.
For instance, in the paper, the planner saw the current state, the previous plans, and the actions taken, and rewrote the step to “libraries near CMU”, which increased the score to 53.94%.
As a result, the failed attempt got replaced in the plan rather than accumulated in the context.
The cost is one planner call per executor step. The authors flag this directly and suggest letting the executor decide when a replan is required.
Nearly all of this is harness design rather than model choice.
You can read the paper here → https://arxiv.org/pdf/2503.09572
A 10-week Roadmap to Run LLMs in Production
As an AI engineer, please learn:
Learn the roofline model and why decode is memory-bound
Deploy vLLM and SGLang, then read their schedulers
Understand paged attention from the code, not the blog post
Build observability before you optimize anything
Track TTFT, inter-token latency, throughput, queue depth
Use Grafana + Prometheus for inference dashboards
Turn on prefix caching and find which workloads it helps
Learn continuous batching and chunked prefill
Run load tests with 1000+ concurrent requests
Report p50, p95, p99, never just the mean
Master quantization tradeoffs (FP8, INT4, AWQ, GPTQ)
Learn speculative decoding and where it stops helping
Set up KV cache eviction for long contexts
Try disaggregated prefill and decode serving
Learn Kubernetes for AI workloads and autoscale on queue depth
Learn how inference costs break unit economics
Build your own model router based on cost, latency, and quality
Create a token budgeting system per request
Build one inference service and benchmark it publicly
Read inference research instead of model release news
Start sharing your optimization benchmarks
We put together a 10-week plan that covers every one of these at 30 minutes a day.
It has 50 sessions, split between reading the theory and building on your own service, and all of them feed one artifact: an inference service you deploy, instrument, load test past 1000 concurrent requests, tune, and publish as a reproducible benchmark.
It is open on GitHub, and contributions are welcome, especially newer sources worth adding.
GitHub repo: http://github.com/patchy631/time-to-first-token
(don’t forget to star it 🌟)
Phases of ML Modeling
Most ML systems don’t jump straight to deep learning. They evolve.
A practical way to think about this evolution is in phases, starting from the simplest possible solution and gradually increasing complexity only when it’s justified. Because unnecessary complexity = low utility.
A staged approach reduces risk, improves debuggability, and aligns naturally with MLOps best practices.
Now, let’s walk through the different phases of ML model development:
Phase 1: Before ML (heuristics and rules):
If you’re solving a problem for the first time, resist the urge to start with a model. Begin with a non-ML baseline: a rule, a heuristic, or a simple deterministic strategy.
For example, in a movie recommendation system, a phase-1 solution could be as simple as recommending the top-10 most popular movies to every user.
This might sound naive, but such heuristics are often surprisingly strong.
These baselines are fast to build, easy to reason about, and set a minimum performance bar. If a complex ML model cannot beat a naive heuristic, something is wrong; either ML isn’t adding value, or there’s a bug in the pipeline.
Here is a conceptual diagram summarizing Phase 1:
Conceptually, Phase 1 looks like a direct mapping from input to output using rules, without any learning component.
Phase 2: The simplest ML model
Once a heuristic baseline exists (or once it’s clear that heuristics aren’t enough), the next step is not a deep model.
It’s the simplest possible ML model.
Think logistic regression, a shallow decision tree, k-nearest neighbors, or a basic linear model; something easy to train, interpret, and deploy.
The goal here is not peak accuracy. This phase answers foundational questions:
Can we train on historical data and get sensible predictions?
Are the features informative?
Does the model generalize better than the heuristic?
This is where you validate the end-to-end ML pipeline: data ingestion, feature extraction, training, evaluation, and serving.
Conceptually, Phase 2 introduces learning, but keeps the model and serving logic minimal.
Phase 3: Optimizing the simple model:
Once the basic model works, there’s often significant performance left on the table, without changing the model class at all.
Phase 3 focuses on extracting as much value as possible from the existing approach.
Typical levers include:
Feature engineering: creating better representations of the input data.
Hyperparameter tuning: systematically searching for better configurations.
More data: expanding the dataset or improving data quality.
This phase is where returns on investment are often highest.
You’re working with models that are easy to understand, cheap to train, and simple to serve, while still achieving meaningful gains.
Many real-world ML systems stop here. A well-tuned logistic regression, gradient boosted tree, or modest ensemble can meet production requirements without the complexity of deep learning.
Here’s the entire thing summarized as a diagram:
Hence, phase 3 overall looks like a refinement loop around the same model family, not a shift in paradigm.
Phase 4: Complex models:
Only after simpler approaches are exhausted should you move to fundamentally more complex models.
This includes deep neural networks, transformers, or large pretrained architectures, depending on the domain.
Complex models bring capacity, but also cost. The decision to enter Phase 4 should be evidence-driven.
Conceptually, Phase 4 introduces higher model expressiveness alongside increased engineering complexity.
A key point to keep in mind is that, at every phase, the previous phase’s best model becomes the baseline.
This phased approach encourages incremental progress and disciplined decision-making.
If you want to learn more about these real-world ML practices and start your MLOps, we have already covered MLOps from an engineering perspective in our 18-part MLOps course.
It covers foundations, ML system lifecycle, reproducibility, versioning, data and pipeline engineering, model compression, deployment, Docker and Kubernetes, cloud fundamentals, virtualization, a deep dive into AWS EKS, monitoring, and CI/CD in production.
Start with the MLOps course here →
👉 Over to you: Which phase do most of your models actually live in today?
Good day!