5 Techniques to Optimize LLMs in Production
...explained visually.
Checkpoint any method automatically using CrewAI
Long-running agent flows fail mid-run, and in most frameworks, that means restarting from zero and burning the tokens again.
CrewAI just shipped checkpointing in v1.14. Every flow method becomes a recovery point, written automatically when an event you specify fires (e.g., method_execution_finished).
You can resume in one line and fork from any saved state into a new branch with full lineage tracking. An async TUI lets you browse checkpoints, inspect events, and resume or fork from the UI.
The pipelines become resumable, inspectable, and branchable processes, with zero extra infra.
You can find the CrewAI repo here →
5 techniques to optimize LLMs in production
A tricky LLM interview question:
You’re serving an LLM in FP16 on an A100.
Token generation is quite slow, so you upgrade to an H100, which has over 3x the FP16 compute.
But upgrading the GPU didn’t improve the token generation speed much.
Why did this happen?
LLM token generation is a memory bandwidth problem disguised as a compute problem.
An A100 can do about 300 trillion FP16 operations per second. But it can only move ~2 TB per second between HBM and its compute units.
During generation, every new token requires reading the model weights and the entire KV cache from memory. While the computation process finishes almost instantly, the chip still has to spend time waiting for data.
This is why the highest-impact serving optimizations all try to tackle the same two things:
how much data moves
and how often the GPU sits idle
The visual below covers five of them.
1) Flash attention:
Standard attention writes the full attention matrix in HBM, and it grows quadratically with sequence length.
Flash Attention splits Q, K, and V into tiles, computes attention inside on-chip SRAM, and writes only the final output back.
The output is mathematically identical, but the quadratic intermediate never exists in memory, so long sequences become relatively manageable.
2) Paged attention:
Naive serving reserves one contiguous memory chunk per request, sized for the maximum possible sequence length.
vLLM paper measured 60-80% of KV cache memory wasted this way.
Paged attention is inspired by virtual memory from OS concepts.
The cache stays in small fixed-size blocks mapped through a block table, allocated as generation proceeds and freed on completion, which reduces waste to under 4%.
3) Continuous batching:
Static batching waits for the slowest request in a batch before admitting anything new. So one long generation keeps every finished slot idle until it completes.
Continuous batching makes scheduling decisions after every decode step instead of after every batch.
When a request finishes, a queued request takes its slot at the next decode step, and the GPU stays saturated.
4) Speculative decoding:
A small draft model proposes K tokens cheaply, then the large target model verifies all K in a single forward pass, accepting matches and correcting the first mismatch.
The rejection sampling scheme guarantees an output distribution identical to running the large model alone, so the speedup costs zero quality.
5) Kernel fusion:
Every GPU operation runs as its own kernel by default, and each one writes its result to HBM for the next kernel to read back.
Kernel fusion merges LayerNorm, MatMul, and activations into one kernel that keeps intermediates in registers and SRAM.
Compilers like TensorRT-LLM and torch.compile do this automatically from the model graph.
That said, all five techniques above are built for one model owning the whole GPU.
Paged attention and continuous batching fit more concurrent requests when more memory is available for KV cache blocks.
So vLLM pre-allocates nearly all GPU memory at startup and turns it into one big block pool.
But production stacks rarely run just one model.
A typical pipeline also runs several small specialized models like embedders, rerankers, and extractors alongside the LLM.
Serving each one the standard way requires a separate vLLM or TEI process, and every process claims its memory upfront with no visibility into the others.
So a 600M-parameter embedder reserves a GPU it will never fully use, and models that could comfortably share one card end up spread across many.
This is also why moving to small yet better models does not lead to substantial cost savings alone.
Superlinked’s open-source inference engine (SIE) solves this by running all of those models behind one API with shared GPU memory and on-demand model loading, so the card is actually shared instead of claimed.
GitHub repo → https://github.com/superlinked/sie
(don’t forget to star it ⭐ )
We wrote a full breakdown of this small-model serving problem and how SIE solves it.
It covers why switching to small specialized models doesn’t reduce inference costs unless those models share GPUs, and why vLLM’s design makes that impossible.
Good day!
P.S. For those wanting to develop “Industry ML” expertise:
At the end of the day, all businesses care about impact. That’s it!
Can you reduce costs?
Drive revenue?
Can you scale ML models?
Predict trends before they happen?
We have discussed several other topics (with implementations) that align with such topics.
Here are some of them:
Learn everything about MCPs in this crash course with 9 parts →
Learn how to build Agentic systems in a crash course with 14 parts.
Learn how to build real-world RAG apps and evaluate and scale them in this crash course.
Learn sophisticated graph architectures and how to train them on graph data.
So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.
Learn how to run large models on small devices using Quantization techniques.
Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using Conformal Predictions.
Learn how to identify causal relationships and answer business questions using causal inference in this crash course.
Learn how to scale and implement ML model training in this practical guide.
Learn techniques to reliably test new models in production.
Learn how to build privacy-first ML systems using Federated Learning.
Learn 6 techniques with implementation to compress ML models.
All these resources will help you cultivate key skills that businesses and companies care about the most.