In today’s newsletter:
Redis is going all in on AI.
GPU architecture, clearly explained.
Redis is going all in on AI.
They just built Redis Iris, a real-time engine that delivers relevant and accurate context for production agents.
In production, support agents receive the same intent in many phrasings:
“What’s the status of my refund?”
“Has my refund gone through yet?”
A semantic cache matches these paraphrases and returns the stored response, which works well when the answer is static.
But, at times, context is not static over time (like refund status). The stored response might indicate “processing” even after the refund has gone through, so the cache returns a stale answer.
A plain agent avoids this by running the full reasoning loop on every request, which gives the right answer but spends tokens re-deriving the same steps each time.
Here’s what happens when a request comes in:
The Intent Router recognizes similar requests.
It then pulls the Agent Playbook for that request. The Playbook doesn’t store an answer. It stores the steps to produce one, like “look up the order, check the refund status, reply using this template.”
The agent follows those steps, so it skips re-planning, and the app fetches the current refund status before the reply goes out.
In short, semantic cache reuses the answer while Iris reuses the steps, so the answer is always computed on live data.
This lets the workflow avoid any additional reasoning while still working on the live data.
You can try Redis Iris here: https://redis.io/iris
To test this, we built a small app to compare all three setups (plain agent, Semantic cache, and Iris-backed agent).
We asked about the refund while it was processing, then updated the status to refunded and asked again with a paraphrase.
The video below shows this in action:
Question 1 (status: processing):
Plain agent: 2 LLM calls, 9.4s, correct
Semantic cache: cache miss, so 2 LLM calls, 10.2s, correct
Iris: 1 LLM call, 5.9s, correct
Question 2 (status: refunded, paraphrased):
Plain agent: 2 LLM calls, 12.7s, and the reply didn’t confirm status
Semantic cache: cache hit, 0.5s, but it returned the stale “processing” answer
Iris: 1 LLM call, 5.6s, correct
Iris skips the planning call because the procedure is already stored, so it ran about 2x faster than the plain agent and was the only setup that was right on both turns.
You can try Redis Iris here: https://redis.io/iris
Thanks to the Redis team for partnering today!
GPU architecture, clearly explained
A GPU can perform calculations faster than it can load the data those calculations need.
This often limits token generation. The model reads its weights from memory, uses them for a small amount of arithmetic, then needs more data.
Adding more arithmetic capacity does little to help if memory cannot supply those weights fast enough.
Memory bandwidth measures how much data the GPU can move per second. When generating one response at a time, the rate at which it reads the model’s weights can limit how quickly it produces tokens.
To see why, consider what happens when the model uses a weight.
At 16-bit precision, reading one weight transfers two bytes. Using it involves a multiplication and an addition, giving just two arithmetic operations. The GPU does very little work with each weight before it needs the next one.
Processing several requests together can improve this. The GPU can load a weight and use it across several requests, doing more calculations with the same data.
That is one reason a model serving many users at once can make better use of the GPU than a model generating a single response.
The GPU’s layout also helps it reuse data. It divides work into small groups, each with access to storage near the units doing the calculations.
There are four names to know:
A thread performs a small part of the computation.
A warp groups 32 threads so the GPU can issue instructions to them together.
A thread block contains one or more warps that can cooperate and share data.
An SM, short for streaming multiprocessor, is a compute unit that runs thread blocks. A block’s warps run on the same SM.
Each SM has storage close to its arithmetic units. Keeping a value there lets the GPU reuse it without fetching it from its larger memory again.
The main storage levels are:
Registers hold the values an individual thread is working with.
Shared memory lets threads in a block exchange and reuse data. L1 is a nearby cache that can also hold recently accessed data.
L2 is a larger cache shared across the SMs.
HBM, or High Bandwidth Memory, holds the model’s weights, its stored attention data, and other large arrays. It sits outside the processing chip, so accessing it takes longer than reading local storage.
When an algorithm keeps useful data in local storage, it can avoid repeated trips to HBM. FlashAttention is an example.
A conventional attention implementation creates a large table of attention scores, writes it to HBM, and reads intermediate results back for later calculations.
FlashAttention works on smaller pieces and keeps intermediate results close to the computation. It avoids storing the full table in HBM. It still loads inputs and saves outputs, but moves less data between those steps.
This is why a GPU’s compute rating alone cannot predict token speed. Performance also depends on how much data the model moves and how much work it does before loading that data again.
Our full guide explains how these parts fit together, with diagrams of GPU execution and memory.
Read the full guide on how a GPU actually works →
We also put together a 10-week roadmap for engineers who want to run LLM inference in production, with 30 minutes of work per day.
Explore the time-to-first-token roadmap →
If you find it useful, star the repository.
👉 Over to you: Is your current inference workload limited by memory bandwidth or compute?
Good day!
P.S. For those wanting to develop “Industry ML” expertise:
At the end of the day, all businesses care about impact. That’s it!
Can you reduce costs?
Drive revenue?
Can you scale ML models?
Predict trends before they happen?
We have discussed several other topics (with implementations) that align with such topics.
Here are some of them:
Learn everything about MCPs in this crash course with 9 parts →
Learn how to build Agentic systems in a crash course with 14 parts.
Learn how to build real-world RAG apps and evaluate and scale them in this crash course.
Learn sophisticated graph architectures and how to train them on graph data.
So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.
Learn how to run large models on small devices using Quantization techniques.
Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using Conformal Predictions.
Learn how to identify causal relationships and answer business questions using causal inference in this crash course.
Learn how to scale and implement ML model training in this practical guide.
Learn techniques to reliably test new models in production.
Learn how to build privacy-first ML systems using Federated Learning.
Learn 6 techniques with implementation to compress ML models.
All these resources will help you cultivate key skills that businesses and companies care about the most.












