Bringing production context into coding agents
One limitation of coding agents does not get discussed enough.
They can inspect every file in your repository, but they cannot see how that code behaves after deployment. So when an endpoint slows down or errors spike, the agent suggests fixes from static code while the useful evidence sits in production traces and logs.
Dynatrace has open-sourced a repository that brings this runtime context into coding agents.
The MCP server provides access to live Dynatrace data. The included skills teach agents how to query and interpret that data, while reusable prompts define complete investigations.
For instance, its performance regression prompt compares P95 latency, error rate, and throughput before and after a deployment. It then finds the slowest trace, maps the bottleneck span back to the relevant workspace code, and recommends either a rollback or a targeted hotfix.
The skills work with Claude Code, Cursor, GitHub Copilot, OpenCode, Gemini CLI, and other compatible agents.
(don’t forget to star 🌟)
Thanks to Dynatrace for partnering today!
4 speculative decoding variants
Large-model decoding often spends one full model run producing a single token, even when the next few tokens are predictable.
Speculative decoding tries to get several useful tokens from that run instead of just one.
A cheaper path drafts a short continuation. The large model checks those tokens together, keeps the accepted prefix, and corrects the first mismatch. Draft tokens never reach the output without verification from the large model.
The main difference between speculative decoding variants is how they produce that draft. Some use a second model, while others build drafting into the target model itself.
Let’s look at the four approaches and the engineering tradeoffs behind each one.
To dive deeper into the full LLMOps lifecycle, we have covered every bit of this in the LLMOps course, starting from fundamentals to production.
The common speculative decoding loop
Every variant has two stages: draft several tokens cheaply, then verify them with the target model in parallel.
Suppose the drafter proposes five tokens. If the target accepts the first three and rejects the fourth, the first three move to the output. The target supplies the replacement at position four, while the remaining draft is discarded.
When all five tokens are accepted, the verifier can add one more token from the same pass. The sequence advances by six tokens after one target-model run.
For greedy decoding, this means checking whether the predicted tokens match. Sampling requires an acceptance and correction rule based on both probability distributions.
The original algorithm preserves the target model’s distribution, so acceleration does not require changing the result.
1. Two-model speculative decoding
The original approach pairs a small draft model with a larger target model. The small model generates a few tokens sequentially, then the target verifies the full block at once.
This is still the first setup to test. It works with an unchanged target model, and most inference stacks already support some form of assisted generation. The original paper reported 2x to 3x acceleration on T5-XXL with identical outputs.
The draft model needs to be cheap enough to justify its work but accurate enough to earn a useful acceptance rate. A larger drafter often predicts better, yet its added latency can reduce the final speedup.
There is also a clear memory cost. The server holds another set of weights and a separate KV cache. Closely matched model families are usually easier to serve because their tokenizers and output behavior align better.
2. EAGLE drafts from hidden states
EAGLE removes the independent draft language model. It trains a lightweight module to predict the target model’s second-to-top-layer features, then converts those predicted features into candidate tokens.
Instead of asking a separately trained model to approximate the target, EAGLE drafts from the target model’s own internal representation.
Feature prediction has an ambiguity problem because different tokens can produce similar hidden states. EAGLE also feeds in the token sequence shifted forward by one position, giving the draft module enough information to make the feature prediction more precise.
The paper reported 2.7x to 3.5x latency speedups for LLaMA2-Chat 70B and roughly doubled throughput in its evaluated setup. In practice, EAGLE fits best when the checkpoint and serving engine can be managed together because its draft module is trained for a specific target.
3. Medusa uses parallel prediction heads
Medusa adds several small decoding heads to the target model. One head predicts the next token, another predicts two positions ahead, and later heads predict positions farther into the continuation.
All heads run from the same model state. This makes drafting parallel, but it also means one head cannot condition on what another head predicted. Their individual guesses do not always combine into a consistent sequence.
Medusa handles this by arranging alternatives into a candidate tree. Tree attention lets the target verify several possible continuations in one pass, and the verified branch determines the accepted prefix.
Medusa-1 freezes the backbone and trains only the added heads. Its paper reports more than 2.2x speedup without changing the backbone’s generation quality. Medusa-2 jointly tunes the heads and backbone, reporting 2.3x to 3.6x, but it requires a more involved training recipe.
Consider Medusa when a second model is undesirable and training small heads is feasible. Candidate-tree width becomes the main serving control because wider trees improve coverage while increasing verification work and temporary memory.
4. LayerSkip exits from early layers
LayerSkip uses the target model as both drafter and verifier. Early transformer layers produce draft tokens, while the remaining layers check and correct them.
This removes the second model and added prediction heads. Drafting and verification also share weights, vocabulary, activations, and part of the computation.
LayerSkip does require the right checkpoint. Its training recipe uses layer dropout and an early-exit loss so intermediate layers learn to produce useful predictions. A compatible checkpoint needs no separate drafter or added heads, but an arbitrary pretrained model will not provide reliable early exits automatically.
The paper reported speedups up to 2.16x on CNN/DailyMail summarization, 1.82x on coding, and 2.0x on TOPv2 semantic parsing. I would treat LayerSkip as a training decision made before deployment, not an optimization applied later to any model.
Choosing between the four variants
The number that matters is accepted tokens per target-model pass after accounting for drafting time and verification overhead.
Two-model decoding is the easiest place to begin because it leaves the target untouched.
EAGLE and Medusa reduce the cost of carrying another model but introduce target-specific training and serving code.
LayerSkip has the fewest inference-time components, provided the checkpoint was trained for early exits.
The benchmark should reproduce the real workload. Low-temperature code generation usually produces more agreement than high-temperature writing. Heavy batching can also reduce the gain because the GPU is already doing more useful work during each decode step.
All four methods still rely on KV caching. Speculative decoding reduces the number of target-model runs, while the KV cache prevents accepted context from being recomputed during the next iteration.
To dive deeper into the full LLMOps lifecycle, we have covered every bit of it in the LLMOps course, starting from fundamentals to production:
Read Part 2 on understanding the core building blocks of LLMs →
Read Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming →
👉 Over to you: Which speculative decoding approach best matches the models you can modify and the memory available in your serving stack?
Good day!