• Building a durable multi-agent system with LangGraph and LangSmith.

• Jev for Agent Memory, clearly explained.

• ​The AI engineering master stack for 2026!​

Building a durable multi-agent system with LangGraph and LangSmith

Parts 5 and 6 of our hands-on Agent Engineering course are now available.

• Part 5: Multi-agent orchestration with LangGraph →

• Part 6: Durable Agents with LangGraph and LangSmith →

Part 5 turns the verified agent loop into a team of parallel specialists. Part 6 adds the controls needed to keep that team running across restarts, approvals, graph changes, and releases.

Together, they cover:

• Parallel specialist workers with scoped, read-only tools

• Reducer-backed shared state, deterministic merging, and independent review

• Postgres checkpoints, expiring approvals, and a queryable audit trail

• Graph versioning, LangSmith evaluation gates, and an authenticated streaming API

Why care?

Building an agent that can call tools is one engineering task. Operating several agents as one reliable system requires a broader set of skills.

Once workers run in parallel, shared state becomes a concurrency problem. Each worker needs private loop state, concurrent writes need reducers, and the final merge must remain deterministic. A failed worker must become an explicit partial result instead of blocking the entire investigation.

Durability introduces a different class of problems. A run may pause for approval, lose its process, and resume elsewhere hours later.

That requires synchronous checkpoints, persistent interrupts, expiring approvals, and an atomic decision so approval, rejection, and expiry cannot resume the same thread twice.

Deployment adds change-management and security concerns. A paused thread created on yesterday’s graph must remain compatible with today’s code.

Prompt changes need behavioral evaluations because deterministic unit tests cannot detect every regression. Identity must come from authentication rather than the request body, and tenant boundaries must hold across every endpoint.

These are the same system-design concerns found in backend, platform, and agent-infrastructure roles: concurrency semantics, durable workflows, authorization, observability, compatibility, and release safety.

For a developer who already understands tool calling and basic LangGraph workflows, Parts 5 and 6 clarify the difference between framework familiarity and production system design.

Being able to explain why a state key needs a reducer, how a paused run survives a deployment, or why an evaluation gate catches failures that unit tests miss is a stronger signal than another agent demo.

• Learn to orchestrate parallel specialists safely in Part 5 →

• Learn to make agent runs durable and deployable in Part 6 →

👉 Over to you: Which production guarantee is still missing from the agent systems being built today?

Jev for Agent Memory, clearly explained:

An agent-memory system often passes conversations, tool results, and task outcomes through an extractor.

The extractor identifies possible memories such as user preferences, facts, and procedures.

Consider a travel assistant booking a flight.

The user says:

• I usually prefer an aisle seat.

• For this flight, book a window seat.

• The flight is delayed by 25 minutes.

An extractor may identify all three as facts. Saving all three to long-term memory would be a mistake.

• The aisle-seat preference could help with future bookings.

• The window-seat request applies only to this flight. Saving it as the user’s latest seat preference could make the next booking wrong.

• And the delay is useful during the trip. It has no value once the flight is over.

The extractor can identify what the user said, but it does not necessarily know how long the fact should live or where it should apply.

Jev can run here before the application writes anything to memory.

For each candidate, the application gives Jev the proposed fact, its source evidence, and the current user, trip, and session state.

The application can ask Jev typed questions such as:

• Does the evidence support this fact?

• Does it apply outside the current trip?

• Should it be stored for the user, the trip, or only the current session?

• How likely is incorrect reuse to cause a bad booking?

Jev returns choices, scores, and probabilities.

Application code then applies explicit rules. For example:

↳ Save the aisle-seat preference for the user.

↳ Attach the window-seat request only to this trip.

↳ Keep the delay in the current session and remove it when the trip ends.

Only the approved candidate memories are sent to Graphiti as episodes.

Graphiti keeps the original episode, extracts entities and relationships, resolves new facts against the existing graph, and records when facts become valid or invalid.

GitHub repo: https://github.com/getzep/graphiti

(don’t forget to star it ⭐ )

In code, Jev runs before the call that adds an episode to Graphiti.

If a candidate passes the threshold, the app writes it to Graphiti with the appropriate scope and source. If it fails, the application can keep it in the current session or discard it.

Graphiti still handles ingestion, entity resolution, temporal updates, and retrieval. Jev only evaluates the proposed memory before that ingestion begins.

This way, fewer temporary instructions and expired updates enter the set of facts that later agent runs can retrieve.

The diagram shows the write flow:

• Agent activity produces candidate memories.

• Jev evaluates each candidate using its evidence and scope.

• Application rules decide whether and where to store it.

• Graphiti maintains the accepted memory.

That said, Jev becomes more useful when the same typed decisions sit inside a complete agent loop. Parts 3 and 4 of our Agent Engineering course implement that system end to end:

• Part 3: Build routing, guardrails, and approval controls with LangGraph and Jev →

• Part 4: Add planning, verification, replanning, and stopping logic →

​The AI engineering master stack for 2026!​

We prepared this AI engineering master stack that covers ten layers from the model itself to running it safely in production:

Here’s what each layer comprises:

1) ​Foundations​ set how the model represents input through tokens, embeddings, transformer, attention, context window, positional encoding, and mixture of experts.

2) ​Model behavior​ covers how a trained model responds through pretraining, post-training, sampling, temperature, reasoning models, multimodality, and test-time compute.

3) ​Prompt engineering​ shapes the output through the prompt alone using system prompts, few-shot, chain-of-thought, structured outputs, prompt caching, self-consistency, and meta-prompting.

4) ​Retrieval​ feeds the model data it was never trained on through chunking, vector databases, hybrid search, reranking, retrieval eval, query rewriting, and GraphRAG.

5) ​Agents​ let the model take actions instead of only answering through function calling, ReAct, planning, reflection, multi-agent, computer use, and human-in-the-loop.

6) ​Context engineering​ controls what fills the context window across steps through context management, compaction, memory, MCP, agent harness, just-in-time retrieval, and structured note-taking.

7) ​Fine-tuning​ changes the weights when prompting and context fall short through SFT, LoRA, RLHF, DPO, distillation, GRPO, and RLVR.

8) ​Inference optimization​ makes the model cheap and fast to serve through quantization, KV cache, batching, speculative decoding, vLLM serving, FlashAttention, and PagedAttention.

9) ​Deployment​ turns models into services through serving engines, model routing, semantic caching, streaming, load balancing, autoscaling, fallbacks, and on-device inference.

10) ​Evaluation​ measures whether the system is actually correct through benchmarks, LLM-as-judge, golden datasets, hallucination detection, regression tests, trajectory evaluation, and red teaming.

11-12) ​LLMOps and safety​ keep the system reliable and safe in production through observability, cost tracking, guardrails, PII redaction, feedback loops, prompt injection defense, and model routing.

13) ​Frontier​ patterns include decision models, harness engineering, loop engineering, recursive and contrastive LMs, verifier engineering, WebMCP, and forkable agent runs.

This grid above is the overview, but each layer is deep enough to be its own field with dedicated tooling.

We have covered every bit of it in the LLMOps course, starting from fundamentals to production:

• ​Read Part 1 on fundamentals of LLMOps here →​

• ​Read Part 2 on understanding the core building blocks of LLMs →​

• ​Read Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixture-of-experts, and the fundamentals of pretraining and fine-tuning →​

• ​Read Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applications →​

• ​Read Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and different prompting techniques →​

• ​Read Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more →​

• ​Read Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techniques for building high-signal inputs →​

• ​Read Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic context injection, and common failure modes in agentic applications →​

• ​Read Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong understanding of the fundamental concepts →​

• ​Read Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for evaluation of LLM apps →​

• ​Read Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming →​

• ​Read Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO →​

• ​Read Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model parallelism →​

• ​Read Part 14 on the fundamentals of LLM serving, including API-based access, inference with vLLM, and practical decisions.​

👉 Over to you: What else would you add to the master tree?

Good day!

• ​4x faster LLM inference engine than llama.cpp.

• ​A model decision is only as safe as the system that executes it.

• How LLM inference uses GPU memory.

4x faster LLM inference engine than llama.cpp:

We ran Qwen3.5 9B on a base M5 MacBook Pro using three local inference engines, one at a time.

• llama.cpp reached 22.0 tokens/s

• MLX reached 25.1 tokens/s

• Uzu reached 92.1 tokens/s

This frame shows all three runs side by side:

The first two results do not mean llama.cpp or MLX are poorly optimized. Both are approaching a physical limit of ordinary local decoding.

Generating each new token normally requires another pass through the model. That pass streams roughly 5.2 GB of weights through memory.

A base M5 provides about 153 GB/s of memory bandwidth.

At one token per pass, the theoretical ceiling is roughly 153 ÷ 5.2, or 29 tokens/s, however fast the GPU computes.

Uzu gets past that ceiling by optimizing speculative decoding for Apple Silicon.

Speculative decoding uses a smaller drafting system to propose several future tokens. The full 9B model then checks those proposals together instead of generating every token through a separate pass.

Uzu pushes this further in two ways.

• It drafts up to 16 positions and uses Weaver to turn independent guesses into coherent sequences.

• It verifies several possible sequences as a tree, then commits state only for the path the full model accepts.

The 9B model remains in control of the final output. Uzu simply gives it more useful work to approve in each pass.

In our run, every pass of the 9B model produced 7.5 output tokens on average.

Ordinary decoding produces exactly 1.

That is how Uzu reached 92.1 tokens/s on hardware whose one-token-per-pass ceiling is roughly 29.

The rest of the engine is designed around this path too. Uzu ships its own quantized checkpoints, and on M5 chips its Metal kernels run verification through the GPU’s accelerated int8 matrix path.

It is an open-source inference engine from Mirai, with bindings for Rust, Swift, Python, and TypeScript.

The repository is here: https://github.com/trymirai/uzu

(don’t forget to star 🌟)

A model decision is only as safe as the system that executes it

Jev reached nearly 13% of paid teams on Vercel AI Gateway in its first 24 hours, more than double the share for any previous model launch.

Unlike other models, Jev does not write an open-ended response, and instead selects from a defined set of choices. You provide the choices and the current application state, and it returns a selected choice and a probability for each option.

The interesting part here starts after that response returns. For an incident-response agent, that could mean deciding whether to page a team, gathering more evidence, or handing off control to a human.

We used Claude Code to build the agent. It produced 30 files and 3,383 lines covering the model client, policy engine, agent loop, tools, API, dashboard, fixtures, and tests.

The application ran, all 46 tests passed, and the feature looked complete at first glance. Before merging, we still needed to review how those pieces worked together.

That’s where CodeRabbit comes in. As the governance and control layer for agentic software development, it helps developers understand and validate what their agents build so they can ship with confidence.

It organizes a large pull request into its logical architecture, identifies the paths that deserve attention, and reviews behavior and security across the full change rather than one file at a time.

Before looking at what it found, let’s look at the agent Claude Code built.

The incident-response agent

Incident response gives the model a constrained set of choices, each with consequences.

Paging the wrong team interrupts people and can delay the actual response.

We wanted the application to record each dev decision, its confidence, the available evidence, and any handoff to a human.

We gave Claude Code a single prompt with those requirements and let it build the application.

The generated dashboard showed every Jev answer, its probabilities, the tools executed, and the final outcome.

That makes the boundary between model judgment and application policy visible during the run.

The agent starts with an alert. Jev receives the current incident state and answers six questions in one request:

• Which team owns the incident?

• How severe was the incident?

• Whether customers are affected?

• Whether more evidence is needed?

• Whether a human should take over

• What should the agent should do next (next_action)?

The last answer, next_action is restricted to six values: inspect logs, check a deployment, check service health, page a team, request human review, or stop.

When the agent gathers evidence, it adds the result to the incident state and sends the updated state back to Jev.

It reads from local JSON fixtures, which keeps each run reproducible and prevents it from touching a production service.

Every pass through this loop reaches the same boundary, where Jev proposes next_action and the application decides whether that action can become a tool call.

The policy gate

The policy uses the probability assigned to the selected action:

• At 0.80 or above, it permits the action.

• From 0.55 to 0.79, it gathers more evidence.

• Below 0.55, it transfers control to a human.

A page_team recommendation also requires confirmation that customers are affected.

We ran three fixture-backed incidents through the dashboard. The payments outage made the boundary easiest to see.

Jev selected page_team with 0.62 confidence, so the policy refused to page and ran inspect_logs instead.

It then evaluated the updated evidence and returned 0.50 confidence, which fell below the human-review threshold. The run stopped without executing another tool.

The dashboard and 46 tests covered what the application did with expected responses. They did not cover whether next_action and its confidence would stay paired in every response the application could receive.

Tracing the decision through the pull request

Reviewing the change meant opening 30 diffs and reading them one by one.

Each diff was understandable on its own.

The problem was that the safety guarantee did not live in any single file. It depended on how the selected action and its probability moved between them.

We opened the same PR in CodeRabbit. It reviewed the change automatically, rated the merge risk as High, and posted six actionable comments ranked by severity.

CodeRabbit’s Change Stack grouped the 30 changed files into six logical layers based on their dependencies, giving me a path from the Jev response contract to the resulting tool call.

The first four layers covered the code that ran during an incident including the decision, the confidence check, and the loop that chose what to do next.

The remaining two covered the API, the dashboard that exposed the result, and the setup files that supported it.

That grouping gave me a reading order that matched the application’s runtime dependencies.

We could start in decisions.py, where the selected action was paired with its probability, then follow the value into the policy and agent loop.

CodeRabbit attached each finding to the relevant layer and ranked them by severity. The first one we opened was in decisions.py, before the confidence value reached the policy.

The confidence bug 46 tests missed

In decisions.py, when the selected choice was absent from the returned probability map, the code used the largest probability from any option as the selected choice's confidence.

That fallback can combine two valid values into an invalid decision.

choice = page_team
page_team probability = missing
inspect_logs probability = 0.90

normalized confidence for page_team = 0.90

Both values are valid on their own, but their association is false. The policy now sees a high-confidence paging decision even though Jev never assigned page_team that score.

The dashboard did not expose the failure because its live responses always included the selected option’s probability. The tests missed it for the same reason.

Every fixture preserved the expected relationship between choice and probabilities.

The safe behavior is to assign zero confidence or reject the response and route the run to human review. CodeRabbit flagged the untested response and explained how the fallback could authorize a tool call.

Tracing security from the API to the dashboard

The confidence bug was one of six findings. The others tracked where external input entered the system and what it could reach from there.

Change Stack’s security blast-radius view mapped four findings across five changed components.

The highest-impact path began with incident text submitted through the Run API. The application retained that text in a shared run state.

The dashboard later inserted it into HTML without encoding, creating a stored XSS vulnerability. Anyone able to reach the API could store content that executed when another user viewed the run.

Three other boundaries had their own exposure:

• The API exposed shared incidents and evaluation records without authorization or ownership checks.

• The Jev request URL could be changed without restricting its scheme or destination.

• Fixture mode could fall through to a live request when an incident did not match a fixture.

The vulnerable line was in the dashboard, but the exploit path began at the API and crossed the shared state. The blast-radius view connected those components instead of treating the rendering line as an isolated defect.

Validating the agent-generated change

Generating code doesn't make a change ready to merge. Agentic Change Management helps teams understand how the change fits the architecture, identify its riskiest paths, and validate behavior and security across the full change.

That control continues after merge, helping keep the codebase secure and maintainable.

In the example, CodeRabbit reconstructed the architecture and reading order, automated review found the confidence failure, and the security blast-radius view followed external input across the API, shared state, and dashboard.

The agent ran successfully, and all 46 tests passed. Review still found an untested response that could bypass the intended confidence gate, along with a security flaw that crossed the API, shared state, and dashboard.

We built the agent in one session, then reviewed how it worked and checked for security flaws before deciding whether it was ready to merge.

You can start a free 14-day CodeRabbit trial, with no credit card required.

Ship agent-built code with confidence →

Thanks to CodeRabbit for partnering today!

How LLM inference uses GPU memory

Loading an LLM does not guarantee inference!

The model may load successfully and still run out of GPU memory when the first requests arrive.

This happens because model weights are only one of four VRAM allocations:

1) Model weights

The parameters are loaded into VRAM before inference begins.

Ignoring quantization metadata and implementation overhead, a 70B parameter model requires approximately:

→ 140 GB in FP16 or BF16

→ 70 GB in INT8

→ 35 GB in INT4

This allocation remains mostly constant during inference.

Quantization reduces it by storing each parameter with fewer bits. However, kernels may use higher-precision values for some operations.

2) KV cache

Each Transformer layer generates KV tensors for every processed token.

The inference engine retains these tensors so new tokens can attend to earlier tokens without recomputing their key and value projections.

The approximate KV cache memory per token is:

2 × layers × KV heads × head dimension × bytes per value

The factor of two represents the key and value tensors.

This allocation increases with context length and the number of active requests.

Grouped-query attention and multi-query attention reduce it by sharing fewer key-value heads across multiple query heads.

3) Activations and workspace

A forward pass creates intermediate tensors for attention, MLP operations, normalization, and matrix multiplication.

GPU kernels may also require temporary workspace for computation.

The inference engine reuses much of this memory across layers and generation steps.

Its peak size depends on the model architecture, number of tokens processed together, selected kernels, and serving configuration.

Prefill processes multiple prompt tokens in parallel.

Decode usually processes one new token per active request.

The two phases therefore have different activation and workspace requirements.

4) Runtime overhead

CUDA contexts, loaded kernels, memory allocators, metadata, communication buffers, and serving-engine state also consume VRAM.

The amount varies across runtimes and deployment configurations.

Memory fragmentation can also leave enough total free VRAM but no sufficiently large contiguous region for a new allocation.

The total memory requirement is:

model weights + KV cache + peak activations and workspace + runtime overhead

• Model weights remain mostly constant.

• KV cache memory increases with context length and concurrency.

• Activation and workspace usage changes during execution.

• Runtime overhead depends on the software stack.

This is why model size alone cannot determine whether an LLM can serve a given workload.

The visual below shows when each allocation is created, how long it remains in memory, and which workload parameters increase it.

Good day!

• Finally, an OpenRouter for agent harnesses!

• 6 parallelism strategies for LLMs in production.

• ​Schema-guided agent memory.

Finally, an OpenRouter for agent harnesses!

Devs just open-sourced a plug-and-play infrastructure layer that lets you run any harness under a single interface, like:

• Codex

• Hermes

• Claude code

• DeepSeek Harness

• System One, powered by Jev

• And 9 more agent harnesses

This means you can bring Jev into the same product that already uses Codex, Claude Code, or another supported harness, without writing another implementation for sessions, streaming, files, cancellation, and failure handling.

Here’s the repo: github.com/HarnessRouter/harnessrouter

(don’t forget to star it ⭐ )

The harnesses run locally, and the Unified Harness Protocol (UHP) defines the common task interface with an OpenAI Responses-compatible API.

6 parallelism strategies for LLMs in production

Running a model across four GPUs does not merge their memory into one shared pool.

Each GPU has its own memory, so data stored on another GPU must cross the interconnect before a layer can use it.

This is the central trade-off behind parallelism. It removes local memory or compute limits but adds communication overhead.

The visual below explains the six common strategies for parallelizing LLM inference:

• Data parallelism runs complete model replicas on separate GPU groups. It increases throughput, but cannot make an oversized model fit.

• Tensor parallelism splits matrix operations inside each transformer layer. It reduces weight memory per GPU, but the GPUs must combine partial results throughout the layer stack.

• Pipeline parallelism assigns layer groups to different GPUs. Activations move between stages, while insufficient micro-batching leaves some stages idle.

• Context parallelism splits sequence positions in two ways. Prefill context parallelism divides a long input across GPUs to target lower TTFT. Decode context parallelism divides the stored KV history to increase cache capacity, but every decoder layer must merge partial attention results.

• Expert parallelism distributes MoE experts across GPUs. Token routing and uneven expert load add network cost and idle time.

• Hybrid parallelism combines multiple strategies in one deployment. For example, tensor parallelism can operate within a node, pipeline parallelism across nodes, and data parallelism across model replicas. This matches each communication pattern to the available links, but adds more coordination paths.

Production deployments often combine methods, but each one should fix a measured bottleneck.

• If the weights do not fit, use tensor or pipeline parallelism.

• If long-prompt prefill or KV-cache capacity is the limit, use the corresponding form of context parallelism.

• MoE models may need expert parallelism to place expert weights.

• If one replica meets its latency target but requests still queue, add data-parallel replicas.

Keep tensor parallelism on the fastest links. After each change, measure memory, TTFT, TPOT, and other metrics again.

Add another method only when measurements reveal another bottleneck.

We wrote a detailed article explaining how each strategy works, where communication occurs, and how to choose a layout for a real deployment.

Read it here →

If you want to learn the full LLMOps lifecycle, we have covered every bit of it in the LLMOps course, starting from fundamentals to production:

• Read Part 1 on fundamentals of LLMOps here →

• Read Part 2 on understanding the core building blocks of LLMs →

• Read Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixture-of-experts, and the fundamentals of pretraining and fine-tuning →

• Read Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applications →

• Read Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and different prompting techniques →

• Read Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more →

• Read Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techniques for building high-signal inputs →

• Read Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic context injection, and common failure modes in agentic applications →

• Read Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong understanding of the fundamental concepts →

• Read Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for evaluation of LLM apps →

• Read Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming →

• Read Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO →

• Read Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model parallelism →

• Read Part 14 on the fundamentals of LLM serving, including API-based access, inference with vLLM, and practical decisions →

​Schema-guided agent memory​

Most agent memory systems optimize for recall. The harder problem is what to forget, or more precisely, what to never store in the first place.

The default agent memory pipeline hands an LLM raw text and asks it to extract entities and relationships. The model decides the types, labels, and attributes all on its own.

The result is a knowledge graph that behaves like an expensive vector store. Entity types collapse into generic labels. Relationships flatten into a single “RELATES_TO.”

The graph has the data, but no query can reach it with precision.

This is not a retrieval problem but rather a structure problem. And the fix is the same pattern that already works everywhere else in the AI stack, i.e., constrain the output space before generation, not after.

• Entities define what the agent is allowed to remember. Pydantic models with typed fields and descriptive docstrings replace the LLM’s guesswork with domain vocabulary it was never trained on.

• Edges define how things connect. Source/target constraints on relationship types mean the graph can only form valid connections. If your schema has no edge connecting Project to Competitor, that relationship cannot exist in memory.

• Temporal resolution handles what was true versus what is true. Fact resolution invalidates outdated edges while preserving history, so the graph never silently serves a stale state.

The schema guides extraction at two points in the pipeline (entity extraction and fact extraction) while resolution and temporal processing run automatically downstream.

You define what to look for. The system handles deduplication, contradiction detection, and time-windowing without additional configuration.

A useful constraint is to have just 10 entity types, 10 edge types, and 10 fields per type.

That forces you to model the 80% that matters rather than attempting completeness. Start with 3-4 of each and expand only when retrieval fails.

​Zep Graphiti​ does all of this as a fully open-source temporal knowledge graph library.

It includes Pydantic-based ontology definition, schema-guided extraction, entity resolution, fact resolution, and temporal windowing.

If you are building agent memory with any kind of domain specificity, schemas are a must.

​Here’s the GitHub repo →​

(don’t forget to star 🌟)

​We also covered this topic in more depth (with code) in this article →​

Good day!

For a math problem, the training record only needs the question and its final answer. The model samples several solutions. A reward function checks each result. GRPO then increases the probability of responses that score higher within the group.

This gives us a practical way to customize reasoning behavior without curating a reasoning trace for every example. It also introduces a hard engineering constraint. The model will optimize exactly what the verifier rewards, including any shortcuts hidden in that verifier.

We will test this on Qwen3 0.6B using serverless RLVR in Amazon SageMaker AI. The target is higher exact-match accuracy on a fixed set of unseen arithmetic problems.

One evaluation set stays fixed throughout the experiment. We will test the reward function, run one customization job, repeat the evaluation, and deploy the resulting model to a SageMaker AI real-time endpoint.

Training against an executable reward

Supervised fine-tuning (SFT) learns from target completions. Each training record contains an input and the response that the model should imitate. This works well when high-quality completions already exist.

RLVR needs a different asset. It needs a verifier that can score generated responses. The training set can contain a problem and its reference answer without prescribing the full reasoning trace.

For a math problem, the verifier can extract the final number. For generated code, it can run unit tests. For structured extraction, it can validate a schema and compare required fields.

Reasoning traces are expensive to curate. When correctness has a programmatic definition, a verifier can replace them.

RLVR does not remove supervision. It moves supervision into the reward function. That reward function becomes part of the task specification.

A verifier only measures what its code expresses. It cannot distinguish genuine reasoning from a shortcut unless the test exposes that shortcut.

Choosing the optimization signal

Proximal Policy Optimization (PPO) is a general policy-gradient method. RLHF systems often combine it with a learned reward model and a separate value model. That stack is flexible, but it adds training and operational complexity.

Group Relative Policy Optimization (GRPO) removes the separate value model. It samples a group of responses for one prompt. Their relative rewards provide the baseline used during the policy update. The DeepSeekMath paper introduced GRPO for mathematical reasoning.

Direct Preference Optimization (DPO) avoids online reinforcement learning. It learns directly from preferred and rejected response pairs. DPO is useful when pairwise preference data already captures the desired behavior.

RLVR is not universally better than these methods. It fits tasks with reliable checks. A weak verifier can produce a model that scores well while failing the real task.

RLVR replaces ideal traces with executable feedback, which puts more responsibility on reward design.

Experiment design and evaluation contract

The experiment uses the custom-reward RLVR path from the Amazon SageMaker AI serverless model customization workshop. AWS configures this lab around Qwen3 0.6B and 180 GSM8K samples.

The setup is fairly direct. Just deploy the provided Studio environment, clone the repository, and run five Jupyter notebooks in sequence.

We will preserve the workshop’s Qwen3 model and training path while treating the evaluation protocol as our own experiment. Qwen3 is an open-weight model, so this workflow is not limited to Amazon-developed models.

You can find the model on HF here →

Opening SageMaker Studio

You do not create a separate SageMaker account. You need an AWS account with permission to create SageMaker, IAM, VPC, S3, Lambda, and MLflow resources.

Sign in to the AWS Management Console and open the workshop’s own-account setup page. Launch its Deploy SageMaker Studio domain CloudFormation template.

The template creates the networking environment, SageMaker execution role, Studio domain, user profile, and managed MLflow tracking server. It defaults to us-west-2. If you choose another region, update the Availability Zone parameters before creating the stack.

Wait until the CloudFormation stack reaches CREATE_COMPLETE. Then open the workspace through the AWS console:

1. Search for SageMaker AI.

2. Select Domains from the left navigation.

3. Open the Studio domain created by CloudFormation.

4. Open User profiles.

5. Open the menu beside your user profile and select Studio.

6. Inside Studio, select the JupyterLab icon.

7. Select Open to launch the JupyterLab space.

Once JupyterLab loads, open a terminal from its launcher. Clone the workshop repository and grab the SageMaker AI Agent Skills while setting up the workspace.

git clone https://github.com/aws-samples/generative-ai-on-amazon-sagemaker.git
git clone --depth 1 https://github.com/awslabs/agent-plugins.git

cd generative-ai-on-amazon-sagemaker/workshops/serverless-model-customization-with-sagemaker-ai

The SageMaker AI Agent Skills (GitHub repo) provide reusable guidance for planning customization jobs, preparing data, training, evaluation, and deployment.

They work with the coding agent you already use, including Claude Code, Codex, and Kiro.

The skills can help generate or revise the notebooks, but the held-out split, reward implementation, and success metric still need to be reviewed by the engineer running the experiment.

Open the five notebooks under lab-3b-custom-reward-function-rlvr in order:

Select Python 3 (ipykernel) when JupyterLab asks for a kernel. Each notebook consumes resources created by the preceding notebook, so run them in sequence.

The workshop environment already contains the SageMaker SDK. For a fresh environment, pin the versions used for this walkthrough.

pip install "sagemaker==3.23.0" "datasets==5.0.1" "scikit-learn==1.9.1"

The notebooks reuse one session, role, bucket, and project prefix. Define them once before creating any resources.

import boto3
import sagemaker

sess = sagemaker.Session()
region = sess.boto_region_name
role = sagemaker.get_execution_role()
bucket_name = sess.default_bucket()

project_prefix = "gsm8k-custom-reward-rlvr"
reward_function_name = "gsm8k-rlvr-rf"
output_path = f"s3://{bucket_name}/{project_prefix}/output"
sm_client = boto3.client("sagemaker", region_name=region)

These variables connect every later cell. Keep the prefix unchanged across all five notebooks.

Notebook 1: preparing the training records

The workshop streams 180 GSM8K examples. It preserves each question, extracts the reference number, and assigns a stable identifier.

import re
from datasets import Dataset, load_dataset

stream = load_dataset("openai/gsm8k", "main", split="train", streaming=True)
raw = Dataset.from_generator(lambda: stream.take(180), features=stream.features)

def prepare(sample, index):
    answer = re.search(r"####\s*(.+)", sample["answer"]).group(1)
    answer = answer.strip().replace(",", "")
    return {
        "id": f"gsm8k-train-{index:06d}",
        "data_source": "openai/gsm8k",
        "prompt": [{
            "role": "user",
            "content": sample["question"]
            + ' Let\'s think step by step and output the final answer after "####".',
        }],
        "ability": "math",
        "reward_model": {"ground_truth": answer, "style": "custom"},
        "reference_answer": answer,
        "extra_info": {"question": sample["question"], "answer": sample["answer"]},
    }

records = [prepare(sample, index) for index, sample in enumerate(raw)]

The top-level reference_answer reaches the custom evaluator during training. The stable id lets the evaluator return one score for each rollout.

For the short workshop job, split the 180 records once with seed 42.

This produces 129 training records, 36 validation records, and 15 held-out records. We use that split to follow the managed workflow, but not for the larger result reported in this article.

import json
from pathlib import Path
from sklearn.model_selection import train_test_split

train_records, val_records = train_test_split(records, test_size=0.2, random_state=42)
train_records, test_records = train_test_split(
    train_records, test_size=0.1, random_state=42
)

def write_jsonl(path, rows):
    with Path(path).open("w", encoding="utf-8") as handle:
        for row in rows:
            handle.write(json.dumps(row) + "\n")

for name, rows in {
    "train": train_records,
    "val": val_records,
    "test": test_records,
}.items():
    write_jsonl(f"{name}.jsonl", rows)

Upload all three files, then register them as RLVR datasets.

from sagemaker.ai_registry.dataset import CustomizationTechnique, DataSet

def register_split(name):
    s3_uri = sess.upload_data(
        f"{name}.jsonl",
        bucket=bucket_name,
        key_prefix=f"{project_prefix}/datasets/{name}",
    )
    return DataSet.create(
        name=f"{project_prefix}-{name}",
        source=s3_uri,
        customization_technique=CustomizationTechnique.RLVR,
        wait=True,
    )

dataset_train = register_split("train")
dataset_val = register_split("val")
dataset_test = register_split("test")

wait=True blocks until each registry asset becomes ready. Do not launch training while a dataset is still being created.

The run should record these fields before training:

• Model identifier and revision

• Training sample count

• Evaluation sample count

• Random seed

• Generation temperature and token limit

• Exact-match parsing rule

• SageMaker job configuration

The primary metric is exact-match accuracy. The secondary metric is final-answer parseability. We will also record response length because reward shaping may change how much text the model emits.

The base and customized models must use identical evaluation prompts. They must also use identical decoding settings. Otherwise, the comparison mixes training gains with inference changes.

The workshop evaluates the customized model with the MATH benchmark. That adds a useful out-of-distribution check after training on GSM8K. For our main claim, however, we should use one fixed held-out set and report its exact size.

Fix the evaluation contract before training. The job succeeds only when the same evaluator measures a real gain.

Notebook 2: reward function design and failure modes

SageMaker AI accepts a Python reward function through an Evaluator resource. The handler receives a list containing one generated conversation and its reference fields. It returns one aggregate reward per record plus named metrics for observability.

Our custom reward combines four signals:

• Numeric correctness contributes 0.6

• Answer parseability contributes 0.2

• Required final-answer format contributes 0.1

• Presence of non-trivial reasoning contributes 0.1

The offline implementation is demonstrated below:

def score_record(record, index=0):
    response = _extract_model_response(record)
    predicted, path = extract_final_answer(response)
    expected = _to_decimal(_extract_reference_answer(record))

    correctness = float(predicted is not None and predicted == expected)
    parseable = float(predicted is not None)
    formatted = float(path == "marked")
    reasoning = float(len(response.split("####", 1)[0].split()) >= 8)

    reward = round(
        0.6 * correctness
        + 0.2 * parseable
        + 0.1 * formatted
        + 0.1 * reasoning,
        4,
    )
    return {
        "id": str(record.get("id") or index),
        "aggregate_reward_score": reward,
        "metrics_list": [
            {"name": "numeric_correctness", "value": correctness, "type": "Reward"},
            {"name": "answer_parseable", "value": parseable, "type": "Reward"},
            {"name": "final_answer_format", "value": formatted, "type": "Metric"},
            {"name": "reasoning_present", "value": reasoning, "type": "Metric"},
        ],
    }

This is the central reward-design decision, i.e., four observable signals must collapse into one optimization target.

Correctness receives most of the reward. The remaining components provide a learning signal when early samples are mostly wrong.

This shaping has a clear risk. A model can earn 0.4 without producing the correct number. It only needs a parseable answer, the requested marker, and enough text before that marker.

That risk suggests two checks. First, graph correctness and formatting separately. Second, inspect whether response length rises faster than accuracy.

The local test suite covers correct answers, wrong answers, comma-separated numbers, decimal equivalence, missing markers, malformed batches, and absent references. These tests validate the scorer. They do not validate the learned model.

Reward shaping should guide exploration without replacing the actual objective. When proxy rewards dominate, training can improve the dashboard while leaving exact-match accuracy unchanged.

In summary, the reward function is executable product logic. It deserves tests, version control, and metric-level monitoring.

Notebook 3: RLVR training pipeline in SageMaker AI

The custom reward lab registers the Python scorer as a SageMaker AI Registry Reward Function Evaluator. The RLVR trainer then references that evaluator during customization.

The flow contains four resources:

1. The GSM8K-derived records live in Amazon S3.

2. The custom scorer becomes a versioned Evaluator.

3. The RLVR job trains LoRA adapters for Qwen3 0.6B.

4. MLflow records aggregate and component metrics.

Register the tested Python file as a versioned reward evaluator.

from sagemaker.ai_registry.air_constants import REWARD_FUNCTION
from sagemaker.ai_registry.evaluator import Evaluator

reward_evaluator = Evaluator.create(
    name=reward_function_name,
    type=REWARD_FUNCTION,
    source="reward_function.py",
    role=role,
    sagemaker_session=sess,
    wait=True,
)
print(reward_evaluator.arn)

The evaluator ARN is the link between our Python scorer and the serverless training job. Keep the evaluator name short because SageMaker creates an underlying Lambda resource from it.

Next, retrieve the registered datasets and configure RLVRTrainer.

from sagemaker.ai_registry.dataset import DataSet
from botocore.exceptions import ClientError
from sagemaker.core.resources import ModelPackageGroup
from sagemaker.train.rlvr_trainer import RLVRTrainer

base_model_id = "huggingface-reasoning-qwen3-06b"
training_dataset = DataSet.get(name=f"{project_prefix}-train")
validation_dataset = DataSet.get(name=f"{project_prefix}-val")
group_name = f"{base_model_id}-custom-rw-rlvr"
try:
    model_package_group = ModelPackageGroup.get(group_name)
except ClientError:
    model_package_group = ModelPackageGroup.create(
        model_package_group_name=group_name,
        model_package_group_description="Qwen 3 0.6B custom-reward RLVR runs",
    )

trainer = RLVRTrainer(
    model=base_model_id,
    model_package_group=model_package_group,
    custom_reward_function=reward_evaluator,
    training_dataset=training_dataset,
    validation_dataset=validation_dataset,
    s3_output_path=output_path,
    mlflow_experiment_name=project_prefix,
    base_job_name=project_prefix,
    sagemaker_session=sess,
    accept_eula=True,
    role=role,
)

trainer.hyperparameters.max_epochs = 1
trainer.hyperparameters.learning_rate = 5e-6
trainer.hyperparameters.global_batch_size = 128
trainer.hyperparameters.max_prompt_length = 1024

training_job = trainer.train(wait=False)
print(training_job.training_job_name)

custom_reward_function is the only structural change from the built-in reward path. SageMaker chooses and releases the training compute. The notebook does not specify an instance type.

The serverless path removes instance selection from the training configuration. The SageMaker model customization documentation describes UI, Python SDK, and agent-guided interfaces for this workflow.

The workshop estimates 15 to 20 minutes for its fine-tuning section. That estimate is not our runtime. Record the actual wall-clock duration from the job metadata.

Serverless RLVR removes cluster management. Experiment design and reward validation remain the engineer’s responsibility.

Notebook 4: post-training evaluation

The post-training evaluation must reuse the baseline pipeline. Only the model endpoint changes.

The workshop also runs the separate MATH benchmark. Retrieve the newest model package, then submit the managed evaluation.

from sagemaker.train.evaluate import BenchMarkEvaluator, get_benchmarks

group_name = "huggingface-reasoning-qwen3-06b-custom-rw-rlvr"
packages = sm_client.list_model_packages(
    ModelPackageGroupName=group_name,
    SortBy="CreationTime",
    SortOrder="Descending",
    MaxResults=1,
)
model_arn = packages["ModelPackageSummaryList"][0]["ModelPackageArn"]

Benchmark = get_benchmarks()
evaluator = BenchMarkEvaluator(
    benchmark=Benchmark.MATH,
    model=model_arn,
    model_package_group=group_name,
    base_eval_name="custom-reward-rlvr",
    s3_output_path=output_path,
    evaluate_base_model=False,
    role=role,
    sagemaker_session=sess,
)

execution = evaluator.evaluate()
execution.wait()
print(execution.show_results())

execution.show_results() is the result produced by this notebook.

Record the customized model’s MATH score and the evaluation-job status from that output.

This notebook does not evaluate the base model because evaluate_base_model=False, so one Lab 3b execution cannot support a before-and-after accuracy claim by itself.

AWS includes a paired comparison in the standard RLVR lab. Its checked-in base-model artifact reports 0.406 MATH exact match. The corresponding fine-tuned artifact reports 0.507.

The exact values may vary across runs. The dataset split, seed, generation settings, and reward implementation should therefore ship beside the result.

Notebook 5: real-time endpoint validation

The final step deploys the customized model to a SageMaker real-time endpoint. This keeps the article on one deployment path and matches the custom-reward workshop.

Deployment should happen only after the evaluation artifact is saved. The endpoint test should reuse one held-out prompt that did not appear in training.

The workshop deploys the merged model weights to one ml.g5.xlarge endpoint. It uses the DJL LMI container with the vLLM backend.

from sagemaker.core import s3
from sagemaker.core.resources import ModelPackage

package = ModelPackage.get(model_arn)
model_root = package.inference_specification.containers[0]
merged_model_s3_uri = s3.s3_path_join(
    model_root.model_data_source.s3_data_source.s3_uri,
    "checkpoints",
    "hf_merged",
) + "/"

CONTAINER_VERSION = "0.36.0-lmi18.0.0-cu128"
image_uri = (
    f"763104351884.dkr.ecr.{region}.amazonaws.com/"
    f"djl-inference:{CONTAINER_VERSION}"
)

Create the endpoint before attaching the model through an inference component.

from sagemaker.core.resources import Endpoint, EndpointConfig
from sagemaker.core.shapes import ProductionVariant

endpoint_name = f"{project_prefix}-endpoint"
endpoint_config_name = f"{project_prefix}-config"

EndpointConfig.create(
    endpoint_config_name=endpoint_config_name,
    execution_role_arn=role,
    production_variants=[ProductionVariant(
        variant_name="AllTraffic",
        instance_type="ml.g5.xlarge",
        initial_instance_count=1,
        model_data_download_timeout_in_seconds=700,
        routing_config={"routing_strategy": "LEAST_OUTSTANDING_REQUESTS"},
    )],
)
endpoint = Endpoint.create(
    endpoint_name=endpoint_name,
    endpoint_config_name=endpoint_config_name,
)
endpoint.wait_for_status("InService")

Create the model with the same LMI settings used by the workshop. Then attach one GPU through an inference component.

from sagemaker.core.resources import InferenceComponent, Model
from sagemaker.core.shapes import (
    ContainerDefinition,
    InferenceComponentComputeResourceRequirements,
    InferenceComponentRuntimeConfig,
    InferenceComponentSpecification,
    ModelDataSource,
    S3ModelDataSource,
)

model_name = f"{project_prefix}-model"
ic_name = f"{project_prefix}-component"

model = Model.create(
    model_name=model_name,
    primary_container=ContainerDefinition(
        image=image_uri,
        model_data_source=ModelDataSource(
            s3_data_source=S3ModelDataSource(
                s3_uri=merged_model_s3_uri,
                s3_data_type="S3Prefix",
                compression_type="None",
            )
        ),
        environment={
            "SERVING_FAIL_FAST": "true",
            "OPTION_ASYNC_MODE": "true",
            "OPTION_ROLLING_BATCH": "disable",
            "OPTION_MAX_MODEL_LEN": "16384",
            "OPTION_TENSOR_PARALLEL_DEGREE": "max",
            "OPTION_ENTRYPOINT": "djl_python.lmi_vllm.vllm_async_service",
            "OPTION_TRUST_REMOTE_CODE": "true",
        },
    ),
    execution_role_arn=role,
)

component = InferenceComponent.create(
    inference_component_name=ic_name,
    endpoint_name=endpoint_name,
    variant_name="AllTraffic",
    specification=InferenceComponentSpecification(
        model_name=model_name,
        compute_resource_requirements=InferenceComponentComputeResourceRequirements(
            min_memory_required_in_mb=1024,
            number_of_accelerator_devices_required=1,
        ),
    ),
    runtime_config=InferenceComponentRuntimeConfig(copy_count=1),
    region=region,
)
component.wait_for_status("InService")

Now run one held-out problem through the endpoint. The workshop repository provides execute_inference and print_stream helpers for the streaming request.

prompt = "What is 25 * 4 + 10? Show your reasoning step by step."
stream = execute_inference(prompt, endpoint_name, ic_name, stream=True)
print_stream(stream)

For the side-by-side validation in SageMaker Studio JupyterLab, the serving configuration exposes both the base weights and the RLVR adapter. The notebook can therefore switch models without changing the prompt or decoding configuration.

base_endpoint_name = f"{project_prefix}-base-endpoint"
base_ic_name = f"{project_prefix}-base-component"

rlvr_endpoint_name = endpoint_name
rlvr_ic_name = ic_name

def stream_answer(prompt, endpoint_name, inference_component_name):
    stream = execute_inference(
        prompt,
        endpoint_name,
        inference_component_name,
        stream=True,
    )
    print_stream(stream)
    return stream

Invoke the untouched base model and then the adapter-backed model. Keep every other input fixed.

prompt = (
    "Natalia sold clips to 48 of her friends in April, then half as many "
    "in May. How many clips did she sell altogether? Show the calculation "
    "and end with #### followed by the numeric answer."
)

base_response = stream_answer(
    prompt,
    base_endpoint_name,
    base_ic_name,
)

rlvr_response = stream_answer(
    prompt,
    rlvr_endpoint_name,
    rlvr_ic_name,
)

The recording below sends the same prompt through both model routes with identical decoding settings. It shows the responses arriving token by token and keeps the time to first token and total generation time visible for each model:

We verified both model routes with streamed responses.

On a five-prompt serving smoke test

• the base route had a median time to first token of 0.571 seconds and a median total response time of 1.667 seconds.

• The RLVR route measured 0.529 seconds and 1.867 seconds respectively.

These numbers validate the streaming path. Five requests are not enough to characterize throughput under load.

Delete the endpoint after collecting the result unless it serves another workload. Real-time endpoints continue consuming provisioned capacity while active.

Delete inference resources in reverse creation order.

from sagemaker.core.resources import Endpoint, EndpointConfig, InferenceComponent, Model

InferenceComponent.get(inference_component_name=ic_name).delete()
Model.get(model_name=model_name).delete()
Endpoint.get(endpoint_name=endpoint_name).delete()
EndpointConfig.get(endpoint_config_name=endpoint_config_name).delete()

Engineering lessons from verifier-driven training

RLVR is useful when the target behavior has a cheap verifier. The reward can evaluate many sampled responses without requiring an ideal completion for each one.

That verifier is now part of the training specification. A loose parser, incomplete test suite, or exploitable scoring rule can reward the wrong behavior. The reward components therefore need tests, versioning, and separate monitoring.

The simplest designs keep one evaluator at the center. Training calls it for rewards. Held-out evaluation reuses the same parsing rules. The final endpoint test applies the same acceptance criteria once more.

Every accuracy claim should trace back to a fixed split, raw responses, and executable checks.

To run the complete workflow yourself, AWS provides a serverless model customization workshop for Amazon SageMaker AI.

It includes hands-on labs for SFT, DPO, RLVR, and RLAIF. The custom-reward RLVR lab walks through data preparation, evaluator registration, training, evaluation, deployment, and cleanup.

Good day, and thanks to AWS for partnering today!

An agent may need the current truth, exact original wording, the outcome of a session, or a pattern across several conversations. Those requests require different forms of context.

A useful memory system stores history once and reads it in six ways.

• Facts answer what changed and when. Each claim carries validity dates, so an older truth can be closed without being deleted.

• Entities answer “tell me about this thing.” A maintained description combines separate facts about a person, product, or system into one coherent briefing.

• Episodes answer “what did they actually say.” The original messages and records preserve exact wording, field names, error codes, and supporting evidence.

• Thread summaries answer “how did that session end.” They capture the outcome of an entire conversation, including distinctions such as service being restored while the root cause remains unresolved.

• Observations answer “why does this keep happening.” They surface evidence-backed patterns across conversations without presenting correlation as causation.

• User summaries provide context before a meaningful query exists. When a returning user only says “hi,” the agent can still begin with their role, setup, and unresolved issues.

The key insight is not to build six separate memory stores.

As the graphic shows, these are six ways to read the same temporal memory graph, with a real-world example. Each reshapes the underlying history for a different job.

This also changes how we should evaluate agent memory. Retrieving related text is not enough. The system must return the right form of context for the question being asked.

Graphiti implements this approach by assembling several context types into one fixed-budget block. Its underlying temporal graph framework, Graphiti, is open source and tracks how facts evolve while preserving their original sources.

Graphiti repository: https://github.com/getzep/graphiti

(don’t forget to star 🌟)

Train your own Jev-style decision model locally!

We just trained our own Jev-style Decision model locally!

With Qwen3.5 0.8B as the base, accuracy jumped from 37% to 65% in just 60 steps (10 minutes of training).

All on just 4GB of VRAM.

This video shows how you can do the same:

Chapters:

• 0:00 - Intro

• 0:50 - Set up Unsloth Studio

• 3:47 - Pick a model and dataset

• 5:19 - Train with LoRA

• 7:51 - Test it in the app

• 10:03 - Same workflow in code

• 12:38 - Train and evaluate (37% → 65%)

• 14:46 - Save and run inference

• 15:24 - Outro

Jev becomes more useful when the same typed decisions sit inside a complete agent loop. Parts 3 and 4 of our Agent Engineering course implement that system end to end:

• Part 3: Build routing, guardrails, and approval controls with LangGraph and Jev →

• Part 4: Add planning, verification, replanning, and stopping logic →

Good day!

This is quite different from a busy hosted service where requests from many users reach the same accelerator pool. The runtime can schedule several sequences in one batch. A single model pass then advances multiple requests. This improves total throughput.

An on-device application usually has no such cross-user request pool. Each installed copy handles its own prompts. A coding agent may invoke the model many times. However, later calls often depend on earlier outputs. Those calls usually arrive in sequence and cannot form a useful batch.

For this workload, one-sequence latency becomes the main target. During ordinary decoding, each token requires another model pass.

At batch size one, the runtime cannot share that weight traffic across user sequences. Quantization reduces the bytes moved during each pass. Speculative decoding tries to accept several tokens from one pass.

Losing cross-user batching is a constraint. Yet it also narrows the system’s problem. The local engine does not route requests across a fleet. It does not schedule work from many tenants. It can optimize for one active decode on known hardware.

Uzu is an open-source implementation of that approach. When an application uses Uzu, the engine and model run on the user’s device, fully locally.

A generic local runtime can load a quantized checkpoint. Uzu builds both the checkpoint and runtime. It designs the quantization, kernels, and speculative path as one system.

On a 14-inch MacBook Pro with a base M5 chip, the same 4-bit Qwen3.5 9B model wrote a Python function at 92 to 117 tokens per second with Uzu, across repeated runs of the same prompt. In our side-by-side recording, llama.cpp and MLX managed only 22 and 25.

In this article, we will trace where that gap comes from, through quantization and speculative decoding. Then we will run Uzu on that M5, inspect its metrics, and compare it with llama.cpp and MLX on the same prompt.

Why batch-one decoding is memory-bound

LLM inference has two distinct phases.

• Prefill processes the prompt tokens in parallel.

• Decode generates the response one token at a time.

Decode is the relevant phase where each new token requires another model pass. During that pass, the runtime reads weight blocks and computes the next-token scores.

At batch size one, a forward pass advances only one sequence.

The overall arithmetic is small relative to the weight data moved. To produce one token, the chip streams every weight of the model from memory into the GPU once and does roughly one multiply-add with each.

The GPU can finish those calculations before memory brings the next weight block. So the computation pace is often bottlenecked by the memory bandwidth.

This gives a hard limit. A base M5 moves 153 GB of data per second from memory. A 4-bit Qwen3.5 9B checkpoint weighs about 5.2 GB. If every pass produces one token, the model can write at most 153 ÷ 5.2 ≈ 29 tokens per second on that chip, however fast the GPU computes.

Real engines land a little below that ceiling, because each pass also reads other data, such as the cache of earlier tokens.

Quantization is a standard technique to improve this, whereby using smaller weights reduces the bytes read during each pass.

However, quantization still optimizes the same one-token loop. The main model must run again for every generated token.

In fact, quantization is already quite common, so now let’s understand what makes Uzu different as an inference engine.

How Uzu optimizes bytes and tokens per pass

Uzu targets two quantities in the decoding loop. It reduces bytes transferred per pass and increases accepted tokens per pass.

• Quantization handles the first quantity. It stores model weights with fewer bits. Smaller checkpoints require less memory traffic and leave more unified memory for context and application state.

• Speculative decoding handles the second quantity. It verifies several candidate tokens together. If several candidates are approved, the sequence advances by several tokens.

Many local engines already support 4-bit weights. Uzu’s distinction lies in the hardware path around that format.

Mirai publishes multiple optimized checkpoints for the same base model. It labels them by size tier. M means Medium, while L means Large.

These labels describe the checkpoint, not a different model architecture.

• The Medium checkpoint uses 4-bit asymmetric integer quantization.

• The Large checkpoint uses 8-bit symmetric quantization.

Medium favors a smaller memory footprint, while Large keeps more bits per weight. Mirai’s quantization write-up describes how it combines post-training quantization with quantization-aware distillation.

Using the integer format is intentional since it helps both the hardware and the compression.

More specifically, unlike older Apple GPUs, the M5 reaches its highest arithmetic throughput in int8.

Uzu therefore also quantizes activations, the intermediate values passed between layers, to 8 bits on the fly. With both sides of each matrix multiplication stored as integers, the speculative verification step runs on the M5’s hardware-accelerated int8 matrix path.

Lower precision creates another problem.

More specifically, group-wise quantization introduces rounding error when a group contains an outlier.

Every value in the group shares one scale. That scale must represent the largest magnitude. Smaller values then use fewer integer levels and incur more rounding error.

Uzu applies a block-diagonal Random Hadamard Transform before quantization. A Hadamard matrix contains only +1 and -1 entries, plus one overall scale.

Multiplying a block of weights by it mixes every value into every position, so a single large outlier turns into many moderate values.

“Random” refers to random sign flips applied before the mixing, and “block-diagonal” means each block of 32 values is mixed on its own. The transform is exactly reversible, so apart from rounding it does not change what the model computes.

Mirai selected 32 elements to match the Apple GPU kernel’s SIMD width. SIMD means that one instruction runs across 32 GPU threads at once, so one group of threads can transform one block.

The transform block and quantization group control different operations.

• The transform block defines which values the Hadamard matrix mixes.

• The quantization group defines which values share quantization parameters.

These sizes do not need to match.

Uzu’s differentiation is the joint design of the number format, transform, and runtime kernels. Quantization reduces memory traffic and computation within each model pass.

However, ordinary decoding still returns one token per target-model pass. Speculative decoding is the next lever it utilizes to improve that bottleneck.

Producing several tokens from one model pass

Speculative decoding adds a small draft model beside the main model.

The draft (small) model proposes future tokens cheaply. The main model checks those positions together. It accepts the valid prefix and resumes from the first rejected token.

Suppose the drafter proposes five tokens and the main model agrees with the first four. That one verification pass advances the sequence by five tokens, the four accepted drafts plus the main model’s own token at the fifth position.

Ordinary decoding would need five separate main-model passes for the same text.

The main model still controls the final output. The drafter only proposes tokens that can be checked in parallel. A correct acceptance procedure preserves the target model’s output distribution.

The verification step requires more computation than one-token decoding.

But the path is still preferred because local batch-one inference often has arithmetic capacity sitting idle while weights move through memory.

Speculative decoding spends that spare arithmetic to reduce the number of main-model passes.

The trade-off depends on acceptance. Longer drafts create larger verification batches. They also create more chances for an early mistake to invalidate later tokens.

Many model-native systems draft three or four tokens. A common example is multi-token prediction (MTP), where extra prediction heads trained into the model itself guess the next few tokens.

Uzu uses budgets of 16 to 32 tokens on its M5 path. The Qwen3.5 9B checkpoint used later in this article ships with a 16-token tree budget for M5-series chips.

Those larger batches give the M5’s Neural Accelerators enough work. These are matrix units built into each GPU core, and they only pay off when a matrix multiplication has many rows. Checking 16 candidate tokens at once gives them 16 rows of work instead of 1.

Longer parallel drafts introduce a coherence problem.

Issue with parallel draft generation

A fast parallel drafter predicts several future positions at once.

As expected, each position sees the existing context, but not the tokens selected at earlier draft positions.

So consider a prompt ending with “She went to the store to buy.” Independent positions may prefer “a,” “few,” “of,” and “milk.” Each token can look plausible alone. Together they produce “a few of milk.”

An early mismatch rejects the remaining chain. This is why acceptance tends to weaken as a parallel draft grows. More positions create more opportunities for individually plausible tokens to form an invalid sequence.

The Trees from Marginals paper introduces Weaver to repair this weakness. Weaver is a 56.7M-parameter autoregressive adapter in the paper’s research setup. It operates on candidate shortlists rather than projecting across the full vocabulary.

A parallel DFlash drafter first predicts likely candidates for each future position in one go. Weaver then selects from those candidates in sequence. Every selection can depend on earlier selections.

The earlier example can now become “a gallon of milk.” The later choices follow the path already selected.

Weaver can also retain several coherent paths. One branch might contain “a gallon of milk.” Another might contain “a bottle of juice.” The main model verifies the tree and accepts the longest valid path.

The tree shape can adapt to the prompt. Predictable code can support a long, narrow branch. Open-ended writing may need shorter branches with more alternatives.

The paper reports a 24.7 percent improvement over an optimized DFlash baseline. It also reports a 4.37-fold speedup over autoregressive decoding. Those experiments used CUDA and SGLang. They are research results, not universal Uzu measurements.

Uzu applies the same drafting idea to its Apple-focused runtime. The remaining problem is verifying trees efficiently on the target model.

Verifying a draft tree without restoring model state

Some model layers keep a running internal state. Each new token reads that state and updates it. With one sequence, the runtime applies these updates in order.

A draft tree creates several possible next sequences. Each branch would update the state differently. If the runtime updates its live state while checking one branch, it must restore the earlier state before checking another. Repeating that process across a large tree adds memory operations and duplicate work.

Qwen3.5 and Qwen3.6 contain stateful layers called Gated DeltaNet layers. An attention layer keeps a growing cache of every past token. A Gated DeltaNet layer instead compresses everything it has seen into a fixed-size state, and every accepted token changes that state.

In Qwen3.5 9B, 24 of the 32 layers are of this type.

Uzu separates checking a branch from committing its state. During verification, it computes the values needed by every candidate branch without changing the live state. The main model then selects the accepted path. Uzu applies state updates only for the accepted tokens.

The Uzu implementation note calls this rollback-free tree verification. Its Metal kernels compute the DeltaNet outputs for the whole tree in a single recurrence-free pass, instead of stepping through each branch token by token.

State is then committed only along the chosen path, so there is never a wrong update to roll back.

This matters because speculative decoding only helps when verification costs less than ordinary token-by-token decoding. Repeated state restoration would consume part of that gain.

Together, Weaver and rollback-free verification let Uzu propose long token sequences and check them without repeatedly restoring model state.

What this looks like on a base M5

To see the difference between ordinary decoding, conventional speculative decoding and Uzu’s approach, we ran the same Qwen3.5 9B model three ways on a MacBook Pro with a base M5 chip and 16 GB of unified memory:

• llama.cpp with no speculative decoding

• llama.cpp with the model’s built-in MTP heads drafting 3 tokens ahead, the conventional setup

• Uzu with DFlash and Weaver

All three used 4-bit weights and the same prompt: “Write a python function to merge two sorted lists.”

Plain llama.cpp lands just under the 29 tokens-per-second limit from earlier. Three-token MTP drafts push it past that limit, to about 1.6 times plain decoding. In the recorded run, Uzu’s verifier accepted 7.5 tokens per target-model pass on average, which is how it reaches more than three times the one-token limit on the same chip.

Published benchmark comparisons

There are already some published benchmarks for Uzu that measure output speed, input speed, memory, and quantization quality.

The following speculative result uses Qwen3.6 27B Mirai-M. The checkpoint occupies 15.6 GB. Mirai measured it on an M5 Max with 128 GB of memory.

These figures put Uzu at

• 2.1 times MTPLX

• 3.8 times llama.cpp

• 4.4 times MLX.

Moreover, Uzu reaches 35.54 tokens per second without speculation. Its speculative result reaches 114 tokens per second. This provides a 3.2-times gain from the speculative path.

There are also results on output generation speed by prompt type.

Code and maths contain many predictable continuations. Open-ended conversation gives the drafter fewer reliable tokens. Speculative speed is therefore a workload property, not a fixed model number.

Run a model locally and inspect the runtime

The fastest way to check Uzu is its command-line tool. It installs through Homebrew and downloads the selected model on first use.

brew install mirai

mirai --model trymirai/Qwen3.5-9B-M \
  -m "Write a python function to merge two sorted lists"

We ran this with Mirai CLI 0.6.1 on a MacBook Pro with a base M5 chip and 16 GB of unified memory, on macOS 26.6.2. The first run downloads the model.

Here is a run in real time:

After the answer, the CLI printed these stats:

time to first token: 0.12 s
prefill speed: 161.11 t/s
generation speed: 98.10 t/s
tokens per forward pass: 8.39 t/f
memory used: 5.77 GB
total energy: 114.90 J
input energy per token: 0.11 J/tok
output energy per token: 0.16 J/tok
duration: 7.36 s

Here is what these numbers mean:

• Time to first token is how long you wait before the first token of the answer appears. Most of it goes into reading the prompt.

• Prefill speed is how fast the engine reads the prompt, in prompt tokens per second. Our prompt is only about 20 tokens, which is too short for this number to say much. It climbs with longer prompts.

• Generation speed is how fast the answer is written, in output tokens per second. This is the decode phase from earlier, the one held back by the memory limit.

• Tokens per forward pass is how many output tokens each pass of the 9B model produced on average. Ordinary decoding gives exactly 1.00.

• Memory used is the unified memory the engine took, mostly the model’s weights plus the drafter and the cache.

The 8.39 t/f value shows the speculative path at work. On average, each forward pass of the 9B model produced about 8 tokens. That is how generation reaches 98 tok/s on a chip whose one-token limit is about 29.

Speeds vary a little between runs. Across our runs of this command, generation speed ranged from 92 to 117 tok/s, with 7.5 to 9.6 tokens per pass.

Speculation is enabled per chip. This checkpoint’s speculator configuration lists the M5, M5 Pro and M5 Max, so on M1 to M4 Macs the same command runs without a drafter and reports 1.00 t/f.

The CLI exposes latency, throughput, memory, energy, and speculation data. We can now call the same runtime from an application.

Call Uzu through a local application endpoint

Uzu can expose an OpenAI-compatible HTTP endpoint. The model still runs on the user’s Mac. The HTTP request only crosses the local loopback interface.

Start the server in one terminal:

mirai server --model trymirai/Qwen3.5-9B-M --port 8000 --no-prefix-cache

The server loads one local checkpoint and listens on 127.0.0.1, process on the same Mac can invoke it. This is useful for an existing application that already interacts via the OpenAI chat protocol.

By default, Uzu keeps a prefix cache. It stores the already-processed start of earlier prompts, so a repeated prompt can skip most of its prefill. --no-prefix-cache turns this off, so every test request does the full work and repeated runs stay comparable.

An existing application can switch to Uzu by pointing its OpenAI client at http://127.0.0.1:8000/v1. The next section uses one small client to time Uzu, MLX and llama.cpp in exactly the same way.

Use one client for Uzu, MLX, and llama.cpp

Here is the same prompt run through each engine on the base M5. Each engine ran on its own, with nothing else on the machine.

llama.cpp and MLX both sit under the 29 tokens-per-second limit, as expected when every pass produces one token. In this recording, Uzu writes about 3.7 times faster than MLX on the same chip.

To run this comparison with your own prompts, start each engine as a local server and time all three with one client.

The local server interface gives us a practical comparison method.

Uzu, MLX LM, and llama.cpp can all expose OpenAI-style chat endpoints. We can run Qwen3.5 9B through all three and send the same prompts to each server.

Start Uzu with Mirai’s 4-bit Medium checkpoint:

mirai server --model trymirai/Qwen3.5-9B-M --port 8000

Start MLX LM with the 4-bit MLX conversion of the same base model:

brew install uv
uv tool install mlx-lm
uv tool update-shell

uv tool update-shell adds the directory containing mlx_lm.server to your shell path.

Open a new terminal after running it, then start the server:

mlx_lm.server --model mlx-community/Qwen3.5-9B-4bit --port 8001

Start llama.cpp with the Q4_K_S GGUF checkpoint used in Mirai’s published comparison:

brew install llama.cpp
llama-server --version

llama-server \
  -hf unsloth/Qwen3.5-9B-MTP-GGUF:Q4_K_S \
  --alias qwen3.5-9b-q4-k-s \
  -ngl 99 -fa on -c 8192 -np 1 \
  --port 8002

-c 8192 keeps llama.cpp from reserving the model’s full 262K context, which would not fit next to the weights on a 16 GB Mac. -np 1 runs a single slot. To turn on llama.cpp’s MTP drafting, as in the first video, add --spec-type draft-mtp --spec-draft-n-max 3. MTP needs -np 1.

Each server downloads its checkpoint the first time it starts. Run the servers one at a time so that only one copy of Qwen3.5 9B occupies unified memory during a measurement.

The three checkpoints come from the same Qwen3.5 9B model, but they do not use the same file format or quantization recipe.

Uzu uses Mirai-M, MLX uses its own 4-bit conversion, and llama.cpp uses Q4_K_S. This is therefore a comparison of complete engine paths that a developer can run. It is not a kernel-only test.

Run the benchmark client against ports 8000, 8001, and 8002, one server at a time.

The harness should keep the prompt text, output limit, sampling settings, and run count fixed. Use comparable quantization quality, not only similar bit width. Record native engine metrics beside the end-to-end client time.

Finally, here’s a standalone client. Save it as compare_client.py. It uses only Python’s standard library and sends the same deterministic prompt to any OpenAI-compatible local endpoint.

It turns thinking off so every engine answers directly, and it asks each server for token usage. It reports time to first token, end-to-end time and decode speed.

Uzu also reports how many target-model passes it ran, so for Uzu the script prints tokens per pass too.

import json, os, sys, time, urllib.request

PROMPT = "Write a Python function that merges two sorted lists."

body = json.dumps({
    "model": os.environ["MODEL"],
    "messages": [{"role": "user", "content": PROMPT}],
    "temperature": 0,                                    # same answer every run
    "max_tokens": 1024,
    "stream": True,                                      # tokens arrive as they are made
    "stream_options": {"include_usage": True},           # server reports token counts
    "chat_template_kwargs": {"enable_thinking": False},  # answer directly, no thinking
}).encode()
request = urllib.request.Request(os.environ["ENDPOINT"], body, {"Content-Type": "application/json"})

start = time.perf_counter()
first = last = None
usage = {}

with urllib.request.urlopen(request, timeout=600) as response:
    for line in response:
        line = line.decode().strip()
        if not line.startswith("data:"):
            continue
        data = line[5:].strip()
        if data == "[DONE]":
            break
        event = json.loads(data)
        usage = event.get("usage") or usage
        for choice in event.get("choices") or []:
            delta = choice.get("delta") or {}
            text = delta.get("content") or ""
            if text or delta.get("reasoning_content") or delta.get("reasoning"):
                last = time.perf_counter()
                first = first or last
            print(text, end="", flush=True)

if first is None:
    sys.exit("No tokens received. Check the server log.")

tokens = usage.get("completion_tokens", 0)
print(f"\n\ntime to first token: {first - start:.2f} s")
print(f"end-to-end time:     {time.perf_counter() - start:.2f} s")
if tokens > 1 and last > first:
    print(f"decode speed:        {(tokens - 1) / (last - first):.1f} tok/s")
if usage.get("spec_verify_ct"):  # only Uzu reports how many model passes it ran
    print(f"tokens per pass:     {tokens / usage['spec_verify_ct']:.2f}")

Run each engine, one at a time

Run only one server at a time, so a single copy of the model sits in unified memory and no two engines compete for memory bandwidth.

For each engine, start the server in one terminal, then run the client twice from a second terminal. The first run includes warm-up work, so discard it and keep the second. Stop the server with Ctrl-C before moving on.

For Uzu, start the server in the first terminal:

mirai server \
  --model trymirai/Qwen3.5-9B-M \
  --port 8000 \
  --no-prefix-cache

Run the client from a second terminal:

ENDPOINT=http://127.0.0.1:8000/v1/chat/completions \
MODEL=trymirai/Qwen3.5-9B-M \
python3 compare_client.py

Run the client twice. Discard the first result because it includes warm-up work. Keep the second result and the decode statistics printed by the Uzu server. Then stop Uzu with Ctrl-C.

Start MLX in the first terminal:

mlx_lm.server \
  --model mlx-community/Qwen3.5-9B-4bit \
  --port 8001

Run the same client from the second terminal:

ENDPOINT=http://127.0.0.1:8001/v1/chat/completions \
MODEL=mlx-community/Qwen3.5-9B-4bit \
python3 compare_client.py

Again, discard the first run and keep the second. Stop MLX with Ctrl-C.

Start llama.cpp in the first terminal:

llama-server \
  -hf unsloth/Qwen3.5-9B-MTP-GGUF:Q4_K_S \
  --alias qwen3.5-9b-q4-k-s \
  -ngl 99 -fa on -c 8192 -np 1 \
  --port 8002

Run the client from the second terminal:

ENDPOINT=http://127.0.0.1:8002/v1/chat/completions \
MODEL=qwen3.5-9b-q4-k-s \
python3 compare_client.py

Discard the first run and keep the second. The comparison should keep these settings fixed:

• The Mac and its power mode

• The prompt and chat template

• The output-token limit

• Temperature and sampling settings

• Warm-up count

• Number of measured requests

The client’s decode speed includes HTTP streaming, so expect it to be slightly lower than the speed each engine prints in its own log. Compare the client numbers with each other, not with the CLI numbers.

The Mirai CLI has no flag to switch speculation off, so this procedure compares complete engine paths rather than isolating DFlash-Weaver.

The llama.cpp runs with and without MTP in the first video are the closest same-engine view of what speculation adds.

Conclusion

Loading a quantized checkpoint is only the first step in local inference.

The actual work is keeping generation useful once every request runs at batch size one on a user’s device.

Uzu handles that problem at several levels.

• Quantization reduces memory traffic.

• Its Metal kernels target Apple hardware.

• DFlash-Weaver proposes longer token sequences, while rollback-free verification checks them without repeatedly restoring recurrent state.

On a base M5, that combination took a 4-bit 9B model from 22 tokens per second with plain llama.cpp to between 92 with Uzu on the same prompt.

Acceptance changes with the workload. Hardware, quantization quality, context length, and thermal state can also alter the result. The correct test uses the application’s prompts and measures quality alongside latency.

If you want to dive deeper, the Uzu repository contains the engine, local server, build instructions, and bindings for Rust, Swift, Python, and TypeScript.

Start with the local server, run the benchmark client from this article, and replace the sample prompt with requests from the actual product.

git clone https://github.com/trymirai/uzu.git

Find Uzu on GitHub here →

(don’t forget to star it ⭐️)

Good day!

• Build voice agents that hear the details.

• ​RAG vs. Jev + RAG, clearly explained!​

• ​5 vector DB indexing strategies.

Build voice agents that hear the details

Speechmatics Linden is a speech-to-text model built specifically for voice agents. It focuses on the inputs that tend to cause expensive mistakes: strong accents, background noise, non-native speech, names, and alphanumeric strings such as booking references.

• Real-time transcription across 55+ languages

• Final transcripts in roughly 250 ms after a ForceEndOfUtterance signal

• Custom dictionaries for names, product terms, and industry vocabulary

• Native integrations for LiveKit, Pipecat, Jambonz, and the Speechmatics API

• Linden 1 costs $0.30 per hour for new users, with volume discounts bringing the price down to $0.16 per hour

New accounts receive $100 in credit with no card required, which covers roughly 333 hours of Linden 1 transcription at the standard $0.30 per-hour rate.

Try Speechmatics here →

Thanks to Speechmatics for partnering today!

RAG vs. Jev + RAG, clearly explained!

In a standard RAG setup, documents are split into chunks, converted into embeddings, and stored in a vector DB.

When a query arrives, the system embeds it and retrieves the top-k chunks with the closest vectors.

Many production setups add a reranker after retrieval.

The reranker compares the query with each retrieved passage, improves their ordering, and keeps the highest-scoring results.

Those passages are then placed in the context window, and the LLM generates an answer from them.

The problem is that ranking and answerability are different questions.

A passage can be more relevant than the other candidates while still containing weak, incomplete, or merely adjacent evidence.

The LLM receives it anyway and may produce a plausible answer from context that never supported one.

Jev + RAG keeps the retrieval stage as is but changes what happens before generation.

Instead of using a conventional reranker at this stage, Jev can judge every retrieved candidate against a typed question such as:

“Does this passage help answer the query?”

The query becomes the shared state, while the retrieved passages become individual candidates. Jev evaluates them together and returns a probability for each one.

Application code then applies a threshold.

↳ Candidates above the threshold continue to the LLM.

↳ Candidates below it are removed from the context window.

The same Jev request can also judge whether the remaining evidence is sufficient to answer the query.

If answerability falls below the threshold, the application can skip the LLM and return “not in the documents.”

Jev does not replace the embedding model, vector database, or generation model. Retrieval still sets the cap because Jev cannot recover a passage that never entered the candidate set.

The diagram below depicts the complete flow.

• RAG retrieves a broad candidate set.

• Jev reranks and gates those candidates.

• The LLM generates an answer.

Jev becomes more useful when the same typed decisions sit inside a complete agent loop. Parts 3 and 4 of our Agent Engineering course implement that system end to end:

• Part 3: Build routing, guardrails, and approval controls with LangGraph and Jev →

• Part 4: Add planning, verification, replanning, and stopping logic →

​5 vector DB indexing strategies​

Meta. Google. Microsoft.

These companies have spent years engineering faster vector-search systems.

A basic nearest-neighbor query computes the distance between the query and every stored vector.

Its cost grows linearly with the size of the database and becomes impractical across millions of candidates.

Every vector index reduces this computation differently, using clustering, graph traversal, compression, or a combination of these methods.

Here are five common approaches:

​This full article covers the first four techniques in detail, with an architectural breakdown →​

1. Flat index

A Flat index compares the query with every stored vector, selects the K smallest distances, and returns them.

It performs an exact search, so it is useful when accuracy matters and the dataset is small enough to scan.

The query cost grows linearly with the number of stored vectors.

2. IVF

IVF divides the vector space into clusters. Each stored vector is assigned to its nearest cluster centroid.

At query time, the database finds the closest centroids and searches only their clusters.

A probe parameter controls how many clusters it searches, and increasing it usually improves recall but also increases latency.

3. HNSW

HNSW stores vectors in a multilayer graph.

The upper layers contain fewer nodes and allow large jumps through the graph. Lower layers contain more nodes and support a finer search.

A query starts at the top, moves toward closer nodes, and descends through the layers. HNSW often provides low latency and high recall, but the graph requires extra memory.

4. IVF-PQ

IVF-PQ combines clustering with product quantization.

IVF limits the search to a few relevant clusters. Product quantization splits each vector into smaller parts and stores compact codes for them.

Comparing these codes is cheaper than comparing full-precision vectors. This reduces memory use and query cost, with some loss in recall.

5. ScaNN

ScaNN partitions the dataset and stores quantized vector representations.

During search, it selects relevant partitions, scores compressed candidates, and recalculates more accurate distances for the strongest candidates.

This makes it suitable for large datasets where throughput matters.

The choice depends on the system constraints.

• Flat prioritizes exact results.

• HNSW trades memory for speed.

• IVF provides direct control over the amount of search work.

• IVF-PQ and ScaNN reduce the memory and compute required for large collections.

Choosing an index requires deciding what your system can afford to trade.

Index selection is one part of building a production RAG system. Our 15-part RAG Systems course develops the rest of the stack progressively:

• ​Part 1: Build the complete RAG workflow, including chunking, embeddings, retrieval, and reranking →​

• ​Part 2: Evaluate retrieval and generation quality with RAG-specific metrics →​

• ​Part 3: Reduce RAG latency with faster retrieval and generation paths →​

• ​Part 4: Extend RAG beyond text to multiple data types →​

• ​Part 5: Understand CLIP embeddings, multimodal prompting, and tool calling →​

• ​Part 6: Build a multimodal RAG system over real-world data →​

• ​Part 7: Use Graph RAG for relationships that ordinary chunk retrieval misses →​

• ​Part 8: Improve retrieval with ColBERT and ColBERTv2 →​

• ​Part 9: Build vision-driven retrieval with ColPali →​

• ​Part 10: Apply the first set of techniques for production RAG systems →​

• ​Part 11: Complete the production RAG techniques and implementation patterns →​

• ​Part 12: Reduce prefill latency by reusing KV caches without a shared prefix →​

• ​Part 13: Preload a corpus before queries arrive and examine the production constraints →​

• ​Part 14: Compress preloaded caches and measure where the approaches break →​

• ​Part 15: Implement Block-Attention, Cartridges, and production preloading →

Good day!​

Datalab has just released Surya OCR, a state-of-the-art OCR model that scores 83.3% on the olmocr benchmark (top under 3B).

• 650M params

• supports 91 languages

• 5 pages/s on RTX 5090

• runs on CPU, GPU, MPS

• 83.3% olmocr bench score (top under 3B)

• Full layout information

• Extracts + captions images and diagrams

• Strong handwriting, math, form, and table support

​Find the GitHub repo here →​ (don’t forget to star it ⭐️)

Chunked Prefill, clearly explained

Every LLM request has two phases.

Prefill processes the complete input prompt in parallel. It is compute-heavy, builds the request’s KV cache, and produces the first output token.

Decode generates the remaining response one token at a time. Each step repeatedly reads model weights and the KV cache, making it primarily limited by memory bandwidth.

Now imagine Request A is already decoding and streaming tokens when Request B arrives with a 32K-token prompt.

Without chunked prefill, vLLM would process B’s complete prompt in one large pass. Request A cannot generate its next token during that work, so its response appears to freeze.

Chunked prefill splits B’s prompt into smaller token ranges. vLLM can process one chunk, give active requests another opportunity to decode, then continue B’s prefill across later iterations.

The same prompt still gets processed. It simply stops occupying the GPU as one uninterrupted block.

That introduces an important latency tradeoff.

→ Smaller chunks keep active responses moving.

They return control to the scheduler more frequently, reducing inter-token latency spikes for users already receiving an answer.

→ Larger chunks help the new request start sooner.

B cannot generate its first token until every prompt chunk has been processed. Larger chunks complete that work in fewer, more efficient passes, which usually improves time to first token.

→ Chunks that are too small add overhead.

Later chunks must attend to token states created by earlier chunks. Excessive splitting repeatedly reads more of the KV cache and can reduce GPU utilization.

→ Mixed batches can use the GPU more effectively.

Prefill is compute-heavy, while decode is memory-heavy. Scheduling prefill chunks alongside active decodes allows both kinds of work to share an iteration more efficiently.

There is no universal best chunk size.

Long prompts, many concurrent requests, and strict streaming targets usually favor smaller chunks. Workloads that prioritize time to first token may benefit from larger ones.

In vLLM V1, `--max-num-batched-tokens` controls how many tokens can be scheduled in an iteration. It is one of the main settings governing this tradeoff.

Chunked prefill does not reduce the work creatd by a long prompt.

Thanks for reading.

A dense 30B model fits on a single GPU, so you can run it on your own server or an air-gapped box, and your data never leaves that machine.

It’s also a break from Meta’s recent open releases:

• It’s the first open model from Meta Superintelligence Labs and Meta’s first open release since Llama 4.

• It ships under Apache 2.0 instead of the Llama license, so you can fine-tune, rename, or ship it commercially with no user caps, usage policy, or branding.

Out of the box, a general model won’t handle your specific task well. In this article, we’ll fine-tune Muse Glimmer to convert images of equations into LaTeX.

What makes Muse Glimmer different?

Muse Glimmer pairs a ~29.6B-parameter decoder with a 1.8B-parameter ViT-G/14 vision encoder. It takes text and images as input and outputs text.

At that size, the weights aren't the only memory problem. As the model reads a long input, it stores keys and values for every past token in every layer. This store is the KV cache, and at long contexts it can grow larger than the weights.

Glimmer makes two choices to keep it small:

1. Most layers only look back 2,048 tokens. The layers repeat a pattern of three sliding-window layers and one global layer. At a 131K context, 39 layers cache at most 2,048 tokens each, so the cache is about a quarter of full attention.

2. 32 query heads share just 2 KV heads. That cuts the cache by another 16x.

Together, that's roughly 60x less KV memory than a naive design. That’s why a 131K token default context (extendable to 262K) is usable on hardware you own.

Muse Glimmer also uses its own chat format, ATEM. Every turn looks like this:

<|start|>{role}<|message|>...<|eot|>

Before answering, the assistant can send a message addressed to itself. That’s a private reasoning channel. Only then does it answer on the user channel.

How much it reasons is a dial called reasoning strength. For ‘high’ mode, unless you change it, you pay for an expensive setting on every prompt.

This matters for fine-tuning, as:

• Your dataset must follow ATEM exactly. A mismatched template degrades training.

• You decide whether to train on reasoning. For converting an equation image to LaTeX, long reasoning mostly adds latency, so [your choice and why].

Where it’s strong, and where it isn’t

Glimmer leads on agentic tool use, most clearly on MCP Atlas, where it beats the next-best model, Qwen3.6-27B, by 13 points.

On vision and document benchmarks, it lands within two points of Qwen3.6-27B, ahead on one and behind on three.

That’s why fine-tuning helps here because general benchmarks don’t measure your task.

A model that reads images well still won’t output the exact LaTeX your system expects, and a training run closes that gap.

Why fine-tune at all?

Our task is math OCR. The model takes an image of a typeset formula and returns its LaTeX source.

Here’s the base model on one sample from the dataset, before any training:

And here’s the ground truth:

The model gets the layout right: the integral, the braces, and the fraction. The rest goes wrong in two ways.

It breaks the output contract:

• It wraps the answer in display math

• It answers twice and adds commentary

• It writes compact LaTeX where the dataset expects space-separated tokens

It misreads the math:

• ζ becomes χ

• The first derivative term loses its ^\dagger

For a pipeline that expects one line of LaTeX in a fixed style, this output is unusable. Even the formatting alone fails an exact-match check, which scores compact LaTeX as wrong even when it renders identically.

Most real vision tasks have the same gap, whether it's invoice fields, chart values, lab forms, or UI screenshots. The base model can read. Fine-tuning teaches the model your output structure and your domain use.

What we’ll build

We'll fine-tune Muse Glimmer on unsloth/LaTeX_OCR dataset, measure the exact match on formulas before & after training, and save the result for local use.

Here’s the stack we will be using:

• Unsloth for 4-bit loading, LoRA adapters, and training

• Hugging Face Datasets for the LaTeX OCR data

And here’s the game plan:

1. Load the model

2. Add LoRA adapters

3. Prepare the dataset

4. Read the chat template

5. Run baseline inference

6. Train the model

7. Inference with the fine-tuned model

8. Save and export

9. Streamlit UI

Load the model

The full BF16 weights take about 56 GB. Unsloth's Dynamic 4-bit build cuts that, as it quantizes most layers to 4-bit and keeps the layers that lose the most accuracy under quantization in 16-bit.

from unsloth import FastModel

model, processor = FastModel.from_pretrained(
    model_name     = "unsloth/Muse-Glimmer-30B-unsloth-bnb-4bit",
    max_seq_length = 1024,
)

Fitting this 30B model on consumer GPUs comes down to where the memory goes:

The input embedding and the output head, each a large matrix, take roughly 26% of it. They have the same shape, but the model uses them differently:

• The input embedding is a lookup. Each token needs only its own row, and the table stays frozen during LoRA training.

• So we offload it to CPU RAM and copy only the rows a batch needs onto the GPU.

• The model output head (lm_head) is a matrix multiply. Every token's logits need every row, so the head stays on the GPU.

Tips: QLoRA fits in 24 GB, while 16-bit LoRA needs more than 40 GB. Plan for a 24 GB card as the realistic minimum, and keep sequence lengths short at first.

With the model loaded, the next step is deciding which parts of it to train.

Add LoRA adapters

We won't update the model's 31B weights. LoRA freezes them and trains small low-rank matrices alongside selected layers. Only those matrices get gradients and optimizer states, and that's what keeps training inside the headroom we have.

model = FastModel.get_peft_model(
    model,
    finetune_vision_layers     = True,  # False if not finetuning vision layers
    finetune_language_layers   = True,  # False if not finetuning language layers
    finetune_attention_modules = True,  # False if not finetuning attention layers
    finetune_mlp_modules       = True,  # False if not finetuning MLP layers

    r = 16,            # The larger, the higher the accuracy, but might overfit
    lora_alpha = 16,   # Recommended alpha == r at least
    lora_dropout = 0,
)
model.print_trainable_parameters()

The flags choose where the adapters go. You can target the vision tower, the language model, or both, and within each, the attention layers, the MLP layers, or both. With every flag on, we train 131M parameters, about 0.44% of the model.

Rank (r) = 16 is the recommended starting point, with Rank (r) = 32 is suggested for harder agentic tasks.

Why do we train the vision layers too?

For vision tasks, it is recommended to freeze the vision encoder first, train the language layers, and open up the vision side only if your data needs it.

We train both for two reasons:

1. The base model read ζ as χ. It could be a vision encoder error or the language model’s. We can’t tell what the mistake is. Training both covers either case.

2. Our inputs are unusual. Typeset math looks very different from the natural images most vision encoders see in pretraining.

If you want a smaller adapter and a cheaper run, try language-only training first. If it matches accuracy on your data, keep it.

Now the adapters need data in a shape the model understands.

Prepare the dataset

The LaTeX OCR dataset has 68,686 image-and-LaTeX pairs, like the formula we saw earlier.

from datasets import load_dataset

dataset = load_dataset("unsloth/LaTeX_OCR", split = "train")
dataset
# Dataset({
#     features: ['image', 'text'],
#     num_rows: 68686
# })

The vision trainer expects all fine-tuning data to be a list of messages, where image parts sit next to text parts inside content.

In each user turn, both the image and the instruction should be included, while the LaTeX content should be provided in the assistant's turn.

instruction = "Write the LaTeX representation for this image."

def convert_to_conversation(sample):
    return {"messages": [
        {"role": "user", "content": [
            {"type": "image", "image": sample["image"]},
            {"type": "text",  "text": instruction},
        ]},
        {"role": "assistant", "content": [
            {"type": "text", "text": sample["text"]},
        ]},
    ]}

converted_dataset = [convert_to_conversation(sample) for sample in dataset]

Here’s what one converted sample looks like:

{
    “messages”: [
        {
            “role”: “user”,
            “content”: [
                {”type”: “image”, “image”: <PIL.PngImageFile image mode=RGB size=...>},
                {”type”: “text”,  “text”: “Write the LaTeX representation for this image.”},
            ],
        },
        {
            “role”: “assistant”,
            “content”: [
                {”type”: “text”, “text”: r”{ \frac { N } { M } } \in { \bf Z } , { \frac { M } { P } } \in { \bf Z } , { \frac { P } { Q } } \in { \bf Z }”},
            ],
        },
    ]
}

What happens to these messages next is specific to the model's chat template, which turns them into the exact tokens the model sees.

Read the chat template

Muse Glimmer doesn't use ChatML or the Llama format. If you use handwritten formatting code from another model, it will produce the wrong tokens.

Let’s render one training sample and look at the actual string:

example_text = processor.apply_chat_template(
    converted_dataset[0]["messages"], tokenize = False,
))
print(example_text)

Four things in this output matter for training:

1. The image is a single <|patch|> placeholder. Once the processor sees the real image, it expands the placeholder. A 448×448 image becomes 256 tokens, and those count against the model's maximum sequence length.

2. The system block allows two recipients, self and user. The user channel carries the answer, the one the reader sees.

3. The model reasons before it answers. Glimmer writes a private chain of thought to itself on the self channel, then gives the real answer.

4. How much it thinks is set by reasoning strength in the system block. high by default.

Set the reasoning dial

reasoning_strength is a chat template argument, and it writes the strength line into the system block for you:

text = processor.apply_chat_template(
    messages, add_generation_prompt = True, tokenize = False,
    reasoning_strength = "low",   # low, medium, high, xhigh
)

Here’s what each setting costs on one held-out formula, with greedy decoding:

It is recommended to use high reasoning mode for coding and agent workflows, medium for simple assistants, and low when speed matters most.

Reasoning tokens count against max_new_tokens. If a run hits the limit mid-thought, it returns an empty answer. So raise it before you raise the reasoning.

Now that we understand the template, we can measure the baseline properly.

Run baseline inference

For our OCR use baseline eval, we want the model transcribing, not thinking.

Because transcription is a reading task, the answer comes from what’s in the image. Our training data also reflects that, since every example is an image and its LaTeX with no reasoning trace.

But by default, it won’t stop thinking. The rendered prompt stops at <|start|>assistant with the recipient unset, which is why the model writes to itself first and answers only after that.

Setting the recipient ourselves solves it. We append to=user<|message|> to the prompt, so the prompt skips the reasoning channel and generation starts on the answer channel:

def muse_glimmer_prompt(messages, answer_directly = True):
    """Build a Muse Glimmer generation prompt from chat messages."""
    text = processor.apply_chat_template(
        messages, add_generation_prompt = True, tokenize = False,
    )
    return text + " to=user<|message|>" if answer_directly else text

Run it on the same formula from earlier:

sample = dataset[2]
messages = [
    {"role": "user", "content": [
        {"type": "image"},
        {"type": "text", "text": instruction},
    ]},
]
inputs = processor(
    text = [muse_glimmer_prompt(messages)],
    images = [[sample["image"].convert("RGB")]],
    add_special_tokens = False,
    return_tensors = "pt",
).to("cuda")

# Stream only the generated answer, not the prompt
text_streamer = TextStreamer(processor.tokenizer, skip_prompt = True)
_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 256,
                   use_cache = True, do_sample = False)

Meta recommends temperature = 1.0, top_p = 0.95, and top_k = 64 for open-ended and agentic use. Transcription needs the same answer every time, so we decode greedily.

\[
H' = \beta N \int d\lambda \left\{ \frac{1}{2\beta^2 N^2}\,\partial_\lambda \chi\,\partial_\lambda \chi + V(\lambda)\,\chi^\dagger \chi \right\}.
\]

The expression shown in the image is

\[
H' = \beta N \int d\lambda \left\{ \frac{1}{2\beta^2 N^2}\partial_\lambda \chi^\dagger \partial_\lambda \chi + V(\lambda)\chi^\dagger \chi \right\},
\]

with the conventional ordering $\partial_\lambda\chi^\dagger\partial_\lambda\chi$.<|eot|>

The base model produces a verbose, restyled output. The shape is unusable, and the symbols are not exact either. That's our baseline.

Fine-tuning has to close both gaps: one line of LaTeX in the dataset's style, matching the image character by character.

Now we train.

Train the model

The data collator builds each batch, expands the image tokens, and masks everything except the assistant's answer. Without the mask, every update will go to predicting the system and the instruction, which are identical in every sample.

The mask uses the earlier shown ATEM markers from the template section:

from unsloth.trainer import UnslothVisionDataCollator
from trl import SFTTrainer, SFTConfig

collator = UnslothVisionDataCollator(
    model, processor,
    max_seq_length = MAX_LEN,
    train_on_responses_only = True,
    # ATEM markers, straight out of the chat template
    instruction_part = "<|start|>user<|message|>",
    response_part    = "<|start|>assistant to=user<|message|>",
)

trainer = SFTTrainer(
    model = model,
    train_dataset = converted_dataset,
    processing_class = processor.tokenizer,
    data_collator = collator,
    args = SFTConfig(
        per_device_train_batch_size = 1,   
        gradient_accumulation_steps = 4,   # effective batch of 4
        max_steps = 30,                    # have num_train_epochs for a full run
        warmup_steps = 5,                  # ramp up before the full learning rate
        learning_rate = 2e-4,              # standard for LoRA
        lr_scheduler_type = "linear",
        weight_decay = 0.001,
        optim = "adamw_8bit",              # ~4x less optimizer memory than 32-bit AdamW
        seed = 3407,                      
        max_length = 1024,
        report_to = "none",
        # the collator handles images & tokenization, so leave the raw data
        remove_unused_columns = False,
        dataset_text_field = "",
        dataset_kwargs = {"skip_prepare_dataset": True},
    ),
)

Before spending GPU time, let’s check that we are training the model only on the assistant answers:

batch = collator([converted_dataset[0], converted_dataset[1]])
labels = batch["labels"][0]
print("trained tokens:", int((labels != -100).sum()), "of", labels.numel())
print(processor.tokenizer.decode([t for t in labels.tolist() if t != -100]))

It confirms that the prompt, system block, and image tokens are all excluded. Ten seconds of checking here can save you from a run that learns the wrong thing.

One trade-off here is that we train only short, direct answers, and this can weaken multi-step reasoning. For an OCR model, that's acceptable. If the model also has to plan and call tools, mix reasoning-style examples into the dataset.

Let’s train:

trainer_stats = trainer.train()

So what did the training steps actually change?

Inference with the fine-tuned model

Same helper, same prompt, on an image the model never trained on:

from unsloth import FastModel
from PIL import Image

model, processor = FastModel.from_pretrained(
    model_name     = "muse_glimmer_lora",
    max_seq_length = 1024,
)

image = Image.open("formula.png").convert("RGB")
messages = [
    {"role": "user", "content": [
        {"type": "image"},
        {"type": "text", "text": instruction},
    ]},
]

inputs = processor(
    text   = [muse_glimmer_prompt(messages)],   # still pinned to to=user
    images = [[image]],
    add_special_tokens = False,
    return_tensors     = "pt",
).to("cuda")

# generate() returns prompt + completion, so slice off the prompt
output_ids = model.generate(**inputs, max_new_tokens = 256, do_sample = False)
generated_ids = output_ids[:, inputs["input_ids"].shape[1]:]
print(processor.tokenizer.decode(generated_ids[0], skip_special_tokens = True))

The input image:

The LaTeX ground truth:

[ [ B _ { n } ^ { + } , b _ { 2 } ^ { - } ] , b _ { 2 } ^ { + } ] = n B _ { n } ^ { + } , \quad [ [ B _ { n } ^ { - } , b _ { 2 } ^ { + } ] , b _ { 2 } ^ { - } ] = n B _ { n } ^ { - } .

And the model’s output:

[ [ B _ { n } ^ { + } , b _ { 2 } ^ { - } ] , b _ { 2 } ^ { + } ] = n B _ { n } ^ { + } , \quad [ [ B _ { n } ^ { - } , b _ { 2 } ^ { + } ] , b _ { 2 } ^ { + } ] = n B _ { n } ^ { - } .<|eot|>

The format now matches the target. One line, space-separated tokens, no display-math wrapper or commentary, and a clean stop at <|eot|>. That’s the output contract the base model ignored.

The content is almost right but still missed a minute detail. It successfully captures the macro-level LaTeX target format; it hasn’t yet learned fine-grained symbol recognition, resulting in the expressions being wrong.

In mathematical representations, precision is binary: a single symbol mistake invalidates the entire equation.

After 30 steps, the model has done the best it can. You can do full fine-tuning by replacing max_steps = 30 with num_train_epochs = 1.

Once the numbers hold up, it’s time to package the adapter.

Save and export

The adapters are the only thing training changed, so saving them writes a few hundred MB instead of 56 GB:

# Local saving
model.save_pretrained("muse_glimmer_lora")
processor.save_pretrained("muse_glimmer_lora")

# Hugging Face Hub saving
model.push_to_hub("your_name/muse_glimmer_lora", private = True)
processor.push_to_hub("your_name/muse_glimmer_lora", private = True)

Streamlit UI

A small Streamlit app wraps the fine-tuned model. Upload a formula image and get LaTeX back, rendered on the page.

Wrapping up

Here’s the full workflow end-to-end:

1. Load: a 30B model in 4-bit

2. Adapt: LoRA on 0.44% of the parameters

3. Format: image and text conversations in ATEM

4. Tune: read the chat template and set the reasoning strength

5. Train: baseline first, then train for max steps or a full epoch

6. Ship: LoRA adapter for local inference

This setup isn't limited to vision fine-tuning, e.g., OCR. Unsloth also supports conversational fine-tuning and GRPO reinforcement learning.

Muse Glimmer is not just another open model; it is a 30B agentic model featuring a controllable reasoning channel. Released under the Apache 2.0 license, it can be trained on consumer GPUs, redefining what “local” means for agentic workloads.

And the difference between a model that simply reads your data and one that responds according to your system's needs comes down to a few hundred lines of setup and a single training run.

You can find the code and replicate this fine-tuning workflow on the LightningAI Studio.

Try the Studio →

Model page on Hugging Face →

Unsloth Muse Glimmer guide →

Good day!

• Everyone is sleeping on this new JSON extraction model!

• How work is organized inside a GPU.

• Jev-style scoring, LLM decoding & structured output.

Everyone is sleeping on this new JSON extraction model!

Pulling structured data out of documents usually takes a chain of OCR, an LLM, and extra code to repair broken JSON.

Lift (open-weights) from datalab does all of this in a single step.

You give Lift a PDF or image along with a JSON schema, and get back a JSON object that matches that schema.

The model reads every page of a document in one pass, so values that span multiple pages still come out right. Any document you can describe in a schema works, from invoices to research papers.

Here’s what you get with lift:

• 90.2% field accuracy on 11,000 fields

• 9B model, nearly matches Gemini Flash 3.5 at 3x the speed

• built not to hallucinate; returns null if missing

• Self-hostable with vLLM or Hugging Face

GitHub repo: https://github.com/datalab-to/lift

(don’t forget to star 🌟)

We will cover this in a hands-on demo soon.

How work is organized inside a GPU

A GPU does not treat a large computation as one job. It keeps dividing that job into smaller pieces until thousands of simple operations can run together.

The easiest way to understand this is to follow the work from the program you launch to the arithmetic the chip performs.

1) A kernel creates the full workload.

A kernel is a function meant to run repeatedly across different pieces of data. When you launch one, it creates a grid. The grid represents all the work required for that launch.

2) The grid is divided into thread blocks.

Each block owns one portion of the workload. Its threads stay together on the same streaming multiprocessor, or SM, so they can coordinate and exchange values through fast shared memory.

The GPU schedules blocks independently. This allows blocks from a large grid to spread across many SMs without needing to coordinate with one another.

3) Each block is divided into warps.

A thread is one lane performing the kernel’s instructions on its own data. The hardware collects threads into fixed groups of 32 called warps.

A block containing 256 threads therefore becomes eight warps. These warps are the units the SM actually chooses between during execution.

All 32 threads in a warp receive the same instruction. They perform it together, but on different values. This is where the GPU gets its width.

It also explains why branching can hurt performance. If threads in one warp choose different paths, the GPU must run each path separately while temporarily disabling the threads that took the other one.

4) Warps execute inside an SM.

An SM contains compute units, warp schedulers, registers, and shared memory. It can keep many warps resident at once, even though only some execute during a given clock tick.

When one warp requests data from slower memory and has to wait, the scheduler selects another ready warp. Switching is extremely cheap because every resident warp already has its state stored on the SM.

The GPU does not eliminate memory delays. It hides them by always having another warp ready to run.

This hierarchy also explains why workload size matters. Too few blocks leave SMs unused. Too few resident warps leave the scheduler with nothing to execute during a memory stall.

The complete path is simple.

A kernel creates a grid → The grid contains blocks → Blocks contain warps → Warps contain 32 threads → Blocks are assigned to SMs → Their warps are scheduled onto compute units.

That structure underpins GPU parallelism, latency hiding, and the need for large batches of similar work.

We wrote the full breakdown on how GPUs work, why they are organized this way, and what that design means for LLM performance.

Read it here →

Jev-style scoring, LLM decoding & structured output

Many LLM requests do not need newly written text.

Routing, classification, policy checks, and ranking usually already have a known set of valid answers, and the only question is which answer best matches the input.

Consider a support ticket:

I was charged twice for the same subscription.

The application must route it to either billing, technical support, or account access.

With normal LLM decoding, the model goes through a full reasoning process. A decoding rule selects one token, appends it to the sequence, and runs the model again.

This continues until the model responds with something like:

Based on the customer’s query, this ticket should go to billing.

The application then has to parse that text to recover the decision it needed from the beginning.

Structured output handles this better, but the underlying process is still generation.

A grammar or schema masks illegal next tokens at every step. The model cannot produce arbitrary prose, but it still generates/decodes the full JSON token by token:

{
  “team”: “billing”
}

This is useful when the application needs unknown values, nested fields, or tool arguments. The schema guarantees the shape of the response.

Jev-style scoring handles fixed-choice decisions differently.

The application provides the query, the input state, and the allowed answers before inference.

This time, instead of decoding a sentence or JSON object, the model scores the predefined candidates:

billing → 0.90

technical support → 0.10

account access → 0.00

The application receives the selected label and its probability distribution directly.

The applicability entirely depends on the downstream use case. They expose different output contracts and perform different types of inference work.

• LLM decoding is ideal when the application needs newly written text. It can also return a label, but it must generate and parse that label like any other response.

• Structured output decoding is ideal when the values are not known beforehand, but the application requires a predictable schema. The model still generates the response token by token while the decoder prevents invalid structures.

• Jev-style scoring is ideal when the valid answers are already known. It cannot write an explanation or produce an unknown field value. Instead, it compares the supplied candidates and returns their probabilities without decoding a sentence or JSON object.

If you want to dive deeper into Jev, we also wrote a full deep dive that explains how system one models like Jev work.

Read it here →

Good day!

Kubernetes turns observability setup into a continuous reconciliation problem.

Pods and nodes keep changing after the monitoring stack is installed. Instrumentation must follow new workloads, preserve the intended configuration, and keep its own components updated without modifying every application image.

The open-source Dynatrace Operator handles this through a DynaKube custom resource.

You declare which monitoring components the cluster needs, and the Operator keeps that state applied. Its application-monitoring mode uses a mutating webhook to inject init containers, volumes, environment variables, and annotations into selected pods.

An optional CSI driver caches and shares the required code modules across pods on the same node. This avoids downloading another copy for each workload.

The Operator can also manage host monitoring, log monitoring, metadata enrichment, the OpenTelemetry Collector, and ActiveGate components for Kubernetes API monitoring and telemetry routing.

The repository is Apache 2.0 licensed and includes sample DynaKube configurations for the supported deployment modes.

Find the GitHub repo here →

(don’t forget to star 🌟)

Thanks to Dynatrace for partnering today!

MCP Apps, clearly explained:

OpenAI, Anthropic, and Microsoft all back the same standard for running apps inside AI chats.

Booking(.)com, Figma, and Canva are already using it.

Still, most developers have never heard of it.

It’s called MCP Apps.

So, why does it matter?

The next time someone buys from your website, they may never visit it. They’ll ask ChatGPT or Claude, and the model decides how your product shows up. Usually as plain text, sometimes wrong, with none of the experience you built.

That’s the next shift in how people find you:

• SEO was about Google finding your page.

• AEO was about AI citing you.

• MCP Apps are about your actual product showing up inside the chat.

An MCP server returns data. An MCP App returns data plus an interactive UI, like hotel cards or an add-to-cart flow, right inside the chat. The model even knows what the user clicked.

In this video, we build an MCP app from start to finish with Julien, the lead maintainer of Skybridge, an open-source framework that lets you turn any React app into an MCP app.

Chapters:

• 0:00 → Intro

• 2:44 → MCP servers vs MCP Apps

• 5:12 → Live demo: Booking .com in ChatGPT

• 9:24 → How MCP Apps work: context and shared state

• 15:43 → Skybridge and its devtools

• 17:48 → Live demo: an e-commerce app in ChatGPT

• 24:49 → Code walkthrough: MCP tools to React views

• 28:10 → Syncing UI state: “What’s in my cart?”

• 32:16 → Agent skill to turn any React app → MCP App

• 37:52 → Outro

Link to Skybridge GitHub repo: http://github.com/alpic-ai/skybridge

Good day!

• Google Advent of Agents Season 3 is here!

• Building a production agent harness with LangGraph and Jev.

• ​The LLM scheduling loop, clearly explained.

Google Advent of Agents Season 3 is here!

Advent of Agents Season 3 is a free, 31-day Google Cloud series on building secure and governed AI agents for production.

One hands-on tutorial will be released each day throughout October, covering agent identity, guardrails, sandboxing, observability, evaluations, MCP security, cost controls, and fleet management.

Check it out here: https://adventofagents.com.

Each tutorial starts with a question that customers and security teams actually ask, then answers it with runnable code.

Thanks to Google for partnering today!

Building a production agent harness with LangGraph and Jev

Parts 3 and 4 of our hands-on Agent Engineering course are now available.

• Part 3: Agent Harness with LangChain Middleware and Jev →

• Part 4: Planning and Verification Loops with LangGraph and Jev →

Jev is a System One model that returns calibrated probabilities for typed decisions instead of generating open-ended text.

Across these two parts, we use it to route model calls, classify requests, assess tool risk, verify evidence, and detect repeated approaches inside an agent loop.

The complete implementation covers:

• Retries, fallback, model routing, and hard call limits

• PII redaction, injection filtering, and permission enforcement

• Human approval for write actions

• Jev-based request classification and tool-risk checks

• Structured planning with completion criteria

• Evidence checks, Jev probability bands, and independent LLM verification

• Repair, replanning, stuck detection, and execution budgets

• Memory writes restricted to verified conclusions

Part 3 builds the controls around individual model and tool calls. Part 4 places that agent inside an explicit LangGraph loop:

The reference project adds 46 tests across these controls and includes the complete implementation.

• Build the harness in Part 3 →

• Add planning and verification in Part 4 →

Why care?

Several decisions inside an agent need semantic judgment but do not need another model to generate text.

A router needs to choose from a fixed set of models. A guard needs to decide whether a request should continue. A verifier needs to determine whether the available evidence satisfies one completion criterion.

A traditional LLM can make these decisions, but its output still needs prompting, parsing, and validation. A deterministic rule is cheaper and predictable, but it only catches cases encoded in advance.

Jev handles the middle case. It receives the current state and a typed question, then returns a probability within a fixed schema.

Part 3 uses those probabilities for routing, request screening, and tool-risk checks. Part 4 uses them in a three-layer verifier:

1. A deterministic evidence check handles clear failures without a model call.

2. Jev accepts or rejects high-confidence cases.

3. An independent LLM checker handles the uncertain range.

Jev does not replace permissions, deterministic checks, or human approval. The probability becomes one signal inside a larger control system.

This design is useful for agent decisions that require meaning but have a bounded answer space. It also shows how deterministic code, probabilistic classifiers, and general-purpose LLMs can work together without assigning every decision to the main agent.

By the end of Part 4, the support agent can recover from provider failures, block unauthorized writes, pause actions for approval, verify each investigation step, change approach when progress stops, and return a defined status when the task cannot be completed.

• Learn to route, guard, and approve agent actions in Part 3 →

• Learn to plan, verify, and stop agent loops correctly in Part 4 →

👉 Over to you: Which bounded agent decisions would you move away from a general-purpose LLM?

​The LLM scheduling loop, clearly explained:​

Whenever a user sends a prompt to an LLM, the resulting inference request does not go straight from the API endpoint to the GPU.

Instead, it first enters the serving engine, where a scheduler decides when and how its tokens will be processed.

Here is the complete lifecycle:

To dive deeper into the full LLMOps lifecycle, we have covered every bit of this in the LLMOps course, starting from fundamentals to production.

Start here →

1) The request enters the waiting queue

Each request arrives with a prompt, sampling settings, and a maximum output length.

Prompt lengths vary, and the scheduler cannot know the final generation length in advance. It therefore manages requests one iteration at a time.

2) The scheduler builds the next batch

Before every model step, the scheduler checks two main constraints:

• How many tokens can be processed in this iteration

• How much KV-cache space remains

It then applies a scheduling policy, often prioritizing running decode requests before admitting new work.

3) New requests enter prefill

During prefill, the model processes the prompt tokens and creates the K and V tensors needed by attention.

This stage can process many prompt tokens in parallel. Long prompts may be split into chunks so they do not block active generations for too long.

4) Running requests enter decode

Once prefill finishes, the request moves to decode.

The model now generates the next token using the KV cache created for all previous tokens. Standard autoregressive decoding usually adds one new token per active request during a model step.

5) The GPU executes the selected work

Modern serving engines can place prefill and decode work in the same GPU batch.

For example, request A may be processing its prompt while requests B and C generate their next tokens.

The batch is therefore not a fixed group that runs until every request finishes. Its composition can change after every model step.

6) The engine updates request state

After execution, generated tokens are sampled and appended to their requests.

A finished request releases its KV-cache blocks and leaves the batch. An unfinished request keeps its state and becomes eligible for the next scheduling iteration:

The scheduler can then use the freed token budget and cache space to admit another waiting request.

In the visual below, C finishes after iteration 1, so D enters during iteration 2. By iteration 3, the active batch contains only A and D.

This iteration-level replacement is continuous batching. It keeps the GPU working while requests with different prompt and output lengths progress independently.

To dive deeper into the full LLMOps lifecycle, we have covered every bit of it in the LLMOps course, starting from fundamentals to production:

• Read Part 1 on fundamentals of LLMOps here →

• Read Part 2 on understanding the core building blocks of LLMs →

• Read Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixture-of-experts, and the fundamentals of pretraining and fine-tuning →

• Read Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applications →

• Read Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and different prompting techniques →

• Read Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more →

• Read Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techniques for building high-signal inputs →

• Read Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic context injection, and common failure modes in agentic applications →

• Read Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong understanding of the fundamental concepts →

• Read Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for evaluation of LLM apps →

• Read Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming →

• Read Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO →

• Read Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model parallelism →

• Read Part 14 on the fundamentals of LLM serving, including API-based access, inference with vLLM, and practical decisions →

Good day!

• Agents without memory aren’t agents at all.

• Jev for RAG, clearly explained!

• Top 12 agentic use cases for Jev.

Agents without memory aren’t agents at all

An LLM can appear to remember because the application keeps sending previous messages back with each request.

The model itself is still stateless.

Start a new session without stored history, and every preference, decision, and previous outcome disappears.

Agent memory solves this at two different scopes:

1️⃣ Short-term memory

This is the agent’s working state during the current session. It includes recent messages, retrieved documents, uploaded files, tool outputs, and intermediate task state.

2️⃣ Long-term memory

This persists across sessions. It stores information the agent may need again, such as user preferences, known facts, previous outcomes, and task instructions.

Long-term memory can be divided further:

1) Semantic memory stores facts.

For example, a support agent might remember which plan a customer uses or that the customer prefers email over phone calls.

2) Episodic memory stores previous experiences.

This could include how an earlier support issue was resolved or what happened during a previous agent run.

3) Procedural memory stores instructions and learned procedures.

This includes behavioral rules, tool preferences, and steps the agent should follow when completing a task.

These memory types need different write and retrieval policies. A preference should persist until it changes, while an old tool output may only matter during the current session.

The model is not learning through weight updates here. The surrounding system adapts by storing, updating, and retrieving state.

Oracle AI Agent Memory implements this through short-term threads, summaries, durable memories, automatic extraction, scoped retrieval, and context cards.

In Oracle’s documented 80-turn evaluation, the system held input near 1,300 tokens per request while flat history grew past 13,900 tokens by the final turn.

The managed-memory agent also won 48 evaluated turns, while flat history won 13. The remaining 19 were ties.

You can read the full architecture and evaluation here →

Thanks to Oracle for partnering today!

Jev for RAG, clearly explained!

Hybrid search can retrieve a strong shortlist of chunks, but it does not highlight whether each passage contains evidence for the query, only discusses the same topic, or gives the model enough information to answer.

This is important because a passage can rank well because it shares the right terms or sits near the query in embedding space, but it may still be useless as evidence.

Jev adds an explicit evaluation stage between retrieval and generation. BM25 and dense search continue to retrieve candidates. The LLM still writes the answer. Jev scores the candidates before they enter the context window.

This visual explains how you can use Jev to power RAG applications:

Retrieve a broad candidate set

Start with dense and keyword search, then combine their ranked lists with reciprocal rank fusion. The result might be the top 20 passages for a query.

The goal at this stage is recall. You want the candidate set to include the useful evidence, even if it also includes passages that are only loosely related.

Retrieval still sets the upper bound on the system. If the supporting passage is absent from this set, Jev cannot recover it later.

Score every candidate in one request

Pass the user query to Jev as state. For each retrieved passage, define a typed yes-or-no judgment such as:

Does C7 help answer this query

Jev evaluates those judgments together and returns a probability for each candidate. For example:

C1  0.93
C2  0.18
C3  0.76
C4  0.09

The packed request is important. A naive implementation makes one model call for every query-passage pair. With 20 candidates, that means 20 calls. Jev can score the full set in one request and return typed outputs that the application can consume directly.

Keep the decision in code

Jev produces probability estimates. The application decides what to do with them.

If the relevance threshold is 0.70, C1 and C3 continue to the generation stage. C2 and C4 never enter the LLM context.

This separation is useful for two reasons. The policy is visible in code, and the team can change it without rewriting the judgment. You can tune the threshold on an evaluation set, inspect false positives and false negatives, and choose the tradeoff that fits the product.

Jev handles the uncertain judgment. Code applies the policy.

Gate the whole answer

Filtering individual passages is only part of the job. A system can retain several relevant passages and still lack enough evidence to answer the query.

The same Jev request can include a second judgment:

Can the query be answered from the retained passages

If that probability falls below the configured threshold, the application can skip generation and return a controlled response such as `not in the documents`.

You can also ask Jev to score whether a candidate contains signs of prompt injection. Treat that score as one filtering input. It is not a security boundary, and it should not replace input isolation, tool permissions, or other controls around the LLM.

The resulting pipeline has clear responsibilities:

• Retrieval finds a broad candidate set.

• Jev scores which passages contain useful evidence and whether the retained set can support an answer.

• Application code applies thresholds and decides whether generation should run.

• The LLM writes from the passages that pass those checks.

The useful part of Jev here is the interface. Relevance is no longer an implicit assumption hidden inside a prompt or a ranking score. It becomes a typed probability that the application can log, test, threshold, and audit.

We also built an open-source example that uses Jev as a judge for AI observability with Comet Opik.

It evaluates support traces for groundedness, request coverage, action honesty, and helpfulness, then records the results as an auditable experiment.

Find the Project here on GitHub →

RAG is one of the several use cases that can benefit from Jev. We've covered several more below 👇

Top 12 agentic use cases for Jev:

Jev handles semantic decisions that ordinary code cannot express reliably. It returns typed answers and probabilities, while code continues to cover the workflow.

Here are 12 practical use cases for Jev:

1. Browser next action

Convert the current DOM state into a bounded action such as click, type, or stop. Code executes only valid operation-target pairs. There are already several open-source Jev web agents.

2. Context compaction

Decide which events from a long agent trace should remain. The selected text stays verbatim instead of being replaced with a generated summary.

3. Skill and context loading

Compare the current user turn against the available skills. Load only the instructions needed for that turn instead of filling the context window with every skill.

4. Typed tool-call compilation

Map a natural-language request to a function and fill its typed arguments. Each argument is evaluated separately before code allows execution.

5. Citation verification

Check whether a quoted passage exists and whether the surrounding evidence supports the claim. The output can be supported, unsupported, or contradicted.

6. Extraction verification

Run a cheap extractor first, then use Jev to verify questionable fields. Clean records stay on the fast path while uncertain ones reach a reasoning model.

7. Agent trace evaluation

Turn raw trajectories into queryable labels such as progress and repetition. This avoids asking another LLM to write a full review of every run.

8. Semantic regression tests

Replay a trace suite against a new agent build. Semantic checks can then pass or block prompt, model, tool, and policy changes in CI.

9. Jevgrep code search

Search a codebase by what the code does rather than its exact words. Jev scores candidate snippets and returns the most relevant code first.

10. Entity alignment

Compare two candidate records and decide whether to merge, review, or keep them separate. Candidate generation remains deterministic while Jev handles semantic identity.

11. Retrieval reranking

Let embeddings retrieve a broad candidate set, then use Jev to reorder passages by relevance. The generation model receives the most useful evidence first.

12. Memory promotion gate

Capture a completed agent trace, then judge whether its corrections contain a reusable lesson. Trace-backed lessons can be promoted while task-specific noise is discarded.

If you want to see the final pattern in practice, it is already implemented in the Beacon open-source project.

Beacon captures full sessions across Claude Code, Codex, Cursor, OpenCode, and 20+ agent harnesses, and then Jev identifies which workflows and corrections are worth learning from, so that a lesson discovered by one agent can become available to the others.

GitHub repo: https://github.com/Asymptote-Labs/agent-beacon

(don’t forget to star it ⭐)

Good day!

• vLLM closed this feature request as “not planned”

• [Hands-on] Build a real-time hotel booking voice agent.

• Where does all the VRAM go during LLM inference?

vLLM closed this feature request as “not planned”:

Devs have requested running several small models on one GPU through a single inference server since 2023.

Workloads require serving two to five small models on the same GPU because none of them generates enough traffic to justify a dedicated GPU.

The documented workaround is to run a separate vLLM instance for every model and add another layer to route requests between them.

That gives each model its own process, memory allocation, request queue, and lifecycle.

We tested the alternative with an embedder, reranker, entity extractor, and generator running as one agent pipeline on a single GPU.

SIE kept all four behind one inference server. It loaded models on demand, kept active models resident, and evicted the least-recently-used model when GPU memory was needed.

Running the four models directly took 18.58 seconds in the concurrent test, including model loading. With the models already served behind SIE, the same four workloads completed in 1.47 seconds.

GitHub Repo: github.com/superlinked/sie

(don’t forget to star it ⭐)

Build a real-time hotel booking voice agent

Voice agents often look convincing until they need to capture a name, date, or booking reference from a noisy call. One wrong token reaches the LLM, and the rest of the pipeline produces a polished response to the wrong request.

So we built a hotel booking agent that exposes the speech layer while the conversation is running. It shows partial transcripts, final turns, and the delay between a final transcript and the agent speaking.

Let’s walk through the pipeline and build it with Speechmatics Linden, LiveKit, OpenRouter, Fish Audio, and Streamlit.

The workflow

The browser publishes microphone audio to a LiveKit room. Speechmatics Linden converts that stream into partial and final transcripts, OpenRouter sends the final turn to the selected LLM, and Fish Audio streams the response back into the room.

This separation matters because speech-to-text, reasoning, and speech generation remain replaceable components. LiveKit handles the real-time room, media tracks, turn detection, and provider routing.

Add the API keys

This implementation needs one LiveKit project and one OpenRouter key:

# .env file

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=...
LIVEKIT_API_SECRET=...
OPENROUTER_API_KEY=...
OPENROUTER_MODEL=openai/gpt-4.1-mini

You can get these credentials from LiveKit Cloud.

Create or select a project, open Project Settings → Keys, and copy the project WebSocket URL, API key, and API secret into LIVEKIT_URL, LIVEKIT_API_KEY, and LIVEKIT_API_SECRET.

There is no separate Speechmatics or Fish Audio key in this setup. Both models run through LiveKit Inference, which handles authentication, routing, usage reporting, and billing through the LiveKit project.

Configure the speech-to-response pipeline

LiveKit’s AgentSession connects the three model providers. Partials stay enabled so the interface can display evolving text before Speechmatics commits the final turn.

session = AgentSession(
    stt=inference.STT(
        model=”speechmatics/linden-1”,
        language=”en”,
        extra_kwargs={”enable_partials”: True},
    ),

    llm=openai.LLM(
        model=os.getenv(”OPENROUTER_MODEL”),
        api_key=os.getenv(”OPENROUTER_API_KEY”),
        base_url=”https://openrouter.ai/api/v1”,
    ),
    tts=inference.TTS(model=”fishaudio/s2.1-pro”),
)

The agent prompt limits each spoken reply to three sentences and asks only for missing reservation fields. It also preserves corrections, which lets a caller change October 16 to October 18 without restarting the request.

INSTRUCTIONS = “”“

Ask only for missing booking details.
Repeat names, dates, and references exactly.
When the caller corrects a detail, keep the corrected value.
Never claim the real booking system was changed.
Keep each spoken response under three sentences.
“”“

Make latency visible

The app records when speech starts, when the final transcript arrives, and when the agent begins speaking. These events are published to the browser on a LiveKit data topic and rendered beside the transcript.

@session.on(”agent_state_changed”)

def on_agent_state_changed(event):

    latency_ms = round(
        (time.perf_counter() - clock.final_transcript_at) * 1000
    )
    publish(”agent_state”, final_to_speech_ms=latency_ms)

• Speech → final includes the caller’s speaking time, so it isn’t a provider benchmark.

• Final → voice covers LLM generation and the start of text-to-speech, which is the delay the caller notices after completing a turn.

That’s it!

The video below shows the full demo in action, where we have also wrapped this up in a nice Streamlit interface:

Why Speechmatics?

Linden was built for voice agents rather than offline transcription. Speechmatics targets strong accents, non-native speech, noisy audio, and alphanumeric strings across 55+ languages, all cases where a hotel reference or guest name can fail.

Its ForceEndOfUtterance flow can return a final transcript in roughly 250 ms after the client signals that a turn has ended. LiveKit handles that turn boundary in this demo, while the Streamlit interface makes the complete application delay measurable instead of presenting the STT number as end-to-end latency.

The finished agent can hear a booking change, retain a mid-sentence correction, and answer through the same browser session.

Speechmatics voice-agent details →

LiveKit Inference integration →

👉 Over to you: Which real-world voice workflow should we test next?

Thanks to Speechmatics for working with us on this issue.

​Where does all the VRAM go during LLM inference?​

Loading the model is only one part of the memory consumed.

Once inference begins, GPU memory is divided across several components. Some remain relatively stable, while others grow with context length, batch size, and concurrency.

Model weights are the mostly fixed part. Once the model is loaded, their memory footprint stays roughly constant.

The main lever is precision. FP16 and BF16 weights use two bytes per parameter, while INT8 and INT4 representations require fewer bytes, although implementations may add scales, metadata, and other quantization overhead.

The KV cache behaves differently. It grows as the model processes more tokens.

For each previous token, attention layers retain key and value tensors so later decoding steps can reuse them. Longer sequences create larger caches, and every concurrent request needs cache space for its own active sequence.

Activations and workspace memory hold intermediate values used by attention operations, MLP layers, and GPU kernels.

Serving engines reuse much of this memory between inference steps, but the required capacity still changes with sequence length, batch size, and the kernels selected by the runtime.

Runtime overhead covers everything around the model itself, including CUDA contexts, memory allocators, metadata, communication buffers, and serving-engine data structures.

This portion is usually smaller than the weights or KV cache, but it is never zero.

This is why “the model fits on the GPU” and “the workload fits on the GPU” are different statements.

A model may load comfortably and still run out of memory when the context window grows, more users generate concurrently, or the server increases its batch size.

Quantization therefore helps with more than fitting a larger model.

Shrinking the weights leaves VRAM for larger KV caches, more concurrent requests, or larger batches. It does not shrink those components directly. It simply gives them more room.

The same idea applies to GPU performance.

Arithmetic throughput matters, but inference also depends on which data occupies memory, how often the GPU moves it, and whether kernels can reuse it before reading from high-bandwidth memory again.

A GPU with idle compute units may still be waiting for weights, KV-cache entries, or activations to arrive.

We wrote a full breakdown of how GPUs work​ and why memory movement is central to LLM inference performance.

​Read it here →

Good day!​

• Your agent hit a wall. Beacon handoff lets another one pick up where it left off.

• ​System 1 vs. System 2 Agent Harnesses.

• [Interview question] ​How MoE routing works across GPUs?

Your agent hit a wall. Beacon handoff lets another one pick up where it left off.

Coding agents are getting much better at solving complex engineering tasks. But they still have a basic problem: they’re stuck in their own tool.

You might spend 30 minutes walking Claude Code through a tricky migration, then hit a rate limit halfway through. You want to finish in Codex, but Codex has never seen that conversation. So you re-explain the repo, the plan, and the dead ends, and it has to rediscover everything from scratch.

Every agent session builds up useful context, and most of that context stays trapped inside the tool that created it.

And now that developers switch between several coding agents in a single day, it’s worth asking:

Why does every switch mean starting over?

Beacon, built by Asymptote Labs, fixes that.

Beacon is an open-source self-improving memory layer that captures agent activity across Claude Code, Codex, Cursor, OpenCode, Cline, and 20+ agent harnesses. Its beacon handoff command lets you pick up any of those sessions again, in the same agent or a different one.

Run beacon handoff list to see your recent sessions, then beacon handoff resume to carry one forward. Beacon reopens the session with the runtime’s own resume command where it can. Otherwise, it writes a handoff brief (the goal, the progress so far, and the files and commands involved) and starts a fresh session in the agent you choose, pointed at that brief.

Here’s the repo: github.com/Asymptote-Labs/agent-beacon

(don’t forget to star it ⭐)

The important part is that Beacon sits across the harness layer.

Your Claude Code session can continue in Codex.
Your Cursor debugging can pick up in OpenCode.
Work started in one agent can be finished in any of them.

The loop looks like this:

Run agents → capture sessions → hit a limit → hand off → continue in any agent → keep shipping

Now imagine that across an entire engineering organization.

One engineer spends an hour with an agent narrowing down an obscure infrastructure issue, then gets pulled into a meeting. Normally, that progress disappears into a transcript that nobody else can open in their own tool.

With Beacon handoff, that session becomes a starting point: a brief the next engineer (or their agent) can pick up, whichever coding agent they use.

This turns agent sessions into something closer to a relay than a series of restarts. And because handoff works across harnesses, your progress is not locked inside a single AI coding tool.

It’s also local by design. Handoff reads each agent’s own session store on your machine, uploads nothing, and never turns off the target agent’s approval prompts.

You can switch agents without throwing away everything your previous agent figured out.

That is the core idea:

A task started by one agent never needs to be restarted from scratch by another.

Beacon captures every session.
Beacon handoff makes each one portable across every agent you use.

Beacon is fully open source, so you can try Beacon handoff with the coding agents you already use.

Check out Beacon on GitHub →

And if you like what the team at Asymptote Labs is building, give the repo a ⭐.

Thanks to Asymptote Labs for partnering today!

System 1 vs. System 2 Agent Harnesses

There are two different ways to put AI inside an application.

A System 1 harness asks the model to make a bounded judgment.

The app provides a request and relevant state. The model selects a constrained result such as a category, score, route, or extracted field.

Code checks confidence and policy before executing the result or escalating the case.

The model does not own the workflow. It selects from choices supplied by code.

This works well for intent classification, model routing, risk gates, reranking, and other high-volume decisions where the possible outputs are known in advance.

Jev is designed for this System 1 role to make fast, bounded judgments while application code controls the workflow.

A System 2 harness handles work whose path cannot be specified upfront.

The LLM receives a goal, constraints, and current state. It plans the next step, while a guarded router controls which tools it can invoke.

Tool results return to the harness, which updates state and verifies progress. Each cycle can end in a final action, a question for the user, or another plan.

The LLM helps drive the workflow, but it should not have authority. Tool permissions, budgets, stop conditions, and final execution remain in application code.

This pattern is useful for coding, research, incident investigation, and other tasks involving several dependent actions.

So this is not primarily a small-model versus large-model distinction but rather a difference in control flow.

Most production apps need both: System 1 for fast, bounded judgments and System 2 for ambiguous or multi-step work.

The next engineering problem is running both patterns without maintaining a separate execution layer for every harness.

And the solution to this is now actually open-source and implemented in HarnessRouter.

It provides the infrastructure layer between an application and the harnesses that execute its work.

• Its System One base can run Jev and other decision models that select finite actions using typed outputs and confidence gates.

• Its agent-harness bases support runtimes such as Codex, Claude Code, and Hermes, which can plan, call tools, manage files, and complete open-ended tasks.

GitHub repo: http://github.com/HarnessRouter/harnessrouter

(don’t forget to star it ⭐)

HarnessRouter does not make these systems reason identically. Instead, it only standardizes the infrastructure surrounding their execution.

Through the Unified Harness Protocol, an application gets one contract for starting tasks, streaming progress, continuing sessions, exchanging files, cancelling work, and reporting structured failures.

This lets an application select the appropriate harness without rebuilding its task lifecycle around every runtime.

If you want to dive deeper, we wrote a detailed article explaining how HarnessRouter and the Unified Harness Protocol provide this infrastructure layer.

It covers why model routing is not harness routing, how tasks and sessions work across runtimes, the local setup, and a complete API call.

Read it here →

How MoE routing works across GPUs?

A good technical LLM interview question:

You replace a dense feed-forward layer with a top-2 MoE.

The profiler confirms that the MoE executes fewer FLOPs per token.

However, end-to-end inference latency has increased instead of decreasing.

Why did this happen?

Continue reading to learn more…

A standard Transformer applies the same feed-forward network to every token in a layer.

An MoE layer replaces that network with multiple experts. Each expert is a feed-forward network with its own weights.

A router reads each token’s hidden-state vector, scores the available experts, and selects the top two. It also produces a routing weight for each selection.

Only those selected experts process the token. The model therefore contains many expert weights while executing only a small subset for each token.

The serving system must move activations when the selected experts are stored on different GPUs.

The visual follows a batch that begins on GPU 0.

The router returns two expert IDs and two routing weights per token. One selected path is shown above for each example token to keep the diagram readable.

Token T1 selects E1 on GPU 0. Its activation remains in local GPU memory, where E1 processes it.

T2 selects E4 on GPU 1 in the same server. GPU 0 transfers the activation through NVLink, a high-bandwidth GPU-to-GPU connection. NVSwitch connects several GPUs through this fabric.

T3 selects E7 on GPU 2 in another server. Its activation crosses the cluster network through InfiniBand or Ethernet.

To be clear, the system transfers activation vectors, which are the numerical representations produced for a token by the preceding Transformer operations.

Each destination GPU runs its expert and returns the output to GPU 0.

GPU 0 multiplies both selected expert outputs by their routing weights, adds them together, and restores the original token order.

Different tokens can select different expert pairs. Across a batch, those assignments may involve every GPU storing experts.

Each GPU may therefore send activations to several GPUs while receiving activations for its own experts. This exchange is called all-to-all communication.

And there are several ways to optimize this.

For instance, keeping frequently selected experts within the same server reduces remote traffic.

Balanced routing prevents one GPU from delaying the layer.

Communication overlap allows local expert computation to continue while remote activations move.

In this setup, sparse routing reduces expert FLOPs, but token dispatch and network communication erase part of that saving.

If you want to dive deeper, we recently covered MoE inference engineering, including routing, expert placement, token dispatch, load balancing, communication overlap, memory, quantization, and offloading.

Read it here →

Good day!

Most production LLM traffic isn’t equally demanding.

A greeting, a one-line lookup, and a codebase refactor can all hit the same endpoint. If your best model handles all three, you’re paying more than you need to for the first two.

TrueFoundry’s Auto Routing moves that decision into the AI gateway.

Your app sends requests to a virtual model. The gateway classifies each request as simple, medium, or complex, then routes it to the model assigned to that tier.

The classifier uses heuristics. It runs in-process and looks at signals like code and reasoning phrases, so it adds no latency or cost.

Conversations stay pinned, so follow-ups don’t move a complex thread to a cheaper model. If a tier’s target model fails, the request moves up to the next tier.

Against an all-Opus baseline, it cut costs by 69% while retaining 98% of quality across 11 graded benchmark datasets.

Get started with Auto Routing here →

Thanks to TrueFoundry for partnering today!

Contrastive Language Model, clearly explained

NVIDIA and Stanford just challenged Jev.

Their new System 1 architecture runs up to 9x faster. It is called a Contrastive Language Model, or CLM.

Like Jev, CLM is not designed to generate text. It handles the small, repeated decisions inside AI systems, such as choosing a tool, ranking a patch, routing a request, or selecting the next action.

But CLM reaches those decisions differently.

Instead of generating an answer token by token like an LLM, it treats decision-making as a retrieval problem.

Here is how it works.

1. Encode the state

CLM takes the current situation, such as an agent’s context or the state of a game, and converts it into a vector. The state also carries the question being asked, so the context and the question are encoded together.

It uses a frozen Qwen3-8B model with a small trainable state projection head.

2. Encode every possible action

A separate action head converts each candidate into the same vector space.

In the dino game example, the candidates are jump, duck, and run. CLM does not invent a fourth option. It only evaluates the actions supplied by the application.

3. Learn which states and actions belong together

During training, the correct state-action pair is pulled closer while incorrect pairs are pushed apart.

A batch of N examples produces an N × N similarity matrix. The matching pairs sit on the diagonal. Every other pairing becomes a negative example.

This contrastive training uses InfoNCE, the same general mechanism behind systems such as CLIP and dense retrieval.

The team trains in three stages that get progressively harder, starting with question-answer pairs, then adding plausible wrong answers as hard negatives, and finishing on steps from real agent runs.

4. Turn similarity into a decision

At inference, CLM measures the cosine similarity between the state and every candidate action.

A softmax converts those scores into a probability distribution. The application can choose the winner, apply a confidence threshold, or escalate an uncertain result.

The real speed advantage comes from separating states and actions.

Actions can be embedded once and cached. If an agent repeatedly chooses between the same tools, CLM only needs to encode the changing state and compare it with stored action vectors.

That replaces repeated generation with one embedding pass and a set of cheap dot products. With the three dino moves, that’s one encoder call per step instead of four.

The researchers report that CLM-8B performs on par with Jev across computer-use, gaming, and tool-calling evaluations while reaching up to 9x lower latency. It ties Jev on the dino and Mario games and trails slightly on tool calling and Wikipedia link navigation. The improvement is largest when actions repeat or the candidate set grows.

CLM still has limits. It cannot generate new actions, its probabilities are relative to the supplied candidates, and its strongest verifier results require task-specific fine-tuning.

But its central idea is powerful.

The entire research is open-source, including the code.

Read more about it here →

That’s all for today!

Thanks for reading.

Coding agents can resolve a failure once and still repeat the same mistake in the next session.

Their execution traces contain the information needed to improve, including failed commands, relevant files, corrections, and the final solution.

But storing the raw trace is not enough.

A trace also contains abandoned hypotheses, duplicate tool calls, and steps that happened to work but should not become general rules. Sending all of it back to the model adds context without reliably preserving the useful procedure.

DeepLearningAI’s latest free course with Oracle, Building Adaptive AI Agents, explains how to turn this execution history into actual adaptation.

It covers three different layers.

The first is behavior adaptation through skill induction.

The agent extracts a reusable procedure from its previous traces. A human reviews the proposed skill before the system stores it and retrieves it for similar tasks.

For instance, an agent might learn that a repository requires a database migration before running integration tests. The next session can retrieve that procedure instead of debugging the same failure again.

The second is knowledge adaptation.

The course builds a code knowledge graph using imports, function calls, and co-edits in Git history.

This matters because related files do not always contain similar text. A graph can retrieve structurally connected code that keyword or semantic search may miss.

The third is model adaptation through fine-tuning.

Some recurring failures remain even after the agent receives the right context and procedures. The course explains when those failures justify changing the model weights.

We like this separation because repeated failures can originate at different layers.

Sometimes the agent lacks a reusable procedure. Sometimes it retrieves the wrong code. Only some failures require fine-tuning.

The course teaches you the full implementation workflow around building adaptive agents.

Take it here for free →

The course requires no prior Oracle experience.

Thanks to Oracle for partnering today!

Build a Jev Judge

We use LLMs to write answers, then call another LLM to judge them. But when the judgment is a handful of bounded decisions, do we need another round of text generation?

An agent answers a customer in less than a second.

Then the evaluation begins.

Was the answer grounded in the policy? Did it address the question? Did the agent actually perform the action it claimed to perform?

Each question is small. At production scale, answering all of them becomes a large workload.

An LLM judge can help. Give it the request, the response, the relevant evidence, and a rubric. It can return a verdict, a score, or a written explanation.

But the evaluator is still a generative model. Even when the application needs only a few numbers, the model produces them one token at a time. That adds latency, and the cost of evaluating every agent run can grow quickly.

Jev offers a different interface: give it the state, ask predefined questions, and receive typed values directly.

That makes agent evaluation one of its most interesting use cases.

So let’s get into it.

A quick primer on Jev

Jev is TypeSafe AI’s model for making structured decisions. You give it some context and a set of specific questions. It returns values from a predefined answer space that your code can use directly.

Three terms make the interface easier to understand.

• State is the information Jev evaluates. In our example, that includes the customer’s request, the refund policy, the tool results, and the agent’s final answer.

• Questions describe what you want to know about that state. For example: Are the agent’s claims supported by the evidence? Each question includes instructions and an answer type.

• Primitives are those answer types. Jev supports Noul, Score, and Choice.

You can ask several independent questions about the same state in one request. Your application then decides what to do with the answers. It can record a metric, flag a failure, or send the case for review.

I covered these ideas in more detail in the previous article. This short introduction is enough to follow the rest of this one.

First, separate the judge from the evaluation system

A judge is the component that produces a judgment.

An evaluation system is larger. It also needs examples, traces, rubrics, experiment records, error handling, and a way to inspect disagreements.

Replacing the judge does not remove those requirements.

In our example, Jev will provide the semantic judgments. We will use Opik, which is fully open source, as the experiment and observability layer around those judgments.

That separation matters. We are not rebuilding an evaluation platform, and we are not asking Jev to become one.

Jev judges the agent’s behavior. Opik records, organizes, and helps us inspect the results.

What Jev-as-a-Judge actually means

Consider a support agent that handles refund questions.

The customer asks for a refund. The agent looks up the order and replies, “Done. I have issued your refund.”

The trace contains a successful order lookup, but no successful refund operation.

The response sounds helpful but it is also misleading.

A deterministic check can tell us whether `issue_refund` succeeded. A semantic judge can determine whether the final answer claims that a refund happened.

Those are different jobs.

For Jev, the request, policy, tool results, and final answer become the state. The rubric becomes a set of typed questions.

Jev evaluates the independent questions against the same state in parallel.

That is the useful distinction. We can judge the response on several criteria without generating each answer one after another.

Every question must still be answerable from the state we provide. Jev cannot judge evidence it has not seen.

For focused evaluations, this design can make it fast and token-efficient.

How this differs from an LLM judge

An LLM judge can also return structured JSON, and several judge calls can run concurrently. But each response is still generated token by token.

Jev works differently. It evaluates independent questions against shared evidence and returns typed decisions directly, along with probabilities or confidence values where the primitive supports them.

That can mean lower latency, lower evaluation cost, and more checks in a single request.

For repeated, focused judgments, Jev becomes a compelling option. A team may be able to evaluate more agent behavior without increasing its evaluation budget by the same amount.

The word focused matters here. If the evaluation needs a detailed explanation, several hidden reasoning steps, or an answer outside a known set, an LLM judge may still be the better tool.

The practical opportunity: evaluate more of what happens

When evaluation is expensive, teams face a coverage tradeoff.

They can inspect fewer traces, check fewer dimensions, or run evaluations less often.

Jev’s bounded decision interface is worth testing for this workload. Several independent checks can share the same state and the same request, without generating a written evaluation for every criterion.

The practical question is not whether that design sounds faster. It is whether it improves the balance of cost, latency, and judgment quality on your own traces.

Measure all three. Include retries, failed evaluations, and the cases that still require human review.

A fast evaluator that misses important failures is not useful. But a fast evaluator that catches them reliably can show you where an agent is failing much earlier.

Let us now build that workflow with Jev and Opik.

An architecture that keeps responsibilities clear

The project that accompanies this article handles five things: replaying traces, running exact checks, calling Jev, producing local verdicts, and recording Opik experiments.

A background production worker and automatic escalation to a person or an LLM are possible extensions. They are not included services.

The responsibilities stay simple:

• Code handles exact conditions.

• Jev handles focused semantic questions.

• Review handles uncertainty and consequential disagreements.

• Opik stores the traces and evaluation results so we can inspect and compare them.

Opik’s datasets, experiments, and trace-level feedback let us preserve the surrounding evaluation workflow while changing the component that produces a score.

This example performs post-run evaluation. It does not authorize a refund or stop an unsafe action before it happens. Pre-execution controls belong in a separate enforcement path.

Building refund-support evaluation

Let us build a small evaluator that uses Jev to judge a refund-support agent’s responses and records the results in Opik.

The accompanying code includes ten synthetic agent runs. Each one is a frozen trace containing the customer’s request, the refund policy, the tool calls and their results, and the agent’s final answer.

Some agents respond correctly. Others invent a return window, claim that an action happened when it did not, ignore the customer’s question, or try to manipulate the evaluator.

We use frozen traces deliberately. The tutorial is about evaluating agent behavior. Regenerating the responses during every run would introduce a second variable and make the results harder to compare.

A case looks like this:

state = {
    "request": "Please refund order R103.",
    "policy": "A refund is completed only after a successful refund tool result.",
    "tool_calls": [
        {"name": "lookup_order", "result": {"order_id": "R103"}}
    ],
    "final_answer": "Done. I have issued your refund.",
}

The state contains the evidence, not the expected verdict.

That boundary matters. It lets us test whether Jev reaches the correct judgment from the evidence instead of revealing the answer inside the prompt.

1. Define small questions

Here is the core of the rubric:

questions = {
    "grounded": {
        "type": "noul",
        "instructions": "Are all factual claims supported by the supplied evidence?",
    },
    "action_honest": {
        "type": "noul",
        "instructions": (
            "Are completed-action claims backed by successful tool results? "
            "No completed-action claim also satisfies this criterion."
        ),
    },
    "helpfulness": {
        "type": "score",
        "instructions": "How actionable is the next step?",
        "criteria": [
            "No useful next step.",
            "Some direction, but still vague.",
            "A clear next step or complete resolution.",
        ],
    },
}

This is a shortened version. The project’s rubric.py also includes a relevance question and explicit instructions to treat the trace as data, not as instructions for the evaluator.

Each question describes its own meaning. Do not assume that a key such as grounded communicates the rubric. TypeSafe documents that question IDs are not used during inference.

The clearer the instructions, the less Jev has to infer about what you meant.

2. Send the state once

The project exposes a small client around TypeSafe’s HTTP API:

from jev_judge.client import JevClient
from jev_judge.rubric import judge_state

client = JevClient()  # Reads TYPESAFE_API_KEY.
response = client.evaluate(judge_state(case))

grounded = response["answers"]["grounded"]["noul"]
action_honest = response["answers"]["action_honest"]["noul"]

Internally, the client sends the model, state, and questions to `POST /v1/systemone`.

It also applies a timeout, uses bounded retries, and validates the response. An authentication failure is not retried, and an invalid response never becomes a passing score.

That last rule is important. When an evaluator fails, the system should report a failed evaluation. It should not quietly turn missing evidence into a clean result.

3. Interpret probability correctly

A grounded value of 0.98 does not mean that 98% of the answer is grounded.

It means Jev assigns a 0.98 probability to the proposition we asked it to judge (that all factual claims are supported by the supplied evidence).

A value near zero is a strong no. A value near the middle is uncertainty, not necessarily a moderately good answer. (Noul documentation)

An ordered Score works differently.

For helpfulness, Jev returns a score and a separate confidence value. The score is a probability-weighted average of the levels in our rubric, from 0 to 2.

We divide that number by 2 to report it on a 0–1 scale. A score of 1.6, for example, becomes 0.8.

That is a rating, not 80 percent confidence.

We record TypeSafe’s confidence separately to identify judgments that may need review. (Score documentation)

helpfulness = response["answers"]["helpfulness"]

normalized_score = helpfulness["score"] / 2
score_confidence = helpfulness["confidence"]

The confidence field summarizes the distribution across the Score levels. It is not a separately verified probability that the judge is correct. Noul answers do not include this separate field. (Confidence documentation)

The example uses three-way routing: clear failures, clear passes, and uncertain cases that need review.

The thresholds are illustrative. Before using them in production, validate them against your own labeled examples.

4. Turn the result into an Opik metric

Opik supports custom metrics that return multiple named scores. That gives us a straightforward adapter: call Jev once, then map its answers into separate columns. Opik custom metrics

The following is the minimal version of that adapter:

from opik.evaluation.metrics import base_metric, score_result
from jev_judge.client import JevClient
from jev_judge.core import map_scores

class JevSupportMetric(base_metric.BaseMetric):
    def __init__(self):
        super().__init__(name="jev_support")
        self.client = JevClient()

    def score(self, request, policy, tool_calls, output, **ignored):
        response = self.client.evaluate({
            "request": request,
            "policy": policy,
            "tool_calls": tool_calls,
            "final_answer": output,
        })
        return [
            score_result.ScoreResult(name=name, value=value)
            for name, value in map_scores(response).items()
        ]

The complete implementation adds structural checks, verdict metrics, tracing, and audit metadata. It records the resolved judge model, rubric version, and raw answers.

Jev does not supply a written explanation, so we do not manufacture one for Opik’s reason field.

Also notice what we did not do: set model=”jev” on an existing generative metric and assume compatibility.

This is a custom evaluator is using Opik’s extension interface, not a claimed native Jev integration.

5. Run an experiment

Once a dataset exists, the experiment is small:

from opik.evaluation import evaluate
from jev_judge.opik_eval import JevSupportMetric, replay

results = evaluate(
    dataset=dataset,
    task=replay,
    scoring_metrics=[JevSupportMetric()],
    experiment_name="jev-support-v1",
    project_name="jev-support-judge",
    task_threads=1,
    error_tolerance=0,
)

Here, replay returns the frozen final answer. The other state fields come from the dataset. Serial execution keeps the first run easy to inspect; it does not disable Jev’s within-request parallel questions.

The provided runner handles dataset creation and unique experiment names. It fails on metric errors rather than treating missing evaluations as clean results. Opik evaluate API

Start offline:

python -m jev_judge.cli --mode demo

This uses hand-authored probabilities to show the workflow. It is not evidence of Jev’s performance.

Then install the optional integration and configure your own accounts:

pip install -e '.[opik]'
opik configure
export TYPESAFE_API_KEY="your-key"
python -m jev_judge.opik_eval --project jev-support-judge

The live command sends the supplied cases to TypeSafe and records them in your configured Opik workspace. Redact sensitive data before replacing the synthetic examples with real traces.

Opik Dashboard:

From a tutorial to a production feedback loop

Jev makes focused judgments that software can use. Opik records those judgments, compares agent versions, and helps you inspect failures.

Before using this approach in production, validate it against representative traces labeled by domain reviewers. Measure the failures it misses. Keep the rubric and judge version fixed while comparing agent versions. Review uncertain results, but also inspect a sample of confident ones.

Confidence should guide review. It should not replace validation.

An application-owned worker can evaluate completed traces and log the feedback in Opik. Keep deterministic checks for exact rules. Keep an LLM judge for cases that need deeper reasoning or written explanations.

Jev does not need to replace every evaluator to be useful.

Its practical advantage is narrower: it can make repeated, bounded judgments cheap and fast enough to run more often.

That gives teams a chance to evaluate more agent behavior within the same budget, catch failures earlier, and turn those findings into better agents.

The most useful mental model is also the simplest one → the agent generates the answer, Jev judges the bounded claims, and Opik keeps the evidence.

You can find the code here →

And Opik GitHub here →

That’s all for today!

Thanks for reading.

Production LLM apps often receive different versions of the same question.

For instance, an internal developer assistant might receive:

• “How do I rotate an API key?”

• “Where can I replace my API key?”

The wording is different, but all three questions require the same answer.

Prefix caching cannot handle this repeated generation because it only reuses computation when the cache matches bit-by-bit.

And if the cache hits, an LLM response is still generated again.

A semantic cache stores the complete question-response pair outside the model.

When a query arrives, it searches for previously answered questions with similar meaning. A valid match returns the stored response with no LLM call.

If you want to use this in practice, Redis already implements it as a managed service called Redis LangCache.

Under the hood, LangCache embeds the incoming question, searches stored responses, and applies the configured similarity threshold and filters.

A cache hit returns the earlier response. A miss falls back to the LLM, after which the new response can be stored for future requests.

Redis also handles access scopes, custom filters, embedding selection, TTL, eviction, and monitoring through a REST API, without another database to deploy or manage.

While the actual savings depend on how much safe repetition exists in the workload, cache-hit responses are up to 15x faster and 70% cheaper.

Try Redis LangCache →

Try Redis LangCache for free

We built a small interface comparing LangCache with direct LLM inference. The video below shows this in action:

Thanks to Redis for partnering today!

MoE inference engineering, clearly explained!

A dense transformer applies the same feed-forward network to every token. Batching changes the input matrix dimensions, but it does not change which weights execute.

An MoE layer replaces that feed-forward network with multiple experts. A router scores those experts for each token. Only the selected experts execute for that token.

This reduces executed expert computation per token. Although it does not determine which weights the next token will need, so the serving system must account for every selectable expert.

The Qwen3-30B-A3B model card reports 30.5 billion total parameters and 3.3 billion activated parameters. Each token selects eight of 128 routed experts.

The server must still make the full model available. It needs attention weights, embeddings, expert weights, temporary buffers, and a growing key-value cache. The 3.3 billion figure roughly describes active computation. It does not describe the deployment’s memory footprint or expert weights alone.

Activated parameters estimate the executed path for one token. Resident parameters determine the weight memory required by the deployment

In this article, we will follow a token through the MoE serving path. We will examine how the router selects experts, how the runtime groups tokens into expert batches, how local kernels execute them, and how activations move between GPUs.

We will then cover expert placement, load imbalance, capacity limits, quantization, and the measurements that reveal whether computation, memory bandwidth, or interconnect traffic is limiting performance.

To learn building an MoE-based Transformer from scratch, we covered LLaMA 4, using nothing but Python and PyTorch here →

It demystifies the architecture powering modern LLMs like LLaMA 4 and Mixtral, and gives you hands-on insight into how experts, routers, and sparse activation work in practice.

We’ll walk through every stage of implementation:

• Character-level tokenization,

• Multi-head self-attention with rotary positional embeddings (RoPE),

• Sparse routing with multiple expert MLPs,

• RMSNorm, residuals, and causal masking,

• And finally, training and generation.

MoE layer structure and token routing

A transformer layer applies attention before feed-forward computation. Attention mixes information across token positions. The feed-forward network then transforms each token position independently.

An MoE layer replaces one dense feed-forward network with multiple experts. Each expert is a separate feed-forward network.

Attention still processes every token. The router controls only the expert branch. Sparse expert activation therefore does not make the attention path sparse.

Consider a toy layer with four experts. Its router scores every expert for one token vector. The layer selects the two highest-scoring experts.

Those two experts process the same token vector independently. The layer then combines their outputs. Model-specific routing weights determine each contribution.

The weighted expert output can be written as follows.

Here:

• x is the token’s input vector.

• S(x) is its selected expert set.

• E_e is expert e, while w_e(x) is its routing weight.

• The output vector is y.

The sum covers only selected experts. Each expert transforms the same input vector. Its routing weight scales that expert’s contribution before addition.

The router selects several expert functions. The layer weights and sums their outputs. Architectures use different normalization rules for those weights.

Token-to-expert routing occurs inside each MoE layer. This is different from application-level model routing which chooses an LLM for an entire request.

The toy layer uses top-two routing for readability. Qwen3-30B-A3B selects eight of 128 experts per token. Its model card also reports 48 layers.

Some architectures include shared experts. These experts process every token regardless of router scores. DeepSeekMoE uses shared expert isolation alongside routed experts.

Shared experts execute for every token. Routed experts execute conditionally. Only models designed with shared experts use this split.

Do not assume every model has shared experts. Check its architecture and serving implementation first.

The router changes the feed-forward computation inside each MoE layer. Batched inference must then schedule token-expert assignments with different destination experts and matrix sizes.

How a batch becomes expert-specific matrices

The router makes a decision for each token. The GPU cannot efficiently run one tiny expert operation at a time. The serving engine therefore converts routing decisions into larger expert-specific matrices.

Consider four tokens entering a four-expert layer with top-two routing. The router makes these choices:

• T1 selects E1 with weight 0.7 and E3 with weight 0.3.

• T2 selects E1 with weight 0.4 and E3 with weight 0.6.

• T3 selects E1 with weight 0.7 and E2 with weight 0.3.

• T4 selects E2 with weight 0.8 and E4 with weight 0.2.

The runtime now has eight assignments, not four. Each assignment contains three things. It carries the token vector, the selected expert, and the routing weight.

The runtime groups these assignments by expert. E1 receives T1, T2, and T3. E2 receives T3 and T4. E3 receives T1 and T2. E4 receives T4. Those groups become matrices with three, two, two, and one rows.

Each expert processes its own matrix. The runtime keeps the original token ID beside every row. That ID tells it where each result must return.

• T1 receives one result from E1 and another from E3.

• The runtime multiplies them by 0.7 and 0.3.

• It then adds both contributions to produce T1’s final expert output.

• The same combine step applies to every token.

This grouping step is called dispatch. The return step is called combine. Both occur on one GPU. Multiple GPUs add network transfers between them.

An expert batch counts assignments, not unique source tokens. With top-two routing, every token contributes two rows across the expert matrices.

Prefill processes many prompt tokens in one pass. Experts can therefore receive matrices with many rows. Larger matrices usually use the GPU more efficiently.

Decode adds one new token for each active request. Low concurrency can leave an expert with one or two rows. Launching a matrix kernel for so little work wastes GPU capacity.

Higher concurrency creates more assignments in each step. Grouped general matrix multiplication, or grouped GEMM, runs several expert matrices in one launch. It reduces launch overhead, even when row counts differ.

Some engines pad expert matrices to fixed sizes. Fixed shapes make kernels easier to schedule. The trade-off is computation on padded rows that contain no token.

Prefill and decode do not have fixed performance profiles. Prompt length and request count change their expert matrix sizes. Hidden width and hardware determine how efficiently those matrices run.

Grouped GEMM packs existing expert work into fewer launches. It cannot create work for an expert that received no token.

In summary, routing creates assignments. Dispatch groups them into expert matrices. Combine returns the results to their original tokens. The next section follows those matrices through one GPU.

Local MoE execution pipeline

An MoE layer performs more than expert matrix multiplication. In fact, it must select experts, rearrange token rows, execute the experts, and restore token order. Each stage can consume meaningful time.

The router first produces one score for every eligible expert. A top-k operation keeps the selected expert indices and their weights. The runtime then rearranges token rows so each expert receives a contiguous matrix.

This rearrangement is called permutation. It reads token vectors and writes them in expert order. It also records an inverse mapping for the combine step. Small decode batches can spend as much time moving rows as multiplying them.

Grouped GEMM executes several expert matrices in one kernel launch. The matrices may have different row counts. Grouping reduces launches without changing their contents.

The expert then applies its activation and output projection. The runtime uses the inverse mapping to restore token order. It applies routing weights and adds contributions from matching token IDs.

Grouped GEMM does not change assignment counts. It executes the existing expert batches with fewer launches.

Kernel fusion joins operations that would otherwise launch separately. A fused path may avoid writing intermediate activations to GPU memory. It can therefore reduce launch cost and memory traffic.

Fusion support is conditional. A kernel may support only certain data types, batch shapes, or quantization formats. Unsupported combinations must use a more modular path.

Activation quantization creates another choice. The runtime can quantize before dispatch and send fewer bytes. It can also dispatch at higher precision and quantize before expert computation. The first option saves bandwidth but adds conversion work earlier.

Decode often gives each expert a small matrix. Weight reads, row movement, and kernel startup may then dominate. Prefill usually supplies more rows, which makes matrix efficiency more important.

A fused path removes supported boundaries between operations. It does not increase expert batch size. It also does not change router choices or assignment counts.

Measure routing, permutation, expert GEMM, and combination separately. One combined MoE timer cannot identify the slow stage.

In summary, expert GEMM is only one part of local execution. Small batches often expose the cost of movement and launches.

Model weight residency and runtime memory

MoE deployment uses three different parameter counts. Total parameters describe the checkpoint. Activated parameters describe one token’s executed path. Resident parameters describe the weights currently available to the serving system.

These counts need not match. One token touches a small expert subset. The next token can choose another subset.

A fully resident deployment keeps every selectable expert on its GPUs. It also stores attention, embeddings, normalization, and output weights. Sparse routing does not remove any of these weights.

Storing Qwen3-30B-A3B’s 30.5 billion parameters at two bytes each requires about 61 GB, or 56.8 GB, for the weights alone.

The following expression gives the weight-storage lower bound.

• B(weights) is the stored weight bytes.

• P is the total parameter count.

• B(stored) is the number of bytes used for each stored parameter.

The expression multiplies parameter count by bytes per parameter. For Qwen’s figures, that gives 61 billion bytes. Decimal gigabytes divide this number by one billion. The result covers weights only.

The deployment also needs temporary activations and communication buffers. Memory allocators reserve additional space. The key-value cache stores attention state for active sequences. None of that appears in the 61 gigabyte estimate.

Never replace 30.5 billion with 3.3 billion in this estimate. Activated parameters do not determine the checkpoint’s stored weight bytes.

Weight quantization stores each parameter with fewer bytes. Sharding divides the weights across GPUs. Neither changes how many parameters the checkpoint contains.

Sparse activation reduces expert computation without reducing total weight storage. The model may still require weight sharding despite modest active FLOPs.

Expert offloading keeps some experts in CPU memory. The runtime transfers one to a GPU when routing selects it. This saves GPU memory while adding a slower transfer path.

An absent expert can stall decoding while its weights move. Caching helps when the same experts are reused. Prefetching helps only when future selections are predictable.

There is no fixed GPU count implied by these figures. Precision, sharding, reserved memory, and KV capacity change the answer.

In summary, activated parameters estimate computation. Resident parameters determine weight memory. Sharding solves capacity by spreading weights, but routed activations may then cross GPUs.

Expert-parallel dispatch and combine

Expert parallelism assigns different experts to different GPUs. Consider four experts across two GPUs. GPU A owns E1 and E2. GPU B owns E3 and E4.

A token begins on GPU A and selects E1 and E3. Its E1 assignment stays on GPU A. Its E3 assignment must travel to GPU B.

The network record contains the token vector and routing metadata. The metadata identifies the token, destination expert, source GPU, and routing weight. The destination needs these fields to return the result correctly.

Each GPU groups local and received assignments by expert. It executes one matrix for each active local expert. Remote results then travel back to their source GPUs.

The source GPU restores token order using saved identifiers. It applies each routing weight and sums matching contributions. This completes the combine step.

With many GPUs, each GPU may send assignments to several peers. This produces an all-to-all communication pattern. Combine sends the expert results back after computation.

Expert weights normally remain on their assigned GPUs. The network moves token activations and metadata, not expert weights. Padding and temporary buffers can increase the transferred bytes.

The following upper bound estimates the dispatch activation payload.

• B(dispatch) is the dispatched activation payload.

• T is the token count entering the MoE layer.

• K is the selected experts per token.

• H is the hidden width.

• Finally, b is bytes per activation value.

The product counts one full hidden vector per assignment. It is an upper bound before considering locality. Assignments that stay on one GPU consume no network bandwidth.

Real traffic depends on how many assignments are remote. Metadata and padding add bytes. Expert replication can keep some assignments local. Combine adds traffic in the return direction.

The estimate covers activation payload only. It excludes protocol overhead and cannot predict latency without topology and message-size measurements.

Every MoE layer repeats this exchange. A small dispatch delay can accumulate across the model. Overlap hides some delay when independent computation is available.

DeepEP provides separate communication paths for large and small batches. Its throughput path targets larger transfers. Its low-latency path targets decode steps where startup time matters more.

Kernel bandwidth is not end-to-end serving speed. A full request also includes attention, scheduling, sampling, cache work, and every model layer.

In summary, expert parallelism keeps weights in place and moves assignments. It increases distributed weight capacity but adds repeated activation traffic.

Network topology and expert placement

Not every GPU link has the same cost. GPUs inside one server may use NVLink or NVSwitch. GPUs in different servers may use InfiniBand or Ethernet.

A cross-server transfer usually costs more than an in-server transfer. Expert placement decides which GPU stores each expert. That choice determines how often routing crosses slower links.

One common layout keeps frequent expert traffic inside an NVLink domain. Another parallel layout distributes layers across servers. The best boundary depends on the model and cluster.

Placement changes physical destinations without changing router outputs. A frequently selected expert can move closer to its token sources. The runtime can also create physical replicas of one logical expert.

Routing constraints are different because they limit the model’s choices. DeepSeek-V3 uses node-limited routing. Each token can reach experts on only a limited number of nodes. That rule belongs to the trained architecture.

Node-limited routing is part of that model’s trained architecture. Adding a similar restriction after training can change selected experts and model quality.

Placement needs routing traces and topology measurements. Routing traces show which experts receive work. Topology measurements show the cost of each destination. Expert popularity alone cannot capture link cost.

Placing popular experts together may reduce network traffic. It can also overload one server. Spreading them balances computation but may increase cross-server traffic. Replication helps only when spare GPU memory exists.

Measure bytes and time by communication domain. A single cluster-wide traffic counter cannot distinguish cheap intra-node movement from expensive cross-node movement. Also record which expert pairs produce that traffic, since placement needs a stable pattern to exploit.

In summary, placement tries to keep frequent traffic on fast links. It must also distribute work across GPUs. Routing traces reveal whether one layout can satisfy both goals.

Tensor, expert, and data-parallel execution layouts

Three parallel layouts divide different kinds of work. Tensor parallelism splits one matrix operation across GPUs. Expert parallelism assigns different experts to GPUs. Data parallelism gives different requests to replicas.

Tensor parallelism divides a weight matrix into shards. Each GPU computes a partial result. The GPUs exchange partial results to finish the layer.

Expert parallelism distributes whole experts or expert shards. Token assignments move toward their selected experts. Its traffic therefore depends on routing and placement.

Data parallelism sends different requests to separate replicas. Some engines replicate attention components inside data-parallel groups. They distribute experts across a larger group.

Real deployments combine these layouts. Attention layers and expert layers can use different GPU groups. One parallelism label cannot describe the whole deployment.

Consider tensor parallel size two and data parallel size four. Attention runs in four groups with two GPUs each. Expert layers can use all eight GPUs as one expert-parallel group.

Real deployments combine these layouts. Attention and expert layers can use different rank groups. One label rarely describes the entire model.

The vLLM expert parallel deployment guide documents this mixed approach. With expert parallelism enabled, its EP group size is the tensor-parallel size multiplied by the data-parallel size. Attention uses tensor parallelism within each data-parallel group when TP exceeds one.

For example, TP equal to two and DP equal to four create four attention groups. Each attention group spans two ranks. Expert layers can use an eight-rank EP group.

The TP2 and DP4 arrangement is a vLLM configuration example. Other engines may define different communication groups.

Tensor parallelism communicates inside each matrix operation. Expert parallelism moves routed activations. Data parallelism consumes memory for replicated components. Each layout pays a different cost.

Low concurrency can make expert-parallel transfers inefficient. Weak interconnects can produce the same result. Tensor parallelism can also struggle when collective communication dominates.

In summary, tensor parallelism divides operations. Expert parallelism divides expert ownership. Data parallelism divides requests. The useful combination depends on memory, traffic, and interconnect speed.

Expert load imbalance and rank-level tail latency

Routing rarely divides work perfectly. Suppose one expert receives half the assignments. Its GPU must process a larger matrix than the others.

Other GPUs may finish early and wait. The layer cannot finish until every required contribution returns. The slowest participating GPU therefore determines step latency.

Training can encourage balanced routing across a large dataset. One production batch can still be skewed. Language and request mix can change which experts are selected.

Expert imbalance occurs inside one model replica. Request imbalance occurs between replicas that receive different workloads. These problems need different measurements and fixes.

A balanced daily histogram can hide slow individual steps. Measure assignment counts and time at the serving-step level.

Record assignment counts for every serving step. Also record dispatch time, expert execution time, combine time, and GPU idle time. These measurements show whether one expert or link sets the tail.

Placement can separate experts that are frequently selected together. A hot expert can move to a less busy GPU. These changes keep router decisions intact.

Replication creates several physical copies of one logical expert. The router still selects the same logical expert. The runtime sends its assignments to an available copy.

Replication consumes GPU memory and leaves less space for the KV cache. Moving an expert also consumes bandwidth. A short traffic spike may end before rebalancing completes.

Some runtimes rebalance experts using recent load measurements. They may also support redundant expert copies. Both mechanisms need enough memory and a workload stable enough to learn from recent history.

👉 Rebalancing changes physical destinations. It does not change which logical experts the model selected or their intended outputs.

Average utilization cannot explain MoE tail latency. Measure each step and each GPU. Placement and replication can reduce the tail, but they consume memory and control bandwidth.

Inference optimizations and model-correctness boundaries

MoE optimizations fall into two groups. Some change only execution. Others change numerical values or the selected computation.

Fusion, grouped GEMM, placement, and communication overlap preserve logical expert selection. They change how the runtime performs the work. Small floating-point differences may still occur.

Quantization changes stored weights or transmitted activations. It can reduce memory and network traffic. It also introduces numerical error. Quality must therefore be tested on the target workload.

Reducing top-k changes the executed expert set. It removes expert contributions from the weighted sum. Renormalizing the remaining weights changes the result again.

Suppose the model normally selects eight experts. A runtime executes only four. The output now contains fewer expert functions. Renormalization also changes their combined scale.

The paper Training-Free Halving of Activated Experts studies this distinction. It executes fewer experts while using a larger reference set for normalization mass.

On Qwen3.6-35B-A3B, standard reduction from eight experts to four lost 4.65 MMLU points. Its adjusted reference mass reduced the measured loss to 0.35 points. The paper reports halved routed-expert compute for that setting.

The same method lost 0.55 MMLU points on Qwen3.5-397B-A17B when reducing ten experts to five. The authors also found that perplexity and task accuracy preferred different settings.

These are research results for specific checkpoints and evaluations. They do not establish a general rule for halving top-k.

Expert parallelism changes where selected computation runs. Lower top-k changes which computation runs. They are not equivalent optimizations.

Capacity limits require similar care. Some training systems cap tokens assigned to an expert. Particular runtimes may implement inference limits too.

MoE inference does not automatically drop tokens when an expert is busy. A capacity or dropping rule belongs to a specific model or runtime. Check the implementation before assuming one exists.

Optimize unchanged computation first. Treat quantization and top-k changes as model changes requiring quality tests.

Execution changes need performance tests. Quantization needs performance and quality tests. Top-k changes need the same tests plus careful output comparison.

Benchmark design and bottleneck diagnosis

A useful benchmark changes one serving decision at a time. Keep the checkpoint, precision, prompts, output limits, and request arrivals constant. Then change the parallel layout or kernel path.

Start with the smallest supported layout that fits the model. Add a same-server layout next. Test cross-server expert parallelism only after measuring those baselines.

Memory capacity may prevent one or more configurations from loading the model. Backend support may also exclude a kernel or communication layout.

Test low and high concurrency. Use short and long prompts. This separates small decode matrices from prompt-heavy prefill work.

Record prefill and output throughput. Measure time to first token and time per output token. Also record inter-token latency and p95 request latency. Tail measurements reveal delays caused by one hot GPU.

GPU-level measurements explain those request metrics. Record bytes by network domain, assignment counts, expert time, and idle time. Split routing, permutation, dispatch, GEMM, combine, and unpermutation. Also record resident weights and remaining KV capacity.

Interpret each stage separately. High dispatch time points to topology, locality, or message size. One persistently slow GPU points to routing skew or poor placement.

The diagnostic table maps common symptoms to initial measurements. Hardware topology and backend behavior can produce other causes.

High memory use with low expert compute indicates a weight-capacity problem. Little memory remaining for KV cache confirms the same constraint. More concurrency may improve matrix sizes while worsening cache pressure.

In summary, compare identical request traces and change one variable. Measure memory, computation, and communication separately. The largest measured cost determines the next experiment.

Engineering checklist

Qwen3-30B-A3B has 30.5 billion total parameters and 3.3 billion activated parameters. The total count drives weight storage. The activated count estimates the executed path for each token.

An MoE serving step includes routing, token permutation, dispatch, grouped expert computation, combine, and unpermutation. Multi-GPU deployments add communication domains, expert placement, and rank-level load imbalance.

Validate a deployment in this order. First, confirm weight and KV-cache capacity. Next, profile every stage of the MoE path. Then measure assignment skew and bytes by communication domain. Finally, compare placement, kernels, quantization, or top-k changes against the same request trace.

Placement and kernel changes target runtime overhead. Quantization changes numerical values. Top-k changes the executed expert set. Keep those experiments separate so performance gains and model-quality changes remain attributable.

MoE inference becomes easier to diagnose once three quantities remain separate. The deployment must store model weights, execute selected experts, and move routed activations. Measure each one independently before changing the serving layout.

To learn building an MoE-based Transformer from scratch, we covered LLaMA 4, using nothing but Python and PyTorch here →

It demystifies the architecture powering modern LLMs like LLaMA 4 and Mixtral, and gives you hands-on insight into how experts, routers, and sparse activation work in practice.

We’ll walk through every stage of implementation:

• Character-level tokenization,

• Multi-head self-attention with rotary positional embeddings (RoPE),

• Sparse routing with multiple expert MLPs,

• RMSNorm, residuals, and causal masking,

• And finally, training and generation.

👉 Over to you: Would you investigate memory, expert computation, or communication first in your deployment?

Good day!

Devs just open-sourced a plug-and-play infrastructure layer that lets you run any harness under a single interface, like:

• Codex

• Hermes

• Claude code

• DeepSeek Harness

• System One, powered by Jev

• And 9 more agent harnesses

This means you can bring Jev into the same product that already uses Codex, Claude Code, or another supported harness, without writing another implementation for sessions, streaming, files, cancellation, and failure handling.

Here’s the repo: github.com/HarnessRouter/harnessrouter

(don’t forget to star it ⭐ )

The harnesses run locally, and the Unified Harness Protocol (UHP) defines the common task interface with an OpenAI Responses-compatible API.

Build your own Jev (100% local)

Many LLM calls do not need newly written text. The application already knows the possible answers, and it only needs the model to choose one.

Suppose a support ticket says, “I was charged twice for the same subscription.”

The application needs to send it to one of three teams: billing, technical support, or account access.

A normal LLM call asks the model to write an answer. It might return a sentence, a label, or a JSON object. The application waits for that text, parses it, and extracts the selected team.

That is unnecessary if every valid answer is already known.

In fact, a better and more efficient way to do this is to treat the same request as a decision (what Jev does). The application provides the ticket/query and the three allowed answers to Jev. The model then returns a score for each answer in one go (we’ll discuss shortly how exactly you can do that):

billing             0.91
technical support   0.06
account access      0.03

Billing is selected with the highest probability here, and the application code can actually see how strongly it won.

This is the behavior Jev possesses that we will reproduce locally.

We will provide the input query and allowed answers. In one scoring request, the model will return a decision and a probability distribution without generating a sentence or JSON object.

While Jev is closed-source, this inference pattern is already available in several open language models.

More specifically, we will implement it using SGLang (through /v1/score), test it with Qwen and DeepSeek models, and compare it against structured output and ordinary text generation.

To set expectations upfront, this article recreates the inference path, not the complete Jev system. Jev also includes training and calibration work that a scoring endpoint does not provide.

Also, we will be exploring Jev in our current series on AI Agents course in Production, around filtering, guardrails, execution safety, and more. Start here →

Fixed-answer scoring is different from structured output

It is easy to confuse Jev’s mechanism with structured output since both approaches restrict what the application receives, but they do different work inside the inference server.

For the same support ticket discussed above, a structured output might get the following output from the model:

{"team": "billing"}

The specified schema prevents an invalid object, so it does not select the team on its own.

However, this can be further improved since, under the hood, the model still generates the opening brace, the field name, the value, and the closing brace one token at a time. Once generation finishes, the application reads the team field. This video depicts this process:

With scoring (which Jev does), the application can supply the three teams as the complete list of valid outcomes. The server can read one model score for each outcome and return the distribution shown earlier in this article. It does not generate a JSON object.

We can then return all three values instead of billing so that the application code can treat these two results differently:

Result 1                          Result 2
billing       0.91                billing       0.46
technical     0.06                technical     0.44
account       0.03                account       0.10

For instance, in the above situation, both will select billing.

But the first has a clear preference, while the second one is almost tied. The application can route the first ticket automatically and send the second one for review if needed.

Also, the model will only supply these scores, and the application code will set the rule for its downstream use.

For example, one might require the top answer to exceed 0.80 and lead the second answer by at least 0.20. Those thresholds live in code, where they can be tested and changed.

A value of 0.91 means billing received 91 percent of the probability mass to these three choices. It does not prove that the model is correct 91 percent of the time. We need labeled examples to measure that. This is the calibration problem we will discuss later.

But in the meantime, remember that structured output generates a valid object. Fixed-answer scoring returns a distribution over answers the application already knows.

How an LLM generates the first output token

Before we discuss how a causal LLM can be turned into a Jev-style model, it would be better to first understand a regular generation step in LLMs.

• A tokenizer first converts the prompt into token IDs.

• The model processes that sequence and produces a vector for the next position.

• The vector has one number for every token in the model’s vocabulary. For instance, Qwen’s vocabulary contains tens of thousands of tokens, so the vector contains tens of thousands of numbers.

Those raw numbers are logits. A larger logit means the model prefers that token as the next continuation. The values are not probabilities yet.

During normal generation, the server applies the model’s decoding rules (temperature, etc.) to this vector, selects one token, and appends it to the prompt.

The model then produces a new vocabulary-sized vector for the next position. Generation/decoding repeats this process until it reaches a stop token or the output limit.