6 Parallelism Strategies for LLM Inference in Production

(popular LLM interview question)

In today’s newsletter:

• Finally, an OpenRouter for agent harnesses!

• 6 parallelism strategies for LLMs in production.

• ​Schema-guided agent memory.

---

Finally, an OpenRouter for agent harnesses!

Devs just open-sourced a plug-and-play infrastructure layer that lets you run any harness under a single interface, like:

• Codex

• Hermes

• Claude code

• DeepSeek Harness

• System One, powered by Jev

• And 9 more agent harnesses

This means you can bring Jev into the same product that already uses Codex, Claude Code, or another supported harness, without writing another implementation for sessions, streaming, files, cancellation, and failure handling.

Here’s the repo: github.com/HarnessRouter/harnessrouter

(don’t forget to star it ⭐ )

The harnesses run locally, and the Unified Harness Protocol (UHP) defines the common task interface with an OpenAI Responses-compatible API.

---

6 parallelism strategies for LLMs in production

Running a model across four GPUs does not merge their memory into one shared pool.

Each GPU has its own memory, so data stored on another GPU must cross the interconnect before a layer can use it.

This is the central trade-off behind parallelism. It removes local memory or compute limits but adds communication overhead.

The visual below explains the six common strategies for parallelizing LLM inference:

• Data parallelism runs complete model replicas on separate GPU groups. It increases throughput, but cannot make an oversized model fit.

• Tensor parallelism splits matrix operations inside each transformer layer. It reduces weight memory per GPU, but the GPUs must combine partial results throughout the layer stack.

• Pipeline parallelism assigns layer groups to different GPUs. Activations move between stages, while insufficient micro-batching leaves some stages idle.

• Context parallelism splits sequence positions in two ways. Prefill context parallelism divides a long input across GPUs to target lower TTFT. Decode context parallelism divides the stored KV history to increase cache capacity, but every decoder layer must merge partial attention results.

• Expert parallelism distributes MoE experts across GPUs. Token routing and uneven expert load add network cost and idle time.

• Hybrid parallelism combines multiple strategies in one deployment. For example, tensor parallelism can operate within a node, pipeline parallelism across nodes, and data parallelism across model replicas. This matches each communication pattern to the available links, but adds more coordination paths.

Production deployments often combine methods, but each one should fix a measured bottleneck.

• If the weights do not fit, use tensor or pipeline parallelism.

• If long-prompt prefill or KV-cache capacity is the limit, use the corresponding form of context parallelism.

• MoE models may need expert parallelism to place expert weights.

• If one replica meets its latency target but requests still queue, add data-parallel replicas.

Keep tensor parallelism on the fastest links. After each change, measure memory, TTFT, TPOT, and other metrics again.

Add another method only when measurements reveal another bottleneck.

We wrote a detailed article explaining how each strategy works, where communication occurs, and how to choose a layout for a real deployment.

Read it here →

If you want to learn the full LLMOps lifecycle, we have covered every bit of it in the LLMOps course, starting from fundamentals to production:

• Read Part 1 on fundamentals of LLMOps here →

• Read Part 2 on understanding the core building blocks of LLMs →

• Read Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixture-of-experts, and the fundamentals of pretraining and fine-tuning →

• Read Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applications →

• Read Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and different prompting techniques →

• Read Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more →

• Read Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techniques for building high-signal inputs →

• Read Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic context injection, and common failure modes in agentic applications →

• Read Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong understanding of the fundamental concepts →

• Read Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for evaluation of LLM apps →

• Read Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming →

• Read Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO →

• Read Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model parallelism →

• Read Part 14 on the fundamentals of LLM serving, including API-based access, inference with vLLM, and practical decisions →

---

​Schema-guided agent memory​

Most agent memory systems optimize for recall. The harder problem is what to forget, or more precisely, what to never store in the first place.

The default agent memory pipeline hands an LLM raw text and asks it to extract entities and relationships. The model decides the types, labels, and attributes all on its own.

The result is a knowledge graph that behaves like an expensive vector store. Entity types collapse into generic labels. Relationships flatten into a single “RELATES_TO.”

The graph has the data, but no query can reach it with precision.

This is not a retrieval problem but rather a structure problem. And the fix is the same pattern that already works everywhere else in the AI stack, i.e., constrain the output space before generation, not after.

• Entities define what the agent is allowed to remember. Pydantic models with typed fields and descriptive docstrings replace the LLM’s guesswork with domain vocabulary it was never trained on.

• Edges define how things connect. Source/target constraints on relationship types mean the graph can only form valid connections. If your schema has no edge connecting Project to Competitor, that relationship cannot exist in memory.

• Temporal resolution handles what was true versus what is true. Fact resolution invalidates outdated edges while preserving history, so the graph never silently serves a stale state.

The schema guides extraction at two points in the pipeline (entity extraction and fact extraction) while resolution and temporal processing run automatically downstream.

You define what to look for. The system handles deduplication, contradiction detection, and time-windowing without additional configuration.

A useful constraint is to have just 10 entity types, 10 edge types, and 10 fields per type.

That forces you to model the 80% that matters rather than attempting completeness. Start with 3-4 of each and expand only when retrieval fails.

​Zep Graphiti​ does all of this as a fully open-source temporal knowledge graph library.

It includes Pydantic-based ontology definition, schema-guided extraction, entity resolution, fact resolution, and temporal windowing.

If you are building agent memory with any kind of domain specificity, schemas are a must.

​Here’s the GitHub repo →​

(don’t forget to star 🌟)

​We also covered this topic in more depth (with code) in this article →​

Good day!