In today's newsletter:
An open-source alternative to Claude (33k+ stars)!
7 MoE routing strategies in production, clearly explained.
As an AI engineer, please learn.
An open-source alternative to Claude (33k+ stars)!
Onyx is a self-hostable AI chat platform that works with any LLM, like Claude, GPT, Gemini, Llama, or any open-weight model you want.
Here’s what it ships with:
Agents that chain multiple tools in sequence
RAG with full indexing across 40+ connectors (Slack, Drive, Confluence, Jira, GitHub, email, call transcripts)
Deep research ranked #1 on DeepResearchBench, above every proprietary alternative
MCP support for connecting to external systems
Code interpreter for data analysis and file generation
Self-host on your own infrastructure via Docker in a few mins
Unlike Claude’s MCP-based connectors that query your tools at runtime, Onyx actually indexes all your data. That means faster, more reliable search across everything your team has ever written.
The entire code is open-source under the MIT license, so you can see the full implementation on GitHub and try it yourself.
Find the GitHub repo here → (don’t forget to star it ⭐️)
7 MoE routing strategies in production, clearly explained
Sending every token to its highest-scoring experts can make an MoE model slower.
The problem is that many tokens can prefer the same experts. Those experts become overloaded while others remain underused.
With fixed capacity and a dropping rule, some token assignments may be dropped. Across GPUs, uneven workloads and communication overhead can reduce the compute savings.
MoE routing has to select useful experts while distributing work efficiently. The seven strategies in the visual handle that tradeoff differently.
One caveat before we start. These are routing designs, not seven interchangeable settings in a serving engine. Some come from research on training or vision models and need extra care for autoregressive generation.
1. Token-choice routing
Each token selects its top-k experts using router scores. If we choose two experts per token, four tokens create eight token-expert assignments.
That fixes the expert count per token. It does not fix how much work each expert receives.
Say all four tokens select the same two experts in a four-expert layer. Two experts receive four assignments each, while the other two receive none.
Load-balancing mechanisms help prevent traffic from concentrating on a few experts. A training-time balancing objective still does not guarantee that every serving batch will be balanced.
2. Expert-choice routing
Here, we reverse the selection. Each expert selects its highest-scoring tokens within a fixed capacity.
Suppose two experts can each take two tokens. Both experts do the same number of token assignments, but they can select an overlapping set of tokens.
One token may reach both experts. Another may reach neither. Balanced expert workloads therefore do not mean every token receives the same amount of computation.
The original Expert Choice paper also flags limitations for autoregressive generation and very small serving batches. Selecting across a token group needs care when future tokens are unavailable.
3. Dynamic routing
The number of selected experts varies across tokens. Confidence-based variants allocate more experts when routing scores are less decisive.
One approach selects experts in descending score order until their cumulative router probability reaches a threshold.
If one expert already carries enough probability, the router can stop there. If probability is spread across experts, it selects more of them.
Router confidence does not directly measure token difficulty, though. We should check quality and actual runtime rather than assuming that more experts always means harder input or better answers.
4. Balanced assignment
A global matching procedure assigns tokens under expert load constraints. Instead of making every decision independently, we consider the token group together.
The assignment tries to keep high token-expert affinity while respecting how much work each expert can take.
Some tokens may miss their highest-scoring expert to achieve a more balanced allocation. Computing that assignment also adds overhead.
BASE Layers is a research example of this idea. Its balanced assignment should not be read as a promise that every production decoder runs a global matching solver at each step.
5. Hash-based routing
A deterministic hash maps tokens to experts without a learned gate. In a token-ID-based version, the mapping stays fixed even when the surrounding context changes.
The expert still processes a contextual token representation. It is the expert selection that does not adapt through a learned router.
That removes router learning and makes selection simple. The tradeoff is less flexibility to adapt expert choice to context.
A fixed mapping also does not automatically balance real traffic. Frequent tokens can concentrate assignments, so the hash design and workload still matter.
6. Soft MoE routing
Experts process weighted mixtures of token representations through slots. Each slot can combine information from multiple tokens.
We first form those mixtures, pass each slot through its assigned expert, and then combine the expert outputs back into output token representations.
This changes what an expert processes. It is different from sending an individual token to a discrete top-k set of experts.
The original Soft MoE work focuses on visual recognition and identifies autoregressive decoding as a limitation. Mixing tokens must preserve causality before we can use the same idea in a decoder.
7. Group-limited routing
The router restricts eligible expert groups before selecting experts. When groups reflect hardware placement, this can reduce communication across devices or nodes.
We first narrow the destinations, then select experts within those eligible groups.
For example, DeepSeek-V3 uses node-limited routing that sends each token to at most four nodes. That restriction limits cross-node destinations while allowing expert selection within them.
The hardware mapping matters. Arbitrary logical groups do not guarantee less network traffic, and adding a routing restriction to an already trained model can change its selected computation.
These choices affect how serving engines batch expert computation and distribute work across GPUs.
Evenly distributing tokens does not guarantee low latency. Expert batches may still be small, and moving tokens between devices takes time.
When we profile an MoE deployment, we need to separate routing, token movement, expert computation, and combination. The slowest stage tells us what to investigate next.
We covered the rest of this path in our MoE inference engineering article, including expert parallelism, grouped GEMM, topology-aware placement, memory requirements, and inference optimizations.
Read the MoE inference engineering breakdown here →
As an AI engineer, please learn:
Knowing a model’s benchmark score does not tell us how it will behave behind a busy API.
We prepared a practical learning checklist for the work that comes after picking a model:
Learn the roofline model and why small-batch decode is often memory-bound.
Deploy vLLM and SGLang, then read their schedulers.
Understand PagedAttention from the code, not only a blog post.
Build observability before optimizing anything.
Track time to first token, inter-token latency, throughput, and queue depth.
Use Grafana and Prometheus for inference dashboards.
Turn on prefix caching and find which workloads benefit. It reuses shared-prefix computation, not the generation of new tokens.
Learn continuous batching and chunked prefill.
Run load tests that sweep concurrency, including 1,000-plus concurrent requests where the hardware and workload allow it.
Report p50, p95, and p99, not only the mean.
Study quantization tradeoffs across FP8 and INT4 formats, and methods such as AWQ and GPTQ. Measure quality alongside memory use and speed.
Learn speculative decoding and where it stops helping.
Test a supported KV-cache eviction strategy for long contexts. Distinguish evicting reusable cached prefixes from discarding attention history that an active request still needs.
Try disaggregated prefill and decode serving.
Learn Kubernetes for AI workloads and use queue depth as an autoscaling signal.
Learn how inference costs affect unit economics.
Build a model router that considers cost, latency, and quality.
Create a token budget for each request.
Build one inference service and benchmark it publicly.
Read inference research rather than relying on model-release news.
Share your optimization benchmarks.
We have turned this into a 10-week roadmap on GitHub. It has 50 sessions of 30 minutes each, with five sessions and two buffer days per week.
The reading and build sessions feed one project. We deploy the service, instrument it, load test it, tune it, add a cost-aware router, and publish a reproducible benchmark.
Treat the timing as a study plan, not a guarantee that every deployment or load test fits into half an hour. The 1,000-request target is a load-testing exercise, not a claim that any GPU can serve that workload within a useful latency budget.
The repo welcomes contributions, including newer sources and benchmark writeups. The useful outcome is a service whose behavior we can measure and explain.
Find our inference-serving roadmap here →
Good day!