Discussion about this post

User's avatar
Unpursued AI's avatar

The 25% to 45% slowdown for SGLang's deterministic mode turns this into a product decision, not just an eval setting. In the 2026 Stack Overflow Developer Survey, 48% of developers said "I trust it when I can easily verify the output," and getting the same answer twice is a big part of verifying it. Do you see teams paying for batch invariance in production, or keeping it for evals and RL only?

Nan Deng's avatar

The number I would put next to any temperature-0 golden set is where the tie broke. All 1,000 Qwen3-235B Feynman runs agreed for 102 tokens, then 8 of 1,000 took "New York City" instead of "Queens" after a batch-size change moved two logits by about 0.0002. That is not sampler noise: the same batch twice is bitwise identical, and the split only changes when other requests change how the H100 splits the reduction. I would turn on the batch-invariant flag for evals and RL rollouts, and log the batch size beside the completion, because SGLang's deterministic path is the one LMSYS measured at roughly 25% to 45% slower and you need to know which runs paid that.

1 more comment...

Ready for more?