3 Comments
User's avatar
Unpursued AI's avatar

The 25% to 45% slowdown for SGLang's deterministic mode turns this into a product decision, not just an eval setting. In the 2026 Stack Overflow Developer Survey, 48% of developers said "I trust it when I can easily verify the output," and getting the same answer twice is a big part of verifying it. Do you see teams paying for batch invariance in production, or keeping it for evals and RL only?

Nan Deng's avatar

The number I would put next to any temperature-0 golden set is where the tie broke. All 1,000 Qwen3-235B Feynman runs agreed for 102 tokens, then 8 of 1,000 took "New York City" instead of "Queens" after a batch-size change moved two logits by about 0.0002. That is not sampler noise: the same batch twice is bitwise identical, and the split only changes when other requests change how the H100 splits the reduction. I would turn on the batch-invariant flag for evals and RL rollouts, and log the batch size beside the completion, because SGLang's deterministic path is the one LMSYS measured at roughly 25% to 45% slower and you need to know which runs paid that.

Abhishek Pan's avatar

That's why an exact-match diff on a temperature-0 golden set makes a flaky regression gate on a shared endpoint. Traffic alone can flip a near-tied token and the rest of the answer follows, so compare graded scores over a few repeats, or run the eval on a replica with a fixed batch size.