The batch-invariance problem has a quiet parallel in retirement planning. Two clients with the same salary and the same SIP amount can run the same projection and land on different numbers, because the invisible context — cashflow order, when the market sequence hits, when you measure — changes the reduction.
That's why I insist a plan be re-runnable: same inputs, same answer every time, or the client stops trusting the number.
The Feynman runs all agreed for the first 102 tokens and then split at Queens versus New York City, which is the near-tie this piece describes, not a different prompt. Replaying that one request later will not reconstruct the batch that set the split-K and split-KV sizes, so the logits on the retry are a new reduction order. I would store the batch size and whether the kernel used a fixed split size next to the completion. Otherwise a later mismatch looks like the model changed when only the other traffic on the server changed.
Since the flip only happens at near ties, logging top-2 logprobs on the eval set tells you fast whether a score move is real. If the first diverging token between two runs always sits on a tiny top-2 gap, that's batch noise, and the eval should run on its own replica with batch invariance on before anyone calls a one-point drop a regression.
The batch-invariance problem has a quiet parallel in retirement planning. Two clients with the same salary and the same SIP amount can run the same projection and land on different numbers, because the invisible context — cashflow order, when the market sequence hits, when you measure — changes the reduction.
That's why I insist a plan be re-runnable: same inputs, same answer every time, or the client stops trusting the number.
The Feynman runs all agreed for the first 102 tokens and then split at Queens versus New York City, which is the near-tie this piece describes, not a different prompt. Replaying that one request later will not reconstruct the batch that set the split-K and split-KV sizes, so the logits on the retry are a new reduction order. I would store the batch size and whether the kernel used a fixed split size next to the completion. Otherwise a later mismatch looks like the model changed when only the other traffic on the server changed.
never knew batch size could change outputs at temperature 0. does running on a single replica fix most of it?
Since the flip only happens at near ties, logging top-2 logprobs on the eval set tells you fast whether a score move is real. If the first diverging token between two runs always sits on a tiny top-2 gap, that's batch noise, and the eval should run on its own replica with batch invariance on before anyone calls a one-point drop a regression.