Discussion about this post

User's avatar
Immanuel Santosh's avatar

The batch-invariance problem has a quiet parallel in retirement planning. Two clients with the same salary and the same SIP amount can run the same projection and land on different numbers, because the invisible context — cashflow order, when the market sequence hits, when you measure — changes the reduction.

That's why I insist a plan be re-runnable: same inputs, same answer every time, or the client stops trusting the number.

Nan Deng's avatar

The Feynman runs all agreed for the first 102 tokens and then split at Queens versus New York City, which is the near-tie this piece describes, not a different prompt. Replaying that one request later will not reconstruct the batch that set the split-K and split-KV sizes, so the logits on the retry are a new reduction order. I would store the batch size and whether the kernel used a fixed split size next to the completion. Otherwise a later mismatch looks like the model changed when only the other traffic on the server changed.

2 more comments...

Ready for more?