Discussion about this post

User's avatar
Dima Shvets's avatar

Thank you for the detailed analysis, Avi

Nan Deng's avatar

The 153 / 5.2 ≈ 29 tok/s ceiling is the part worth internalizing: once decode is bandwidth-bound, accepted tokens per pass is the only lever left on-device, which is why the draft model has to be co-designed with the quantization. I'd be curious how the 92–117 tok/s holds up on longer generations where the draft model's quality drifts, since speculative decoding tends to degrade exactly when output stops looking like predictable code.

1 more comment...

Ready for more?