The 153 / 5.2 ≈ 29 tok/s ceiling is the part worth internalizing: once decode is bandwidth-bound, accepted tokens per pass is the only lever left on-device, which is why the draft model has to be co-designed with the quantization. I'd be curious how the 92–117 tok/s holds up on longer generations where the draft model's quality drifts, since speculative decoding tends to degrade exactly when output stops looking like predictable code.
With that 29 tok/s bandwidth ceiling, the 92 to 117 number is mostly accepted draft tokens. A Python function is about the best case for that since code is so predictable, so I'd want the same run on plain prose or a long tool-call chain before trusting the speedup.
Thank you for the detailed analysis, Avi
The 153 / 5.2 ≈ 29 tok/s ceiling is the part worth internalizing: once decode is bandwidth-bound, accepted tokens per pass is the only lever left on-device, which is why the draft model has to be co-designed with the quantization. I'd be curious how the 92–117 tok/s holds up on longer generations where the draft model's quality drifts, since speculative decoding tends to degrade exactly when output stops looking like predictable code.
With that 29 tok/s bandwidth ceiling, the 92 to 117 number is mostly accepted draft tokens. A Python function is about the best case for that since code is so predictable, so I'd want the same run on plain prose or a long tool-call chain before trusting the speedup.