The 153 / 5.2 ≈ 29 tok/s ceiling is the part worth internalizing: once decode is bandwidth-bound, accepted tokens per pass is the only lever left on-device, which is why the draft model has to be co-designed with the quantization. I'd be curious how the 92–117 tok/s holds up on longer generations where the draft model's quality drifts, since speculative decoding tends to degrade exactly when output stops looking like predictable code.
With that 29 tok/s bandwidth ceiling, the 92 to 117 number is mostly accepted draft tokens. A Python function is about the best case for that since code is so predictable, so I'd want the same run on plain prose or a long tool-call chain before trusting the speedup.
The 153 / 5.2 ≈ 29 tok/s ceiling is the part worth internalizing: once decode is bandwidth-bound, accepted tokens per pass is the only lever left on-device, which is why the draft model has to be co-designed with the quantization. I'd be curious how the 92–117 tok/s holds up on longer generations where the draft model's quality drifts, since speculative decoding tends to degrade exactly when output stops looking like predictable code.
Thank you for the detailed analysis, Avi
With that 29 tok/s bandwidth ceiling, the 92 to 117 number is mostly accepted draft tokens. A Python function is about the best case for that since code is so predictable, so I'd want the same run on plain prose or a long tool-call chain before trusting the speedup.