Pull down to go back
Why does speculative decoding give wildly different speed boosts across AI models? (665% for some, barely 40% for others)

Why does speculative decoding give wildly different speed boosts across AI models? (665% for some, barely 40% for others)

推測解碼為什麼在不同 AI 模型上快 665% 或慢 40%?差異在哪裡

Someone's testing speculative decoding in llama.cpp—a technique that predicts multiple tokens at once to speed things up—and getting bizarre results. With the same settings, Gemma 4 31b doubles speed (100% boost), Qwen 3.6 barely crawls at 40% faster, but Devstrall small absolutely rockets at 665% speed increase. The question: what's actually different about these models that makes speculative decoding work so differently? They've also found tweaking the repeat penalty and spec type helps Qwen catch up. It's a deep dive into why the same optimization trick doesn't work equally for everyone.