Why Is Speculative Decoding Fast?

This is my post! Happy to answer any follow-up questions people have.

I wrote this to try and clear up what I think is a common misconception (speculative decoding is a free lunch), when the picture is more nuanced (and imo, interesting).

we have action chunking in robotics for the same reason so VLAs can analyze faster and come to actions, the latency and inference are both slow so many steps need to come together for this to happen correctly

has anyone tried different approaches to this?

[flagged]