Tag: hardware
All the articles with the tag "hardware".
The Idle Machine
Published: at 02:22 AMAutoregressive decoding leaves almost all of an accelerator's arithmetic idle while it waits on memory bandwidth. Batching and speculative decoding fill that idle budget by the same mechanism, and the acceptance rate tells you what the mechanism is worth.