The Idle Machine

The first time a generation endpoint got faster without getting better, I did not trust the latency chart. Nothing about the model had changed. Same weights, same prompts, same outputs, checked token for token against the previous build. The p50 for a company assessment went from about four seconds to under two.

What had changed was which part of the machine was busy.

Two budgets

Every accelerator has two throughput numbers, and they are not the same kind of number. One is arithmetic: how many floating point operations it can retire per second. The other is bandwidth: how many bytes it can pull out of memory per second. Williams, Waterman and Patterson put both on the same chart in 2009 and called it the roofline. The chart has a ridge where the two meet, and the position of that ridge is just the ratio of the two numbers.

Take an A100 80GB SXM. The datasheet gives 2,039 GB/s of HBM2e bandwidth and 312 teraFLOPS of dense BF16 tensor core throughput, so the ridge sits at 153 floating point operations for every byte the memory system delivers. On an H100 SXM it is 3.35 TB/s against 989 teraFLOPS dense, which puts the ridge at 295.

Which side of the ridge you land on depends on one property of the workload: how many FLOPs you do for each byte you move. That ratio has a name, arithmetic intensity, and nearly every performance problem I have run into in the last two years is a story about it.

What one token costs

Take a single dense layer holding N weights in bf16. Push one token through it. Each weight is used once in a multiply-accumulate, so the layer does 2N FLOPs. Every one of those weights has to arrive from HBM first, and at two bytes each that is 2N bytes. Divide the two and the arithmetic intensity of a decode step is one FLOP per byte.

Against the A100’s ridge of 153, that is 0.65% of the tensor cores. The other 99.35% of the arithmetic sits there doing nothing while the memory bus runs flat out.

This is not something a better kernel fixes, because the weights have to move. Shazeer diagnosed it in 2019, in the paper that introduced multi-query attention, and his sentence is more precise than most of what has been written about inference since: incremental inference is slow “due to the memory-bandwidth cost of repeatedly loading the large keys and values tensors”. Look at what his fix does. Multi-query attention shares one key and value head across all the query heads, which cuts the number of bytes that move per step. It is a bandwidth fix that shipped as an architecture change.

More work per byte

There are two ways to raise arithmetic intensity: move fewer bytes, or do more with each byte you move. Batching is the second one. B sequences share a single read of the weights, so the same 2N bytes now support 2NB FLOPs and the intensity goes up by roughly the batch size. To reach the A100’s ridge you would want a batch near 153.

Real numbers for that tradeoff come from Pope and colleagues, who measured PaLM 540B and reported 29 milliseconds per token at low batch size and 76% of peak model FLOPs utilisation when the same hardware is fed a large batch. Same model, same chips, and the utilisation is set almost entirely by how many positions are in flight at once.

A single request cannot batch with itself. That is why a chat endpoint with one user feels slow and a batch job feels fast, and it is why serving infrastructure is largely a machine for assembling batches out of strangers’ requests.

Then something unrelated turns up and caps the batch. The KV cache for an 8B model in a Llama-3-8B style configuration, grouped-query attention with 8 key-value heads of width 128 across 32 layers, costs 131,072 bytes per token, so a quarter of a megabyte per thousand tokens and a full gigabyte for a single 8,192-token sequence. On an 80 GB card with 14.9 GiB of bf16 weights, 65 GiB is left for cache, which caps the batch at 65 sequences at that context length.

What the PagedAttention result actually buys is batch size, and batch size is arithmetic intensity. Kwon and colleagues showed that fragmented KV memory limits the batch size, and that near-zero waste plus sharing bought 2x to 4x throughput at the same latency. The paper reads as memory management. The lever it pulls is arithmetic intensity, which is the same lever batching pulls.

Batching across time

There is a second way to get more positions out of one read of the weights, and it does not need other users.

A forward pass is already parallel over positions. That is all prefill is. Decode cannot use that parallelism because token t+1 does not exist until token t has been chosen. So guess it.

That is speculative decoding. A small draft model proposes k tokens, the target model scores all k+1 positions in a single forward pass, you keep the prefix the target agrees with, and you resample the first position where it disagrees. Leviathan, Kalman and Matias introduced it in 2022 and measured 2x to 3x on T5-XXL with identical outputs. Chen and colleagues at DeepMind published the same idea independently in early 2023, 2x to 2.5x on Chinchilla 70B, with the sampling rule that keeps the target’s distribution intact “within hardware numerics”.

Batching across users buys the extra positions from other requests. Speculation buys them from a guess. The arithmetic is identical: one read of the weights, k+1 positions scored. What differs is the failure mode. A wasted slot in a batch is a slot that simply is not there. A wasted position in speculation is scored in full and then thrown away.

The acceptance rate is a divergence

Write p for the target’s next-token distribution at a position and q for the draft’s. The draft proposes a token from q. The target accepts it with probability min(1, p/q), and on rejection resamples from the normalized max(0, p - q). The acceptance rate works out to

alpha = sum_x q(x) * min(1, p(x)/q(x)) = 1 - TV(p, q)

where TV is total variation distance. I checked the algebra on paper and then checked it in code: both forms give 0.558157 on the same pair of distributions.

The correction term is the whole trick, and it is easy to get wrong. I built a target over 64 tokens and a draft that keeps the top 8 and renormalizes, giving TV = 0.4418 and an acceptance rate of 0.5582, then drew 400,000 samples three ways:

The first row returns the draft’s own distribution, so it is off by 0.4418 in total variation. The second is the tempting fix and it is still biased, because mixing accepted draft tokens with fresh target tokens does not reconstruct the target. Only the third is exact, and 400,000 draws land within 0.00085 of p, which is the noise floor for that sample size. Chen and colleagues’ phrase for this is that the distribution is preserved within hardware numerics, and that is exactly what it means: the sampler is exact and the floating point is not.

So the acceptance rate is a measurement, not a parameter. It is the fraction of the target’s next-token distribution that the draft has captured at that position, and it says how much of a big model’s behaviour a small model reproduces. I have started treating it that way. Speculative decoding is as much an instrument as an optimisation.

What the yield is worth

With k draft tokens and a per-token acceptance rate a, the expected number of tokens produced per target forward pass is

(1 - a^(k+1)) / (1 - a)

I verified the formula against Monte Carlo. At a = 0.9 and k = 7 it predicts 5.695 tokens per pass and the simulation gives 5.697.

Two things fall out of it. The first is a floor. The expression is smallest at k = 1, where it equals 1 + a, and it rises from there. The floor underneath the whole scheme is 1, reached when the draft is useless: a complete rejection still hands you one token, resampled from the target, which is exactly what plain decoding would have produced from the same forward pass. You cannot lose tokens to speculation. The only thing you can waste is the draft’s own compute, and the price of that is the cost ratio c, one draft forward over one target forward.

The second is that the best k is set by a and c and nothing else. Divide the yield by 1 + kc and take the best k:

The table runs the other way too, and that is the use I actually get out of it. My own generation path measured 2.4x to 2.8x and I never had a direct number for the acceptance rate. If the draft costs a tenth of a target forward, that speedup implies an acceptance rate of at least 0.79 for the 2.4x case and at least 0.84 for the 2.8x case. If the draft costs a fifth, the same two numbers are 0.90 and 0.94. The latency figure I already had was carrying the acceptance rate inside it the whole time.

The waste costs nothing

At an acceptance rate of 0.8 with k = 8, the yield is 4.33 tokens per forward pass, which means 52% of the positions scored in that pass are rejected. Half the arithmetic goes into tokens that get thrown away.

On a machine sitting at 0.65% of its arithmetic capacity during decode, that is free. What you are doing is spending a resource that was idle in order to stop waiting on a resource that was saturated, and the idle budget is enormous, roughly 150x the work you are actually asking for. The draft model’s job is not to be accurate. Its job is to be cheap enough that its mistakes fit inside a budget nobody was using.

Where it stops working

The acceptance rate belongs to the draft and the target together, at that position. It is not a function of how large the draft is, and it is not a number you can read off a model card.

If the draft’s limitation is capacity over the tail of the distribution, the entropy of the target decides how much you get. Same target logits, temperature varied from 0.2 to 3.0, draft keeping the top 8:

A draft that can only represent the head of the distribution is nearly perfect where the target is confident and nearly useless where it is not.

I ran a second draft model to see whether that survives a different failure mode: the target’s logits perturbed by fixed-scale noise, averaged over eight draws. It does not reproduce the collapse at all. The acceptance rate stays between 0.94 and 0.96 from temperature 0.35 upward, because a flat target is easy to match with anything broad.

Both results are real and they disagree, which is the useful part. Top-k truncation models a draft that cannot represent the tail. Logit noise models a draft that gets the shape roughly right and the head wrong. Which failure mode you actually have decides whether a high-entropy target hurts you, and you cannot get that from the draft’s parameter count. You measure the acceptance rate where you deploy, by counting acceptances.

What I take from it

The pattern underneath all of this is a rule I now apply well outside hardware. When something is slow, find out which resource is saturated and which one is idle, then move work to the idle one. Removing work is the obvious move and often the wrong one, because the work was rarely what was costing you.

It is the same shape as the thing I keep building. Sentinel runs hundreds of deterministic checks before anything ships, and a deterministic check is the cheapest verifier there is: microseconds to run, exact verdict, no model in the loop. Glide’s extraction and generation paths are the same propose-then-check structure. A cheap wrong answer plus an exact check beats an expensive careful answer, whenever the check is cheaper than being careful. Speculative decoding is that same arrangement with the check being a single forward pass of the model you were going to run anyway.

A week ago I wrote that the plastic timeline prices an edit by its distance into the past. The price here is set by something else: how often the guess is right. Distance there, accuracy here. In both cases the price of being wrong is set by something nobody chose on purpose.

The practical version is short. Count your acceptances and let the measured rate pick k, because k is not a taste decision. Put speculation where the target’s own distribution is peaked, since that is where a small model’s mistakes are cheapest. And when a system is slow, check the utilisation before touching the algorithm. The machine that takes four seconds for a token is almost always waiting on memory.