Local LLM tokens per second: how to compute your ceiling
Decode speed is memory bandwidth divided by the bytes read per token. The formula, a verified table for common hardware, and how to find your real limit.
Someone posts that they get 30 tokens per second on a 27B model and asks whether that is good. The replies arrive as a list of things to change: switch backends, try a different quant, turn on speculative decoding, reduce the context. Occasionally one of them helps. None of them answer the question, which was whether 30 is close to what the hardware can physically do.
That question has an arithmetic answer, and it takes about ten seconds to work out.
Key takeaways
- For single-stream decode, the ceiling is roughly memory bandwidth divided by the bytes the machine reads per token. On a dense model those bytes are essentially the weight file.
- So a 16.5 GB quantised 27B on a 960 GB/s card tops out near 58 tokens per second, and no amount of backend tuning goes past it.
- Most real setups land somewhere between half and three quarters of that number. Below half is worth investigating; above the ceiling means you measured something else.
- Mixture-of-experts models read only their active parameters per token, so a 35B model with 3B active behaves like a 3B for this calculation.
- Context length changes the KV cache, not the weights. If your tokens per second is flat across very different context sizes, the cache is not what is limiting you.
- Prefill is a different regime. It is compute bound, so bandwidth arithmetic does not describe how fast your prompt gets read.
Why is memory bandwidth the limit and not the GPU?
Generating one token with a dense transformer requires reading every weight once. There is no way around it: the arithmetic for a single token is tiny, and the machine spends its time moving parameters from memory into the compute units rather than doing math on them. At batch size one, which is what running a model locally means, that read is the bottleneck.
This is why a graphics card with enormous compute can feel slow on a large model, and why two cards with similar compute but different memory can differ by a factor of two on the same model. The number that predicts decode speed is bandwidth.
What is the formula for tokens per second?
Take the memory bandwidth of the device and divide it by the bytes read per token:
tokens/sec ceiling ≈ memory bandwidth (GB/s) ÷ bytes read per token (GB)
For a dense model, bytes read per token is close to the size of the weight file on disk. A 27B model at Q4_K_M is roughly 16.5 GB, so on a card with 960 GB/s:
960 ÷ 16.5 ≈ 58 tokens/sec
That is the speed of light for that pairing. It is not a target and not a promise. It is the number that nothing you install can beat, because beating it would mean reading the weights faster than the memory bus can deliver them.
How many tokens per second should my hardware get?
Bandwidth figures below are the manufacturers' published specifications. The tokens-per-second columns are the ceiling from the formula above, using approximate Q4_K_M file sizes. Treat them as the top of the range, not the expected result.
| Hardware | Bandwidth | 7B Q4_K_M (~4.4 GB) | 27B Q4_K_M (~16.5 GB) | 70B Q4_K_M (~42 GB) |
|---|---|---|---|---|
| Apple M4 Pro | 273 GB/s | ~62 tok/s | ~17 tok/s | ~6 tok/s |
| Apple M4 Max (40-core GPU) | 546 GB/s | ~124 tok/s | ~33 tok/s | ~13 tok/s |
| Apple M3 Ultra | 819 GB/s | ~186 tok/s | ~50 tok/s | ~20 tok/s |
| Radeon RX 7900 XTX | 960 GB/s | ~218 tok/s | ~58 tok/s | does not fit in 24 GB |
| GeForce RTX 4090 | 1008 GB/s | ~229 tok/s | ~61 tok/s | does not fit in 24 GB |
| GeForce RTX 5090 | 1792 GB/s | ~407 tok/s | ~109 tok/s | does not fit in 32 GB |
Two things this table makes obvious. A 70B at Q4 does not fit on any consumer card listed, which is the actual reason people run those models on Apple silicon with large unified memory despite the lower bandwidth. And the gap between an M4 Pro and an M3 Ultra on the same model is almost exactly the gap in their bandwidth, because that is the only variable that matters here.
Why am I not hitting the ceiling?
Because the ceiling assumes perfect memory utilisation, and nothing achieves that. Kernels have overhead, the KV cache competes for the same bus, and the backend may not be issuing reads efficiently for your particular hardware.
As a rough band, most working setups land somewhere between half and three quarters of the theoretical number. On a 960 GB/s card running a 16.5 GB model, that puts the realistic range around 32 to 44 tokens per second against a 58 ceiling. Someone reporting 30 on that pairing is at the bottom of the band but not obviously broken, and someone reporting 75 has measured something other than sustained decode, usually a short burst or a cached prefix.
That distinction is the useful part. If you are at 55 percent of the ceiling, tuning might buy you something. If you are at 20 percent, something is wrong in a way tuning will not fix, and if you are above the ceiling, your measurement method is the thing to check first.
Does context length change tokens per second?
It does, but through a different mechanism, and less than people expect. The weights are re-read for every token regardless of context. What grows with context is the KV cache, which adds to the bytes read per token as the sequence gets longer.
This gives you a clean diagnostic. Run the same model at a short context and a long one. If tokens per second is roughly flat between them, the cache is not your limiting factor and the ceiling is being set by how fast the weights stream. If it degrades noticeably at long context, the cache is now a meaningful share of the bytes per token, and quantising the KV cache is the lever that will actually move it.
How do I tell whether bandwidth is really my limit?
Change the weight size instead of the context, which is the one variable the formula says should move the result proportionally.
Run a smaller quant of the same model, or a much smaller model, and watch whether tokens per second moves roughly inversely with the file size. If halving the weights roughly doubles the speed, you are bandwidth bound, and multiplying your measured tokens per second by the weight size tells you the bandwidth you are actually achieving. If the speed barely moves when the weights get much smaller, bandwidth is not what is holding you, and changing quants will keep disappointing you.
That second case points at the backend, the driver, or a configuration problem, and it is worth knowing before you spend an evening downloading quants.
What about mixture-of-experts models?
MoE changes the input to the formula rather than the formula. A sparse model activates a fraction of its parameters per token, so the bytes read per token track the active parameters, not the total. A 35B model with 3B active at 8-bit reads roughly 3 GB per token, which is why such a model can feel dramatically faster than its parameter count suggests while still needing enough memory to hold all of it.
This is also why comparing a sparse model to a dense one by total parameter count produces confusion in both directions. Compare active parameters when you are reasoning about speed, and total parameters when you are reasoning about whether it fits.
Where this fits with recal
We build recal, a local-first assistant for macOS, so the arithmetic above is how we decide what can run on a given machine before we promise anything. It is genuinely constraining, and the honest version is that a model which needs to feel instant has to be small enough that its weights stream in a fraction of a second, whatever the marketing around a chip says. Being able to compute the ceiling in advance saves a lot of benchmarking that would only have confirmed physics.
Frequently asked questions
Does this formula work for prompt processing too?
No. Prefill processes many tokens at once, which turns the work compute bound rather than bandwidth bound. A machine can be fast at decode and slow at prefill, which is the usual explanation when a long prompt takes a while before generation starts.
Does quantisation improve speed, or only memory use?
Both, and for the same reason. A smaller quant is a smaller weight file, so there are fewer bytes to read per token. The speed gain is roughly proportional to the size reduction, which is exactly what the formula predicts.
Why do two people with the same GPU report different numbers?
Backend and configuration, mostly. The ceiling is fixed by hardware, but the fraction of it you achieve depends on the inference engine, the build, and how the model is loaded. That is the range between half and three quarters, and it is wide enough to explain most disagreements without either person being wrong.
Is more VRAM or more bandwidth better for local models?
They answer different questions. Capacity decides which models you can run at all, and bandwidth decides how fast the ones that fit will generate. A machine with a lot of slow memory runs large models slowly, and a fast card with little memory runs small models quickly.
Methodology and sources
Bandwidth figures are the manufacturers' published specifications, checked on 2026-08-11: Apple's M4 Pro and M4 Max announcement and M3 Ultra announcement, NVIDIA's RTX 5090 and RTX 4090 product pages, and AMD's RX 7900 XTX specifications. Quantised file sizes are approximate and vary between model releases and quantisation tools, so the derived tokens-per-second figures are order-of-magnitude guidance rather than benchmarks. The half-to-three-quarters band is an observation from reported figures and our own measurements, not a specification, and it is the softest number on this page. We have not benchmarked every device in the table; the ceilings are computed from published bandwidth, which is the point of the method.
This post was researched and drafted with AI assistance, then verified against primary documentation and edited by the recal team.