Dynamic batching: more throughput without wrecking the tail
Batching requests is the cheapest way to raise GPU utilisation and the easiest way to ruin latency. Here is how Terrane decides when to wait and when to fire.
A single request rarely fills an accelerator. Batching several together raises utilisation, sometimes from 30% to 80%, and cuts cost per token accordingly. The catch is that a batch has to wait for its members to arrive, and every millisecond of waiting lands on the latency of the first request in.
Two knobs, one decision
Every batching system trades on two parameters: how many requests to wait for and how long to wait for them. Static values are always wrong for someone. A max_wait of 20 ms is invisible under heavy load and doubles latency under light load, when the batch never fills and the timer always fires.
Terrane’s batcher replaces the static pair with a target: stay under a latency budget you set, and within that budget, batch as aggressively as possible.
How the scheduler decides
For each runtime, the scheduler maintains a live estimate of two curves:
- Arrival rate: how many requests per millisecond are showing up right now, smoothed over the last second.
- Execution cost by batch size: measured, not modelled. A batch of 8 on an H200 does not cost eight times a batch of 1, and the ratio changes with sequence length.
Given a latency budget of, say, 60 ms, the scheduler computes the largest batch it can form and execute within the remaining budget for the oldest request waiting. If arrivals are slow, batches stay small and fire fast. If arrivals are fast, batches grow because they fill before the budget runs out.
What operators see
The batching panel in the runtime view shows three things per runtime: current batch size distribution, budget utilisation (how close batches are running to your latency ceiling) and the throughput gain versus unbatched execution. When budget utilisation sits at 95% for more than a minute, the panel says so, because that usually means the budget is too tight for the traffic and you are leaving throughput on the table.
Numbers from a customer workload
An embedding API on 4× L40S, 1,200 requests per second at peak, 70 ms p99 budget:
| Static batching (8 / 15 ms) | Dynamic batching | |
|---|---|---|
| Peak throughput | 1,180 req/s | 1,910 req/s |
| p99 at peak | 96 ms (over budget) | 66 ms |
| p99 at 10% load | 31 ms | 19 ms |
| GPU utilisation at peak | 61% | 84% |
The static configuration was tuned by hand for peak and still missed the budget; off-peak it added 15 ms of pure waiting to every request. The dynamic scheduler needed one number.
Caveats
Batching only helps when requests are similar enough to share an execution. Mixed workloads, a 20-token classification next to a 4,000-token summary, batch badly. The runtime detects this and splits by shape automatically, but if your traffic is truly heterogeneous, separate deployments will always beat one clever batcher.