Cutting p99 latency in half with region-aware routing
How the Terrane router picks a runtime path using live health, queue depth and network distance, and what it did to our tail latency across 24 regions.
Category
Engineering
Deep dives into how the Terrane runtime is built: routing, scheduling, failover and the numbers behind them.
Notes from the runtime group. Expect latency histograms, design trade-offs and the occasional post-mortem.
Articles
2 articles
Dynamic batching: more throughput without wrecking the tail
Batching requests is the cheapest way to raise GPU utilisation and the easiest way to ruin latency. Here is how Terrane decides when to wait and when to fire.