Cutting p99 latency in half with region-aware routing
How the Terrane router picks a runtime path using live health, queue depth and network distance, and what it did to our tail latency across 24 regions.
Averages lie. An inference API with a 42 ms median can still feel broken if one request in a hundred takes 900 ms, because that request is the one your user is waiting on. This post is about the tail, and about the routing changes that brought our p99 from 188 ms down to 91 ms across the fleet.
The problem with “nearest region”
The naive approach routes every request to the geographically closest region. It is easy to reason about and it is wrong often enough to matter. Three things break it:
- Capacity is not uniform. A region can be close and saturated at the same time. Sending a request into a queue that is already 400 deep saves 12 ms of network and costs 300 ms of waiting.
- Health is not binary. A runtime can be “up” while one accelerator is thermally throttled and answering at a third of its normal rate.
- Network distance changes. Peering shifts, undersea cables get cut, a transit provider has a bad afternoon. Static distance tables go stale within weeks.
What the router looks at now
Every runtime publishes a small health record every 250 ms: accepted requests per second, current queue depth, p50 and p95 for the last window, and a hardware status flag. The router combines that with a continuously refreshed latency map measured from the edge nodes themselves.
The scoring function is intentionally boring:
score = rtt_ms
+ queue_depth * per_request_cost_ms
+ penalty(hardware_status)
+ penalty(recent_error_rate)The lowest score wins. There is no machine learning in the hot path. A rule you can explain in one sentence is a rule you can debug at 3 a.m.
Failover without the cliff
The old failover was a switch: healthy or not. Flipping it moved 100% of a region’s traffic in one step and regularly overwhelmed the neighbour. The new behaviour is a ramp. As a region’s score degrades, it receives proportionally less traffic. By the time it is formally marked unhealthy, it is usually already carrying under 10% of its normal load.
We measured this during a real incident in fra on June 19th. The region lost two accelerator nodes within a minute. Traffic to fra dropped from 100% to 14% over 40 seconds while ams and lhr absorbed the rest. Global p99 rose from 91 ms to 117 ms and recovered in under two minutes. Nobody paged.
Results
| Metric | Before | After |
|---|---|---|
| Global p50 | 44 ms | 41 ms |
| Global p99 | 188 ms | 91 ms |
| Cross-region retries / min | 1,240 | 86 |
| Failover events causing alerts (30 days) | 7 | 0 |
The p50 barely moved, which is the point. Median requests were already fine. The work was entirely in the tail.
What we are doing next
Queue depth is a lagging signal. We are experimenting with publishing predicted completion time per runtime, derived from the shapes of requests already in flight, so the router can see a slow batch coming before the queue grows. Early numbers look promising; expect a follow-up once it has survived a month in production.