Cutting p99 latency in half with region-aware routing
How the Terrane router picks a runtime path using live health, queue depth and network distance, and what it did to our tail latency across 24 regions.
Maya leads the runtime group at Terrane: the schedulers, the routing layer and the failover paths that keep inference traffic flowing when a region goes dark. Before Terrane she ran the edge-compute platform at a CDN and, earlier, built the request pipeline for a large-scale search engine.
She writes about the unglamorous parts of production AI, the p99s, the queue depths and the retry storms, because that is where the real engineering lives. Her long-standing rule for the team: if a system needs a hero to stay up, the system is not finished.
Articles
By Maya
2 articles in the journal.
02 Company
Company 2 min read
Why we built a unified deployment surface
Production AI is not a model behind a URL. It is a model, an API in front of it, workers around it and a control plane nobody wanted to write. Here is why we collapsed them into one.