Terrane Runtime 2.0: unified deployment surface, 24 regions, live observability
The second major release of the runtime brings one deployment file for inference, APIs and backend services, six new regions and a rebuilt observability layer.
Journal
Notes from the runtime team
Latency histograms, rollout post-mortems, copy-pasteable tutorials and the occasional opinion on how production AI should be run.
Cutting p99 latency in half with region-aware routing
How the Terrane router picks a runtime path using live health, queue depth and network distance, and what it did to our tail latency across 24 regions.
All articles
The archive
Dynamic batching: more throughput without wrecking the tail
Batching requests is the cheapest way to raise GPU utilisation and the easiest way to ruin latency. Here is how Terrane decides when to wait and when to fire.
Deploying a vLLM endpoint to 24 regions in one afternoon
A start-to-finish walkthrough: package a Llama 3 endpoint with vLLM, deploy it to four regions, verify routing, then roll out globally with a canary.
Why we built a unified deployment surface
Production AI is not a model behind a URL. It is a model, an API in front of it, workers around it and a control plane nobody wanted to write. Here is why we collapsed them into one.
Case study: a semantic search API at 14,000 requests per second
How a marketplace team moved embeddings, reranking and their public search API onto Terrane, cut p99 from 340 ms to 98 ms and stopped running a control plane.
No articles in this category yet.