Case study: a semantic search API at 14,000 requests per second

The team runs product search for a marketplace with 38 million listings. Every query is embedded, matched against a vector index and reranked by a cross-encoder before results go back to the shopper. At peak they see 14,200 requests per second.

Before

Three deployments on three systems: the embedding model on a managed GPU service, the reranker on self-managed Kubernetes with GPU node pools, and the search API on a serverless platform. Each had its own scaling rules, its own dashboard and its own deploy pipeline.

The number that hurt most was p99: 340 ms, dominated by cross-system hops and cold starts on the serverless tier. A second number hurt in a different way: two engineers spent roughly 40% of their time keeping the three systems in step.

What changed

The three services moved into one Terrane deployment across six regions:

services:
  - name: embed
    kind: inference
    model: bge-m3
    hardware: l40s
    latency_budget_ms: 25
  - name: rerank
    kind: inference
    model: bge-reranker-v2
    hardware: l40s
    latency_budget_ms: 40
  - name: search-api
    kind: http
    image: registry.terrane.example/market/search:2026.02

Requests enter through the search API, which calls embed and rerank over the runtime’s internal mesh. Because all three live in the same region and the same rollout, there is no cross-provider hop and no version skew.

Results after 90 days

	Before	After
p50	88 ms	41 ms
p99	340 ms	98 ms
Peak throughput	11,600 req/s	14,200 req/s (no longer capped by the reranker)
GPU utilisation	38%	79%
Infra cost per 1M queries	100 (index)	61
Deploy pipelines maintained	3	1

Dynamic batching did most of the utilisation work: at peak the reranker now runs batches of 12 to 16 and still sits inside its 40 ms budget.

What did not go smoothly

The first canary rollout paused twice because the reranker’s cold start on a freshly scaled node briefly pushed p99 over the budget. The fix was a warm_pool: 1 setting per region, which keeps one node ready ahead of demand. It costs a little idle capacity and removed the problem entirely.

In their words

“We had a search team and an infrastructure team. Now we have a search team.” — Lead engineer, marketplace search