Tutorials 2 min read

Deploying a vLLM endpoint to 24 regions in one afternoon

A start-to-finish walkthrough: package a Llama 3 endpoint with vLLM, deploy it to four regions, verify routing, then roll out globally with a canary.

DR Daniel Reyes Developer Advocate

This tutorial takes a model you can run locally and puts it behind a global endpoint with region-aware routing and autoscaling. Total hands-on time is about two hours; most of it is waiting for the first image build.

Prerequisites

  • A Terrane account with the CLI installed (npm i -g @terrane/cli or brew install terrane)
  • Access to the model weights you intend to serve
  • Docker, if you want to build the image locally rather than in the platform

1. Describe the runtime

Create terrane.yaml in an empty directory:

runtime: edge-inference
regions: [iad, sfo, fra, sin]
autoscale:
  min: 1
  max: 8
  target_utilisation: 0.7
batching: dynamic
observability: full

services:
  - name: llama3-8b
    kind: inference
    engine: vllm
    model: meta-llama/Meta-Llama-3-8B-Instruct
    hardware: h200
    latency_budget_ms: 120

The engine: vllm line tells the runtime to use the managed vLLM image. You can bring your own with image: instead.

2. Deploy to four regions

terrane deploy --dry-run
terrane deploy

The dry run prints the plan: which regions receive capacity, in what order, and the hardware that will be reserved. The real deploy streams build and rollout logs. Expect 6 to 9 minutes for the first region; the others reuse the image and finish in under a minute each.

3. Verify routing

terrane endpoint llama3-8b
# https://llama3-8b.acme.terrane.example

curl -s https://llama3-8b.acme.terrane.example/v1/chat/completions \
  -H "Authorization: Bearer $TERRANE_KEY" \
  -d '{"model":"llama3-8b","messages":[{"role":"user","content":"Which region am I in?"}]}' \
  -D - | grep -i x-terrane-region

The x-terrane-region response header tells you where the request was served. Run the same command from a machine in Europe and you should see fra. Run it enough times from anywhere and you will occasionally see a different region when the nearest one is busy; that is the router doing its job.

4. Watch it under load

Open the runtime view and start a short load test:

terrane loadtest llama3-8b --rps 40 --duration 3m

Watch three panels: batch size distribution should climb as load rises, budget utilisation should stay under 100%, and autoscale events should show at least one scale-up in the busiest region. If budget utilisation pins at 100%, raise latency_budget_ms or lower target_utilisation.

5. Go global with a canary

Edit terrane.yaml:

regions: all
rollout:
  strategy: canary
  steps: [5, 25, 100]
  bake_minutes: 10

Then:

terrane deploy

The rollout moves 5% of traffic to the new regions, bakes for ten minutes while watching error rate and p99, then advances. Any regression pauses the rollout and pages whoever is on call. When it completes, you are serving from 24 regions.

Cleaning up

terrane destroy llama3-8b

Capacity is released within a minute and billing stops with it.

Keep reading

Related notes