Documentation

Everything the runtime does, written down.

Start with the quickstart if you have fifteen minutes. Use the references when you have a specific key, flag or header in front of you.

Start here

Deploy your first endpoint in fifteen minutes.

Three steps: install the CLI, describe the runtime in one file, deploy to two regions. Switch tabs to follow along with a terminal open.

Start on the free tier How it works 
$ npm i -g @terrane/cli
$ terrane login
✓ authenticated as ada@company.com
$ terrane init --template inference
✓ wrote terrane.yaml
Index

Six chapters

Guides are written to be followed with a terminal open. References are written to be searched. Open an entry to see what it covers.

  1. 01 15 min

    Quickstart

    Install the CLI, write your first terrane.yaml, deploy to two regions and read the runtime view.

    • Installation

      npm i -g @terrane/cli, or a standalone binary for macOS, Linux and Windows. Node 18 or newer for the npm route.

    • Authentication

      terrane login opens the browser once and stores a project-scoped token in your keychain. CI uses TERRANE_TOKEN.

    • Your first deployment

      terrane init writes a starter terrane.yaml; terrane deploy --dry-run prints the plan; terrane deploy ships it.

    • Reading the runtime view

      Health per region, request flow, latency percentiles and the deployment timeline, on one screen.

  2. 02 reference

    Runtime configuration

    Every key in the deployment file: services, regions, hardware tiers, autoscaling, batching budgets and rollout policies.

    • services[]

      One entry per service: name, kind (inference, http, worker), model or image, hardware tier, latency budget.

    • regions and all

      A list of region codes, or all. Add pin: to keep tagged data inside a jurisdiction.

    • autoscale

      min, max and target (queue_depth, rps or utilisation). min: 0 scales to zero when the queue is empty.

    • batching and latency_budget_ms

      max_batch and max_wait_ms bound the batch; latency_budget_ms is the promise the scheduler keeps.

    • rollout strategies

      direct, or canary with steps and pause_on gates. A failed gate keeps the previous version serving.

    • warm_pool

      Nodes kept ready ahead of demand, per region. Costs a little idle capacity, removes cold starts.

  3. 03 reference

    CLI reference

    The terrane command, subcommand by subcommand, with flags and exit codes.

    • deploy, --dry-run

      Plan, then apply. Exit code 0 only when every region reports ready.

    • rollout pause / resume / rollback

      Freeze a canary, continue it, or return to the last healthy version in under a minute.

    • endpoint, loadtest

      Print the stable endpoint of a service; replay a traffic profile against it and read the percentiles.

    • logs, metrics

      Tail logs by service, region and window; export metrics as JSON or OpenMetrics.

    • migrate, destroy

      Read a Kubernetes or Compose file and print an equivalent terrane.yaml; tear a deployment down explicitly.

  4. 04 20 min

    Routing and failover

    How requests are scored, how regions shed load, how to pin traffic and how to read the x-terrane-region header.

    • Scoring function

      rtt + queue_depth × per_request_cost + hardware penalty + recent-error penalty. The lowest score wins.

    • Proportional failover

      A region sheds traffic in proportion to its degradation before it is ever marked unhealthy.

    • Region pinning

      Requests tagged with a data class are routed only to the regions pinned for that class.

    • Health records

      What every runtime publishes every 250 ms, and how to read it in the request log.

  5. 05 20 min

    Observability and alerts

    Panels, percentiles, log retention, alert rules and exporting metrics to your own stack.

    • Runtime view

      The four questions: healthy, where, slow, changed. Every panel links to logs for the same window.

    • Alert rules

      Thresholds on any percentile, per service and region, routed to email, Slack or PagerDuty.

    • Log export

      Ship request logs to your own store, with a retention you control.

    • OpenTelemetry endpoint

      Scrape or push metrics into Grafana, Datadog or anything that speaks OTLP.

  6. 06 reference

    SDKs

    Typed clients for Python and TypeScript with per-request batching overrides and streaming helpers.

    • Python

      pip install terrane. Typed client, streaming helpers, per-request batching overrides.

    • TypeScript

      npm i @terrane/sdk. Same surface as Python, first-class in edge runtimes.

    • REST API

      Every SDK call is an HTTP request you can make yourself; the reference lists them all.

    • Authentication tokens

      Project-scoped, role-scoped, expiring. Revoke any of them from the audit log.

Hands-on

Guides from the journal

Longer walkthroughs and engineering notes that explain the why behind the reference.

  1. 01 Engineering Cutting p99 latency in half with region-aware routing How the Terrane router picks a runtime path using live health, queue depth and network distance, and what it did to our tail latency across 24 regions.
  2. 02 Engineering Dynamic batching: more throughput without wrecking the tail Batching requests is the cheapest way to raise GPU utilisation and the easiest way to ruin latency. Here is how Terrane decides when to wait and when to fire.
  3. 03 Tutorials Deploying a vLLM endpoint to 24 regions in one afternoon A start-to-finish walkthrough: package a Llama 3 endpoint with vLLM, deploy it to four regions, verify routing, then roll out globally with a canary.