Case study: transcribing 900 hours of audio a night on capacity that goes to zero by morning

A broadcasting group transcribes everything it airs, in fourteen languages, so that editors can search yesterday’s programmes this morning. The batch runs overnight, it has to be finished before the newsroom arrives, and until last spring it usually was not.

Before

The pipeline was a queue, a fleet of GPU workers that ran all night whether the queue was full or not, and a spreadsheet. The spreadsheet listed the jobs that had failed and needed a human to resubmit them, about 6% each night. Workers were sized for the worst night of the year, so on an average night they ran at 30% utilisation and still finished after 9 a.m., because a handful of very long files went through one at a time.

The team had tried to solve this with autoscaling twice. Both attempts were abandoned after a month of tuning thresholds that were right for one night and wrong for the next.

What changed

Transcription became a batch service on Terrane with a latency budget measured in hours rather than milliseconds, dynamic batching turned up, and scale-to-zero when the queue is empty:

services:
  - name: transcribe
    kind: inference
    model: whisper-large-v3
    hardware: l40s
    batching:
      max_batch: 24
      max_wait_ms: 30000
    scale:
      min: 0
      max: 40
      target: queue_depth
    retries: 3
    latency_budget_ms: 7200000
regions:
  serve: [fra, iad, syd]

Three settings did most of the work. max_wait_ms: 30000 lets the runtime hold a batch open for thirty seconds so it can fill, which is irrelevant for a chat API and exactly right for a nightly job. scale.min: 0 means nothing runs by 4 a.m. when the queue drains. And retries: 3 moved the spreadsheet into the runtime: a job that fails on a node is retried elsewhere, and only jobs that fail three times reach a person.

Long files were split at silence boundaries by a small pre-processing service, so no single file blocks a worker for forty minutes any more.

Results after 90 days

	Before	After
Nightly backlog cleared in	9.5 h	3.1 h
GPU hours per night	210	96
Average GPU utilisation during the run	30%	84%
Cost per audio hour (index)	100	43
Jobs re-run by hand per night	~54 (6%)	0
Editors who noticed	—	all of them

The cost figure includes the extra region: running in fra, iad and syd lets the batch follow the cheapest available capacity across the night.

What did not go smoothly

Scale-to-zero has a cold start, and the first night the queue had two thousand jobs waiting while the first node came up. The team set a schedule: min: 4 from 22:00 to 23:00, then back to zero. It is a small compromise on the elegance of the idea and it removed the only complaint they had.

In their words

“I deleted the spreadsheet in June. Nobody has asked for it since.” — Archive platform lead