Case study: fraud scoring on every card authorisation, inside a 40 ms budget

Every card authorisation this provider processes goes through a fraud model before the bank answers. The card network gives them a hard budget: if the risk decision is not back in 40 ms, the authorisation continues without it. In the industry this is called failing open, and every failed-open decision is a transaction nobody scored.

Before

The model ran in a single US region on a managed inference service. European and Brazilian traffic crossed an ocean to reach it, so the median was fine and the tail was not: the p99 sat at 71 ms, and on a normal day 0.8% of decisions timed out and failed open. Fridays were worse.

Two things made this hard to fix in place. The card data of European and Brazilian issuers is not allowed to leave its jurisdiction, so “add a region” meant a second and third copy of the whole serving stack with its own deploy path. And the model is retrained weekly, so any multi-region setup had to roll a new version out everywhere without ever leaving a region on the old one for long.

What changed

The model, its feature-lookup service and the decision API moved into one Terrane deployment pinned to six regions, two of them residency-restricted:

services:
  - name: risk-features
    kind: http
    image: registry.internal/risk/features:2026.05
    latency_budget_ms: 8
  - name: risk-model
    kind: inference
    model: risk-xgb-ensemble-v41
    hardware: cpu
    latency_budget_ms: 20
  - name: decision-api
    kind: http
    image: registry.internal/risk/decision:2026.05
regions:
  serve: [iad, sfo, fra, ams, gru, sin]
  pin:
    fra: { data: eu-card }
    ams: { data: eu-card }
    gru: { data: br-card }
rollout:
  strategy: canary
  steps: [5, 25, 100]
  pause_on: { p99_ms: 36 }

Region pinning does the residency work: requests carrying EU card data can only be routed to fra or ams, and the runtime refuses a rollout that would leave those two regions without a healthy copy. The weekly retrain is a canary that pauses itself if the new version pushes p99 past 36 ms in any region, well inside the 40 ms line.

The model runs on CPU on purpose. At two thousand decisions a second the ensemble is cheap to evaluate, and a CPU tier in six regions costs less than accelerators in one.

Results after 120 days

	Before	After
p50 decision time	29 ms	18 ms
p99 decision time	71 ms	34 ms
Decisions failed open	0.8%	0.02%
Regions serving traffic	1	6
Weekly model rollout	4 hours, manual	35 minutes, automatic canary
Pages to on-call per month	14	2

The remaining failed-open decisions are almost entirely issuer-side timeouts that no serving change would recover.

What did not go smoothly

The feature-lookup service depended on a cache that was warm in the US and cold everywhere else. The first week in gru showed a p99 twice the budget while the cache filled. The team added a warm_pool: 1 node per region and a pre-warm job that replays the previous hour of traffic against a new region before it takes live requests. The runtime did not solve this; it made the problem visible per region so the fix was obvious.

In their words

“We spent two years explaining to the network why 0.8% of our decisions were blank. Now the conversation is about the model, which is the conversation we wanted.” — Head of risk engineering