Dynamic batching: more throughput without wrecking the tail
Batching requests is the cheapest way to raise GPU utilisation and the easiest way to ruin latency. Here is how Terrane decides when to wait and when to fire.
2 articles in the journal.
The rest of the team behind the journal. Every article carries a byline you can follow.