loading
loading
Throughput is total output volume over time, usually tokens/second across all concurrent requests — the metric that decides your cost per million tokens. It trades off against per-request latency: batching more requests together raises throughput but can slow any single one. A GPU might do 50 tok/s for one user but 5000 tok/s aggregate when batching 200. Care about this when you're serving at scale or self-hosting and paying for the hardware.
Plainly
Think of Throughput as roads and power for the app city. How many tokens (or requests) a server can crank out per second.
In practice
Use it when local behavior needs to become a reachable, reliable deployed service. In practice, define the owner, input, output, and failure mode before you rely on it.