loading
loading
Running many requests through the model at once to use the GPU efficiently.
GPUs are massively parallel, so processing one request wastes most of the chip — batching packs many requests into a single forward pass to amortize the cost. Modern serving uses continuous (in-flight) batching, where new requests slot into the batch as soon as others finish a token, instead of waiting for the whole batch to complete. This is the single biggest lever on throughput and cost. The tradeoff: a packed batch can add latency to any individual request.
Plainly
Think of Batching as roads and power for the app city. Running many requests through the model at once to use the GPU efficiently.
In practice
Use it when local behavior needs to become a reachable, reliable deployed service. In practice, define the owner, input, output, and failure mode before you rely on it.