loading
loading
Storing a model's weights at lower numerical precision (e.g. 8-bit or 4-bit instead of 16-bit) to shrink memory and speed up inference.
A 70B-parameter model in 16-bit floats needs ~140GB of memory; quantizing the weights to 8-bit roughly halves that and 4-bit halves it again, which is often the difference between a model fitting on one GPU or not. The tradeoff is a small, usually-tolerable loss in output quality that grows as precision drops — 8-bit is typically near-lossless, aggressive 4-bit schemes can degrade reasoning. It's a deployment-time lever for self-hosted open models; with a hosted API like Claude's you don't manage it, but it's why the same open model can run at very different speeds and quality depending on how it was packaged.
Plainly
Think of Quantization as roads and power for the app city. Storing a model's weights at lower numerical precision (e.g. 8-bit or 4-bit instead of 16-bit) to shrink memory and speed up inference.
In practice
Use it when local behavior needs to become a reachable, reliable deployed service. In practice, define the owner, input, output, and failure mode before you rely on it.