loading
loading
The delay between sending a prompt and receiving the model's response.
Latency in LLM systems has two components: time-to-first-token (TTFT), which affects how quickly users see a response start, and total generation time, which depends on output length. Streaming solves the perception problem for TTFT. Choosing a smaller, faster model for simple tasks and reserving large models for complex ones is the most impactful latency optimization.
Plainly
Think of Latency as the brain part that guesses or decides. The delay between sending a prompt and receiving the model's response.
In practice
Use it when model choice, prompts, latency, cost, or quality affect the product result. In practice, define the owner, input, output, and failure mode before you rely on it.