loading
loading
The first, most expensive phase where a model learns language and world knowledge by predicting the next token across a huge chunk of the internet.
Pretraining runs self-supervised next-token prediction over trillions of tokens of text and code, with no human labels — the model just learns to continue text, and in doing so absorbs grammar, facts, reasoning patterns, and coding idioms. This is where almost all the compute and cost goes (months on thousands of GPUs), and it produces a raw 'base model' that completes text but won't follow instructions or chat. Everything after — instruction tuning, RLHF — is cheap polish on top of this foundation.
Plainly
Think of Pretraining as the brain part that guesses or decides. The first, most expensive phase where a model learns language and world knowledge by predicting the next token across a huge chunk of the internet.
In practice
Use it when model choice, prompts, latency, cost, or quality affect the product result. In practice, define the owner, input, output, and failure mode before you rely on it.