loading
loading
The component that chops text into tokens the model can actually read, and stitches them back into text.
Models don't see characters or words — they see token IDs, and the tokenizer is the lookup table that converts between them. It splits text into subword chunks (common words are one token, rare words split into several) using algorithms like BPE, which is why 'cat' is one token but 'antidisestablishmentarianism' is many. This is why token counts differ from word counts, why non-English text often costs more tokens, and why weird spacing can quietly inflate your bill.
Plainly
Think of Tokenizer as the brain part that guesses or decides. The component that chops text into tokens the model can actually read, and stitches them back into text.
In practice
Use it when model choice, prompts, latency, cost, or quality affect the product result. In practice, define the owner, input, output, and failure mode before you rely on it.