loading
loading
A model that takes in or produces more than just text — images, audio, video, or PDFs alongside words.
Multimodal models encode non-text inputs (an image, a screenshot, an audio clip) into the same token/embedding space the language model already understands, so it can reason across modalities in one pass. Practically, this is what lets you paste a screenshot of a bug, hand it a design mockup to turn into code, or feed it a PDF without OCR plumbing. Costs are usually counted by converting images to a token-equivalent, so a high-res screenshot can quietly be thousands of tokens.
Plainly
Think of Multimodal as the brain part that guesses or decides. A model that takes in or produces more than just text — images, audio, video, or PDFs alongside words.
In practice
Use it when model choice, prompts, latency, cost, or quality affect the product result. In practice, define the owner, input, output, and failure mode before you rely on it.