loading
loading
When a model scores high on a benchmark because the test data leaked into its training set.
If the benchmark's questions and answers were scraped from the public web before training, the model may have memorized them, so the score measures recall, not capability — and it tricks you into picking the wrong model. It's also a trap in your own evals: never let your golden dataset end up in fine-tuning data or public repos. Example: a model aces a coding benchmark whose solutions sit in a popular GitHub repo it trained on, then flops on your real tasks.
Plainly
Think of Benchmark Contamination as the brain part that guesses or decides. When a model scores high on a benchmark because the test data leaked into its training set.
In practice
Use it when model choice, prompts, latency, cost, or quality affect the product result. In practice, define the owner, input, output, and failure mode before you rely on it.