loading
loading
The test rig that runs your prompt or agent against a fixed dataset and scores the outputs so you can measure changes objectively.
An eval harness is the plumbing around your evals: it loads test cases, runs them through your system, collects outputs, and applies scoring (exact match, an LLM judge, or a custom checker) to produce a number. Without one you're shipping prompt changes on vibes; with one you can prove a tweak made things 8% better or quietly worse. Treat it like a CI suite for your model behavior — run it on every prompt or model change.
Plainly
Think of Eval Harness as the checklist that keeps the app open for real people. The test rig that runs your prompt or agent against a fixed dataset and scores the outputs so you can measure changes objectively.
In practice
Use it when a change has to survive deploys, users, incidents, analytics, or billing reality. In practice, define the owner, input, output, and failure mode before you rely on it.