loading
loading
A separate model trained to score how good a response is, used as the optimization target during RLHF.
You train it on human preference data — pairs where a labeler said 'A is better than B' — and it learns to output a scalar reward predicting which answers humans like. During RLHF the main model is tuned to maximize this score, so the reward model is effectively a stand-in for human judgment at scale. Its blind spots become the policy's blind spots: if the reward model loves long verbose answers, the trained model will pad — classic reward hacking and a source of slop.
Plainly
Think of Reward Model as a simple recipe for doing the work better. A separate model trained to score how good a response is, used as the optimization target during RLHF.
In practice
Use it when you need a repeatable method instead of guessing from vibes. In practice, define the owner, input, output, and failure mode before you rely on it.