Reward model
A learned stand-in for the human judge — and everything the policy will do to exploit it.
Definition
A reward model is a model trained to score how good an output is. It takes a prompt and a candidate response and returns a scalar the training process can optimize against. It is fit on human preference data — typically pairs where a person marked one response as better than the other — using a loss that pushes the score of the preferred response above the score of the rejected one. Its purpose is economic as much as technical: human judgment is far too slow to consult at every step of reinforcement learning, so the reward model is trained once to stand in for the judges, then queried millions of times.
Because it is a learned approximation, it can be wrong in ways the policy will find. Optimizing hard against an imperfect scorer produces reward hacking — outputs that score well and satisfy nobody: excessive length, hedging, flattery, confident assertion in place of correct reasoning. Practitioners counter this with a divergence penalty that keeps the policy near its starting distribution, with ensembles of reward models, and by periodically refreshing preference data collected on the policy's current behaviour rather than on an older snapshot.
Evaluating one is its own discipline. The usual instrument is a held-out preference set the model never saw during training, scored on how often its ranking matches the humans'. Aggregate accuracy hides the interesting failures, so held-out sets are normally broken out by domain and by difficulty: a reward model can be near-perfect on everyday requests and no better than chance on specialist material, which is exactly the regime frontier training cares about. Building those held-out sets is expert authoring work in itself, and the ceiling on it is finding people qualified to say which of two specialist answers is actually correct.
How DeepenSkill approaches it
The preference pairs a reward model is fit to, and the held-out sets it is tested against, are authored by people — which is where DeepenSkill sits. Everyone clears the same bar before starting: credentials verified, a standardized domain test passed, then a practical task scored against gold reference data. On live work the team measuring the experts is separate from the team running them — the platform rejects a validation report authored by the party that delivered the work — agreement statistics are reported with the method named, the item and rater counts stated and a 95% interval attached, and every judgment traces to a verified contributor, a task spec and a timestamp. See expert RLHF work, expert model evaluation, and a sample verification bundle you can recompute yourself.
FAQ
What is a reward model in RLHF?
A separate model trained on human preference data to score how good a candidate response is, used in place of the human judges to supply a reward signal during reinforcement learning.
What is reward hacking?
Behaviour where a policy finds outputs that score highly against an imperfect reward model without being better — excessive length, hedging, flattery, or confident assertion in place of correct reasoning.
How is a reward model evaluated?
Against a held-out preference set it never saw in training, scored on how often its ranking matches the humans', and broken out by domain and difficulty so a strong average cannot hide weakness on specialist material.