Model evaluation

Definition

Model evaluation is the practice of establishing what a model can and cannot do, by assessing its outputs against a standard of correctness, quality or safety. It divides into automated evaluation — scoring against a held-out set with a programmatic metric — and human evaluation, where people judge outputs directly, either by comparing candidates or by scoring against a rubric. A third form, model-graded evaluation, uses a language model as the judge; it is fast and cheap, but it inherits the judge model's blind spots and has to be validated against human judgment before its scores mean anything.

The hard part is not running an evaluation but designing it. An evaluation is a claim about a population of tasks, and it is only as good as the sample standing in for that population. Three failure modes quietly invalidate results: items contaminated by leaking into training, items too easy to discriminate between good models, and items whose supposedly correct answer is itself wrong. Evaluations also decay — once a set is public and optimized against, its score stops tracking the capability it was built to measure while remaining a headline number.

Reporting is the other half. A number without a method, a sample size and an interval is not a result, and two evaluations of the same model are comparable only if the prompt format, the decoding settings and the scoring rule were held constant. Where humans do the scoring, the same discipline applies to them: independent scoring by more than one judge, agreement measured with a statistic suited to the scale, disagreements arbitrated rather than averaged, and reference items with a known answer used to check the judges themselves.

How DeepenSkill approaches it

DeepenSkill supplies the human judges and applies that reporting discipline to their work as a matter of course. Everyone clears the same bar before starting: credentials verified, a standardized domain test passed, then a practical task scored against gold reference data. On live work the team measuring the experts is separate from the team running them — the platform rejects a validation report authored by the party that delivered the work — agreement statistics are reported with the method named, the item and rater counts stated and a 95% interval attached, and every judgment traces to a verified contributor, a task spec and a timestamp. See expert model evaluation work, the adversarial case in red-teaming, and a sample verification bundle you can recompute yourself.

FAQ

• What is model evaluation?

  Establishing what a model can and cannot do by assessing its outputs against a standard of correctness, quality or safety — with automated metrics, with human judgment, or with both.

• Can a language model be used as the judge?

  Yes, and it is fast and cheap, but it inherits the judge model's blind spots. Model-graded scores have to be validated against human judgment before they mean anything.

• Why do benchmarks lose their value over time?

  Because once a set is public it gets optimized against, directly or through data selection, and its score drifts away from the capability it was built to track while still being quoted as a headline number.