Preference data
The record of which answer a person preferred — and every surface feature that quietly decided it.
Definition
Preference data is the record of human judgments about which model output is better. Its most common form is the pairwise comparison — two responses to the same prompt, one marked preferred — and its common variants are ranked lists over several candidates, ratings on a scale, and rubric scores broken out by dimension such as correctness, helpfulness and clarity. It is the training input for reward models, and the direct input for preference-optimization methods that skip the reward model entirely.
Pairwise comparison is used because it is easier to do reliably than absolute scoring. Judges disagree wildly about what a seven out of ten means, and agree far more about which of two answers is better, so the comparison extracts a cleaner signal from the same person in less time. The trade-off is that comparisons carry no information about magnitude: a pair can be near-identical or one response can be catastrophic, and both are recorded as a single bit. That is why rubric scores are often collected alongside, and why the pairs shown are usually sampled to be informative rather than drawn at random.
The pathologies are well documented. Judges reward length, fluency, formatting and confident tone independently of correctness; presentation order biases which candidate wins; and on technical material a judge who cannot evaluate the substance falls back on surface features entirely. The countermeasures are structural rather than motivational: randomize presentation order, collect independent judgments from more than one person on a sample, measure agreement with a statistic suited to the scale, arbitrate disagreements instead of averaging them, and check the judges against reference items whose correct answer is already established.
How DeepenSkill approaches it
Those countermeasures describe how DeepenSkill runs preference work, and the results are reported rather than summarized. Everyone clears the same bar before starting: credentials verified, a standardized domain test passed, then a practical task scored against gold reference data. On live work the team measuring the experts is separate from the team running them — the platform rejects a validation report authored by the party that delivered the work — agreement statistics are reported with the method named, the item and rater counts stated and a 95% interval attached, and every judgment traces to a verified contributor, a task spec and a timestamp. See expert RLHF work, the judging discipline in model evaluation, and a sample verification bundle you can recompute yourself.
FAQ
What is preference data?
The record of human judgments about which model output is better — usually pairwise comparisons, sometimes ranked lists, ratings or rubric scores broken out by dimension.
Why are pairwise comparisons preferred to ratings?
Because people disagree about what an absolute score means and agree far more about which of two answers is better. Comparison extracts a cleaner signal from the same judge in less time.
What biases affect preference data?
Judges reward length, fluency, formatting and confident tone independently of correctness, and presentation order biases which candidate wins. Randomized order and independent second judgments are the standard defences.