Pre-registration: an independent re-judging of RewardBench 2

Why publish this before the result

An evaluation number is only as trustworthy as the choices made before it existed. Which items were sampled, which were dropped, which coefficient was reported, whether the interval was computed the way it was planned, whether the sample stopped growing when the answer looked good — every one of those decisions can be made after the fact, and none of them is visible in the number itself.

Pre-registration is the standard answer to that problem in clinical trials and, increasingly, in empirical research generally: fix the design in public, date it, and let anyone check afterwards that the analysis they were shown is the analysis that was planned. We think a measurement offered as evidence of quality deserves the same discipline, so we are applying it to our own runs starting with this one. The full registered text is reproduced verbatim near the bottom of this page, with a content hash you can recompute yourself.

This page makes no claim about a result. There isn't one.

What is being tested

RewardBench 2 is a public benchmark for reward-model evaluation, released under the ODC-BY licence by the Allen Institute for AI (Malik et al., RewardBench 2: Advancing Reward Model Evaluation, arXiv:2506.01937, ICLR 2026). Its test split carries 1,865 rows, each pairing a chosen response with rejected alternatives.

Those labels are used across the field as an answer key. What is missing next to them is a human agreement figure: how much a set of independent expert judges, working blind, land where the published label lands. That is the number this run supplies — measured, not asserted.

Three validators will independently judge every item in a stratified sample of 210 items, each blind to the benchmark's own published choice and blind to the other two validators' judgments. Blindness is enforced by the review software rather than by reviewer discipline: the reviewer-facing view of an item never carries the original label, and each reviewer sees only their own assignment. Because the label has two options and every item carries three judgments, a majority always exists, so no arbiter step can arise.

The exact dataset revision is pinned in the registered text by commit hash and modification date. A re-run against a later revision would be a different measurement, and would have to say so rather than quietly reuse this registration.

The sample, and how its size was fixed

Five of RewardBench 2's six subsets are uniformly best-of-four on every row: one chosen response, three rejected, four completions. Checked row by row across all 1,865 rows rather than on a sample, those five are Factuality (475), Focus (495), Math (183), Precise IF (160) and Safety (450) — 1,763 rows in total.

The sixth subset, Ties (102 rows), is excluded from this run in full. Its chosen and rejected counts vary per row by design, because it exists to test whether a reward model detects a pool of near-equal candidates. There is no honest way to reduce “detect an N-way near-tie” to “pick the better of two responses”, so it is left out and the exclusion is stated up front rather than discovered in a footnote later.

From the remaining population of N = 1,763, n = 210 items are drawn by proportional stratified sampling with largest-remainder apportionment, stratified on the benchmark's own subset field. The allocation follows from that rule and from the recorded seed:

Factuality   475 -> 56
Focus        495 -> 59
Math         183 -> 22
Precise IF   160 -> 19
Safety       450 -> 54
Total       1763 -> 210

The seed — rewardbench2-measurement-1-proof-run-v1 — was recorded before any sample was drawn. The same population, sample size and seed always produce the same sample on any machine, and the seeded generator is specified bit for bit in the published verification bundle.

On the size itself, plainly: n = 210 is a design constant, fixed before the sample was drawn. A simulation-based power analysis of this exact design and these exact statistics — run before any judgment exists, and reproduced in the registered text below — finds it has 80% power to detect a Cohen's κ of 0.19 or a Krippendorff's α of 0.15 as beyond chance; true effects smaller than that are as likely as not to go undetected. A separate, harder, and separately reported question — whether the interval clears those same two values outright, not just chance — only reaches 80% power around κ 0.37 / α 0.28; it is not a falsification test. What the fixed number still buys, independent of either figure, is that the sample cannot be grown or trimmed afterwards to move a coefficient.

The statistics that will be reported

Two coefficients will be reported, and they answer different questions. Conflating them is the single easiest way to mislead with this kind of measurement, so the registration binds us to name which is which every time either appears.

Consensus against the published label — Cohen's κ

RewardBench 2's own published choice is paired against our final label, which is the majority of the three independent judgments. That is two entities, complete data, nominal labels — the case Cohen's κ is defined for, so κ is the reported coefficient. Gwet's AC1 is reported alongside it over the same pairs as a supplementary cross-check, never in place of it. Why those are the right choices for this shape of data, and where each one breaks, is set out in our guide to inter-annotator agreement.

Validators against each other — expected to be Krippendorff's α

A second, separate coefficient is computed over the three validators' own raw labels, with no external reference at all. With three distinct validators per item, the expected method is Krippendorff's α rather than κ. Where an item carries fewer than two judgments the value is reported as null, never fabricated.

These are different claims. The first says how much our consensus resembles one external answer key. The second says how reliable our validators are with each other. A high first number with a low or absent second one is a real possibility — validators disagreeing with each other while still landing on the same majority — and it would be reported as exactly that.

What is deliberately not reported

Percent agreement is not the headline statistic here: every row carries two comparable entities, so a chance-corrected coefficient is admissible, and the observed-agreement term inside the κ computation stays readable in the published record either way.

Reference-item (“gold”) accuracy does not apply to this run, by design. Every item is a standard item and no known-answer item is mixed in, because the benchmark's published labels are the object under audit. Scoring our validators against those same labels as though they were ground truth would be circular.

Defect rate is reported with a Wilson score interval if, and only if, validators log defects during review. No defect taxonomy is required for this run, so that statistic is opportunistic rather than a designed part of the measurement.

Intervals

Every point estimate ships with a 95% interval: a seeded percentile bootstrap at 2,000 resamples for the two coefficients, and a Wilson score interval for proportions. The seed and the resampling procedure travel with the published record, so an interval can be reproduced bit for bit — in another language, on someone else's machine. A coefficient published without its interval and its seed is a claim, not a result.

The registered text, and its content hash

Below is the registered methodology verbatim. Its SHA-256 is printed so that this page can be checked against any later copy of itself: if a single character changes, the hash changes, and the version you are reading is no longer the version that was registered.

SHA-256 of the block below

fca8d3ea917c84490788e0f6d91b7850cd02819817b7d22eaa6b74941a65be9a

Exactly which bytes are hashed. The hashed input is the block below in full: from the first character of the BEGIN marker line through the newline that ends the END marker line, both marker lines included. The two marker lines occur exactly once on this page, as the first and last lines of the block below — anchor any script on whole lines, not substrings. The block is US-ASCII throughout, UTF-8 encoded, with LF line endings, no byte-order mark, exactly one trailing newline and no other trailing whitespace — 252 lines, 15,776 bytes. Save it as a plain text file and run:

Recompute it yourself

$ wc -l < rewardbench-2-preregistration.txt
     252
$ wc -c < rewardbench-2-preregistration.txt
   15776
$ shasum -a 256 rewardbench-2-preregistration.txt
fca8d3ea917c84490788e0f6d91b7850cd02819817b7d22eaa6b74941a65be9a  rewardbench-2-preregistration.txt

# or, without shasum:
$ python3 -c "import hashlib,sys;print(hashlib.sha256(open(sys.argv[1],'rb').read()).hexdigest())" rewardbench-2-preregistration.txt
fca8d3ea917c84490788e0f6d91b7850cd02819817b7d22eaa6b74941a65be9a

If your line and byte counts match and your hash does not, the copy you are holding has been altered. That is the point of printing it.

Registered methodology

-----BEGIN PRE-REGISTRATION METHODOLOGY-----
Measurement: an independent re-judging of a stratified sample of RewardBench 2.
Registered 2026-08-19. Amended 2026-08-22, 2026-08-23 and 2026-08-29 (changelog
below).
Published 2026-08-29. Status at publication: the run has not started, no
judgment of any kind exists, and the three judges are not yet named.

1. DATASET
   Source: RewardBench 2, "test" split, 1865 rows. A public benchmark under the
   ODC-BY licence (Open Data Commons Attribution). Attribution: Malik et al.,
   "RewardBench 2: Advancing Reward Model Evaluation", arXiv:2506.01937,
   ICLR 2026, Allen Institute for AI.
   Revision pinned: commit 7ff08853b0d5686e79b13fda8677024f566a104a, dataset
   last modified 2025-06-04, fetched 2026-08-19 and frozen locally. A re-run
   against any later revision is a different measurement and must say so.
   Checked row by row across all 1865 rows, not on a sample: five of the six
   subsets are uniformly best-of-4 on every row (one chosen response, three
   rejected, four completions) -- Factuality 475, Focus 495, Math 183,
   Precise IF 160, Safety 450, totalling 1763 rows.
   The sixth subset, Ties (102 rows), is excluded from this run in full. Its
   chosen and rejected counts vary per row by design (observed shapes include
   1-vs-37, 26-vs-12, 16-vs-15) because it exists to test detection of a pool
   of near-equal candidates, and there is no honest reduction of "detect an
   N-way near-tie" to "pick the better of two responses".
   Sampling population: N = 1763.

2. SAMPLE
   n = 210, drawn from N = 1763 by proportional stratified sampling with
   largest-remainder apportionment, stratified on the benchmark's own subset
   field.
   Allocation, fixed by that rule and reproducible on any machine:
     Factuality   475 -> 56
     Focus        495 -> 59
     Math         183 -> 22
     Precise IF   160 -> 19
     Safety       450 -> 54
     Total       1763 -> 210
   Seed, recorded before any sample was drawn:
     rewardbench2-measurement-1-proof-run-v1
   The same (population, n, seed) triple always yields the same sample. The
   seeded pseudo-random generator is specified bit for bit in the published
   verification bundle.
   On the size itself: n = 210 is fixed here as a design constant, chosen
   before a power analysis existed for it. Section 2B below now supplies one:
   a detectability figure, not a floor that section 5's falsification test is
   stated against -- that test remains a plain chance-exclusion test.

2B. POWER AND MINIMUM DETECTABLE EFFECT
   Added 2026-08-29, before any judgment exists. No closed-form power formula
   applies to a percentile-bootstrap interval on the coefficients in section 4,
   so power here comes from simulation: many synthetic judgment sets are
   generated with a known true population effect, each run through the same
   coefficient-and-interval procedure section 4 describes, and power is the
   share of those runs whose resulting 95% interval clears zero. (One
   precision note on the simulation's own inputs: the balanced label split
   this simulation assumes is exact across the full population by
   construction, but only approximate within any one drawn 210-item sample.)
   At n = 210, that simulation finds this design has 80% power to detect, as
   beyond chance:
     a Cohen's kappa (4.1) of 0.19 or higher, and
     a Krippendorff's alpha (4.2) of 0.15 or higher.
   Read this precisely: it is a detectability statement, not a pass/fail
   floor. It says the smallest true effect this design can tell apart from
   chance 80% of the time -- it does not say the interval clears that same
   value 80% of the time when the true effect equals it. That second,
   harder question is answered separately below.
   Power to detect a given true value as beyond chance, at n = 210:
     kappa 0.10 -> power 0.29     alpha 0.10 -> power 0.61
     kappa 0.20 -> power 0.85     alpha 0.20 -> power 0.995
     kappa 0.30 -> power 0.99     alpha 0.30 -> power 1.00
     kappa 0.40 -> power 1.00     alpha 0.40 -> power 1.00
     kappa 0.50 -> power 1.00     alpha 0.50 -> power 1.00
     kappa 0.60 -> power 1.00     alpha 0.60 -> power 1.00
   Both already clear 80% power at 0.20, consistent with the two values above.
   The same simulation found the sample size that would reach 80% power to
   detect 0.30 as beyond chance on the first coefficient (about 95 items) and
   0.20 on the second (about 70 items) -- detecting a SMALLER effect costs
   MORE items, not fewer, which is why the planned 210 was kept rather than
   reduced. Growing the sample beyond 210 would lower both detectable values
   further; that tradeoff against additional judge-hours was not taken up for
   this run, and is disclosed here rather than decided silently.
   A second, separate, REPORTED-not-tested figure: this run will also report,
   never as a pass/fail gate, whether the realized interval's lower bound
   clears these same two values outright (kappa above 0.19, alpha above
   0.15) -- a substantially harder bar. At a true effect exactly equal to
   that value, the interval is centered ON it, not above it, so it clears
   only a small fraction of the time. Probability the interval clears 0.19,
   by true kappa:
     0.19 -> 0.045     0.25 -> 0.122     0.30 -> 0.378
     0.35 -> 0.723     0.40 -> 0.892
   Probability the interval clears 0.15, by true alpha:
     0.15 -> 0.025     0.20 -> 0.203     0.25 -> 0.568
     0.29 -> 0.873     0.30 -> 0.905
   This reaches 80% power only around kappa 0.37 / alpha 0.28 -- nearly double
   the detectability values above. No acceptance threshold is set on this
   reported figure either.
   This section describes only what the design can detect. It is not, and
   does not become, a result -- no judgment exists yet.

3. JUDGES AND REPLICATION DESIGN
   Three validators judge every one of the 210 items, independently.
   Each is blind to the benchmark's own published choice and blind to the other
   two validators' judgments. Blindness is enforced by the review software
   rather than by reviewer discipline: the reviewer-facing view of an item
   never carries the original label, and each reviewer sees only their own
   assignment.
   Because the label has two options and every item carries three judgments, a
   majority always exists (3-0 or 2-1) and no arbiter step can arise.
   Judge bar, decided 2026-08-23, before any judgment exists: three internal
   senior annotation and QA leads, each gated on (1) a written qualification
   statement on file stating role and years of evaluation or QA experience, and
   (2) a 20-item gold-reference qualification exam taken before any of their
   judgments counts. That exam includes about five mathematics items, and at
   least one of the three judges must hold a quantitative background, because
   the sample carries 22 Math items seen by all three.
   Permitted description of this panel: internal senior validators, gated on a
   documented bar. Not permitted: "independent external experts".

4. STATISTICS THAT WILL BE REPORTED
   4.1 Consensus against published label. The benchmark's own published choice
       is paired against our final label, which is the majority of the three
       independent judgments. Two entities, complete data, nominal labels, so
       the reported coefficient is COHEN'S KAPPA. Gwet's AC1 is reported
       alongside it over the same pairs as a supplementary cross-check, never
       in place of it.
   4.2 Validator against validator. A second, separate coefficient is computed
       over the three validators' own raw labels, with no external reference at
       all. On this design -- three distinct validators per item -- the
       expected method is KRIPPENDORFF'S ALPHA. Where an item carries fewer
       than two judgments the value is reported as null, never fabricated.
   4.3 These two are different claims and will never be collapsed into one
       sentence. 4.1 says how much our consensus resembles one external answer
       key. 4.2 says how reliable our validators are with each other. A high
       4.1 with a low or absent 4.2 is a real possibility and would be
       reported as exactly that.
   4.4 Percent agreement is not the headline statistic for this run, because
       every row carries two comparable entities and a chance-corrected
       coefficient is therefore admissible. The observed-agreement term inside
       the kappa computation stays readable in the published record.
   4.5 Reference-item ("gold") accuracy does not apply to this run, by design.
       Every item is a standard item and no known-answer item is mixed in. The
       benchmark's published labels are the object under audit, so scoring our
       validators against those same labels as though they were ground truth
       would be circular.
   4.6 Defect rate is reported with a Wilson score interval if, and only if,
       validators log defects during review. No defect taxonomy is required
       for this run, so that statistic is opportunistic and is not a designed
       part of the measurement.
   4.7 Intervals. Every point estimate ships with a 95% interval: a seeded
       percentile bootstrap at 2000 resamples for the coefficients in 4.1 and
       4.2, a Wilson score interval for proportions. The seed and the
       resampling procedure travel with the published record, so an interval
       can be reproduced bit for bit in another language.

5. HYPOTHESES, AS FALSIFIABLE STATEMENTS
   H1. Our three-validator consensus agrees with the benchmark's published
       labels beyond chance. Falsified if the 95% interval on the Cohen's
       kappa of 4.1 includes zero.
   H2. Our three validators agree with one another beyond chance. Falsified if
       the 95% interval on the coefficient of 4.2 includes zero, or if that
       coefficient cannot be computed at all.
   H3. Every statistic published from this run is recomputable from the
       published record by a third party, to within 1e-9, with no access to
       our systems. Falsified if the published verification script reports
       anything other than success against the published bundle.
   No directional prediction is made about the magnitude of H1 or H2 beyond
   the chance point, and no acceptance threshold is set -- on this test or on
   either figure reported in section 2B. Choosing a threshold after seeing the
   data would defeat the purpose of registering this in advance.
   Section 2B, read alongside these: this design has 80% power to detect an
   effect of 0.19 (H1) or 0.15 (H2) as beyond chance -- a detectability
   statement, not a version of this falsification test. A second, separate,
   reported-not-tested figure in section 2B additionally shows how much
   harder it is for the interval to clear those same two values outright.
   Neither figure is a pass/fail gate for H1 or H2.

6. STOPPING AND EXCLUSION RULES
   6.1 The sample is drawn once, from the recorded seed, before any judgment.
       Items are never added, replaced or re-drawn to move a number.
   6.2 The Ties subset (102 of 1865 rows, 5.5% of the benchmark) is excluded
       in full, for the reason given in section 1. No other exclusion is
       permitted after the fact.
   6.3 The run stops when all 210 items carry three independent judgments.
       There is no interim look at the statistics and no early stopping:
       the metrics are computed once, after completion.
   6.4 An item carrying fewer than two judgments contributes nothing to 4.2
       and is reported as such rather than imputed.
   6.5 Before any number from this run is published, the published
       verification script is run against the real bundle and must report
       success. If it does not, nothing is published until it does.

7. WHAT IS PUBLISHED, REGARDLESS OF THE RESULT
   7.1 The result, whatever it is: including a low, zero or negative
       coefficient, and including the case where H1 or H2 is falsified.
   7.2 The verification bundle: the sampling record with the exact ordered
       population item-id list, every final judgment as a pseudonymised label
       pair, the computed metrics with their seeds and intervals, and prose
       specifying every algorithm used, including the seeded-generator spec.
   7.3 Per-item disagreement detail, not only the summary coefficient.
   7.4 This registration, unchanged, next to the result.

8. WHAT THIS RUN WILL NOT SUPPORT CLAIMING
   8.1 That we audited RewardBench 2. Ties is excluded and 210 items is a
       fraction of even the 1763-row reduced population. Always state n and N.
   8.2 That we replicated the benchmark's best-of-4 scoring protocol. Each
       item here compares the chosen response against one of its three
       rejected candidates, not against all three at once. A response that
       loses to one rejected candidate might still have beaten another, and
       this run cannot speak to that.
   8.3 That the benchmark is wrong. The claim is narrower: here is a human
       agreement figure for those labels, and here is exactly how to recompute
       it.
   8.4 Anything about validator expertise beyond the documented bar in
       section 3.
   8.5 That the published record proves the judgments are authentic. It proves
       the arithmetic follows from the record. Separating the team producing
       the work from the team measuring it is the control that addresses the
       rest.
   8.6 That section 2B's power analysis, or its reported floor-clearance
       figure, is itself a result, or an acceptance threshold for H1 or H2.
       Both describe what this design can detect or how hard a harder bar
       would be to clear, before any judgment exists -- not a measurement of
       our validators or our consensus, and not a pass/fail gate.

CHANGELOG
   2026-08-19  Initial registration, written before any judgment exists.
   2026-08-22  Method-selection detail recorded for ordinal label types. This
               run is unaffected: its labels are nominal and its coefficient
               is Cohen's kappa, as in 4.1.
   2026-08-22  Validator-against-validator agreement (4.2) added, closing a gap
               this document had originally flagged as not computed, together
               with the standing note that 4.1 and 4.2 are different claims.
   2026-08-23  Judge bar resolved as recorded in section 3, before any judgment
               exists.
   2026-08-29  Section 2B added: a simulation-based power analysis of this
               exact design, run before any judgment exists, because no power
               calculation had been performed for n = 210 until now.
   2026-08-29  Correction, same day: an earlier version of this entry restated
               H1/H2's falsification test using section 2B's 80%-power values
               (kappa 0.19, alpha 0.15) as a floor. That was a statistical
               error, caught before publication: those values are the effect
               size at which the interval clears ZERO 80% of the time: power
               for the interval to clear THAT SAME VALUE is only about 4-5%
               at that effect size, not 80%. Section 5 reverts to the
               original chance-exclusion falsification test. Section 2B now
               states its values strictly as a detectability figure, and adds
               a second, separate, reported-not-tested figure for the harder
               floor-clearing question, with its own true crossing point
               (kappa ~0.37, alpha ~0.28). Also corrected: the drawn n = 210
               sample's own label balance is approximately, not exactly, half
               and half; only the full population is exactly balanced.
-----END PRE-REGISTRATION METHODOLOGY-----

How this fits with the rest of the evidence

Nothing here is new machinery. The measurement described above is the same one set out under how we prove it and shown at work in the sample verification bundle: the party computing the statistics is not the party producing the work, every figure carries a named method and a 95% interval, and a standard-library Python script recomputes all of it on your own machine with no dependency on us. Pre-registration adds the one thing a bundle cannot supply on its own — evidence that the analysis was chosen before the data, rather than after it.

FAQ

• What does it mean to pre-register a benchmark run?

  It means fixing the design in public before any result can influence it: the dataset revision, the sample and its seed, the hypotheses and how each one could be falsified, the statistics and the interval method, the stopping and exclusion rules, and the commitment to publish whatever comes out. Because the text is dated and hashed, a reader can check afterwards that the analysis they were shown is the analysis that was planned, rather than the one that flattered the number.

• What happens if the result is bad?

  It gets published. The registration commits us to reporting the coefficient whatever it is, including a low, zero or negative value, and including the case where one of the hypotheses is falsified, alongside the per-item disagreement detail and the record needed to recompute every figure. A registration that only binds you when the answer is flattering is not a registration.

• Why 210 items, and how was that number chosen?

  n = 210 was fixed as a design constant before the sample was drawn, then allocated proportionally across the five subsets that reduce honestly to a pairwise choice. A simulation-based power analysis of this exact design was run before any judgment exists: it finds 210 items give 80% power to detect a Cohen's kappa of 0.19 or a Krippendorff's alpha of 0.15 as beyond chance, and a separate, harder, and separately reported question -- whether the interval clears those same values outright, not just chance -- only reaches 80% power around kappa 0.37 or alpha 0.28; that second figure is not a falsification test. The registered text below reproduces the full analysis. What the fixed number still buys, independent of either figure, is that the sample cannot be grown or trimmed afterwards to move a coefficient.