The bundle behind the numbers, yours to recompute.
This is a real verification bundle produced by our own review tooling — not a mockup, and not hand-edited afterward. It has been redacted for publication and is fixture data, not a customer's results, but every statistic in it is genuine and every one of them recomputes to an exact match on your own machine.
What's in this bundle
A batch of 16 judged items — 14 drawn from a public, openly licensed benchmark of AI-assistant responses (NVIDIA's HelpSteer2) plus 2 hand-authored reference items with a known-correct answer — each scored against a five-part quality rubric (helpfulness, correctness, coherence, complexity, verbosity, each 0–4) with an overall pass line at 2. Two reviewers scored every item independently. On one item their scores disagreed enough to flip the pass/fail outcome; a third reviewer arbitrated that one item independently, without seeing who had said what.
The bundle carries every judgment, every individual reviewer's response, the one flagged defect, the rubric definition, and the full computed-statistics block — plus a plain-language description of exactly how each number was derived, so the recompute script isn't the only place the method is written down.
The numbers, with their intervals
A point estimate on its own is not a result.
-
Inter-rater agreement — Cohen's κ = 0.818
Between the two reviewers, before arbitration, over the 16 items both scored independently. 95% bootstrap interval [0.25, 1.00] — wide, because 16 items is a small sample, and the interval says so rather than hiding it.
-
Agreement with the original label — 100% (16/16)
The resolved judgment (after arbitration on the one disagreement) matched each item's original label on every item in this batch.
-
Reference-item accuracy — 100% (2/2)
Both hand-authored, known-answer items were scored correctly. 95% Wilson interval [34.2%, 100%] — honestly wide with only two reference items; a production batch carries far more of them.
-
Rubric pass rate — 78.6% (11/14)
Of the 14 non-reference items. 95% interval [52.4%, 92.4%] (Wilson) / [57.1%, 100%] (bootstrap).
-
Defect rate — 6.25% (1/16)
One item was flagged with a major-severity defect during review. 95% Wilson interval [1.1%, 28.3%].
Read more about the statistics: κ, α, AC1 and why the interval matters →
Verify it yourself
Download the bundle and the script below into the same folder, then run:
python3 verify-bundle.py sample-verification-bundle.json
verify-bundle.py is standard-library Python only — nothing to install, no network call, no dependency on us. It recomputes percent agreement, Cohen's κ, Gwet's AC1, gold-set accuracy, defect rate and rubric pass rate — plus every bootstrap and Wilson confidence interval — directly from the raw judgment and defect records the bundle carries, and reports each one against the bundle's own stated value.
Output of the command above, against the file as published here
{
"ok": true,
"checks": [
{
"name": "agreement_value",
"expected": 1,
"actual": 1.0,
"diff": 0.0,
"ok": true
},
{
"name": "intervals.agreement.point",
"expected": 1,
"actual": 1.0,
"diff": 0.0,
"ok": true
},
{
"name": "intervals.agreement.lo",
"expected": 1,
"actual": 1.0,
"diff": 0.0,
"ok": true
},
{
"name": "intervals.agreement.hi",
"expected": 1,
"actual": 1.0,
"diff": 0.0,
"ok": true
},
{
"name": "breakdowns.supplementary_agreement.gwet_ac1",
"expected": 1,
"actual": 1.0,
"diff": 0.0,
"ok": true
},
{
"name": "intervals.gwet_ac1.point",
"expected": 1,
"actual": 1.0,
"diff": 0.0,
"ok": true
},
{
"name": "intervals.gwet_ac1.lo",
"expected": 1,
"actual": 1.0,
"diff": 0.0,
"ok": true
},
{
"name": "intervals.gwet_ac1.hi",
"expected": 1,
"actual": 1.0,
"diff": 0.0,
"ok": true
},
{
"name": "gold_accuracy_pct",
"expected": 100,
"actual": 100.0,
"diff": 0.0,
"ok": true
},
{
"name": "intervals.gold_accuracy.bootstrap.point",
"expected": 100,
"actual": 100.0,
"diff": 0.0,
"ok": true
},
{
"name": "intervals.gold_accuracy.bootstrap.lo",
"expected": 100,
"actual": 100.0,
"diff": 0.0,
"ok": true
},
{
"name": "intervals.gold_accuracy.bootstrap.hi",
"expected": 100,
"actual": 100.0,
"diff": 0.0,
"ok": true
},
{
"name": "intervals.gold_accuracy.wilson.point",
"expected": 100,
"actual": 100.0,
"diff": 0.0,
"ok": true
},
{
"name": "intervals.gold_accuracy.wilson.lo",
"expected": 34.2380227506653,
"actual": 34.2380227506653,
"diff": 0.0,
"ok": true
},
{
"name": "intervals.gold_accuracy.wilson.hi",
"expected": 100,
"actual": 100.0,
"diff": 0.0,
"ok": true
},
{
"name": "intervals.defect_rate.wilson.point",
"expected": 6.25,
"actual": 6.25,
"diff": 0.0,
"ok": true
},
{
"name": "intervals.defect_rate.wilson.lo",
"expected": 1.1119344764642518,
"actual": 1.1119344764642518,
"diff": 0.0,
"ok": true
},
{
"name": "intervals.defect_rate.wilson.hi",
"expected": 28.328737570298944,
"actual": 28.328737570298944,
"diff": 0.0,
"ok": true
},
{
"name": "defect_rate_pct",
"expected": 6.25,
"actual": 6.25,
"diff": 0.0,
"ok": true
},
{
"name": "rubric_pass_rate_pct",
"expected": 78.57142857142857,
"actual": 78.57142857142857,
"diff": 0.0,
"ok": true
},
{
"name": "intervals.rubric_pass_rate.bootstrap.point",
"expected": 78.57142857142857,
"actual": 78.57142857142857,
"diff": 0.0,
"ok": true
},
{
"name": "intervals.rubric_pass_rate.bootstrap.lo",
"expected": 57.14285714285714,
"actual": 57.14285714285714,
"diff": 0.0,
"ok": true
},
{
"name": "intervals.rubric_pass_rate.bootstrap.hi",
"expected": 100,
"actual": 100.0,
"diff": 0.0,
"ok": true
},
{
"name": "intervals.rubric_pass_rate.wilson.point",
"expected": 78.57142857142857,
"actual": 78.57142857142857,
"diff": 0.0,
"ok": true
},
{
"name": "intervals.rubric_pass_rate.wilson.lo",
"expected": 52.410769413399706,
"actual": 52.410769413399706,
"diff": 0.0,
"ok": true
},
{
"name": "intervals.rubric_pass_rate.wilson.hi",
"expected": 92.42861328730683,
"actual": 92.42861328730683,
"diff": 0.0,
"ok": true
},
{
"name": "validator_agreement.value",
"expected": 0.8181818181818182,
"actual": 0.8181818181818182,
"diff": 0.0,
"ok": true
},
{
"name": "validator_agreement.method",
"expected": "COHENS_KAPPA",
"actual": "COHENS_KAPPA",
"diff": null,
"ok": true
},
{
"name": "validator_agreement.n_items",
"expected": 16,
"actual": 16,
"diff": null,
"ok": true
},
{
"name": "validator_agreement.n_validators",
"expected": 2,
"actual": 2,
"diff": null,
"ok": true
},
{
"name": "validator_agreement.interval.point",
"expected": 0.8181818181818182,
"actual": 0.8181818181818182,
"diff": 0.0,
"ok": true
},
{
"name": "validator_agreement.interval.lo",
"expected": 0.2544186046511653,
"actual": 0.2544186046511653,
"diff": 0.0,
"ok": true
},
{
"name": "validator_agreement.interval.hi",
"expected": 1,
"actual": 1.0,
"diff": 0.0,
"ok": true
}
]
}
33 of 33 checks, every diff 0.0. This is the literal output of the command above, run against the exact files linked on this page — nothing trimmed, nothing re-ordered. Download this output as a text file →
A full evidence pack from the platform additionally carries an ES256 signature over a public-safe manifest, checked with a second script (verify-pack.py) against a published public key. That signing step runs against a key held in the platform's own key-management service and isn't reproduced in this standalone sample — what's here is the statistics side: the raw judgment records and the script that recomputes every number from them.
What was redacted, and what wasn't touched
Before publishing, every internal database identifier was replaced with a plain sequential label — item ids, the batch id, the metrics-run id, and each reviewer's internal reference — remapped consistently everywhere they appear so the same item and the same reviewer still line up correctly across the whole bundle. The internal audit trail (who on our side touched the batch and when) was removed entirely; it documents provenance for our own records, but the recompute script never reads it and no statistic depends on it.
Nothing else changed. No label, no score, no defect flag, and no computed statistic was edited, rounded, or removed. That's exactly why the verify output above was captured after redaction, against the same file published on this page — the numbers you can check are the numbers that actually got redacted, not a cleaner set computed before the fact.
The caveat
This is fixture data assembled to demonstrate the method — a small batch, a public benchmark, two reviewers and an arbiter we set up ourselves. It is not a customer's results, and the batch size here (16 items) is far smaller than a production run. What carries over exactly as shown is the methodology: the same statistics, the same named methods, the same confidence intervals, and the same recompute script a buyer would run against a real engagement's numbers.