Inter-annotator agreement: what κ, α and AC1 actually measure

What agreement measures — and what it does not

Inter-annotator agreement (also called inter-rater reliability, or IAA) measures reproducibility: whether a judgment is a property of the item or a property of the person who happened to judge it. If two qualified experts working from the same guidelines reach the same verdict, the verdict is likely about the item. If they don't, the label is carrying the rater's idiosyncrasies into your training data — and a model fitted to it will learn those instead.

Agreement does not measure correctness. Two raters can agree perfectly and both be wrong, which is exactly what happens when guidelines are clear but mistaken, or when both raters share the same blind spot. Correctness is a separate measurement, made against reference items with a known answer. Any quality report that offers you only one of the two is answering half the question. More on that below.

Agreement is also not a substitute for adjudication. A disagreement is a signal that the item, the guidelines or one of the raters needs attention; the coefficient tells you how often that happens, not what to do about it. Whether the number you are shown is computed before or after disagreements were resolved changes its meaning completely, and it is a fair thing to ask.

Why raw percent agreement misleads

Percent agreement is the fraction of items on which every rating is identical. It is the honest starting point, it is the only measure everyone reads the same way, and it should always be reported. Its flaw is that it credits agreement that chance alone would have produced.

Consider a binary pass/fail review where the true pass rate is high. Two raters who never look at the items at all, and simply guess in proportion to how often each label occurs, will still agree most of the time. On a task where nine items in ten pass, blind guessing lands both raters on the same label roughly 82% of the time. A reported 85% agreement on such a task means almost nothing.

Every chance-corrected coefficient in this article is an answer to that problem, and they all have the same shape:

coefficient = (p_o - p_e) / (1 - p_e)

  p_o  observed agreement — the fraction actually agreed
  p_e  chance agreement   — the fraction two raters would
                            have agreed on without reading

The numerator is how much agreement you got beyond chance; the denominator is how much agreement beyond chance was available to get. A value of 1 is perfect agreement, 0 is exactly chance, and negative values mean the raters agreed less than chance would predict. The entire argument between the coefficients — κ, α, AC1 — is an argument about how to estimate p_e. Nothing else about them differs as much.

The measures, and where each applies

Percent agreement

Fraction of items where every rating on the item is identical. No chance correction, no assumptions, nothing to argue about. Report it always, alongside a chance-corrected coefficient — never instead of one. It is also the only honest fallback when there is a single rater per item, because with one rating there is nothing to chance-correct.

Cohen's κ (kappa)

Two raters, nominal categories, every item rated by both. Chance agreement is estimated from the product of the two raters' own marginal distributions: if rater A calls 30% of items FAIL and rater B calls 25% of items FAIL, they would land on FAIL together 7.5% of the time by chance, and the same for every other category. That assumption — each rater has a fixed personal propensity for each label, and they guess independently — is what makes κ both intuitive and fragile. It is the right default for the common case of exactly two reviewers with complete coverage.

Fleiss' κ

Generalizes κ to a fixed number of raters per item, where the raters need not be the same people on each item, estimating chance from one pooled category distribution rather than per-rater marginals. We do not implement Fleiss' κ, so there is no worked example of it on this page. For more than two raters we use Krippendorff's α instead, which covers the same case, does not require an equal number of ratings per item, and tolerates missing ratings — all of which happen in real review work.

Krippendorff's α (alpha)

Any number of raters, missing data allowed, and a choice of difference function for the label space. Rather than pairing raters, α pools every pairable rating into a coincidence matrix and compares observed disagreement to expected disagreement: α = 1 − D_o/D_e. Items rated by fewer than two raters contribute nothing and are excluded — a single rating cannot agree or disagree with anything.

Its ordinal form is the reason to reach for it on scored work. With a nominal difference function, a 4-versus-3 disagreement on a five-point scale counts exactly as much as 4-versus-1. With the ordinal difference function, near-misses cost less than far-misses, which is almost always what you actually mean when you ask whether two experts agreed about quality. Example 2 below shows the same ratings scored both ways, and the gap is large.

Gwet's AC1 and AC2

Two raters, and the answer to the prevalence problem. AC1 keeps the same observed agreement as κ but estimates chance from pooled prevalence across both raters, spread over the categories in use:

p_e(AC1) = (1 / (q - 1)) * sum_k [ pi_k * (1 - pi_k) ]

  q     number of categories in use
  pi_k  pooled prevalence of category k, across both raters

Because that term is largest when the categories are evenly used and smallest when one category dominates, AC1 does not collapse under high prevalence the way κ does. AC2 is its weighted, ordinal generalization: the same pooled-prevalence trick, applied with a distance weighting over the scale's own values so near-misses again cost less than far-misses. Example 3 is the case they exist for.

Weighted κ

Two raters, ordinal scale, complete pairing. Same construction as Cohen's κ, but each disagreement is weighted by how far apart the two ratings are in rank — linearly, or quadratically so that a two-step disagreement costs four times a one-step one. It is the ordinal counterpart to κ in the same way ordinal α is the ordinal counterpart to nominal α.

Confidence intervals: why the interval outranks the point estimate

Every one of these coefficients is computed from a sample of items. Judge 12 items and you get a number; judge a different 12 from the same population and you get a different number. The point estimate alone tells you nothing about how far apart those two numbers would have been. The interval does, and at the batch sizes that reviews actually run at, that spread is usually the whole story.

Wilson score intervals, for proportions

Where the statistic is literally "successes out of n" — reference-item accuracy, defect rate, pass rate — there is a closed form worth using. The Wilson score interval stays inside [0, 1] and behaves sensibly at small n and at extreme proportions, where the textbook normal approximation produces nonsense such as a negative lower bound or an upper bound above 100%:

center = (p + z^2/2n) / (1 + z^2/n)
margin = z * sqrt( p(1-p)/n + z^2/4n^2 ) / (1 + z^2/n)

  p  observed proportion    n  number of observations
  z  two-sided normal critical value (1.959964 at 95%)

Seeded bootstrap, for the coefficients

κ, α, AC1, AC2 and weighted κ have no analytic interval worth trusting at small n. The percentile bootstrap makes no distributional assumption at all: resample the items with replacement, recompute the coefficient on each resample, and read off the 2.5th and 97.5th percentiles of the resulting distribution. The examples on this page use 2,000 resamples at the 95% level.

The bootstrap's random draws must be seeded, and the seed must be published with the number, or the interval is not reproducible and therefore not checkable. Two honest degenerate cases are worth knowing: a bootstrap on a single unit is refused outright rather than reported as a zero-width interval, and a resample that happens to contain no disagreement at all leaves the coefficient undefined — that case is treated as trivial total agreement, which is why small-batch upper bounds so often sit exactly at 1.00.

Worked examples, computed by the code behind our evidence bundles

The tables below are small and obviously synthetic — they exist to make the behaviour visible, not to describe any engagement. What is not synthetic is the arithmetic: each block is the literal, unedited output of running these tables through the same statistics functions that compute the numbers in the sample verification bundle, which is itself recomputable with a published standard-library Python script. Bootstrap intervals throughout: percentile method, 2,000 resamples, 95% level, seed iaa-explainer-2026-08-28. The full rating tables are printed so you can recompute every figure yourself.

Example 1 — two raters, 12 items, nominal labels

The ordinary case: two reviewers, three possible verdicts, complete coverage. All four measures are shown so you can see how little they disagree when prevalence is balanced — and how wide all four intervals are at n = 12.

=== EXAMPLE 1 — 2 raters x 12 items, nominal (PASS / FAIL / REVISE) ===
rater A: PASS PASS FAIL PASS REVISE FAIL PASS REVISE PASS FAIL PASS REVISE
rater B: PASS PASS FAIL REVISE REVISE FAIL PASS PASS PASS FAIL PASS REVISE
method selected (2 raters, no missing): COHENS_KAPPA
percent agreement      0.8333 | point 0.8333  95% CI [0.5833, 1.0000]  (2000 resamples, level 0.95)
Cohen's kappa          0.7333 | point 0.7333  95% CI [0.2590, 1.0000]  (2000 resamples, level 0.95)
Krippendorff's alpha   0.7444 | point 0.7444  95% CI [0.3353, 1.0000]  (2000 resamples, level 0.95)
Gwet's AC1             0.7576 | point 0.7576  95% CI [0.3999, 1.0000]  (2000 resamples, level 0.95)

Read it as: the raters agreed on 10 of 12 items, chance correction pulls that down to roughly 0.73–0.76 whichever coefficient you pick, and the data is consistent with anything from about 0.26 to 1.00. That last clause is the finding. A κ of 0.73 sounds like a strong result and is routinely reported as one; at 12 items it does not rule out mediocre reliability.

Example 1b — the identical pattern, four times over

The same 12-item pattern repeated to 48 items. Nothing about the raters' behaviour changed, so the point estimate does not move. Only the sample size did.

=== EXAMPLE 1b — the same pattern repeated 4x (n=48), same point estimate ===
Cohen's kappa          0.7333 | point 0.7333  95% CI [0.5484, 0.8972]  (2000 resamples, level 0.95)

Same 0.7333, and now a claim you could act on. This is the single most useful thing to internalise about agreement statistics: the point estimate is what people quote, and the sample size is what determines whether the quote means anything. If a vendor's κ moves batch to batch while the intervals overlap almost entirely, nothing has actually changed.

Example 2 — three raters, 10 items, ordinal scores, one rating missing

Cohen's κ cannot be computed here: there are three raters and item 8 was never rated by the third. Krippendorff's α handles both facts natively. The same ratings are scored twice, once treating the scores as unordered categories and once as an ordinal scale.

=== EXAMPLE 2 — 3 raters x 10 items, ordinal 1-4, one missing rating ===
  item  1: [3, 3, 3]
  item  2: [2, 3, 2]
  item  3: [4, 4, 4]
  item  4: [1, 2, 1]
  item  5: [3, 2, 3]
  item  6: [4, 3, 4]
  item  7: [2, 2, 2]
  item  8: [3, 3, -]
  item  9: [1, 1, 2]
  item 10: [4, 4, 3]
method selected (3 raters, missing data): KRIPPENDORFF_ALPHA
ordinal method selected                 : KRIPPENDORFF_ALPHA_ORDINAL
percent agreement (unanimous items) 0.4000
Krippendorff's alpha (nominal)      0.4510 | point 0.4510  95% CI [0.1694, 0.6957]  (2000 resamples, level 0.95)
Krippendorff's alpha (ordinal)      0.8048 | point 0.8048  95% CI [0.5133, 0.8999]  (2000 resamples, level 0.95)

Three numbers, one rating table, and they tell three different stories. Percent agreement says the raters were unanimous on only 4 items in 10 — technically true and badly misleading, because every disagreement here is one point on a four-point scale. Nominal α agrees with that pessimism, because it treats a 3-versus-2 as being exactly as wrong as a 4-versus-1. Ordinal α, which is the correct choice for a scored scale, says the raters were in substantial agreement about quality.

The lesson is not that ordinal α is the flattering choice. It is that the difference function must match the label space, and it must be fixed before the data is in. Choosing between these two numbers after seeing both is not measurement.

Example 3 — high prevalence, and the κ paradox

Twenty items, two raters, and almost everything passes — the normal state of affairs on mature work with a good expert pool. The two reviewers disagreed on 2 items out of 20, and never in the same direction.

=== EXAMPLE 3 — high prevalence, 2 raters x 20 items (the kappa paradox) ===
rater A: PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS FAIL PASS
rater B: PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS FAIL PASS PASS
percent agreement      0.9000 | point 0.9000  95% CI [0.7500, 1.0000]  (2000 resamples, level 0.95)
Cohen's kappa          -0.0526 | point -0.0526  95% CI [-0.1364, 1.0000]  (2000 resamples, level 0.95)
Krippendorff's alpha   -0.0263 | point -0.0263  95% CI [-0.1143, 1.0000]  (2000 resamples, level 0.95)
Gwet's AC1             0.8895 | point 0.8895  95% CI [0.6800, 1.0000]  (2000 resamples, level 0.95)

90% raw agreement, and a κ slightly below zero — nominally "worse than chance". Nothing is broken. Each rater called 19 of their 20 items PASS, so κ's chance term says two raters guessing in proportion to those marginals would already have agreed just over 90% of the time — fractionally more than they actually did. Krippendorff's α, which corrects for chance the same way, lands in the same place. Gwet's AC1, estimating chance from pooled prevalence instead, reports 0.8895.

This is the trap that makes agreement statistics dangerous to quote out of context. On a skewed batch, κ punishes the very quality you were hoping for, and any vendor whose incentives run the other way can exploit exactly the same effect in reverse — reporting a healthy-looking κ on an artificially balanced sample. Two defences: report the category prevalence alongside the coefficient, and report a prevalence-robust coefficient next to κ rather than in place of it.

Note also the upper bounds of 1.0000 on the κ and α intervals. At this prevalence many bootstrap resamples contain no disagreement at all, where the coefficient is undefined and treated as trivial total agreement. The interval is telling you honestly that a 20-item batch with two disagreements cannot distinguish these hypotheses.

Example 4 — ordinal scores, two raters: weighted κ and AC2

A five-point quality score, two reviewers, complete coverage. Unweighted percent agreement treats every disagreement as total; the weighted coefficients do not.

=== EXAMPLE 4 — 2 raters x 12 items, ordinal 1-5: weighted kappa and AC2 ===
rater A: 4 3 5 2 4 3 1 5 4 2 3 4
rater B: 4 4 5 1 4 3 2 5 3 2 3 5
ordinal method selected (2 raters, no missing): WEIGHTED_KAPPA_QUADRATIC
unweighted percent agreement 0.5833
weighted kappa (quadratic)   0.8598 | point 0.8598  95% CI [0.6667, 0.9474]  (2000 resamples, level 0.95)
Gwet's AC2 (quadratic)       0.9050 | point 0.9050  95% CI [0.8471, 0.9722]  (2000 resamples, level 0.95)

The raters matched exactly on 7 items in 12, and every one of the five disagreements is a single point on the scale. Reporting 58% agreement on that basis would misrepresent the work; so would reporting the weighted figure without saying that it is weighted, or without saying which weighting. Quadratic weighting is a strong claim about the scale — it asserts that a two-point miss is four times as serious as a one-point miss — and it has to be declared.

How to read "κ = 0.818, 95% CI [0.25, 1.00]" from a 16-item batch

That is a real line from the sample verification bundle published on this site: Cohen's κ of 0.818 between two reviewers over 16 items, with a 95% bootstrap interval running from 0.25 to 1.00. Here is how to read it in order.

Start with the design, not the number. Two raters, every item rated by both, nominal outcome — so Cohen's κ is the right method, and the report names it. Sixteen items is a spot check, not an estimate. Everything that follows is conditioned on that.

Then read the interval, and only then the point. The lower bound of 0.25 is the operative number: the data does not rule out reliability that would be unacceptable in production. The point estimate of 0.818 is the best single guess, but a best guess from 16 items is a weak instrument, and the upper bound sitting exactly at 1.00 is the tell — as in Example 3, many resamples of a small batch with one disagreement contain no disagreement at all.

Then decide what the number is for. An interval this wide is perfectly adequate as a smoke test — it would have caught a rater who was labelling at random — and inadequate as an acceptance gate. The correct response to a wide interval is either to accept it as a spot check and say so, or to judge more items. What is never correct is to quote 0.818 on a slide and leave the interval in a footnote.

Finally, check what else was reported. An agreement coefficient alone cannot tell you whether the raters were right, only whether they were consistent. That is why the same bundle reports reference-item accuracy separately, with its own interval.

Gold and reference items are a different measurement

A gold or reference item is one whose correct answer is known in advance, seeded into live work so it is indistinguishable from the rest. Scoring against it answers "were they right?", where agreement answers "were they consistent?". You need both, because they fail independently: perfect agreement with poor reference accuracy means the guidelines are wrong, and good reference accuracy with poor agreement means the guidelines are ambiguous or one rater is drifting.

Reference-item accuracy is a plain proportion, so the Wilson interval applies directly — which makes it a clean illustration of how much sample size, rather than the observed rate, determines what you can claim:

=== EXAMPLE 5 — reference-item accuracy: a proportion, not an agreement ===
8 of 8 correct : point 100.0000%  Wilson [67.56%, 100.00%]  bootstrap [100.00%, 100.00%]
9 of 10 correct: point 90.0000%  Wilson [59.58%, 98.21%]  bootstrap [70.00%, 100.00%]
45 of 50 correct: point 90.0000%  Wilson [78.64%, 95.65%]  bootstrap [80.00%, 98.00%]

A perfect 8-out-of-8 is compatible with a true accuracy as low as roughly 68%. The two 90% rows are the same headline figure and completely different claims. Note also the third column: the bootstrap on 8 identical successes returns a zero-width interval, because every resample is 8 successes — a case where the analytic interval is the informative one and the resampled interval is an artefact. Reporting both, and knowing which to trust when, is part of the method.

How DeepenSkill does it

Everything above is vendor-neutral and would be equally true of a lab measuring its own reviewers. Here is where we stand on it, which is only what is already published elsewhere on this site.

The team measuring the experts is separate from the team running them — the platform rejects a validation report authored by the party that delivered the work. Percent agreement, Cohen's κ, Krippendorff's α, Gwet's AC1 and AC2, weighted κ, gold-set accuracy, defect rate and rubric pass rate are each reported with the method named, the item and rater counts stated, and a 95% interval from a seeded bootstrap or a Wilson score interval for proportions.

And the numbers are meant to be checked rather than believed: the sample verification bundle carries the raw judgment records alongside a standard-library Python script that recomputes every statistic and interval in it from scratch, on your machine, with no dependency on us. Related reading: model evaluation, gold standard, provenance and our Trust Center.

FAQ

• What is a good inter-annotator agreement score?

  There is no universal threshold. A coefficient is only interpretable against the label scheme, the number of raters, how prevalent the categories are and how many items were judged — and a point estimate from a small batch is consistent with a very wide range of true values. Ask for the interval, not the number: a κ of 0.82 from 16 items and a κ of 0.82 from 500 items are not the same claim.

• What is the difference between Cohen's kappa and Krippendorff's alpha?

  Cohen's κ is defined for exactly two raters who both rated every item, and it estimates chance agreement from each rater's own marginal distribution. Krippendorff's α takes any number of raters, tolerates missing ratings, pools every pairable rating into one coincidence matrix, and can weight a disagreement by how far apart the two ratings sit on an ordinal scale. On two complete nominal raters the two usually land close together.

• Why can kappa be low when the raters agree on almost every item?

  That is the prevalence paradox. When one category dominates, the chance-agreement term κ subtracts grows toward the observed agreement, so the corrected coefficient collapses — and can go negative — even though the raters disagreed on almost nothing. Gwet's AC1 estimates chance agreement from pooled prevalence instead and stays stable in that regime. Report it alongside κ, never quietly in place of it.

• Why does an agreement score need a confidence interval?

  Because the score is computed from a sample of items, and a small sample only supports a wide claim. The interval is what tells you which values the data actually rules out. For proportions such as reference-item accuracy, the Wilson score interval is the closed-form answer; for chance-corrected coefficients there is no analytic interval worth trusting at small n, so the items are resampled with replacement from a recorded seed and the percentiles of the resampled coefficients are reported.