When you're buying preference data for RLHF, the first metric anyone quotes you is inter-rater agreement — usually presented as a percentage. "Our annotators agree 87% of the time." That sounds reassuring. But it hides more than it reveals.
What agreement actually tells you
Inter-rater agreement (IRA) measures how often two or more independent annotators give the same label, ranking, or preference to the same data point. High agreement means your annotators are consistent with each other. It does not mean they're correct.
The consistency vs. accuracy gap
Five annotators can consistently agree that Response A is better than Response B — and all five can be wrong, because they share the same blind spot. This happens frequently in domains where the annotators lack subject-matter expertise.
High agreement without domain expertise is just consistent noise. It trains your model to be confidently wrong.
What actually matters for RLHF quality
Beyond raw agreement, you should be tracking:
- Domain calibration: Are annotators trained on your specific evaluation rubric, or are they applying generic "better/worse" instincts?
- Edge-case divergence: Where do annotators disagree? Those disagreements are where your model's failure modes live.
- Annotation velocity vs. quality: Raters who are too fast are skimming. Consistent speed without pause-and-think patterns signals shallow evaluation.
- Rubric adherence: Can you trace each preference decision back to a specific criterion in your rubric?
How to audit your preference data
Before feeding preference data into your reward model, run these checks:
- Gold standard injection: Seed your annotation queue with items where the correct answer is known. Measure accuracy, not just agreement.
- Disagreement analysis: Pull the bottom 10% by agreement and review them manually. These edge cases often reveal rubric ambiguity.
- Annotator stratification: Group by annotator and look for outliers — both consistently fast annotators and those whose preferences systematically diverge from the group.
- Cross-domain validation: If your raters handle both coding and creative writing tasks, check if their performance is domain-consistent.
What we do differently
At Mavy EduSolutions, we match annotators to tasks by domain expertise first, throughput capacity second. Every batch ships with:
- Agreement scores per rubric criterion (not just overall)
- Disagreement analysis with edge-case samples
- Gold standard accuracy (not just inter-rater consistency)
- Annotator-level metadata so you can stratify internally
Need RLHF data you can trust? Start a project — tell us your model, your rubric, and your domain, and we'll match annotators with the right expertise.