Medical data labeling vs annotation: ground truth, adjudication and agreement.

Healthcare AI teams use "labelling", "labeling" and "annotation" for overlapping work. The terminology matters far less than the method: who labels the data, how their work is checked, and how you know the result is reliable enough to train or test a model on.

1. Labelling, labeling or annotation?

"Labelling" is the British spelling and "labeling" the American; they mean the same thing. In practice, teams often use the words like this:

• Labelling usually means assigning a category or value to a whole item: this X-ray shows a fracture, this note mentions sepsis, this answer is unsafe.
• Annotation often covers richer marking within an item: outlining a lesion, highlighting a phrase in a report, or linking a finding to its location.

Many projects need both. The useful question is not which word to use, but what exactly each reviewer will record, and how.

2. What ground truth means in medicine

Ground truth is the reference your model is trained on or measured against. In medicine it is rarely absolute. Common reference standards, from weakest to strongest, include:

• a single expert's judgement;
• agreement between independent experts, with disagreements adjudicated;
• consensus from a panel of experts;
• a confirmed outcome, such as a later diagnosis, test result or follow-up, where one is available.

Stronger standards cost more and are not always possible. The right choice depends on what the model is for and how its results will be used. Whatever you choose, write it down, because it defines what your model's performance figures actually mean.

3. Why one label is often not enough

Experienced clinicians looking at the same image or record do not always reach the same conclusion. This variability is well documented across specialties, and it is greatest for borderline cases, subtle findings and graded scales. A model trained on one reviewer's labels learns that reviewer's habits, including their mistakes.

Independent review shows you where that variability is, and gives you a way to resolve it.

4. Choosing a review design

Common designs, in rough order of cost and reliability:

1. Single read with quality checks. One qualified reviewer labels each item, and a sample is checked. Suits clear-cut tasks and early prototypes.
2. Independent double read with adjudication. Two reviewers label each item without seeing each other's work. Where they disagree, a senior reviewer decides. A strong default for training and test data.
3. Panel review. Three or more reviewers per item, with a majority or consensus rule. Suits small, high-stakes test sets.

You do not have to use one design throughout. Many teams single-read most of their training data and double-read the test set, where reliability matters most.

5. Measuring agreement

Agreement statistics tell you how reliable your labels are. Choose the measure to suit the data:

• Percentage agreement: simple, but can look high by chance when one label is very common.
• Cohen's kappa: agreement between two reviewers, corrected for chance.
• Fleiss' kappa: the equivalent for more than two reviewers.
• Weighted kappa or intraclass correlation: for ordered or continuous scales, where a near miss should count for more than a complete disagreement.

Report agreement alongside your results. If reviewers only moderately agree with each other, a model that matches them perfectly is not a realistic target, and very high reported model accuracy deserves a second look.

6. Writing annotation guidelines

Most disagreement comes from unclear instructions rather than poor reviewers. Good guidelines:

• define every label in clinical terms, with examples and counter-examples;
• say what to do with borderline, ungradable or poor-quality items;
• specify exactly how to mark findings, so outputs are consistent;
• are piloted on a small batch, then revised before the main work starts.

7. Questions to ask a provider

1. Who will label our data, and how are their qualifications and registration checked?
2. Which review design do you recommend for our task, and why?
3. How are disagreements resolved, and by whom?
4. Will you measure and report agreement between reviewers?
5. Will the guidelines and criteria be documented and shared with us?
6. How are data access, confidentiality and secure handling agreed?
7. Can the output be delivered in the format our pipeline needs?

DeepLabel matches your data to specialist clinicians, designs the review with you and backs it with analysis from PhD data scientists, including agreement analysis where appropriate. Tell us about your dataset and we will help you choose the right approach.