HLTH 2026Meet DeepLabel at HLTH, Las Vegas, 15–18 Nov, kiosk 127-79Meet us in Las Vegas, 15–18 Nov
deeplabel

Use case · builds on Clinical AI Evaluation

Medical LLM and chatbot evaluation by clinicians.

DeepLabel has specialist clinicians grade your medical LLM or chatbot on the questions real users actually ask, scoring each answer against a rubric for accuracy, safety, completeness and tone, with every issue graded by severity. PhD data scientists measure how often reviewers agree and report the error patterns to fix. This is structured grading of normal use; it pairs with medical red-teaming, which goes looking for the rarer, adversarial failures.

Who it's for
Teams building medical LLMs, health chatbots and AI assistants for patients or clinicians.
The problem
Benchmarks and internal checks rarely show how your product answers the everyday questions your users ask.
Clinical and data science expertise
Clinicians matched to your clinical area and users, with PhD data scientists who design the rubric and analyse the results.
How quality is assessed
Severity grading against an agreed rubric, with blinded double grading on a sample, measured agreement and adjudication where the design includes them.

NDA and data processing agreement on every project. How we handle your data.

What we deliver

What do clinicians grade?

Clinical accuracy

Whether the answer is correct for the question asked and the information given.

Safety

Unsafe advice, missed red flags, and whether the product escalates or declines when it should.

Completeness

Important omissions: the caveat, contraindication or safety-net that should have been there.

Unsupported claims

Confident statements the input or the evidence does not support.

Appropriateness and tone

Whether the answer suits the user, the setting and the reading level, without overstepping.

Handling uncertainty

Whether the product is honest about what it does not know, and says when to see someone.

What you receive · depending on scope
  • Agreed grading rubric
  • Representative test set
  • Graded answers with severity
  • Findings by error type and severity
  • Worked examples with the clinician's reason
  • Reviewer agreement analysis
  • A prioritised report

Looking for adversarial testing rather than structured grading? See medical red-teaming.

Our evaluations provide clinical insight to support product development. They do not certify clinical safety, confer regulatory approval or replace formal validation requirements.

How it works

How does an evaluation run?

  1. 01

    Agree intended use and rubric

    We set who the product is for and what a good answer looks like, then build a rubric with severity levels for the errors that matter.

  2. 02

    Build a representative test set

    We weight the set towards the everyday questions your users actually ask, then deliberately add the rarer, higher-risk cases that decide safety, and whole conversations where relevant.

  3. 03

    Clinicians grade, blinded

    Matched clinicians score each answer against the rubric, blinded to the source where it matters, and grade every issue by severity.

  4. 04

    Measure, adjudicate, report

    Where the design includes double grading of a sample, we measure agreement and adjudicate disagreements; in every case we report error patterns and worked examples you can act on.

A worked example

What does grading look like?

Illustrative example

A model answer is scored against the agreed rubric. The clinician marks what is wrong or missing and grades each issue by severity.

Where the design includes it, a second clinician reviews a sample independently, and PhD data scientists measure how often they agree before the findings are written up.

AI draft · Clinical noteIllustrative example

Start amoxicillin 500 mg three times daily for 5 days.

No allergy check required.

Review if symptoms persist. Missing: red-flag and escalation advice

  • Allergy statusSafety-critical
  • Safety-nettingModerate
  • Dose and durationAccurate
Reviewer B · independent read
  • Allergy status flagged
  • Safety-netting flagged
  • Dose and duration accurate
Reviewer agreementκ [example]
Findings reportIllustrative example
  • Safety-criticalAllergy status. Penicillin allergy must be checked before prescribing.
  • ModerateSafety-netting. Omission: no red-flag symptoms or escalation advice.
  • AccurateDose and duration. Accurate for the stated scenario.
  • AgreementReviewer agreement. κ [example], measured on the double-read sample.
01 · AI draftIllustrative

Start amoxicillin 500 mg three times daily for 5 days.

No allergy check required.

Review if symptoms persist.

02 · Clinician flags

Start amoxicillin 500 mg three times daily for 5 days.

No allergy check required.

Review if symptoms persist. red-flag and escalation advice

03 · Second read
  • ✓ Allergy status flagged
  • ✓ Safety-netting flagged
  • ✓ Dose and duration accurate
Reviewer agreementκ [example]
04 · Findings report
  • Safety-criticalAllergy status. Penicillin allergy must be checked before prescribing.
  • ModerateSafety-netting. Omission: no red-flag symptoms or escalation advice.
  • AccurateDose and duration. Accurate for the stated scenario.

Swipe · 4 frames

Common questions

Medical LLM evaluation, answered.

Anything else, just ask on a scoping call.

How is this different from medical red-teaming?

This is structured grading of normal use: clinicians score your product's answers to the questions real users ask. Red-teaming is adversarial: clinicians deliberately try to make the product fail on edge cases and risky requests. Many teams do both, and we can combine them in one project.

What do clinicians grade against?

A rubric we build with you for your intended use, covering accuracy, safety, completeness, unsupported claims, appropriateness and how the product handles uncertainty. Each issue gets a severity. As an example of the kind of scale we agree, issues might be graded as no issue, minor, moderate, severe or potentially harmful, so you can tell a wording nit from a safety-critical error. The exact scale is set for your product, not fixed.

Can you evaluate multi-turn conversations?

Yes. We can grade whole conversations, not just single answers, including whether the product keeps context, stays consistent and does not become less safe as the conversation goes on.

How do you measure reviewer agreement?

Where the design includes it, typically for a sample of answers, two clinicians grade independently. PhD data scientists report agreement using Cohen's kappa for two reviewers, or Fleiss' kappa for more, and disagreements can be adjudicated against the rubric.

Who does the grading?

Specialist clinicians matched to your clinical area and users, from the BiteLabs network of 30,000+ UK and US clinicians, supported by PhD data scientists. DeepLabel is a sister company to BiteLabs. We confirm the right expertise for your project during scoping.

How do you handle our data and model access?

We offer an NDA and put a data processing agreement in place on every project. Every reviewer signs a confidentiality and data protection agreement. UK and US reviewers, access set per project. Only the clinicians assigned to your project can see your data, and where they are based is agreed with you before work begins. Clinicians can grade inside your own evaluation tools or in ours, and we can deliver the results in your format.

Let's solve your next clinical AI challenge.

Tell us what you're developing and we'll reply the same working day.

Discuss your project