How to evaluate a clinical LLM: accuracy, safety and red-teaming.

A clinical large language model (LLM) can answer a textbook question well and still give advice that would be unsafe in a real consultation. General benchmarks are useful for comparing models, but they say little about how a model behaves with your prompts, your users and the decisions they make. A good clinical evaluation is designed around those things from the start.

1. Start with intended use

Every later decision depends on a clear statement of what the product is for. Write down:

• Who uses it: clinicians, patients, administrators or another system.
• The setting: primary care, emergency care, a specialty clinic, or at home.
• What it produces: answers, summaries, notes, triage suggestions or draft letters.
• What happens next: who acts on the output, and whether a clinician reviews it first.
• What is out of scope: the questions it should decline or redirect.

The same error can be trivial in one setting and serious in another. A missed detail in a draft letter that a clinician always checks is not the same as a missed red flag in advice given directly to a patient.

2. Define what good looks like

Turn intended use into a rubric that clinicians can apply consistently. Most clinical rubrics cover some version of:

• Accuracy: is the content clinically correct?
• Completeness: is anything important missing?
• Safety: could the output lead to harm, including through omission?
• Appropriateness: is the advice right for this patient and setting, including when to escalate?
• Clarity: would the intended reader understand and act on it correctly?

Grade errors by severity as well as counting them. A simple scale, such as no error, minor, moderate and potentially serious, lets you separate cosmetic problems from those that matter clinically. Define each level with examples, and test the rubric on a small pilot set before the main review. Ambiguous definitions are the most common reason reviewers disagree.

3. Build a representative test set

The test set should reflect how the product will really be used, not just the cases that are easy to write. Include:

• common presentations, in proportion to how often they occur;
• rarer but high-risk scenarios, where errors would matter most;
• ambiguous or incomplete inputs, as real users write them;
• variation across age, sex, comorbidities and other relevant patient factors;
• questions the product should decline or escalate.

Keep the test set separate from anything used to develop or tune the model, so results are not inflated. Decide the size by the decisions you need to make: estimating an overall error rate needs fewer cases than comparing two versions on a rare but serious error type.

4. Choose the right reviewers

Reviewers should match the clinical domain and level of the task. A dermatology question needs a dermatologist's judgement; a general triage tool may need emergency medicine and primary care input. Check professional registration and relevant experience.

When comparing models or versions, hide which output came from which system, and randomise the order. Reviewers who know which version is new tend to judge it differently.

5. Measure agreement between reviewers

Clinicians do not always agree, and that is useful information. Have a sample of outputs reviewed independently by two or more clinicians, then measure how often they agree. Common measures include percentage agreement, Cohen's kappa for two reviewers and Fleiss' kappa for more than two. For ordered scales, such as severity grades, a weighted kappa or an intraclass correlation is usually more informative.

Low agreement usually means one of three things: the rubric is ambiguous, the reviewers need calibration, or the cases are genuinely hard. Each calls for a different fix. Where reviewers disagree on individual outputs, a senior reviewer can adjudicate so every case ends with a single agreed judgement.

6. Add medical red-teaming

Standard test sets show typical behaviour. Red-teaming looks for the failures you would not otherwise see. Clinicians deliberately try to make the model go wrong, for example with:

• requests for unsafe advice, or for information the product should not give;
• presentations where a serious diagnosis hides behind common symptoms;
• leading or confidently wrong statements from the user;
• missing context, such as an unstated pregnancy or allergy;
• requests for references or guidelines, to check for invented sources.

Record each successful attack with the prompt, the output and why it matters, so your team can reproduce and fix it.

7. Report findings people can act on

A single accuracy score hides what your team needs to know. A useful report includes:

• error rates by type and severity, with confidence intervals;
• results broken down by scenario, specialty or user group;
• worked examples of the most important errors, with the clinician's explanation;
• agreement between reviewers, so readers can judge how reliable the findings are;
• comparisons with previous versions, where relevant;
• recommended areas for further investigation or improvement.

8. What evaluation is not

A clinical evaluation gives your team evidence about how a model performs and where it falls short. It does not certify clinical safety, confer regulatory approval or replace formal validation. Depending on what your product does and where it is used, it may fall within medical device regulation in the UK, EU or US, and evaluation evidence would then form one part of a wider body of evidence.

9. A short checklist

1. Intended use, users, setting and out-of-scope questions written down.
2. A rubric with defined criteria and severity levels, piloted on a small set.
3. A test set that reflects real use, kept separate from development data.
4. Reviewers matched to the domain, with registration checked.
5. Blinded, randomised review when comparing versions.
6. Independent double review on a sample, with agreement measured and disagreements adjudicated.
7. Red-teaming for high-risk failure modes.
8. A report with error types, severities, examples and confidence intervals.
9. A plan to re-run the evaluation when the model or its context changes.

DeepLabel combines specialist clinicians with PhD data scientists to design and run evaluations like this, from rubric to report. Tell us what you are building and we will help you scope the right approach.