How these are evaluated

Why this page exists

A demonstration shows a tool working on an example chosen by the person who built it. That is worth very little. What is worth something is a stated method, a fixed answer key written before the tool was run, and a record of what happened when it was — including the runs that went badly and the faults found in the measurement itself.

Measurement is fallible. A harness can be wrong in the same way a model can, and it is wrong more quietly, because its output looks like a result. The worked example below is included precisely because the first fault it surfaced was in its own scoring.

Method

Corpus

Eight synthetic patients, 34 documents: GP summaries, discharge summaries, clinic letters, current drug charts, care-home MAR sheets and one epilepsy nurse telephone note. No real patient data is used anywhere in this work.

The corpus is deliberately poor quality, because real documents are. It contains NHS abbreviations rather than structured fields, brand and generic names used interchangeably, a photocopied MAR sheet creased across a dose, a transfer letter missing its second page, and a drug history taken from a patient's memory in an emergency department at 23:40.

Answer key

Forty discrepancies were planted in the documents before the pipeline was ever run, covering all eleven discrepancy types. The key is held outside the corpus directory so that the pipeline cannot read it, and the model is never given it.

Two of the planted cases have no correct answer and are keyed as unresolvable: a clinic letter and a nurse telephone note dated the same day stating different levetiracetam doses, and a creased MAR sheet whose bisoprolol dose could read 1.25mg or 2.5mg. A tool that confidently answers either is doing the wrong thing, and is scored as wrong for doing so.

A further eight entries were added to the key after inspecting a run. They were genuine discrepancies that had been in the documents all along and that I had missed when writing the key. Scoring against a key edited after seeing the output is not a blind test, so the two sets are reported separately and never combined. Only the first forty carry the word blind.

Procedure

Each document is sent to the model on its own. The model transcribes what that document says and attaches a verbatim quote to every value it reports. It is not asked to judge whether anything is a discrepancy. Deterministic code then compares the transcriptions across documents and applies the eleven rules, and a grounding check attempts to locate every quote in the document it was attributed to. Anything that cannot be located is rendered as unverified rather than asserted.

Scoring can be re-run against saved extractions without calling a model, which is how the two failure modes are told apart: change the rules with the extractions fixed and any movement is the rules engine; change the model with the rules fixed and any movement is extraction.

A blind run, and a fault in the scoring

Concord, run against the full corpus. All 34 documents were extracted by a live model call, one call per document, with no failures. The model had not seen the answer key.

The first score

The first scoring pass reported two hallucinated findings — the metric that matters most, because a tool that invents a drug is worse than no tool at all. Both were named:

P006 OMISSION mst 30mg — drug name not found in any document for P006
P007 OMISSION beclometasone with formoterol — drug name not found in any document for P007

What they actually were

Neither was a hallucination, and neither came from the model.

MST Continus 30mg tablets appears in three of the P006 documents. The name parser strips Continus as a formulation word, leaving a normalised key that appears verbatim in no document. The drug was there; the string being searched for was not.

Beclometasone with formoterol is what my own brand map expands Fostair into. It is the correct generic, and it is absent from the documents because my configuration generated it rather than the model reporting it.

The fault

The safety check was grounding on the normalised drug key rather than on the name the model had actually reported from the source document. It was comparing my normalisation against the documents and attributing the mismatch to the model. The headline safety metric was grading its own text as fabrication.

The check now tests the name as written in the supporting sources. Both scoring passes are kept as files, so the sequence can be inspected rather than taken on trust: the same extractions score two hallucinations before the fix and none after it, ninety-four seconds apart.

This is the argument for building the harness at all. Had the metric not been specific enough to name the two findings, the run would have been reported as a two-hallucination result, and the fault would have been attributed to the model.

Result after the correction

Same extractions, corrected scorer. One model, one run, untuned.

Four planted discrepancies were surfaced under a different type from the one keyed.

In the first two the model read the dose inconsistently across documents — three distinct values — and the engine declined to adjudicate between them. That is the intended behaviour, scored as a failure. The fourth is the inverse, and is a genuine error: the creased MAR sheet should not have been resolved.

Each of these four is penalised twice, once as a missed key entry and once as an unexpected finding. A label difference therefore costs the precision figure more than a straightforward miss does. Every one of these drugs still reached the pharmacist with its sources and quotes attached, which is why drug-level and type-exact recall are reported separately rather than collapsed into one number. A wrong label costs a moment. A drug that is never surfaced costs something else.

The twelve unexpected findings

Twelve findings were not in the answer key. Four are the label differences above. The remaining eight are grounded disagreements that were present in the documents and that the key had not anticipated. None is a fabrication, and each is listed individually in the run output rather than summarised.

One of them is a defect rather than a finding. Because the name parser separates MST Continus from morphine sulfate, a single drug produced one commission and two separate omissions. That was unfixed at the time of this run and is visible in the output.

What these numbers do not cover

This is a measurement, not a validation, and the distance between the two is most of the work.

One model, one run. No repeat runs, no second model, no tuning, no confidence interval. A single run tells you the tool is worth measuring properly. It does not tell you how it behaves.

Synthetic documents. The corpus was written to be difficult, but I wrote it knowing what I intended to test with it. Real records are worse in ways this corpus does not model.

The key is mine. An answer key cannot contain a class of discrepancy its author has not thought of. The eight entries added after the first run are direct evidence of that, and there is no reason to think they were the last.

Nothing has been validated clinically. What would be required is shadow-mode running against real reconciliations with pharmacist adjudication, an agreed evidence threshold set before the run, and DCB0129 assurance with a named clinical safety officer. None of that has happened, and none of it should be inferred from the numbers above.

An earlier evaluation of the same tool reported 100% recall and precision. That run replayed extractions written by hand rather than produced by a model, and it validated the comparison layer only. It is superseded by the run described here and is mentioned so that the earlier figure is not quoted as if it were this one.