How to Verify Transcription Accuracy

This is a practical guide to verifying transcription accuracy in automated output — what the error figures actually mean, why fluent text is more dangerous than garbled text, and how to check a page so that the highest-consequence errors surface first. A transcription you can trust is one you have checked against the image, whatever tool produced it.

To verify automated transcription accuracy, work from the manuscript or print itself — not from the model's fluent output — and concentrate your effort where errors do the most damage: proper names, numbers, dates, negations, and any passage your argument will rest on. A headline character error rate tells you how a model performed on someone else's corpus, not how it performed on your page. Verification closes that gap. It is the step that converts a plausible draft into a defensible source, and it is not optional.

This guide sets out a working method: what the accuracy numbers mean, why fluent output is more dangerous than garbled output, and a practical sequence for checking a page. It applies whether you are transcribing an Elizabethan will, a French notarial register, a Dutch parish book, or an early modern printed pamphlet.

First, understand what the accuracy figure is telling you

Before you can verify a transcription, you need to know what claim you are verifying against. Most automated tools report a character error rate (CER): the minimum number of character-level edits — substitutions, insertions, deletions — needed to turn the machine's output into a correct reference transcription, divided by the total number of characters in that reference. The calculation runs on Levenshtein edit distance, and it is worth being precise about what it does and does not say.

A 5% CER does not mean five words in a hundred are wrong. It means roughly five character edits per hundred characters. In practice a 5% CER corresponds to somewhere around 95–99% word accuracy and only 70–80% line accuracy — the ratios shift with script, language, and model, so treat them as orientation, not law. Word error rate (WER) is the same computation at the word level, and it is brittle in a specific way: one misplaced space can throw off every word match that follows it, so a low CER can sit alongside a handful of genuinely unusable words.

Two consequences follow for verification.

First, a headline figure does not transfer to your document. Reported CER is conditioned on the model's training distribution — script, hand, period, language, image quality, layout. This is precisely why benchmark corpora like IMPACT (over 600,000 historical images) and GT4HistOCR (313,173 line pairs) exist: no single test set represents everything. There is no portable adjustment factor that tells you how much accuracy degrades when you move from one collection to another. The number on the vendor's page is a starting hypothesis about your material, not a measurement of it.

Second, the metric normalizes by reference length, which means CER can exceed 100% when the hypothesis runs longer than the reference — a real signal that a model has invented text rather than merely misread it. That failure mode is where verification earns its keep.

Why fluent output is the more dangerous failure

There is a temptation to assume the errors worth hunting are the obvious ones: garbled characters, split words, dropped lines. Those are real, but they are the recoverable kind. You can see them, and a glance at the image fixes them.

The dangerous errors are the ones that read perfectly. General-purpose large language models — ChatGPT, Claude, Gemini and the like — are increasingly reached for as quick transcription tools, and they fail in a characteristic way. They downsample the high-resolution image they are given, then lean on their language model to produce text that fits the layout and the surrounding sense without being grounded in the marks on the page. The result is fluent, plausible, and sometimes entirely fabricated. Vision-language models have been measured hallucinating at rates around a quarter of the time, and the Crosilla, Klic and Colavizza benchmark (2025) found specialist HTR models consistently outperforming general LLMs on handwritten text across the conditions tested — a single-corpus study that, like any benchmark, still requires you to verify against your own material rather than take the ranking on faith.

The mechanism matters for how you verify. A specialist recognition model errs locally — a wrong character, a split word — and those errors sit visibly against the image. A general LLM errs coherently: a plausible name where the manuscript has a different one, a smoothed clause where the scribe wrote something awkward, an entire invented line that reads as period-appropriate prose. Coherent errors survive a casual read precisely because nothing looks wrong. This is the reason ChatGPT struggles with historical handwriting in ways that are hard to catch, and why LLM transcription errors demand a different verification posture than OCR noise. The whole argument for keeping a human in the loop rests here: the more fluent the output, the more disciplined the check must be.

The verification method, step by step

Verification is not re-transcription. It is a targeted comparison of the output against the image, ordered so that the highest-consequence errors surface first. Here is a working sequence.

Read the transcription against the image, not on its own

The most common mistake is proofreading the transcription as text — checking whether it reads sensibly. That check rewards exactly the fluent fabrications you most need to catch. Put the image and the transcription side by side and read the page, line by line, with your eye on the source. You are not asking "does this make sense?" You are asking "is this what is written?"

Triage with confidence scores, but do not trust them

Some engines expose per-character or per-word confidence values. Tesseract returns character-level confidences; Transkribus surfaces them in its validation interface. These are useful for triage — they point you at the tokens the model was least sure of — but they are model posteriors, not calibrated probabilities of correctness. No comprehensive study has established that confidence scores from the major engines are calibrated against actual correctness on difficult historical hands, so treat a high-confidence reading as a place you can look faster, never as a reading you can skip. And note that general LLM transcription typically exposes no per-token confidence at all, removing even this rough triage signal.

Verify the high-consequence tokens first

Not all errors cost the same. Concentrate scrutiny on:

Proper names — people and places, where a single substituted letter changes an identification and where the model has no linguistic prior to lean on.
Numbers and dates — sums in an account, ages in a census, regnal or calendar dates. These carry no redundancy: a wrong digit is undetectable from context.
Negations and quantifiers — "not," "never," "none," "all." A dropped negation inverts meaning while reading perfectly.
Anything your argument will cite. If a clause is going to appear in a footnote, verify it character by character.

Check that historical spelling survived

Ask specifically whether the output preserves what was on the page or has quietly modernized it. Many engines normalize in the background — long s (ſ) mapped to round s, u/v and i/j interchange resolved, "ye" silently turned to "the," abbreviations expanded without notice. This is normalization, not transcription, and it erases evidence that orthographic and dating studies depend on. A published CER can look excellent while the historical record has been modernized out of existence. If your source is printed, the same discipline applies to the long s, ligatures, and the macron or titulus standing in for an omitted nasal — marks that OCR built for modern type routinely drops or "corrects." Reading these correctly is a matter of paleographic literacy as much as tooling; our guides to manuscript abbreviations and ligatures and reading early modern paleography cover the marks you are checking for.

Verify against ground truth where the stakes justify it

For high-value material — a corpus you will publish, a reference edition — the discipline standard for establishing correctness is double-keying, also called four-eyes verification. Two transcribers independently produce the same text, and an adjudicator reconciles the differences; the adjudicated text becomes ground truth. Note that double-keying means independent transcription, not one person checking another's work. And ground-truth quality dominates quantity: roughly twenty carefully corrected pages outperform a hundred pages carrying errors in the reference. If you are measuring a model's real CER on your material — rather than trusting its headline — this is how you build the reference to measure against, and open evaluation tools such as dinglehopper will compute CER and WER transparently once you have it.

Keep interpretation separate from the base transcription

Whatever you do to a transcription after verifying it — correcting residual errors, modernizing spelling, translating, extracting names — should be layered on top of the faithful base reading, not baked into it. Once you overwrite the verbatim transcription with a corrected or modernized version, you have lost the ability to check either against the image, and you have lost the record of what the source actually says. Keep the base reading intact and reversible; this is the throughline in any serious approach to analyzing historical documents.

Where the tool you choose changes the verification burden

Verification is unavoidable, but the tool that produced the draft determines how much of it you do and which errors you are hunting. If a model returns fluent fabrications, verification means re-reading the whole page against the image, because you cannot trust the parts that look right. If a model errs locally and preserves what is on the page, verification narrows to a targeted check of the high-consequence tokens.

This is the ground on which Leo is built. Its transcription engine, ATR-1, is a specialist model for Latin-script material — English, French, German, Dutch, Spanish, Italian, Latin and the other languages written in that alphabet, across manuscript hands and historical print alike — and it is trained to transcribe what is on the page rather than smooth it into modern, plausible prose. It preserves strikethroughs, marginal additions, editorial expansions and archaic orthography rather than silently normalizing them, which is exactly what the spelling check above is watching for. On a randomized 97-image sample of early-modern English manuscripts from the Folger Shakespeare Library, at ATR-1's release, Leo recorded roughly a 5% character error rate — 61% fewer errors than the next-best model tested, with Transkribus's Text Titan I at about 13%, Claude Opus at 23.3%, Gemini 2.5 Pro at 24.8%, and GPT-4.1 at 56.7% (full results here). That figure describes one corpus at one moment. It does not describe your document, and the verification method above still applies in full.

What a lower, character-local error rate changes is the shape of the work: the base transcription in Leo stays faithful to the page, and every transformation — correction, modernization, translation, entity extraction — writes to a separate tab, leaving the verbatim reading untouched and available to check against the image. That is verification made structurally easier, not verification made unnecessary. The image sits beside the transcription so the side-by-side read is the default view, not something you have to assemble. And because corrections users make feed back into training, the model's accuracy compounds across releases — though nothing about that removes the human check at the point of use.

The habit that outlasts any tool

No model, Leo included, absolves you of reading the page. The accuracy figures are useful for choosing where to start and misleading if you treat them as a warrant. What makes a transcription trustworthy is not the number on the vendor's site but the discipline you bring to the check: reading against the image rather than against the model's fluency, spending your attention where errors are undetectable from context, and keeping the verbatim reading intact so that anyone — including you, later — can go back to the source and see for themselves. That habit is the oldest thing in paleography, and the part no automation reaches. Keep it, and the tools become what they should be: fast first drafts, honestly checked.

Frequently Asked Questions

How do you verify transcription accuracy in automated output?

Verify a transcription by working from the manuscript or print itself — not the model's fluent output — and concentrate effort where errors do the most damage: proper names, numbers, dates, negations, and any passage your argument will rest on. Put the image and transcription side by side and read line by line, asking "is this what is written?" rather than "does this make sense?" Confidence scores can help triage, but treat them as pointers, not guarantees. Check that historical spelling survived rather than being quietly modernized. For published or reference material, establish ground truth through double-keying. A headline accuracy figure describes someone else's corpus, not your page.

What is a character error rate (CER) and what does it actually mean?

A character error rate is the minimum number of character-level edits — substitutions, insertions, deletions — needed to turn a machine's output into a correct reference transcription, divided by the total characters in that reference, calculated using Levenshtein edit distance. A 5% CER does not mean five words in a hundred are wrong; it means roughly five character edits per hundred characters, corresponding to somewhere around 95–99% word accuracy and only 70–80% line accuracy. Those ratios shift with script, language and model. A reported figure is conditioned on the model's training material, so it is a starting hypothesis about your document, not a measurement of it.

Why is fluent transcription output more dangerous than garbled output?

Fluent output is more dangerous because its errors read perfectly and survive a casual proofread. Garbled characters, split words and dropped lines are recoverable — you can see them and a glance at the image fixes them. General-purpose large language models fail differently: they downsample the image and lean on their language model to produce text that fits the layout without being grounded in the marks on the page, producing plausible names, smoothed clauses, or entirely invented lines that read as period-appropriate prose. Vision-language models have been measured hallucinating at rates around a quarter of the time. Coherent errors survive because nothing looks wrong, so the check must be more disciplined.

Can I trust confidence scores to find transcription errors?

Confidence scores are useful for triage but should never let you skip a reading. Some engines expose per-character or per-word values — Tesseract returns character-level confidences, and Transkribus surfaces them in its validation interface — and they point you at the tokens the model was least sure of. However, they are model posteriors, not calibrated probabilities of correctness. No comprehensive study has established that confidence scores from the major engines are calibrated against actual correctness on difficult historical hands, so treat a high-confidence reading as a place you can look faster, never one you can skip. General LLM transcription typically exposes no per-token confidence at all, removing even this rough signal.

What is double-keying and when should I use it?

Double-keying, also called four-eyes verification, is the discipline standard for establishing ground truth: two transcribers independently produce the same text, and an adjudicator reconciles the differences, with the adjudicated text becoming the reference. It means independent transcription, not one person checking another's work. Use it for high-value material — a corpus you will publish or a reference edition — or when you need to measure a model's real error rate on your own material rather than trusting its headline figure. Ground-truth quality dominates quantity: roughly twenty carefully corrected pages outperform a hundred pages carrying errors in the reference.