Fluent but Wrong: Understanding LLM Transcription Errors on Historical Documents

This article explains why LLM transcription errors on historical manuscripts are harder to catch than ordinary machine mistakes, what those errors look like on the page, and how to structure your work so fluent fabrications never survive into a finished transcription. If you rely on a general model to read a difficult hand, the risk is not garbled text — it is plausible text that reads correctly and is wrong.

When a large language model transcribes a historical document, its mistakes do not look like mistakes. They read as clean, confident, grammatical prose — which is precisely what makes them dangerous. The most consequential LLM transcription errors are not garbled characters a reviewer would catch at a glance. They are plausible fabrications: a word the model expected to see, an archaic spelling it invented, a passage it smoothed into sense. A specialist recognition model that stumbles produces a visibly broken reading you can correct. A general LLM that stumbles produces a fluent error you may never notice.

That difference — detectability, not raw error count — is the whole of the problem. What follows is an account of why it happens, what the failure looks like on the page, and how to structure your work so fluent errors do not survive into your finished transcription.

Why fluent errors are the ones that hurt

Start with a distinction that most accuracy discussions skip. Two transcription systems can post the same character error rate and pose entirely different risks.

Character error rate (CER) measures surface edit distance against a reference: substitutions, deletions, and insertions divided by the total characters in the reference. It is the standard metric in the field, and it is genuinely useful. But it tells you how many characters differ from the truth, not how catchable those differences are. A garbled OCR output — `rn` where the page reads `m`, a broken ligature, a dropped diacritic — announces itself. Any reviewer glancing between image and transcript will stop on it. A fluent fabrication that scores an identical CER slips past the same reviewer, because nothing about it looks wrong.

This is why the danger ordering runs by fluency, not magnitude. The worst error is not the largest one. It is the one that reads correctly, scores competitively, and embeds a false reading into a dataset that later work will cite. This is the throughline of why keeping a human in the loop still matters: the tools that produce the most confident output are the ones that most need checking, and the checking is hardest exactly where the output is smoothest.

What produces the fluent error

To see why general LLMs fail this way, it helps to know what they are actually doing when they "read" a page.

A vision-language model couples a vision encoder to an autoregressive language model. The image is broken into patches — a Vision Transformer typically processes a 224×224 or 336×336 pixel image as a few hundred patches (patchification is the standard image tokenization approach) — and the language model then generates text one token at a time, optimizing next-token prediction against fluent natural-language targets. Two structural facts follow from this design, and both matter for historical material.

First, the image is downsampled. High-resolution detail — the hairline that distinguishes one minim from two, the faint macron over a vowel, the difference between a long s and an f — is compressed away before the model ever reasons about it. General LLMs hallucinate on historical handwriting because they lean heavily on the text side and work from a downsampled image.

Second, the model is trained to produce fluent language. When the visual evidence is thin, its language prior fills the gap. It completes a partially obscured word, normalizes an archaic spelling, or inserts a word it expects based on context. The OCR-hallucination literature describes this as the production of fluent, plausible text not grounded in the input; OpenAI's own account frames it as a consequence of training and evaluation procedures that reward guessing over acknowledging uncertainty. The KIE-HVQA benchmark documents current multimodal models frequently misinterpreting or ignoring textual visual content across the GPT-4o and Qwen2.5-VL classes.

There is also a specifically historical variant of the failure. Levchenko and colleagues, evaluating twelve multimodal LLMs on eighteenth-century Russian Civil-font texts, named it over-historicization: the model inserts archaic characters and spellings that are period-appropriate to the genre but not present on the page. The transcript looks more authentically old than the source. It is wrong in the most flattering possible direction — which is to say, the direction least likely to trigger suspicion.

The failure on the page

Consider what these mechanisms do to real material.

An early-modern English scribe writes `yo`^`r` — a superscript r standing for your. A faithful transcription preserves the abbreviation or expands it with a visible editorial marker: `yo[u]r`. A fluent model silently renders `your`, dropping the evidence that the page carried a contraction at all. The reading is not wrong in meaning. It is wrong as a record.

A page carries the long s (ſ). Conventional OCR built for modern type reads it as f, producing a visible garble. A general LLM often does something subtler and worse: it recognizes the word from context and "corrects" the spelling to its modern form, so the transcript reads `sister` where the page shows a form your own eye would have flagged. The correction erased the thing you needed to see.

A partially damaged word — ink loss, show-through from the verso, a warped gutter — gives a specialist model a garbled or incomplete reading, and you know to go back to the image. A general LLM completes the word. It supplies the letters it expects. You get a whole, grammatical sentence with a hole in the evidence you cannot see, because it has been seamlessly filled.

These are not exotic edge cases. They are the ordinary texture of reading early modern paleography: the superscripts and brevigraphs of scribal abbreviation, the letterforms of secretary hand, the u/v and i/j interchange that early-modern convention treats as normal and a language model treats as error. Every one of them is an invitation for a fluent model to improve the page into something it was not.

Why "I'll just check it" is not enough

The instinctive reply is that a careful scholar will catch these things on review. The evidence on human oversight is not encouraging.

Automation bias — the tendency of human reviewers to over-trust plausible automated output and under-detect its errors — is one of the better-documented effects in the human-factors literature. A systematic review found it to be a fairly robust and generic effect across research fields, and, strikingly, that decision-making was in some settings more accurate without a decision aid than with a flawed one. The mechanism is exactly the fluency profile of LLM transcription: when the output is coherent and confident, the reviewer's own judgment gets sidelined.

There is an honest limit to note here. That research comes from decision-support contexts — clinical systems, chiefly — and no controlled study has yet isolated the effect of fluency specifically on human proofreading of transcripts. But the direction of the finding is consistent, and the reason is intuitive. You cannot proofread against a reference you do not have. When you are transcribing a difficult hand, the transcript often is your reference; a fluent error becomes the thing you check the image against, rather than the other way around. The fabrication does not merely survive review. It can quietly steer it.

What LLMs are, and are not, good for

None of this makes general LLMs useless for historical work. The critique is targeted, not blanket.

On clean printed text and well-segmented modern handwriting, the general models perform strongly — one benchmark reports GPT-4o-mini at 1.71% CER on the IAM modern-handwriting database. And on a range of downstream tasks performed over an already-faithful transcript — translation, summarization, named-entity extraction, reconciling reading order — they are genuinely valuable. The picture on difficult historical material is contested rather than settled: Transkribus's 2025 benchmark shows specialist recognition winning decisively on handwritten historical documents, while the Levchenko study finds LLMs substantially outperforming traditional OCR on one eighteenth-century corpus when used appropriately — a single-corpus, single-script result that still turns on careful verification. What no benchmark supports is trusting a general LLM as the primary transcriber of a difficult hand and skipping the check.

The safe division of labour follows from that. Use a system built to read the page for the reading; reserve the general models for analysis performed on top of a transcription you already trust. The difference between the two is the whole subject of ChatGPT's specific struggles with historical handwriting, and it is worth internalizing before you build any pipeline around it.

Building the workflow so fluent errors cannot hide

The practical answer is not to abandon machine transcription. It is to choose a tool whose errors are the catchable kind, and to keep interpretation separate from the base record.

This is the design principle behind Leo's recognition model, ATR-1. It is a Transformer trained on images of historical documents, and it weighs high-resolution visual evidence against context rather than matching glyphs to a modern type model or leaning on a language prior to guess. Its errors, when they come, tend to fall on characters and words — the recoverable kind, visible against the image — rather than the fluent, self-consistent fabrications a general LLM produces. And it is built to transcribe what is on the page: the long s stays a long s, the superscript abbreviation is preserved rather than silently modernized, the archaic spelling survives. On a randomized 97-image sample of early-modern English manuscripts from the Folger Shakespeare Library, ATR-1 scored roughly 5% CER at release — 61% fewer errors than the next-best model tested (Transkribus/Text Titan I ≈13%, Claude Opus ≈23.3%, Gemini 2.5 Pro ≈24.8%, GPT-4.1 ≈56.7%), with the full comparison published here.

The second half of the principle matters as much as the first. If you do want a language model's help — to modernize archaic prose, to translate, to reconcile several readings — the transformation should be layered on top of the faithful transcription, in its own tab, leaving the base text untouched. That way the fluent output never overwrites the record; it sits beside it, clearly marked as interpretation, where you can compare it against the source and discard it if it drifts. The base transcription remains the thing you can always return to. Fluent errors are contained precisely because they are kept out of the layer that claims to be the source.

That structure is worth insisting on whatever tools you use. The point is not the product; it is that a faithful base and a separate interpretive layer make fluent errors visible instead of letting them pass for the record.

The habit worth keeping

The appeal of fluent output is that it seems to save you the hardest part of the work — and it does not. It relocates it. A transcript that reads well is not a transcript you have verified; it is a transcript that has made verification feel unnecessary, which is a different and more dangerous thing.

The discipline that protects your scholarship is old and unglamorous: read the transcript against the image, not the image against the transcript. Preserve what the page actually shows before you decide what it means. Treat the machine's confidence as a hypothesis, not a finding. The tools have improved, and they will keep improving as corrections feed back into them — but the judgment about what is really on the page stays with the reader who knows the hand. That is not a limitation to engineer away. It is the work.

Frequently Asked Questions

What are LLM transcription errors on historical documents?

LLM transcription errors on historical documents are plausible fabrications rather than garbled text: a word the model expected to see, an archaic spelling it invented, or a passage it smoothed into sense. Because a general large language model is trained to produce fluent language, it fills thin visual evidence with its language prior instead of reading what is on the page. The result reads as clean, confident, grammatical prose — which is exactly what makes it dangerous. Unlike a broken OCR reading that announces itself, a fluent error scores competitively and slips past review, embedding a false reading into a dataset that later work will cite.

Why are fluent LLM errors harder to catch than OCR mistakes?

Fluent errors are harder to catch because they read correctly, so nothing about them signals a problem. Two systems can post the same character error rate and pose entirely different risks: a garbled OCR output — `rn` for `m`, a broken ligature, a dropped diacritic — announces itself, and any reviewer glancing between image and transcript will stop on it. A fluent fabrication with an identical score does not. The danger runs by fluency, not magnitude. The worst error is not the largest one; it is the one that looks right, which is why detectability, not raw error count, is the real problem.

What is over-historicization in LLM transcription?

Over-historicization is a failure where a language model inserts archaic characters and spellings that are period-appropriate to the genre but not present on the page. Levchenko and colleagues named it while evaluating twelve multimodal LLMs on eighteenth-century Russian Civil-font texts. The transcript ends up looking more authentically old than the source itself. This is wrong in the most flattering direction — the one least likely to trigger suspicion, because it matches what a reader expects an old document to look like. It is a specifically historical variant of hallucination, and it makes faithful proofreading against the image essential rather than optional.

Can I just proofread an LLM transcript to catch the errors?

Proofreading alone is not a reliable safeguard. Automation bias — the tendency to over-trust plausible automated output and under-detect its errors — is one of the better-documented effects in the human-factors literature, found to be robust and generic across research fields; in some settings decision-making was more accurate without a flawed aid than with it. The mechanism matches the fluency profile of LLM transcription. You cannot proofread against a reference you do not have: when transcribing a difficult hand, the transcript often becomes your reference, so a fluent error becomes the thing you check the image against rather than the other way around.

Are LLMs ever useful for historical documents?

Yes — the critique is targeted, not blanket. On clean printed text and well-segmented modern handwriting, general models perform strongly; one benchmark reports GPT-4o-mini at 1.71% CER on the IAM modern-handwriting database. They are also genuinely valuable for tasks performed on top of an already-faithful transcript: translation, summarization, named-entity extraction, and reconciling reading order. The problem is trusting a general LLM as the primary transcriber of a difficult hand and skipping the check. The safe division of labour is to use a system built to read the page for the reading, and reserve general models for analysis layered on a transcription you already trust.