AI in Historical Research: What's Reliable and What Isn't

This is a practical map of where AI belongs in historical research and where it does not. The line runs between tools that transcribe what is on the page and tools that generate what sounds plausible — and knowing which is which is the difference between a recoverable error and a fabrication that survives into your footnotes.

AI is reliable in historical research for the mechanical, verifiable parts of the work: converting page images into searchable text, drafting translations, extracting names and places, and summarizing material you will read yourself. It is unreliable — sometimes dangerously so — for anything where fluent output can pass as fact: unverified citations, quotations, and readings of difficult script that "look right" but were never on the page. The practical rule is to separate tools that transcribe what exists from tools that generate what is plausible, and to verify everything against the original. This piece maps that line task by task.

The distinction matters because the failure modes are not symmetric. A specialist transcription tool that misreads a word leaves a visible, recoverable error — a garbled character you catch against the image. A general chatbot that invents a source leaves a plausible, confident fabrication that can survive into a footnote. Knowing which kind of error a tool tends to make is the first thing a historian working with AI needs to understand, and it sits at the center of the broader question of what AI can and cannot responsibly do across the humanities and cultural heritage.

The two families of AI tool, and why the difference matters

Nearly every AI application in historical work descends from one of two lineages, and they fail in opposite ways.

Recognition tools

Handwritten text recognition (HTR) and optical character recognition (OCR) take an image and return the text on it. They are constrained by the input: their job is to read marks, and when they fail, they fail on the marks. A misread letterform, a dropped abbreviation, a confused ligature. These errors are local, visible, and correctable against the page.

Generative tools

Large language models such as ChatGPT, Claude, and Gemini produce fluent text conditioned on a prompt. Their strength is breadth: they translate, summarize, classify, and draft across many languages. Their characteristic failure is confabulation: confident, well-formed output that is not grounded in any source. The output is designed to be plausible, which is precisely what makes its errors hard to catch.

The reliability question in historical research almost always reduces to matching the task to the right family — and never letting a generative tool do a recognition tool's job. That single confusion is behind most of the trouble. The underlying difference between OCR and HTR, and when each matters, is worth understanding in its own right, because "just use AI" hides a real choice.

Reliable: turning images into searchable text

This is where AI has genuinely changed the discipline. For centuries the record has been effectively unsearchable — you cannot Ctrl+F paper, and transcription by hand is the labor-intensive bottleneck gating billions of records: deeds, registers, inventories, minutes, letters.

Specialist HTR closes that gap. On scripts with adequate training data, accuracy is now genuinely usable. Transkribus community models report a character error rate (CER) around 5.4% on German Kurrent and Sütterlin, 8.2% on nineteenth-century French hands, and 4.04% on Middle Dutch civic records — though these are validation figures on the models' own training distributions, and should be read as best-case rather than what you will see on an unfamiliar hand.

Two caveats keep this honest. First, there is no single HTR accuracy number: reported CERs span an order of magnitude depending on hand, period, language, image quality, and how much labelled data the model has seen. Second, CER hides error type. A wrong character in a common word and a wholly wrong reading both count as one error, and — importantly — a preserved historical spelling variant is indistinguishable, by the metric, from a misread. This is why "good enough" is task-dependent: a low CER is excellent for full-text search and unacceptable for a diplomatic edition.

The reliable conclusion: use recognition tools to make collections searchable and to produce a first-pass transcription, then verify against the image. That workflow is the foundation of any serious effort at making an archive genuinely searchable, and it is the part of AI-in-history that has earned its place.

Unreliable: asking a chatbot to read a manuscript

The most common mistake is the most understandable one. A general LLM is right there, it accepts an image, and it will confidently return a transcription. The problem is that the transcription is often wrong in ways you cannot see.

Benchmarks make the gap concrete. On modern handwriting, top multimodal models do well — GPT-4o-mini reaches 1.71% CER on the modern IAM dataset. On difficult historical material the same class of model collapses: Claude 3.5 Sonnet and GPT-4o land in the 71–77% CER range on the Bentham and READ2016 corpora — roughly an order of magnitude worse than a specialist model, meaning more than seven in ten characters are wrong. And because the output reads fluently, the errors are easy to miss.

There is a live complication here worth stating plainly: the gap between general LLMs and specialist HTR is narrowing, some 2024–2025 benchmarks show general models competitive on certain modern hands, and any single cited figure may already understate current capability — which is exactly why continuous re-benchmarking, on your own material, matters more than any headline number. But on the hard historical hands that actually stall research, the specialist advantage remains large, and it remains verifiable only against the page. The mechanics of why general models struggle — image downsampling and fluent confabulation — are worth understanding before you trust one with a manuscript.

Unreliable: trusting a citation because the AI produced it

If manuscript reading is where LLMs fail quietly, citation is where they fail catastrophically — and where the stakes for published historical work are highest.

The evidence is unambiguous. In a controlled test of scientific-writing references, fabrication rates were 39.6% for GPT-3.5, 28.6% for GPT-4, and 91.4% for Bard — invented sources, quotations, and page numbers presented with complete fluency. This is not a hypothetical risk. Attorneys were fined $5,000 each in the Mata v. Avianca case over fabricated case citations, and a California appellate court issued a $10,000 sanction in September 2025 after 21 of 23 quotations in a brief turned out to be fake. The problem is even leaking into the scholarly record: a 2026 audit reported that 1 in 277 PubMed-indexed papers contained fabricated references, a roughly twelve-fold rise in two years.

Retrieval-augmented generation (RAG) — conditioning the model on passages from a curated corpus with citations exposed — reduces the problem but does not eliminate it; retrieval quality becomes the limiting factor, and history-specific studies of residual fabrication under RAG are sparse. The only safe posture is the old one: never accept a citation you have not resolved to a DOI, URL, or library record. Treat LLM output as a lead, not a finding.

Partly reliable: OCR on historical print

There is a persistent assumption that OCR "handles old books fine." It does not. General-purpose OCR is engineered for clean, modern, high-contrast type, and it degrades sharply on earlier material — mapping the long s (ſ) to f, splitting or dropping ligatures (fi, fl, æ, œ), stumbling on blackletter and Fraktur, and silently normalizing archaic orthography that is not an error. Holley's benchmark study remains instructive: raw OCR accuracy on Australian Trove newspapers was about 71%, climbing to roughly 98% only after manual correction — a gap no fully automatic pipeline closed.

So historical-print OCR is reliable conditionally: with a pipeline built for the material and a human reviewing against the page. The naive assumption that a modern OCR engine will read an early-modern book faithfully is where the errors enter.

Partly reliable: downstream analysis on noisy text

Everything you do after transcription inherits the errors in it. This is easy to forget when the text looks clean. Named-entity recognition — pulling people and places out of a corpus — is a good illustration: person-entity F1 falls from about 87% on clean text to about 63% as CER rises from 7% to 20%, a thirty-point drop from what looks like a modest increase in transcription noise. The lesson is that translation, summarization, and entity extraction are all reasonable AI applications — but their reliability is capped by the quality of the transcription beneath them, and by whether the analysis stays tied to the source rather than drifting into the model's own priors. Keeping each transformation reversible and anchored to the original is what separates defensible analysis from confident guesswork.

Where a purpose-built tool fits

If the reliable path is recognize faithfully, then verify against the image, then analyze on top, the practical question becomes which instrument sits at the recognition stage. Two things distinguish the tools worth trusting there: they preserve what is on the page rather than smoothing it into modern prose, and they make the base transcription easy to check.

This is the stage Leo was built for. Its ATR-1 model is a specialist Transformer trained on images of historical documents, reading Latin-script material — whatever the language on the page, English wills, French notarial records, German parish books, Dutch registers, or Latin treatises alike — with no per-corpus model training required. On a randomized 97-image sample of early-modern English manuscripts from the Folger Shakespeare Library, ATR-1 scored roughly 5% CER at release, about 61% fewer errors than the next-best model in the comparison (the full benchmark, including the LLM and Transkribus figures, is published here). The point of that number is not the number; it is the error type behind it. ATR-1 is built to transcribe what is written — preserving strikethroughs, marginal additions, abbreviation marks, and archaic spelling — rather than resolving them into something plausible, which is exactly the discipline generative tools lack.

Two design choices follow directly from the reliability argument above. First, the original image sits beside the transcription so verification is a glance, not a separate task. Second, the interpretive AI — translate, modernize, summarize, extract entities — runs as a separate layer that writes to a new tab and never overwrites the base transcription, so the faithful reading of the page stays intact underneath any analysis you build on it. That is the whole architecture of trustworthy AI use in one product: a faithful base, checked against the source, with interpretation layered visibly on top rather than baked in.

A working rule for the historian

The honest summary is not "AI is reliable" or "AI is dangerous." It is that reliability tracks the task and the error type. Recognition tools err on marks, recoverably, against a page you can inspect. Generative tools err on facts, plausibly, in ways that survive if you let them. Use the first family to open your sources and the second to help you think about them — but verify the first against the image and never let the second stand in for a source you have not seen.

The emerging disciplinary norms point the same way: compare output against the original, resolve every citation, attach provenance to what you produce, and disclose the tools and the verification behind AI-assisted work, as the American Historical Association's guiding principles for AI in history now recommend. None of this is new to historians. It is the ordinary scholarly obligation to your sources, applied to a new set of instruments — and it is the part of the craft no model will do for you.

Frequently Asked Questions

Is AI reliable for historical research?

AI is reliable in historical research for mechanical, verifiable tasks: converting page images into searchable text, drafting translations, extracting names and places, and summarizing material you will read yourself. It is unreliable for anything where fluent output can pass as fact — unverified citations, quotations, and readings of difficult script that look right but were never on the page. The practical rule is to separate tools that transcribe what exists from tools that generate what is plausible, and to verify everything against the original. Reliability tracks the task and the error type, not the technology as a whole.

What is the difference between recognition AI and generative AI in historical work?

Recognition tools — handwritten text recognition (HTR) and optical character recognition (OCR) — take an image and return the text on it, so their errors are local, visible, and correctable against the page: a misread letterform, a dropped abbreviation, a confused ligature. Generative tools, such as large language models, produce fluent text from a prompt, and their characteristic failure is confabulation: confident, well-formed output not grounded in any source. The errors are designed to look plausible, which makes them hard to catch. Most trouble in historical research comes from letting a generative tool do a recognition tool's job.

Can ChatGPT accurately transcribe historical manuscripts?

No, not for difficult historical material. On modern handwriting top multimodal models do well, but on hard historical hands the same class of model collapses — Claude 3.5 Sonnet and GPT-4o land in the 71–77% CER range on the Bentham and READ2016 corpora, roughly an order of magnitude worse than a specialist model, meaning more than seven in ten characters are wrong. Because the output reads fluently, the errors are easy to miss. The gap between general models and specialist HTR is narrowing on some modern hands, but on the historical hands that actually stall research, the specialist advantage remains large and verifiable only against the page.

Do AI chatbots make up citations?

Yes, and this is where they fail most dangerously for published historical work. In a controlled test of scientific-writing references, fabrication rates reached 39.6% for GPT-3.5, 28.6% for GPT-4, and 91.4% for Bard — invented sources, quotations, and page numbers presented with complete fluency. This is not hypothetical: courts have sanctioned attorneys over fabricated case citations, and one 2026 audit reported fabricated references in 1 in 277 PubMed-indexed papers. Retrieval-augmented generation reduces the problem but does not eliminate it. The only safe posture is to resolve every citation to a DOI, URL, or library record, and treat AI output as a lead, not a finding.

Does OCR work reliably on old books and historical print?

Only conditionally. General-purpose OCR is engineered for clean, modern, high-contrast type and degrades sharply on earlier material — mapping the long s (ſ) to f, splitting or dropping ligatures like fi, fl, æ and œ, stumbling on blackletter and Fraktur, and silently normalizing archaic spelling that is not an error. Holley's benchmark study on Australian Trove newspapers found raw OCR accuracy of about 71%, climbing to roughly 98% only after manual correction — a gap no fully automatic pipeline closed. Historical-print OCR is reliable with a pipeline built for the material and a human reviewing against the page, not with a naive modern engine.