AI Can Finally Read Handwriting Quite Well!

Just a week ago, during a lecture for family history researchers in Valmiermuiža, I said that I was impatiently waiting for the moment when artificial intelligence would finally be able to read handwritten texts. Until now, my experiments with different AI models for reading Latvian genealogical documents had been rather unsuccessful.

But then Gemini 3 Pro arrived!

The model was released exactly on November 18, and it immediately caused a wave of excitement. Of course, new developments in AI appear every day now, and you can’t blindly trust all the hype. Still, what caught my attention were the positive reviews specifically about the model’s ability to understand historical handwriting (you can read more in Mark Humphries’ blog post in English.

I tried it myself, and I can join the excitement – finally, there is a model that can read historical handwriting much better than any previous one.

1. First experiment: my great-grandfather’s student file

I started with what I considered a simpler task — I “fed” the model my great-grandfather’s student file, written mostly in Latvian and in handwriting. It contains various documents:

• an application to enter university,–

• an extract from the baptism register,–

• a passport copy,

• school completion documents, etc.

I asked the model to extract biographical information, and after 74 seconds I received surprisingly usable data: personal details, family status, education, places of residence, etc.

But I also noticed errors — the godmother was listed ainstead of mother. The model had mixed up the columns in the baptism record. It had also interpreted the parish name incorrectly. Still, the majority of the extracted information was correct.

And this, in my opinion, is where the biggest risk lies: the result is so good that people will become lazy and won’t check those few mistakes.

2. The real challenge: church books

Next, I turned to what I consider the most difficult task — transcribing baptism entries from our churchbooks. I chose a reasonably legible handwriting sample and specified that it came from the Mežmuiža Lutheran parish in Latvia.

Again — the result was surprisingly good, but there were errors:

• Runge is not the child’s surname, but the farm name where the father Ansis worked as a farmhand.

• Surname Stintiņ was interpreted as Plintis.

• There were mistakes in the godparents’ surnames as well.

But I didn’t give up. I explained to the model what was what and asked it to transcribe the next entry on the same page.

This time, there was visible progress.Gemini correctly understood that Laure is a farm name, not a surname. It even tried to supplement the surnames and names with their modern forms, such as:

• Osoling = Ozoliņš,

• Anliese = Anlīze.

3. A discussion with the model: Gemini analyzes instead of blindly agreeing

A particularly interesting discussion emerged around the godfather Ansis Ozoliņš, who was noted in the entry as the person holding the child during the baptism. In German, this role is called Haltpathe.

Gemini, however, was convinced that the word was Wolgaling and that the man had been a manorial carpenter (Gutszimmermann) rather than a Haltpathe.

Although I disagreed, I was genuinely impressed by how the model argued:

• it compared the strokes of specific letters to other occurrences on the page;

• it explained why it believed a particular letter looked the way it did.

Those who have worked with ChatGPT know how dangerous it is that it always “agrees”. Gemini is the opposite — it tends to stick to its interpretation, explain, and analyze. I would have gladly continued the discussion and taught it to understand our church books even better, but I had to finish this blog post:)

4. Handwriting in Russian: NKVD case materials

The third experiment involved NKVD (the predecessor of the KGB) criminal case file pages, handwritten and in Russian. Gemini handled these very well too. Of course, there were small mistakes:

• Vitālijs became Vasilij,

• Barkava became Varakļāni, etc.

But overall: the content was correctly understood; it was able to transcribe the document, produce a summary, and even add relevant historical context.

5. Summary: this will change family history research

I have no doubt that this tool will bring major changes. But — as with all AI — there will also be problems.

If a person does not know the original language or handwriting style, they will be unable to catch mistakes. And because the result is good, there will often be little motivation to check.

Why read 30 pages yourself if AI gives you a summary, right?

6. Five recommendations by Gemini itself on how to transcribe handwriting better

1) Crop a specific baptism entry instead of giving the whole church book page.

This prevents the model from wasting time on unrelated information.

2) Provide precise contextual information.

For example:

“This is an 1883 Lutheran church book from Courland (Latvia). Language: German, with Latvian personal names. Note that the father’s name often begins with a farm name (Gesinde), followed by status (Knecht), and then the surname.”

Why: As we saw with “Runge Knechts Ans”, the model initially misunderstands the structure. If you give it the formula [Farm name] + [Status] + [Given name] + [Surname], the number of errors drops dramatically.

3) Ask for a precise transcription (verbatim), not a translation.

Say: “Transcribe exactly what is written, letter by letter (verbatim). Do not expand abbreviations unless they are clear. If a word is illegible, write [?]. Do not guess.”

Why: This forces the model to focus on the visual text (as in “Wolgaling” vs. “Osoling”) rather than trying to guess. Remember that the model is not only a “reader” — it is also a guesser, working with probabilities.

4) Provide a local database.

If you know the most common surnames or farm names in the parish, give them in a list.

Say: “Common surnames in this parish include: Spurdziņš, Veckungs, Ozoliņš. Check whether the unclear word matches any of these.”

Why: LLM models struggle with proper names because they lack semantic meaning. Giving a list reduces hallucination risk.

5) When there are errors, force the model to think visually.

If you see the model making a mistake (like with “Gutszimmermann” vs. “Haltpathe”), it is not enough to say “that word is wrong.”

Say: “You read ‘Osoling’, but I see a letter that looks like ‘W’ and the ending ‘-ling’. Please describe the shape of the first letter and compare it to other ‘W’s on the page.”

Why: This switches the model from text-generation mode to visual recognition mode. Often, the model then “notices” its own mistake and corrects it — as happened with the carpenter entry.

P.S. You can try Gemini 3 Pro for free in Google AI Studio.