Concept

Vision

Town Records makes scanned historical archive documents searchable by meaning, not just keywords. Users ask natural language questions and get answers grounded in the archive, with citations back to the original scanned pages.

Motivation

Town archives contain centuries of handwritten and printed documents in inconsistent formats and scan quality. These are effectively inaccessible without manual reading. An OCR-to-RAG pipeline transforms them into a queryable knowledge base while preserving structure, quality information, document type, metadata, and provenance.

Core Ideas

One canonical format. Different OCR engines produce different output. All output is normalized to a single canonical JSON structure so downstream stages (classification, chunking, indexing, search) work identically regardless of which OCR backend was used.

No single OCR engine wins. Printed books, handwritten letters, ledgers, and invoices each suit different engines. The pipeline supports OCR backend routing and swapping without changing anything downstream.

Quality gates before indexing. Poor OCR produces unreliable search results. Each page is assessed for quality before embedding. Low-quality pages are flagged for review rather than silently polluting the vector database.

Semantic chunking by document type. A ledger row, a meeting minutes motion, and a letter body are different semantic units. Chunking strategy adapts to document classification.

Citations over black boxes. Every RAG answer includes source document references and page numbers so results can be traced back to the original scans.

Workflow Objectives and Data Objects

We want to have a workflow that can process 10,000 documents ideally in less than 72 hours and definitely less than 200 hours. The town-record scans should be turned into several kinds of documents:

Scan-Record Markdown. This document should capture the characters and layout similar to the way a layout program works. So, each sequence of text should have a coordinate X and Y indicating where it is on the page and a size H and W. This will be complicated for something like a table because each item in the table will have its own coordinates. There should only be a single line of text for a given sequence. Things like pictures or illustrations should be identified as a rectangle.

Text-Block Markdown. In this phase the OCR should group blocks of text. So, for example: - Multiple lines are recognized as paragraphs - Illustrations are recognized as including captions - Page numbers are recognized - Footnotes are recognized - Marginal notes are recognized

Page-Record Markdown. This document should have the markup of the processed text blocks. It should include: - The raw ocr'd text in continuous batches (elimination of hyphenation) - Tab characters where the paragraphs have been indented - newline characters where there is a blank space (or the end of a paragraph) - Meta data such as the page in the book/document and the book/document it belongs to

Document Record. The pages need to be assembled into documents.

Name Index. For each document there should be an index of personal names and places including the page number(s) in the document where the name is found.

Document Summary. Each document should have an abstract or summary that briefly describes the document and identifies (1) the date of the document, (2) the number of pages in the document, (3) references the Name Index and other files (like the intermediate scans)and raw image file.

The Document summaries and Document records need to have embeddings generated and stored in a vector database.

Indexes. In order to do hard querying on set metadata like dates, we either need to (1) use an SQL or other database, or (2) make our own indexes and write a join system for those indexes.

There ideally should be a hybrid querying mechanism that allows the user to either do a "hard" query (on specific metadata, like datate) or a soft query (on vague criteria, like documents involving agriculture) or both (for example: "tell me all documents involving agriculture from 1930 to 1940). For a hard query the system needs to translate English language requests into the correct querying commands (whether they are SQL or something else). Obviously we need to protect against prompt jail breaking or user attempts to hack the computer using a crafted query (using escape characters or whatever).

Ultimately we need to return a list of resulting documents (limited to maximums) and allow the user to see the summaries, the full document or the page images.

Non-Goals (v1)

  • Full document routing and automation
  • Handwriting recognition perfection
  • Knowledge graph construction
  • Real-time processing