Architecture

System Overview

Scanned Documents
        │
        ▼
Image Ingestion         — load, assign IDs, preserve source metadata
        │
        ▼
Image Quality Assessment — detect blur, skew, noise, blank pages, damage
        │
        ▼
OCR Engine Selection     — route by document characteristics
        │
        ▼
OCR & Layout Extraction  — text, regions, tables, reading order, confidence
        │
        ▼
Canonical JSON Generation — normalize all OCR output to one format
        │
        ▼
Document Classification   — classify by type (letter, ledger, minutes, etc.)
        │
        ▼
Metadata Extraction       — dates, people, organizations, locations, topics
        │
        ▼
Quality Validation        — gate: is this page safe for RAG?
        │
        ▼
Semantic Chunking         — split by document type into meaningful units
        │
        ▼
Embedding Generation      — vectorize each chunk with metadata payload
        │
        ▼
Qdrant Vector Database    — store embeddings, enable filtered search
        │
        ▼
RAG / LLM Search          — natural language Q&A with citations
                              │
                              ├── Hard path: C engine (names, dates, subjects)
                              ├── Soft path: Qdrant (semantic search)
                              └── Merge: hybrid retrieval with overlap boosting

Two-Layer Architecture

Python Pipeline (10 stages)

All stages read and write canonical JSON files. Each stage adds its section (source, page, quality, ocr, layout, classification, metadata, chunks) as pages move through the pipeline. Atomic writes (temp-file + rename) prevent data loss on crash.

OCR subsystem: multi-engine with auto-routing. Docling for clean printed pages, Surya for handwriting, Qwen3-VL for complex/degraded scans. Repair-first flow: primary OCR → score coherence → repair with alternate engine → escalate to Qwen-VL if still below threshold. Ensemble word-level voting available on heavy hardware. Process-wide model cache eliminates reload overhead.

C99 Query Engine

Fast exact lookups on structured metadata using merge-join on sorted arrays.

Three indexes: - names — personal names and places with page references - dates — date → document index with range overlap queries - subjects — subject heading → document index

Query DSL: JSON plan language with MATCH, CONTAINS, RANGE, INTERSECT, UNION, SORT, SLICE operators. English-to-DSL translation via small LLM.

Execution: O(n) merge-join on (doc_file, page) key. Page=0 doc-level wildcard enables cross-index queries (subjects are document-level, names are page-level). Lazy loading — only indexes referenced by the plan are read.

99/99 tests passing. Build with make. Verify with make verify.

Bridge

Connects the two layers. Reads page JSON from the OCR pipeline and produces the three index files the C engine consumes. Auto-detects input format — handles raw pipeline output, bridge export, and synthetic data without configuration. Also supports incremental index building (per-page, with fcntl locking) and priority-sorted streaming mode.

Subsystems

Ingestion Subsystem

Loads scanned documents from source archive. Produces processing metadata (source path, batch ID, page number, input type). Supports single images, folders, PDFs, and nested document folders.

Quality Subsystem

Two stages. Pre-OCR assessment detects image-level issues (blur, skew, noise) using OpenCV with graceful fallback when unavailable. Post-OCR validation determines whether extracted text is reliable enough for RAG indexing using composite score (image quality × OCR confidence × text coherence).

OCR Subsystem

Evaluates and routes to OCR backends based on handwriting detection, table detection, and image quality. Produces raw output consumed by the canonicalization stage. Backend selection is swappable per document type.

Canonicalization Subsystem

Converts raw OCR output from any engine into the canonical JSON format. This is the stable interface — every downstream stage reads canonical JSON, never raw OCR output. Universal page JSON detection handles any input format.

Classification Subsystem

Classifies documents by type using keyword signals and layout heuristics. Falls back to VLM-based classification for ambiguous cases. Classification drives chunking strategy and metadata extraction rules.

Metadata Subsystem

Extracts structured metadata: dates (4 regex formats with OCR century error correction), people (title patterns with geographic filtering), organizations, locations, monetary amounts, and topics. Enables filtered search in Qdrant.

Chunking Subsystem

Splits canonical documents into semantically meaningful chunks. Strategy varies by document type — article-based for meeting minutes, date-anchored for church records, row-based for ledgers, paragraph-based for books.

Embedding Subsystem

Generates vector embeddings for each chunk. Configurable model — nomic-embed-text (768-dim) or Qwen3-Embedding (up to 4096-dim on heavy hardware). Attaches metadata payloads for filtered retrieval.

Storage Subsystem (Qdrant)

Stores embeddings with payload metadata. Supports ANN search with metadata filtering. Enables queries like "find church records from 1898 mentioning Newington."

Search Subsystem

Accepts natural language queries. Three retrieval paths: hard (C engine, exact metadata), soft (Qdrant, semantic), and hybrid (merged with overlap boosting). English-to-query translation via small LLM. Answers generated with source citations.

Refinement Subsystem

Nightly re-OCR of low-confidence pages. Self-improving — the archive gets more accurate every night. Pages failing 3 attempts are flagged for human review.

Data Flow

All stages read from and write to the canonical JSON format. Each stage adds its section (classification, metadata, quality, chunks) to the document. The canonical JSON accumulates data as it moves through the pipeline. The final document is the complete record.

The bridge converts canonical JSON to index JSON. The C engine reads the indexes. Qdrant reads the canonical chunks. The search subsystem queries both and merges results.

Key Design Decisions

See decisions/index.md for ADRs.