Design

Canonical JSON

The canonical JSON format is the stable interface between all pipeline stages. Every OCR backend's output is normalized to this structure.

{
  "source": {
    "path": "/archive/batch_001/ledger_1898/",
    "filename": "page_003.tif",
    "batch_id": "batch_001",
    "page_number": 3,
    "input_type": "image"
  },
  "page": {
    "width": 2480,
    "height": 3508,
    "dpi": 300
  },
  "ocr": {
    "engine": "docling",
    "version": "2.1",
    "text": "...",
    "markdown": "...",
    "reading_order": "auto"
  },
  "layout": {
    "regions": [
      {
        "region_id": "page_003_region_001",
        "type": "paragraph",
        "text": "...",
        "bbox": {"x0": 120, "y0": 340, "x1": 980, "y1": 420},
        "confidence": 0.92
      }
    ],
    "tables": [],
    "images": []
  },
  "classification": {
    "document_type": "ledger",
    "confidence": 0.87,
    "signals": ["numeric columns", "tabular layout", "monetary amounts detected"]
  },
  "metadata": {
    "dates": ["1898"],
    "people": [],
    "organizations": ["Town of Newington"],
    "locations": ["Newington"],
    "monetary_amounts": ["$12.50"],
    "topics": ["town records", "expenditures"]
  },
  "quality": {
    "image_quality": "good",
    "ocr_quality": "good",
    "safe_for_rag": true,
    "needs_review": false,
    "issues": []
  },
  "processing": {
    "ingested_at": "2026-07-01T10:00:00Z",
    "ocr_completed_at": "2026-07-01T10:00:05Z",
    "classified_at": "2026-07-01T10:00:06Z",
    "pipeline_version": "0.1.0"
  }
}

Stage Details

Image Ingestion

Walks the source archive directory. For each supported file, creates an ingestion record with batch ID (derived from parent folder) and page number (derived from filename ordering). PDFs are split into pages. The ingestion record is the minimum viable canonical JSON — only source and page sections populated.

Output: one canonical JSON file per ingested page.

Image Quality Assessment

Runs before OCR to detect pages that may produce poor results. Uses image processing (OpenCV or similar) to detect:

  • Blur — Laplacian variance below threshold
  • Low contrast — histogram spread below threshold
  • Skew/rotation — Hough line transform
  • Noise — signal-to-noise ratio
  • Bleed-through — ghost text from reverse side
  • Blank pages — low variance, no text regions
  • Damage — edge detection anomalies

Writes quality.image_quality into the canonical JSON. Pages with severe issues are flagged needs_review: true but still proceed to OCR — a human reviewer decides later.

OCR Engine Selection

Routes each page to an OCR backend based on document characteristics detected during quality assessment and any prior classification in the batch. The router is a simple rules engine in v1:

has_tables AND numeric_density > threshold → Docling
handwriting_likely → Qwen Vision or GLM-OCR
printed_text AND clean_scan → Unlimited-OCR
ambiguous → Docling (safe default)

Backend selection is recorded in ocr.engine.

OCR and Layout Extraction

Calls the selected OCR engine. Each engine adapter:

  1. Converts the canonical page image to the engine's expected input format
  2. Calls the engine
  3. Maps engine output to canonical ocr and layout sections

Engine adapters are the only place where engine-specific code lives. Adding a new engine means writing one adapter.

Canonical JSON Generation

After OCR, the document has source, page, ocr, and layout sections. The canonicalization step validates these against the schema and fills in defaults for missing fields. Downstream stages never see raw engine output.

Document Classification

Classifies each page by document type. Two-pass approach:

  1. Rule-based (fast) — keyword matching, layout heuristics, number/table density
  2. VLM-based (slow, fallback) — send page image to vision model when rule-based confidence is low

Classification result is written to classification in the canonical JSON. This drives chunking strategy and metadata extraction rules.

Metadata Extraction

Extracts structured metadata from OCR text using a combination of:

  • Regex patterns — dates, monetary amounts
  • NER model — people, organizations, locations
  • Keyword lists — topics, document types

Metadata is written to metadata in the canonical JSON and later attached to Qdrant payloads for filtered search.

Quality Validation

Post-OCR quality gate. Combines:

  • Image quality scores from pre-OCR assessment
  • OCR confidence scores from engine output
  • Text coherence heuristics (dictionary word ratio, gibberish detection)

Sets quality.safe_for_rag boolean. Pages that fail remain in the canonical JSON (for audit) but are excluded from embedding and Qdrant ingestion. They go to a review queue.

Semantic Chunking

Chunking strategies by document type:

Document Type Strategy
Book page Paragraph boundaries, section headers
Letter Header (date/addressee), body paragraphs, signature block
Meeting minutes Agenda item, topic section, motion, vote tally
Ledger Row-based, preserving column alignment
Invoice/Check Header, line items, totals
Contract Clause boundaries, section headers
Church record Entry-based, date-anchored
Form Field groups, label-value pairs

Each chunk includes: chunk ID, text, document type, page number, source path, metadata subset, and bounding box references.

Embedding Generation

Each chunk is embedded using a configurable embedding model. The embedding + metadata payload is prepared for Qdrant upsert. Chunk text may be augmented with document type and date context before embedding.

Qdrant Ingestion

Embeddings are upserted into Qdrant collections organized by document type or batch. Payload includes all filterable metadata. Point IDs encode source path and chunk ID for traceability.

  1. User query is embedded with the same model
  2. Qdrant retrieves top-k chunks (with optional metadata filters)
  3. Retrieved chunks + query are sent to LLM with a system prompt instructing citation format
  4. Response includes answer text, source citations, page references, and quality warnings

Error Handling

  • Missing files: logged, skipped, batch summary updated
  • OCR engine failures: page marked for retry with alternate engine
  • Empty OCR output: quality flag set, page routed to review queue
  • Classification confidence below threshold: falls back to VLM, then to unknown
  • Qdrant unavailable: embeddings buffered to disk, retried