Implementation

Philosophy

Development proceeds incrementally. Each stage results in a runnable pipeline. At no point should the project require a complete rewrite to continue development. Stages are ordered so each one builds on the output of the previous.

Stages

Stage 1: Ingestion + Canonical JSON Skeleton — DONE

  • Walk source directory, identify supported files
  • Assign batch ID and page number
  • Generate minimal canonical JSON (source + page sections)
  • PDF splitting via pdf2image or similar
  • CLI: tr-ingest /path/to/archive --output /path/to/canonical/

Stage 2: Image Quality Assessment — DONE

  • Implement blur, contrast, skew, noise, blank page detection
  • Write quality.image_quality to canonical JSON
  • Summary report: counts by quality level
  • Graceful OpenCV fallback (returns "unknown" when cv2 unavailable)

Stage 3: OCR Engine Adapter + OCR Execution — DONE

  • Multi-engine adapter interface (Docling, Surya, Qwen3-VL)
  • Auto-routing by page characteristics (handwriting, tables, image quality)
  • Repair-first flow: primary OCR → score → repair → fallback
  • Ensemble word-level voting across multiple Qwen-VL models
  • Per-document-type prompts (ledger, meeting minutes, church record)
  • Process-wide model cache via ModelManager
  • --turbo flag for parallel page processing on heavy hardware
  • CLI: tr-ocr /path/to/canonical/

Stage 4: Engine Evaluation Framework — DONE

  • Multi-engine comparison framework
  • Score by: text accuracy, layout fidelity, table quality, speed
  • Produce evaluation report
  • CLI: tr-evaluate /path/to/canonical/

Stage 5: Document Classification — DONE

  • Rule-based classifier (keyword signals per document type)
  • VLM fallback stub for low-confidence cases
  • Writes classification to canonical JSON
  • CLI: tr-classify /path/to/canonical/

Stage 6: Metadata Extraction — DONE

  • Implement date detection (4 regex formats, OCR century correction)
  • Name extraction (title patterns, capitalization, geographic filtering)
  • Organization, location, monetary amount, and topic extraction
  • Writes metadata to canonical JSON
  • CLI: tr-metadata /path/to/canonical/

Stage 7: Quality Validation — DONE

  • Combines image quality + OCR confidence + text coherence
  • Sets quality.safe_for_rag and needs_review flags
  • Preserves image-quality metadata from earlier stages
  • Generates review queue for flagged pages
  • CLI: tr-validate /path/to/canonical/

Stage 8: Semantic Chunking — DONE

  • Per-document-type chunking strategies (meeting minutes by article, church records by date-anchored entries, ledgers by row)
  • Default: paragraph-boundary splitting
  • Each chunk includes chunk_id, text, document_type, page_number, source_path, metadata, bbox_refs
  • CLI: tr-chunk /path/to/canonical/

Stage 9: Embedding + Qdrant — DONE

  • Configurable embedding model (nomic-embed-text, Qwen3-Embedding)
  • Qdrant upsert with metadata payloads (document type, page number, source path, dates, topics)
  • Filtered search with optional document type filter
  • Heavy hardware: defaults to Qwen3-Embedding-8B (4096-dim)
  • CLI: tr-embed /path/to/canonical/

Stage 10: RAG Query Interface — IN PROGRESS

  • Query embedding → Qdrant retrieval → LLM answer
  • Citation format with source path and page number
  • Quality warnings on answers from low-confidence pages
  • English-to-DSL translation via LLM
  • Hybrid search: C engine hard queries + Qdrant semantic search
  • CLI: tr-search "documents about agriculture from 1930 to 1940"

Query Engine

A C99 query engine provides fast exact lookups on structured metadata:

  • Indexes: names (spec output #4), dates (spec output #5), subjects (spec output #6)
  • Operators: MATCH, CONTAINS, RANGE, INTERSECT, UNION, SORT, SLICE
  • Execution: merge-join on sorted (doc_file, page) keys with O(n) complexity
  • Wildcards: page=0 entries match any page from the same document (doc-level indexes)
  • Lazy loading: only indexes referenced by the query plan are loaded
  • CLI: tr-query query.json indexes/

99/99 tests passing. Build with make. Verify with make verify.

Bridge

The bridge connects OCR pipeline output to C engine indexes:

  • Input: raw pipeline output (page_.json), bridge export (.page.json), or synthetic test data
  • Auto-detection: text, page numbers, and source filenames resolved from any JSON key
  • Output: names.json, dates.json, subjects.json in spec format
  • CLI: tr-bridge /path/to/page-json/ --output indexes/

Also available: tr-convert (canonical JSON to bridge format), tr-incremental (per-page index building with fcntl locking), tr-stream (priority-sorted index building).

Refinement Loop

Nightly re-OCR of low-confidence pages for self-improving archives:

  • Scans canonical JSON for ocr_primary_score < 0.7 or needs_review == true
  • Re-runs through full Qwen ensemble
  • Replaces pages where coherence improves >0.1
  • Flags pages failing 3 nightly attempts for human review
  • CLI: tr-refine /path/to/canonical/ --ensemble