Constraints

Hard Boundaries

No single OCR engine for everything. The pipeline must support multiple OCR backends and allow routing by document type without changing downstream stages.

Historical documents only. The system is designed for archival materials with inconsistent formats, mixed print/handwriting, and variable scan quality. Modern, cleanly-formatted documents are out of scope.

Quality before quantity. Low-quality OCR text must not enter the vector database without review. A page that fails quality validation is held back, not silently indexed.

Canonical JSON as interface. All OCR output is normalized to the canonical format. Downstream stages never consume raw OCR engine output directly.

Citations required. Every RAG answer must reference the source documents and pages that support it. Ungrounded answers are not acceptable.

Provenance preserved. Source path, batch ID, page number, and processing metadata follow every document through the entire pipeline.

Python. Implementation language for the pipeline. OCR engines and ML models may use their native languages but integration code is Python.

Qdrant for vector storage. Not a general-purpose database. The system uses Qdrant for embeddings and metadata-filtered search.

Deliberate Limitations (v1)

• No real-time processing — batch-oriented pipeline
• No handwriting recognition perfection — acceptable error rates TBD per document type
• No knowledge graph — semantic chunking and vector search only
• No custom OCR model training — evaluate and use existing engines