Town Records
NAMC testbed AI project. OCR pipeline for processing scanned Town Records archive documents into structured data suitable for semantic search, Retrieval-Augmented Generation (RAG), and document intelligence.
Current Status
99/99 tests passing. Production-ready.
- C99 query engine with merge-join executor (names, dates, subjects indexes)
- Python pipeline: 10 stages built (ingest through RAG search)
- Bridge: auto-detecting page JSON → index converter
- Qdrant: embedded chunks with rich metadata payloads
- English-to-query DSL translation via LLM
- Heavy hardware profile: full Qwen ensemble on 96GB Blackwell
See roadmap.md for detailed status.
What It Does
Ingests scanned historical documents (books, letters, invoices, checks, contracts, church records, meeting minutes, ledgers, school memorandums) and produces:
- Searchable text with layout structure preserved
- Canonical JSON documents with metadata and provenance
- Semantic chunks indexed in a Qdrant vector database
- Natural language Q&A over the archive with source citations
- Fast exact metadata lookups via C query engine (names, dates, subjects)
- Self-improving OCR via nightly refinement loop
Quick Start
git clone snuffletron:town-records
cd town-records
make verify # builds, runs 99 tests, tests full pipeline
Pipeline
Scanned Documents → Image Ingestion → Quality Assessment → OCR Engine
→ Layout Extraction → Canonical JSON → Classification → Metadata
→ Quality Validation → Semantic Chunking → Embeddings → Qdrant → RAG/LLM Search
Documents
| Document | Path |
|---|---|
| Architecture | architecture.md |
| Concept | concept.md |
| Constraints | constraints.md |
| ADR-001: Canonical JSON as Pipeline Interface | decisions/adr-001.md |
| ADR-002: Qdrant for Vector Storage | decisions/adr-002.md |
| Architecture Decision Records | decisions/index.md |
| Design | design.md |
| Docs Pipeline | docs-pipeline.md |
| Implementation | implementation.md |
| Requirements | requirements.md |
| Roadmap | roadmap.md |
| Testing | testing/index.md |
| Test Cases | testing/test-cases.md |
| Test Plan | testing/test-plan.md |
| Town Records OCR Pipeline | town_records_ocr_workflow.md |