Roadmap

Current status and planned work. See implementation.md for detailed build stages.

Current (2026-07-08)

  • C99 query engine: 99/99 tests passing. Merge-join executor with INTERSECT/UNION/SORT/SLICE on names, dates, and subjects indexes. Page=0 doc-level wildcard for cross-index queries. Lazy loading (only loads indexes referenced by the query plan). snprintf truncation guards. Thread-safety documented.

  • Python pipeline: 10 stages built. Repair-first OCR with multi-engine auto-routing (Docling, Surya, Qwen3-VL). Per-document-type prompts. Ensemble word-level voting for maximum accuracy. Atomic writes with fcntl locking on all stages. Universal page JSON format detection.

  • Bridge: connects pipeline output to C engine indexes. Accepts raw pipeline page JSON, bridge export format, and synthetic test data. Auto-detects text location, page numbering, and scan filenames.

  • Qdrant integration: embeddings with configurable model (nomic-embed-text or Qwen3-Embedding). UUID point IDs. Lazy client initialization.

  • Heavy hardware profile: TOWN_RECORDS_HEAVY=1 enables full Qwen stack resident (72B+8B+2B ensemble, 8B embedding, 32B answer generation).

  • Security: prompt injection hardening, depth limits, stack protector, FORTIFY_SOURCE, PIE, RELRO.

  • Verification: make verify runs full pipeline test on synthetic data. setup.sh handles environment configuration.

Planned

Phase 2: Full Pipeline (Stages 5–8) — COMPLETE

  • Document classification
  • Metadata extraction
  • Quality validation gate
  • Semantic chunking by document type

Phase 3: Search (Stages 9–10) — IN PROGRESS

  • Embedding generation and Qdrant ingestion — DONE
  • RAG query interface with citations — STUBBED (Qdrant search works, LLM answer generation needs ollama)

Phase 4: Refinement

  • Nightly confidence loop (tr_refine.py built, awaiting deployment)
  • Parallel page processing (--turbo flag built)
  • LLM result re-ranking
  • Result enrichment with cross-references and facets