ADR-002: Qdrant for Vector Storage
Status: Accepted
Date: 2026-06-30
Context
The pipeline needs a vector database to store document chunk embeddings and support semantic similarity search with metadata filtering. Requirements:
- ANN (approximate nearest neighbor) search over embeddings
- Metadata filtering (by document type, date range, people, organizations, locations)
- Payload storage for chunk text and source references
- Self-hosted on Snuffletron
Alternatives Considered
| Option | Pros | Cons |
|---|---|---|
| Qdrant | Metadata filtering, payload storage, Rust (fast), REST API, self-hosted | Single purpose, separate service to maintain |
| Chroma | Simple, Python-native, embedded mode | Weaker filtering, less proven at scale |
| Weaviate | Full-featured, GraphQL | Heavier, more complex to operate |
| pgvector | Leverages existing PostgreSQL | Filtering is SQL, ANN performance varies |
| FAISS (no DB) | Fastest ANN | No metadata filtering, no persistence, file management burden |
Decision
Use Qdrant. It provides the best balance of metadata filtering, payload storage, performance, and operational simplicity for this use case.
Consequences
- Positive: Metadata filtering enables precise queries ("church records from 1898 mentioning Newington").
- Positive: Payload storage means chunk text lives alongside embeddings — no separate lookup.
- Positive: REST API simplifies integration from Python pipeline code.
- Negative: Separate service to install and maintain on Snuffletron. Mitigated by Qdrant's simple deployment (single binary or Docker container).
- Negative: Vendor dependency. Mitigated by Qdrant's open-source license and standard embedding format (vectors are portable).