01
Upload
Drop a PDF in the React UI. The backend hashes content, rejects duplicates with HTTP 409, and parses structure with Docling.
Research case study
Local RAG Engine
Offline PDF Knowledge Copilot
For teams that cannot upload contracts and policy PDFs to a cloud LLM — natural-language Q&A with verifiable citations, entirely on the machine that holds the documents.
FastAPIChromaDBSQLite FTS5llama.cppReact
- Role
- AI Systems + Full Stack Engineer
- Domain
- Private Document Intelligence / RAG
- Users
- Open source · local workflows
- Market
- Colombia
- Stack
- Hybrid RRF · Chroma · FTS5 · llama.cpp · pytest
Product demo
Problem
Sensitive PDFs need cited answers — but cloud RAG sends your documents and questions to someone else's API.
Legal, compliance, and policy teams work from contracts, manuals, and regulatory PDFs that cannot be uploaded to OpenAI or similar services. Keyword search fails on paraphrased questions; vector-only RAG misses exact article numbers and defined terms.
The privacy gap
Cloud convenience vs. on-machine control — many document workflows need the second without giving up citations.
The product challenge is trust and speed: people need answers they can defend in an audit, from documents that legally or contractually cannot leave the building. Keyword search breaks on how people actually ask questions; a generic chatbot cannot cite the manual.
I built this as the on-machine counterpart to my cloud Enterprise Knowledge Copilot — same document Q&A job, different constraint: privacy and data residency come first.
See the constraintThen see the local engine
Solution
Turn sensitive PDF libraries into a searchable, cited knowledge layer — on the machine that already holds them.
Upload policy and contract PDFs, ask in natural language, and get answers tied to page-level evidence. Ingestion is duplicate-safe; every response shows which chunks supported it.
02
Index
Chunks embed locally into Chroma; headings and text land in SQLite FTS5. The document library shows indexed files with metadata.
03
Ask
A natural-language question triggers hybrid retrieval — vector + lexical fusion with Spanish stopword filtering and heading boost.
04
Cite
llama.cpp generates from selected chunks only. The UI shows the answer with document/chunk citations you can verify.
What the product delivers
Auditable answers
Every response includes citations to source chunks — document, section, and page — so compliance and legal teams can verify claims.
Finds the right section
Hybrid search combines meaning and exact terms, so paraphrased questions and precise references (article numbers, defined terms) both reach the right evidence.
Safe to re-upload
Content-hash deduplication stops the index from filling with duplicate PDFs — important when teams share the same manual across folders.
Honest when unsure
When the corpus does not contain an answer, the system says so instead of inventing policy — critical for HR, legal, and compliance workflows.
Stays on your infrastructure
PDFs, indexes, and inference run locally. No default path sends document content to a hosted LLM or embedding API.
Built to improve over time
Stage-level timings, golden-set evaluation, and automated tests make retrieval quality something you can regress — not guess after each change.
Impact
Faster policy answers without trading away data control.
The outcome that matters: teams working from sensitive PDFs can ask questions in plain language, trust the citations, and keep documents on infrastructure they control.
01 - Who it serves
Legal, compliance, HR, and operations teams with contracts, manuals, and regulatory PDFs that cannot go to a cloud AI vendor.
02 - What changes
An 80-page policy manual becomes queryable in minutes — with section-level citations — instead of manual search or risky upload to ChatGPT.
03 - Why it exists
Pairs with the cloud Knowledge Copilot in the portfolio: same knowledge-worker problem, different constraint when data residency and auditability require an on-machine path.
Supporting signals
On-machine
Data residency
PDFs, indexes, and default inference stay on the user's machine — no document egress to a hosted LLM in the standard path.
1.8 h/day
Search waste benchmark
McKinsey's estimate for time knowledge workers spend searching or gathering internal information — the job this product targets.
Cited by default
Answer traceability
Responses include source chunks and references so reviewers can verify claims — not a black-box summary.
Sensitive PDFs cannot use cloud RAG
A product-shaped local engine that complements the cloud Knowledge Copilot when privacy and residency are non-negotiable.
Architecture
How the local RAG engine is built.
Hexagonal service boundaries, dual stores, versioned API contracts, and instrumentation on every query path.
RAG Pipeline
Ingestion path into dual stores; query path through hybrid RRF retrieval, rerank, and local generation with observability.
Decisions
Lessons
Proven
What held up in implementation and tuning.
- Hybrid RRF beat vector-only on manuals with exact terminology and named sections.
- Heading boost and FTS heading index were cheap precision wins after the Jul 2026 retrieval pass.
- Hexagonal ports made ingestion dedup and path-safety tests fast without real PDF parsers.
Evolving
What the next iteration targets.
- Desktop packaging with Electron or Tauri for single-install distribution.
- Larger golden set and retrieval profiles per document type.
- Multi-user ACL when moving beyond single-machine scope.
Transfer
What changed my engineering judgment.
- RAG quality is mostly retrieval — citations and trust matter more than model branding.
- Cloud EKA and Local RAG are complementary when the trust boundary is explicit to buyers and hiring managers.
- Golden-set eval pays for itself the first time ranking logic changes.
Next step
Need on-machine document Q&A with citations?
Open the repo, run the local stack, or get in touch to discuss how this fits next to a cloud knowledge copilot in your organization.
Keep exploring