Document-Grounded RAG QA Assistant

Overview

MeChatBot is a document-grounded RAG QA assistant that answers questions strictly from a Markdown knowledge base and refuses to answer when the evidence is insufficient. It was built to help engineers and reviewers query legacy documentation without wading through raw files, while keeping every answer traceable back to a source document.

Architecture

Offline knowledge & indexing: Markdown documents (data/docs/*.md) act as the single source of truth. An ingestion pipeline (ingest.py) chunks documents, attaches metadata (doc_id, title, section, timestamps, category), embeds each chunk with OpenAI or Ollama embeddings, and stores them in Chroma, supporting incremental updates so only new documents need to be re-embedded.

Online serving: A Gradio UI collects the question and displays the answer with citations. LangGraph orchestrates the workflow — state management and conditional branching between retrieval paths (Vector RAG via LlamaIndex + Chroma, or Graph RAG via Neo4j). The retrieval layer fetches top-k relevant context, and the LLM layer (local Ollama or production OpenAI API) generates the final grounded response.

Design principles (SELF-RAG & CRAG-inspired)

• Answer strictly from the retrieved context; outside knowledge is prohibited by the prompt.
• Citations are generated only from state.sources, never invented by the LLM — any citation block the model writes on its own is stripped via regex post-processing.
• A minimum similarity threshold (MIN_SIM_SCORE) filters weak retrieval matches to balance precision and recall.
• When evidence is insufficient, the model is instructed to abstain rather than fabricate an answer.

Problems solved

Pain point	Root cause	Status
Unclear source / missing citation	Grounding failure, lack of traceability	Solved
Content not present in the document	Hallucination, generation drift	Partially solved
Missing info in long documents	Retrieval miss	Solved
Answer drifts from provided evidence	Ungrounded answer	Solved
Conflicting answers across document versions	Conflict, ambiguity	Open — planned GraphRAG conflict resolution
Confident but wrong answers	Lack of uncertainty handling	Partially solved (low temperature + abstention)

Results

On a demo question set spanning unclear-source, hallucination, retrieval-miss, and grounding-failure cases, MeChatBot correctly abstained on out-of-scope questions ("When is Eunbi Yoon's birthday?" → cannot answer) while answering in-scope questions with citations (e.g., "What is MeChatBot's main technique?" → LlamaIndex / LangGraph / Chroma).

Key learnings

• Answers only become trustworthy once citations or supporting sources are attached.
• Explicitly stating "insufficient information" is the key defense against hallucination — more effective than trying to generate plausible-sounding content.
• Because the system uses RAG rather than fine-tuning, answer quality degrades immediately when retrieval fails, so retrieval quality is the main lever.
• Providing evidence and actually grounding the answer in it are different problems — the workflow had to be explicitly designed with LangGraph rather than left to prompting alone.

Code

github.com/EunbiYoon/meChatBot

Tech stack

LlamaIndex, Chroma, Neo4j (optional Graph RAG), LangGraph, Gradio, OpenAI API / Ollama