AI Support Chatbots with RAG — Case Study

This is the case study for the custom RAG chatbots I built for customer-support use cases. Each bot pulled from a specific client's knowledge base — documentation, help-center articles, internal runbooks — and answered support questions grounded in that material rather than hallucinating from the model's general training. Multiple bots shipped across different clients, and the support teams that adopted them continued to use them daily after my engagement.

What was the problem?

Generic chatbots — the ones built on a stock LLM with no retrieval layer — have a recurring failure mode: they sound confident about things they do not actually know. For customer support that is a serious problem. A wrong answer about a refund policy, a billing detail, or a product limitation does real damage to trust. Worse, the failure is silent: the user does not know the bot is making things up until they act on the wrong answer.

The pattern across the support teams I worked with was the same. They had real documentation — sometimes years of it — that contained the right answers. The challenge was making that documentation answerable in natural language, in seconds, without first making a human read every page. They did not need an "AI assistant" in the marketing sense; they needed a fast, grounded retrieval layer on top of what they already knew.

RAG — retrieval-augmented generation — is the right shape for that problem. You index the knowledge base, retrieve the relevant chunks for each query, and ask the LLM to answer using those chunks. The model is no longer guessing; it is summarizing material the team already authored.

What was the approach?

I treated each engagement as a tightly scoped build with three phases.

1. Ingest and index. Start with the knowledge base, not the model. Walk the source documents (HTML, PDF, Markdown, sometimes Notion exports), normalize them, chunk them with a strategy that respected document structure, embed the chunks with a current embedding model, and index them in a vector store. The ingestion step was a one-command operation so the index could be rebuilt whenever the source docs changed.
2. Build the retrieval-and-generation loop. Query embedding, vector search, top-k retrieval, prompt assembly with explicit grounding instructions, generation through OpenAI's API, optional re-ranking, response formatting. The LangChain documentation covers the orchestration primitives that made this loop quick to compose.
3. Build the evaluation harness before going to production. A frozen set of real support questions paired with the correct answers — typically 50–200 pairs per client, drawn from actual tickets or FAQ pages. Every change to the system ran against that set. Retrieval-hit and answer-faithfulness were the two metrics that gated merges.

Only after those three phases existed did the bot get wrapped in a UI (web widget, Slack bot, or whatever the client needed) and deployed. The order matters. Most chatbot projects that fail in production fail because the UI shipped before the evaluation harness existed; once silent regressions started slipping in, nobody could tell what was breaking.

What is in the stack?

• Python. The default language for this kind of ML-adjacent work. Mature libraries, good tooling, easy to deploy as a service or as a serverless function. The right call when the loop is data-processing-heavy.
• LangChain. Orchestration for the RAG pipeline — document loaders, text splitters, embedding wrappers, vector store adapters, prompt templates, chains. Not a magic framework; a structured way to compose the same primitives you would otherwise hand-roll, with sensible defaults that get you to a working baseline quickly.
• OpenAI's API. Embeddings (text-embedding-3 family) and generation (the current chat-completion model at the time of the build). Managed API beat self-hosted models on cost, debuggability, and time-to-ship. OpenAI's documentation covers the API contracts.
• Vector store per client. pgvector when the deployment target already had Postgres, a dedicated managed vector DB when scale or latency required it. The choice was a one-liner change in the LangChain configuration, never a rewrite.
• FastAPI for the service layer where the bot was deployed as an HTTP endpoint. Lightweight, async-friendly, typed.
• Evaluation harness in plain Python. Pytest-driven so it ran in CI. The harness was the most valuable artifact in the entire project — more valuable than the bot itself, because the bot would change and the harness was the constraint that kept the changes safe.

The boring parts of the stack are deliberately boring. The interesting work was in the chunking strategy, the prompt structure, and the evaluation harness — the parts that actually move answer quality.

What was the outcome?

Multiple bots shipped across different clients. The support teams that adopted them used them daily. Answer quality on the evaluation sets was strong enough that the bots reduced first-response time on common questions without producing the kind of confident-wrong answers that erode trust.

Qualitative observations:

• Per-client vector indices were worth the small infrastructure cost. Sharing indices to save money added complexity that was not worth the savings.
• The first ingestion pass on a messy knowledge base was always the slow part. Once the ingestion pipeline existed and the documents were normalized, every subsequent improvement was fast.
• Re-indexing on documentation updates was the operational habit clients had to develop. Some made it automatic from their docs repository; others did it manually on a weekly cadence. Both worked.
• The bots did not replace the support team. They replaced the time the team spent answering the same questions repeatedly, freeing senior staff for the questions that actually needed human judgment.

No usage or deflection numbers go on this page that I cannot independently verify. The honest claim is that the bots shipped, ran, and got used.

What did I learn?

The evaluation harness is the project. I will repeat this because it is the single highest-leverage decision in any RAG build. Without a frozen evaluation set, you cannot tell whether a prompt change improved or regressed the system. With one, every change is measurable. Build the harness in week one, not week eight.

Chunking strategy matters more than the model choice. Within reason. Swapping between current generation chat models tends to change answer quality by a small amount; changing chunking from "paragraph blocks" to "semantic chunks that respect document structure" tends to change retrieval quality by a large amount. Spend the time there.

RAG > fine-tuning for almost every support use case. Fine-tuning is expensive, slow to update, and tends to memorize the wrong things. RAG is cheaper, fresher, and more verifiable. The longer discussion lives in RAG vs fine-tuning for chatbots. For most clients I never even propose fine-tuning.

Grounding is a system-design decision, not just a prompt instruction. "Refuse to answer if the retrieved chunks do not support the question" is a prompt instruction. Making the system actually do that consistently requires the evaluation harness, the retrieval quality, and the prompt all to be working together. None of the three is sufficient alone.

For the comparison, see RAG vs fine-tuning for chatbots. For the stack reasoning across AI-first builds, best stack for AI SaaS in 2026. The RAG chatbot service page has scope and pricing for similar engagements; AI automation covers broader AI integration work.

Email hello@ignax.dev if you have a knowledge base and a support team that could use one. Repository style at github.com/ignaxdev.