Building a Chatbot Knowledge Base That Actually Answers Questions

Most chatbot projects die in the knowledge base. The prompt gets written in an afternoon, the platform gets picked in a week of comparison, and then somebody has to actually put the business's facts somewhere the bot can find them. That part gets rushed, the bot answers a pricing question wrong on day three, and the client loses faith in the whole thing.

I build knowledge bases for voice agents and chat widgets across a mixed set of clients: a jewelry retailer running a shopping assistant on multiple sites, a UK windows-and-doors company, medical clinics, contractors. The mechanics differ by platform. The failure modes are identical everywhere.

Start by scraping the site, because it takes minutes

When I build a demo agent for a prospect, the first thing I do is point the knowledge base ingester at their website. Most modern platforms will crawl a domain, pull the text, chunk it, and embed it without me writing any code. For a typical small-business site of 20 to 60 pages, that is a few minutes of wall time and I have an agent that can answer questions about their services before I have spoken to them.

That is genuinely useful, and it is also a trap if you stop there.

A scraped site gives you the marketing copy. Marketing copy is written to persuade, not to answer. It says "we offer flexible financing options" where a customer asks "do you take payments and what's the minimum credit score". It has the same three-paragraph company boilerplate in the footer of every page, which means every chunk you retrieve is 40 percent boilerplate. And it is silent on exactly the things customers actually call about: whether you service their zip code, what a typical job costs, how soon you can come out, what happens if they need to reschedule.

So the scrape is the floor, not the ceiling. I use it to get something working the same day, then I do the real work.

The real work: writing down what the front desk knows

The highest-value knowledge base content is not on the website at all. It lives in the head of whoever answers the phone.

I ask for three things from every client:

The last 50 questions customers actually asked. Email threads, text messages, whatever they have. If they cannot produce them, I sit with the phone logs. This is the single best input you can get, because it tells you the distribution of real questions, which is never the distribution you would guess.

The answers to those questions in their words. Not polished. I want "we don't do anything under $300, it's not worth the truck roll" because that is the actual policy. I can soften the phrasing in the prompt. What I cannot do is invent the number.

The list of things they refuse to have the bot say. Every business has some. Do not quote prices for commercial jobs. Do not mention the competitor down the road. Do not confirm a technician's arrival window. These become prohibitions in the system prompt, not knowledge base entries, but you find them during this conversation.

From that I write a set of documents. Not one giant document, and not 400 tiny ones.

Structure: one topic per document, question-shaped headings

Retrieval works by similarity between the user's question and your chunks. So write chunks that look like answers to questions.

What that means in practice:

• One coherent topic per document. "Service area and travel fees" is a document. "About us" is not.
• Lead each section with the question as a heading. "Do you service Will County?" beats "Coverage".
• Keep sections between roughly 100 and 400 words. Shorter and there is not enough context to be useful once retrieved in isolation. Longer and the relevant sentence gets diluted by everything around it.
• Repeat the entity name inside each section. The chunk that says "we are open until 6" retrieves badly. The chunk that says "Northside Dental is open until 6pm on weekdays" retrieves well and reads correctly when it lands in the model's context with no surrounding document.
• Put the answer in the first sentence, then elaborate. If the chunk gets truncated, you lose the elaboration rather than the answer.

That last point matters more than people expect. Retrieved chunks arrive in the model's context stripped of their document, their neighbors, and often their heading. Every chunk has to stand alone. I read my own chunks in isolation and ask whether a stranger could answer the customer from that text and nothing else.

What does not belong in the knowledge base

Three categories I keep out.

Anything volatile at the hour scale. Live availability, current inventory, whether the truck is out. Retrieval is a snapshot of a document; it will confidently serve you yesterday's answer. Volatile facts belong behind a tool call that hits the real system. On the jewelry retailer's assistant, product availability comes from a live lookup, not from the KB, precisely because a stale "yes we have that in stock" is worse than "let me check".

Anything the bot must never get wrong. Legal disclaimers, consent language, medical safety statements. Those go in the prompt as fixed text, or in code, because retrieval is probabilistic. If the right chunk does not surface, the disclaimer silently does not happen. On an SMS assistant I built with a hard consent gate, the gate lives in code, not in a document the model might or might not retrieve.

Behavioral instructions. I see people put "always be polite and ask for the customer's phone number" into a knowledge base article. It will only influence the bot on turns where it happens to be retrieved. Behavior goes in the prompt, facts go in the knowledge base. Keeping that line clean saves you a lot of confusing debugging.

Keeping it fresh, which is where most of them rot

A knowledge base is not a project, it is a maintenance commitment. Here is the process I actually run.

Version everything and date it. Every document gets a "last reviewed" date in the text itself, and I keep the source files in git. When a client says the bot quoted an old price, I need to know what the bot was reading and when it changed. Platform KB editors that offer no history make this painful, which is a real reason to author outside the platform and sync in.

Re-crawl on a schedule, diff the result. For clients whose website is the source of truth for services and hours, I re-ingest monthly and look at what changed. The valuable output is the diff, not the crawl. It catches the case where marketing rewrote the services page and nobody told anyone.

Mine the transcripts weekly. This is the highest-yield habit by a wide margin. I pull every conversation where the bot said some version of "I don't have that in front of me", or where the customer rephrased the same question twice, or where a chat ended without resolution. Each one is either a missing document or a badly written chunk. On new deployments this list is long for two weeks and then drops off sharply. That workflow is the core of chatbot analytics as I use it.

Test retrieval, not just answers. When an answer is wrong, find out whether the right chunk was retrieved and ignored, or never retrieved at all. Those have completely different fixes. If retrieval missed it, rewrite the chunk to match how customers phrase the question. If retrieval hit it and the model still answered wrong, that is a prompt problem. Most platforms will show you the retrieved chunks; use that view, it saves hours.

Prune aggressively. Knowledge bases get worse as they get bigger if the additions are low quality. Duplicate documents compete with each other and split the retrieval signal, so two mediocre versions of the pricing page can be worse than one good one. When I add a document I check whether it overlaps something existing, and I delete rather than accumulate.

A realistic timeline

For a small business, here is what this actually costs in effort.

Stage	Time	Output
Scrape the site	Under an hour	A demo-quality bot that knows the marketing copy
Interview the front desk, write core docs	Half a day	10 to 20 documents covering real questions
Test with 30 real questions, fix retrieval	Half a day	Answers that hold up
Weekly transcript review	30 minutes a week	Gaps closed before customers notice

The first hour gets you 60 percent of the value and 100 percent of the demo. The remaining day is what separates a bot the client keeps from one they quietly turn off after a month. I price the maintenance in, because a knowledge base nobody reviews is a liability that generates confidently wrong answers, and that is worse for the business than no bot at all. There is a related judgment call in when not to use a chatbot.

FAQ

Can I just point the bot at my website and be done? For a demo or an internal test, yes, and it will take you minutes. For production, no. Website copy answers the questions marketing wanted to answer, not the ones customers ask. Expect the scrape to cover roughly half of real inbound questions, and plan a day of writing to cover the rest.

How many documents does a small business need? Usually 10 to 25, each covering one topic in 200 to 800 words. I have run effective knowledge bases with 8 documents. If you are past 100 documents for a single-location small business, you almost certainly have duplication that is hurting retrieval rather than helping it.

Should pricing go in the knowledge base? Price ranges yes, firm quotes no. I put published ranges in the KB and add an explicit prohibition in the prompt against committing to a specific number for a specific job. That way the bot can say "installs typically run between X and Y" and still hand off for a real quote, which is what the business wants anyway.

PDFs or plain text? Plain text or markdown, always, if you have the choice. PDF extraction mangles tables, multi-column layouts, and headers, and you end up with chunks containing page numbers and fragments of two unrelated columns. If a client only has PDFs, I convert and clean them once rather than ingesting them raw.

How do I stop the bot answering from general knowledge instead of the knowledge base? Say so explicitly in the prompt, name the knowledge base as the only acceptable source for business facts, and give the bot a specific sentence to use when it finds nothing. Then verify with transcripts, because models will still occasionally fill gaps from training data. The full set of techniques is in stopping chatbot hallucinations.