RAG explained: how an AI answers from your own documents

Retrieval-augmented generation finds the right passages in your documents and hands them to the model with each question. How it works, costs and fails.

By Sohail Hussain17 November 20266 min read

RAG, short for retrieval-augmented generation, is how an AI answers from your documents instead of from what it learned in training. When a question arrives, the system searches your content for the few passages most likely to contain the answer, places them in the prompt next to the question, and instructs the model to answer from them. The model does not learn or store your documents. It reads the relevant pages each time, the way a new employee looks something up before replying.

That is why RAG sits behind most business chatbots and internal assistants, including the ones we build in our AI agents and chatbots work. This post explains how it works, when you can skip it, what decides whether its answers are good, and where it fails. For where RAG fits in the bigger picture, see AI agents for business.

How does RAG work, step by step?

RAG has two halves: preparing your content once, and answering each question.

Preparing the content (indexing).

  1. Collect the documents: help pages, policies, manuals, product sheets, PDFs.
  2. Split each one into chunks of a few hundred words, ideally along headings so each chunk covers one topic.
  3. Turn each chunk into an embedding: a list of numbers that represents its meaning, produced by an embedding model. Chunks about similar things get similar numbers.
  4. Store the chunks, their embeddings and metadata (source, title, date, who may see it) in a searchable index.

Answering a question (retrieval and generation).

  1. Turn the question into an embedding with the same model.
  2. Find the chunks whose embeddings are closest to it. Most good systems combine this with ordinary keyword search, which catches product codes, names and exact phrases that meaning-based search misses.
  3. Optionally rerank the results with a second model that scores how well each chunk answers the question.
  4. Send the top few chunks to the language model with the question and instructions: answer from these passages, cite them, and say so if the answer is not there.

When a document changes, you re-index that document. The next question uses the new version.

Why not just put all your documents in the prompt?

Sometimes you should. As of October 2026, the main models accept about a million tokens of input: Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5 take 1,000,000, GPT-6.1 Sol takes 1,050,000 and Gemini 4 Argon 1,000,000. That is roughly 1,500 pages of text. If your content is a few dozen pages, sending all of it with every question is simpler than building retrieval, and prompt caching makes the repeated part cheap. The context window checker tells you whether your documents fit in each model with room for an answer.

Cost and quality decide where the line falls. Take 50,000 tokens of content (roughly 75 pages) and 10,000 questions a month on Claude Sonnet 5.5, at US$2 per million input tokens:

ApproachInput tokens per questionInput cost per questionPer month at 10,000 questions
All content in the prompt, uncachedabout 50,000about US$0.10about US$1,000
All content in the prompt, cachedabout 50,000, mostly read from cacheabout US$0.005about US$50, plus cache writes
RAG, top five chunksabout 4,000about US$0.008about US$80

The middle row surprises people. On Claude Sonnet 5.5, a cache read costs US$0.10 per million tokens, a twentieth of the normal input price, so at this size a cached prompt can cost less than retrieval. The catch is that the cache has to stay warm: entries last five minutes by default on Anthropic's API, as its prompt caching documentation explains, and every expiry means paying 1.25 times the normal price to write the content again. Quiet hours and weekends add those writes back. And at 500,000 tokens of content, every figure in the first two rows is ten times larger while RAG stays the same.

Quality matters too. A short prompt with five relevant passages is easier for the model to answer from, and easier for you to check, than a long one where the answer is buried on page 312. Retrieval also lets you filter by permissions, so a user never receives a passage they are not allowed to see.

The rule we use: under about 50 pages, with no per-user permissions, put it all in the prompt with caching and test whether answers are good. Above a hundred pages or so, or when different users may see different documents, use RAG.

What decides whether RAG answers well?

The content. RAG retrieves what you have. If two documents disagree, it will retrieve whichever scores higher that day. If the answer is not written down, no retrieval setting will find it. Most of the improvement in a RAG system comes from fixing documents.

The chunks. Chunks that are too small lose the context needed to answer; chunks that are too large blur several topics into one embedding and match vaguely. A few hundred tokens with a small overlap is a common starting point. Keep the document title and section heading with each chunk so it still makes sense alone. The RAG chunk size calculator shows how many chunks your documents produce at different settings, and what embedding them costs.

The search. Combine meaning-based and keyword search. Use metadata filters for product, region, language and date. Retrieve more candidates than you need and rerank them.

The instructions. Tell the model to answer only from the passages, to cite which passage it used, and to say plainly when the passages do not contain the answer. How to stop an AI chatbot making things up covers this in detail.

Measuring retrieval separately. Before judging answers, check that the right passage appears in the top results for a set of real questions. If retrieval misses, the answer will be wrong however good the model is.

What does RAG cost?

Less than people expect for the retrieval part. Embedding is cheap: as of October 2026, OpenAI lists text-embedding-3-small at US$0.02 per million tokens on its pricing page. A thousand documents of 2,500 words each is about 3.3 million tokens, or about seven cents to embed. Anthropic does not make an embedding model; its documentation points to third-party providers.

Storage is a few dollars to a hundred dollars a month at small business scale, or nothing extra if you add pgvector to a PostgreSQL database you already run. The running cost that matters is the model reading the retrieved passages with every question, covered in how to estimate AI API costs. The build cost that matters is preparing and maintaining the content; how much an AI chatbot costs breaks that down.

Where does RAG go wrong?

The question uses different words from the document. A customer asks about "sending it back", the policy says "returns". Meaning-based search usually handles this; product codes and abbreviations often need keyword search alongside it.

The answer is spread across many documents. "Which of our policies mention overtime?" needs every relevant passage, not the top five. RAG is built for finding, not counting or listing. Questions like this need a structured query over your data or a different design.

Tables and scanned PDFs. Text extraction from complex PDFs often scrambles tables. Check what your pipeline actually extracted before indexing it.

A stale index. If a document changes and nothing re-indexes it, the bot answers from the old version. Sync on change, not on a monthly schedule.

Permission leaks. In an internal assistant, retrieval must filter by what the person asking is allowed to see, before the model receives anything. Internal knowledge assistant: a build guide covers how to set that up.

Was this guide useful?

Questions

Frequently asked.

Does RAG train the AI model on my documents?

No. The model's weights never change. Your documents sit in your own index, and the relevant passages are sent with each question, read, and discarded. Delete a document from the index and the AI stops using it on the next question, which is one of the main reasons to prefer RAG over training.

Do I need a vector database for RAG?

Not necessarily. If you already run PostgreSQL, the pgvector extension stores embeddings next to your other data. For a few hundred documents, an in-memory index is enough. A dedicated vector database earns its place at millions of chunks or when you need its filtering and scaling features.

What is the difference between RAG and fine-tuning?

RAG gives the model facts to read at question time. Fine-tuning adjusts the model itself, which is better at changing tone, format or behaviour than at teaching it facts reliably. For business knowledge that changes, such as prices, policies and stock, RAG is the right tool and fine-tuning is usually not needed.

Sohail Hussain

Founder, MSH Technologies

Sohail has designed and built websites, web apps and AI products for over 12 years, for clients from Singapore-based industrial groups to early-stage SaaS startups. He writes about what actually worked.

Keep reading

Related guides.

All posts

Your next move

Like what you see? Tell us about it.

A process that takes too long, a website that undersells you, or an idea ready to take shape. Tell us where you are and we will suggest a useful next step.

India · Working with clients worldwide

Sohail, Founder, replies within 1 working day (Mon–Sat, IST). Free and no obligation.