RAG Explained: How AI Works With Your Own Documents — Instead of Guessing

The most common disappointment when a company first encounters artificial intelligence sounds like this: “The thing makes stuff up.” Someone asked about their own delivery terms, a product’s warranty period, or the content of a framework agreement — and got a fluently worded, convincing-sounding, completely wrong answer.

This isn’t a malfunction. It’s the logical consequence of how a language model works. It has learned to generate a probable continuation from the start of a text. If it doesn’t know your warranty terms, it produces the most probable wording — that is, whatever is typically found in comparable documents. For an answer to a customer, that’s useless.

The usual way out is called Retrieval Augmented Generation, RAG for short. The term sounds more technical than the principle actually is.

The principle in one sentence

Instead of letting the model answer with what it thinks it knows, the system first searches your own documents for the relevant passages and presents them to the model together with the question. The model then formulates the answer using only this material.

The analogy that makes this clearest for most people: you’re not asking someone who has memorized your company by heart. You’re asking an attentive new colleague, and before every question you place the three relevant pages from the file folder on the table in front of them. They’re good at reading and writing — they don’t need to know anything.

What happens technically

Preparing the documents

First, your documents are read in: PDFs, Word files, presentations, intranet pages, emails, spreadsheets. They’re broken down into manageable sections, because a 200-page manual as a whole doesn’t help anyone. This breakdown matters more than it sounds: split it in the wrong places and you sever the context — a table loses its heading.

Converting into vectors

Each section is translated into a string of numbers that captures its meaning. Texts with similar content get similar number strings. The advantage over classic keyword search: the question “How long can I return a device?” also finds a section that talks about a “right of withdrawal period,” even though not a single word matches.

Search and answer

When a question comes in, it too is translated into a string of numbers, the most similar sections are retrieved — usually a handful — and passed to the language model together with the question. Along with this comes an instruction along the lines of: answer only based on these excerpts; if the information isn’t there, say so.

Good systems also state which documents the answer came from. This source citation isn’t a nice extra — it’s the actual value: it makes the answer verifiable.

What RAG solves — and what it doesn’t

RAG substantially lowers the risk of fabricated content, because the model works from material that’s put in front of it. It ensures that new documents take effect immediately, without a model needing to be retrained. And it allows access rights to be taken into account — more on that shortly.

What RAG doesn’t do: it doesn’t make bad documents good. If your file storage has three versions of the same price list and none is marked as the valid one, the system will find one of them — possibly the wrong one. Nor does it answer questions that require analysis across many documents. “How many quotes did we send to customers in Bavaria last quarter?” is a database question, not a search question. That requires a connection to the relevant business system.

The four places it typically snags in practice

1. The quality of your file storage

This is by far the most common reason for unsatisfying results. File storage that has grown over time contains drafts alongside final versions, outdated prices, duplicate files named things like “final_v3_new.” A person can tell from context what’s valid. A search system can’t.

The good news: you don’t have to clean up the entire archive. It’s enough to define a clean area for the first use case — say, the approved product documents and the current price list — and connect only that. Further areas can be added once they’re in order.

2. Permissions

If an assistant searches your documents, it has to follow the same rules a person would. A warehouse employee shouldn’t be able to find the salary list sitting in an HR folder through the assistant. Technically, that means permissions have to be checked on every single request, not just when the result is displayed.

Systems that simply dump everything into one shared search index are a serious problem here. This question should be at the very top of every vendor comparison — and the answer shouldn’t be “we handle that through the prompt.”

3. Tables, scans, and legacy formats

Plain running text is easy. It gets difficult with tables, multi-column layouts, diagrams, and scanned documents. A price table can lose the alignment between row and column during careless extraction — and the result is a price attached to the wrong item. Anyone who works a lot with such material should test exactly these cases, not the easy ones.

4. The expectation that everything can be connected at once

The wish to “connect everything at once” regularly leads to a system that’s mediocre at everything. An assistant that reliably handles product documents and references for sales is more valuable than one that does a little of everything.

What you should prepare

If you’re planning such a connection, four preparatory steps will help more than any technology decision.

Define the use case. Who asks what, how often, and what does that person do with the answer?

Identify the sources. Which folders, which systems, which document types. And who is responsible for keeping them up to date.

Clarify the permissions model. Do clean groups and permissions already exist? If not, that’s the first task — independent of AI.

Collect real test questions. Twenty to thirty questions that are actually asked, along with the correct answer. That’s your benchmark. Without this list, every evaluation stays a matter of feeling.

The difference in everyday use

A general-purpose language model is an excellent drafting assistant. It helps with rewriting, shortening, translating, and structuring. A system connected to your documents is something else: it answers questions about your business, with a source citation, within the requester’s permissions.

The effort for the second step is smaller than many expect — provided the file storage is organized in one area and the permissions are correct. This is usually exactly where our work begins: we look at your existing structure, say honestly what needs tidying up first, and then connect the area that delivers the greatest benefit.

How to recognize a good answer

A simple yardstick helps when evaluating a RAG system. A useful answer meets four conditions: it’s factually correct, it’s complete enough for the task, it names its source, and it says honestly when something isn’t in the material.

The last condition is the most commonly underestimated, and it’s the most important one. A system that responds “I can’t find anything on that in the connected documents” when information is missing is far more valuable in production than one with a higher hit rate that formulates something plausible when in doubt. Test this specifically by asking questions whose answer is definitely not in the corpus.

The effort required to stay current

A point that rarely shows up in proposals: a connected corpus ages. New documents get added, old ones get replaced, prices change. If no one is responsible for it, answer quality declines gradually — and no one notices, because wrong answers sound just as confident as correct ones.

So settle two things before you start: who is responsible for the content of the connected area, and at what interval it’s reviewed. For a manageable corpus, a one-hour check every quarter is enough.

Frequently asked questions

How many documents does it take for this to be worthwhile?

Volume is rarely the deciding factor. What matters is how often something is searched for and how costly it is not to find it. A corpus of two hundred well-maintained documents that’s searched daily is a better starting point than twenty thousand disorganized files.

Does our file storage need to be tidied up first?

Only the part you connect. Define a clean area and start there. Trying to put everything in order first is the surest way to never get started.

Are our documents transferred to the provider?

When generating an answer, the retrieved text excerpts are passed to the language model — that’s technically necessary. What matters, therefore, is where that model runs, whether content is stored, and what the contract says about it. Ask about this explicitly.

Can such a system also calculate or analyze data?

Only to a limited extent. Questions like “How many quotes went to Bavaria last quarter?” are database questions. That requires a connection to the relevant business system, not to the document storage.

Why preparation determines quality

Between “connecting documents” and “getting good answers” lies a step that rarely appears in a proposal and often makes the difference: how the documents are broken down and enriched.

Chunking. A document is split into sections, because a whole manual as an answer basis would be useless. Cut strictly by character count, and it tears connections apart: a table loses its heading, a paragraph starts mid-sentence, a condition gets separated from its exception. Good systems cut along the structure — at headings, paragraphs, table boundaries — and let sections overlap slightly, so nothing gets lost at the seams.

Enrichment. Every section should carry along where it came from: document title, chapter, date, version, ownership. This has two effects. The answer can carry a useful source citation — not “according to a document,” but “according to the service guide, section 4, as of March.” And the system can favor the more recent hit when there are several matches, instead of guessing.

This, incidentally, is exactly why a date in the document or its file properties is worth more than it looks. If three versions of a price list exist and none carries a reliable date, no system can pick the right one.

Evaluating your test list correctly

When you work through your twenty to thirty test questions, the hit rate is the least interesting number. It gets revealing once you sort the misses by cause.

The document was missing from scope. Not a system problem, but a scoping decision. Easy to fix.

The document was outdated or existed in duplicate. A storage problem. Also fixable, but it shows where things need tidying up.

The format was extracted poorly. Typical with tables, multi-column layouts, and scans. This is where comparing several vendors pays off, because quality varies noticeably.

The wrong passage was retrieved. A problem with chunking or search — this is the area where the technical solution really matters.

The passage was correct, the answer was wrong. Only here is the model actually the cause. In our experience, this is the smallest group.

This sorting is why we almost always start projects with a stocktaking rather than a model selection: four out of five causes lie outside the model.

When answers get worse over time

An effect many don’t expect: a system that performed convincingly in testing delivers weaker answers six months later — without anything technical having changed.

The cause is almost always the corpus. New documents were added, old ones weren’t removed, versions were appended instead of replaced. What was a well-maintained area during testing has grown back into a sprawling archive.

The only remedy is a fixed owner and a recurring date. A one-hour check every quarter, where the responsible person verifies whether the connected content is still current, is entirely sufficient for a manageable corpus.