RAG explained: retrieval plus LLM, but no guarantee of accuracy

The explainer frames RAG as a practical response to a basic limitation of large language models: they are trained on large datasets but do not know a company's latest manuals, internal rules, or customer information, and they cannot automatically learn information added after training. RAG addresses this by searching external data before the model answers. A concrete example — an internal rule stating expense reports are due on the 25th of each month — shows how relevant text is retrieved and handed to the LLM.

The mechanics divide into preparation and answering. Documents are split into chunks, converted into vectors by an Embedding model, and stored. When a user asks a question, the question is also vectorized, compared against stored vectors, and the closest passages are passed to the LLM along with the question. RAG is not limited to vector search; keyword search and hybrid methods also count, so equating RAG with Embedding-based search understates its scope.

The cautionary section is the most business-relevant. RAG can fail when retrieval finds nothing useful or when the LLM misreads correct material and adds unsupported specifics — an example is an AI answering "by 5 p.m. on the 25th" when the manual only says the 25th. Suggested safeguards include instructing the model to say when information is unavailable, showing source documents, and adding human review for important decisions. Building RAG requires design across data ingestion, search, permission management, evaluation, and operations — not just choosing a model.