RAG vs Graph RAG vs Agentic RAG: Choosing the Right Retrieval Architecture

A knowledge base proposal that only says “RAG” hides the architecture decision. The word covers three designs, and the one you pick decides whether the system answers the questions your team asks or only the ones it was built to retrieve.

The short answer: standard RAG fits direct lookups, where the answer lives in one passage. Graph RAG fits relationship questions, where the answer has to be assembled from facts connected across several documents. Agentic RAG fits work where the right sources and tools change from one question to the next. Matching the design to your questions is a business decision, and you can make it before any engineering starts.

Standard RAG: direct lookup

Mechanically: documents get split into chunks, a question comes in, the system retrieves the chunks most similar to the question, and the model composes an answer from them.

This wins when the answer lives in one place. “What’s the refund window on annual plans?” “What ports does the installer need?” “What does travel policy say about booking class?” Most support knowledge-base questions are this shape. So are compliance and policy lookups, where the requirement that matters is citation: the system points at the clause instead of improvising.

Where it breaks: similarity search retrieves passages that resemble the question. Nothing in the design connects facts spread across documents. Ask it something that requires stitching three sources together and it answers from whichever fragment scored highest.

Graph RAG: relationship traversal

Graph RAG adds a preprocessing step. The system extracts entities (services, products, contracts, teams) and the relationships between them into a graph, then traverses that graph at query time.

Take a hypothetical. An engineer asks whether checkout is affected by an upcoming maintenance window. No single document answers it: one says checkout depends on the payments API, another says the payments API runs on a particular cluster, a third says that cluster is scheduled for maintenance. Similarity search grabs one fragment and misses the chain. Graph traversal follows the chain and answers the question that was asked, as long as the graph is current.

Service dependency questions are the obvious case. The same shape shows up wherever one change has to be traced through records written separately: which contracts reference a feature you’re deprecating, which policies cite a regulation that just changed, which teams touch a given dataset.

The cost, stated plainly: someone maintains the extraction pipeline, and the graph is only as good as its last refresh. A stale graph gives confident wrong answers about relationships, which is worse than “I don’t know.” Take this on when relationship questions carry real money or real risk. Skip it when they don’t.

Agentic RAG: dynamic multi-source work

The first two designs fix the retrieval path at build time. Agentic RAG puts an LLM agent in the loop that decides, per query, which sources to consult, which tools to call, and in what order.

You need this when the right sources depend on the question. “Give me the state of a priority account before my 2pm” pulls from CRM, recent tickets, billing, and last quarter’s QBR notes. The next question needs a different combination, and a fixed pipeline covers one of them by design and the other by accident.

The bill: every routing decision adds latency and inference cost. Agents pick wrong tools, loop, or stop early. And an agent with access to billing and CRM data needs governance from day one: scoped permissions, a log of which sources it consulted, and evals on routing as well as answers. When query diversity demands this design, it earns its cost. When a fixed pipeline would have done the job, it’s an expensive liability.

Classify your questions, then pick

The table only does work once it holds your questions. Here is a filled one, illustrative only: an internal operations knowledge base at a 200-person software company, 24 questions pulled from the ticket queue and the help channel, invented counts, no client behind it.

The decision writes itself. Ship standard RAG with citations for the seventeen. Hold the graph until someone can say what a wrong answer about contract exposure costs. Schedule the agentic layer after the lookups have earned the users’ trust. One afternoon of sorting, and the proposal that said “RAG” now says which RAG and why.

The working sequence, for your own corpus:

1. Collect 20 to 30 real questions

   Pull actual phrasing from tickets and chat. Idealized queries flatter every architecture.

2. Sort them into three buckets

   Lookup, relationship, multi-source. The counts are the argument.

3. Ship the cheapest design that covers the dominant bucket

   If the clear majority are direct lookups, standard RAG ships first and the rest gets a later phase.

4. Turn the list into your eval suite before launch

   Thirty questions with known-good answers is a regression suite. Without one, you learn about degradation from the users who quietly stopped trusting the system.

One caution. Retrieval techniques attract dramatic efficiency claims, usually without enough context to check. Treat any multiplier you read as unverified until you have reproduced it on your own corpus with your own questions. Your question inventory is the benchmark that matters.

Where Mercury fits

Mercury starts AI knowledge systems from this exercise: question inventory first, architecture second, and an eval suite built from those questions before launch. The sequence holds whether the work is an internal knowledge base, a support-deflection workflow, or a governed agent with scoped access to business systems. If the proposals on your desk all just say “RAG,” bring us your hardest 20 real questions. You get them back sorted, with the architecture each bucket needs, the cost it carries, and the eval set that tells you whether the build worked.