Before you blame the model for a bad answer, check which evidence it was allowed to see, where it looked, and when that evidence was last updated.
This week in my diary, I am working through how to build an Agentic RAG solution when the answer lives across documents, databases, and business applications, not in one convenient vector index.
Consider a hypothetical account team. Maya, a solutions architect, asks its assistant:
“Summarize Northstar’s latest RFP requirements and show this account’s current revenue.”
The assistant produces a polished brief. It cites an RFP and includes a revenue figure. Unfortunately, the RFP is an older revision, and the revenue came from a slide deck rather than the finance system.
The answer is grounded in something. Just not the right thing!
Maya’s team could spend another week comparing models. I would first ask a different question: did the retrieval architecture give any model a fair chance?
In my earlier edition on context engineering, I focused on what belongs in an agent’s working context. This week is about the supply line that gets reliable evidence there.
Think of retrieval as a research desk rather than a filing cabinet. Some questions require searching through existing documents. Others need the latest information directly from the systems that manage it.
When I talk about Agentic RAG, I’m referring to a retrieval approach where an AI model can figure out what information it needs, choose the right tools to retrieve it, and try again if the evidence isn’t strong enough to answer the question.
However, the application still needs to enforce security, access controls, validation, execution limits, and stopping conditions. This is what separates Agentic RAG from a traditional retrieve-then-generate pipeline. And importantly, you don’t need a complex multi-agent system to make it work.
What I’m sharing here is a practical reference architecture and a step-by-step approach to building it. The architecture is a starting point, but getting it production-ready requires additional engineering and governance.
I like to break this down into a six-stage evidence pipeline, inspired by LangChain’s RAG from Scratch, with enterprise security and governance built into the design.
Prepare: Make enterprise data searchable and keep it up to date.
Plan: Understand the question and identify what information is needed.
Route: Choose the right data sources and formulate the queries.
Select: Bring together the results, filter out irrelevant information, and rank the most useful evidence the user is allowed to access.
Assemble: Organize the retrieved information into a concise context while preserving its meaning.
Answer: Generate a response, verify that the evidence supports it, and identify any gaps.
One important distinction is that preparation happens continuously in the background, keeping the data ready for retrieval.
The remaining stages happen when a user asks a question. If the evidence isn’t sufficient, the system can go back, retrieve additional information, and refine its response within defined limits.
Indexed retrieval searches a maintained representation of source content.
Live retrieval queries an authoritative system at request time. Neither is automatically the better choice.
Also, MCP is an interface, not a freshness guarantee. An MCP tool can search an index or call a live service. An ordinary API can do either too. The meaningful distinction is the data path behind the tool.
1. Prepare: Build a reliable foundation for retrieval
Before creating embeddings or search indexes, we need to ensure the underlying data is accurate, secure, and up to date.
For Maya’s RFP, this means tracking document versions, ownership, modification dates, and access permissions.
Why does this matter?
An AI agent shouldn’t retrieve outdated requirements or documents the user isn’t authorized to access.
Tools like Airweave can help continuously sync enterprise data into a searchable layer. However, we still need to validate how permissions, updates, and deletions are handled.
Choosing the right retrieval technique
Different retrieval problems require different approaches.
My approach: Start with structure-aware chunking and hybrid search. Measure retrieval quality using real business questions, then introduce more advanced techniques only when needed.
The goal is simple: Retrieve the right information, preserve context, and enforce access permissions.
Example Tools: Azure AI Search
2. Plan: Understand the question before searching
Before retrieving information, the AI needs to understand what the user is actually asking.
Below are some common query planning techniques.
With (Hypothetical Doc Embedding) HyDE, the hypothetical content is only used to improve search. It should never be treated as factual evidence.
Maya’s question might sound simple, but it requires two different types of information.
If "current revenue" is unclear, the AI should clarify whether Maya means quarterly recognized revenue, annual recurring revenue, or another metric. It shouldn't guess.
My approach: For Maya’s question, decomposition is enough. Split the question into RFP requirements and revenue, preserve important details like “Northstar” and “latest,” and clarify the revenue definition before retrieving data.
The goal is to make the question easier to answer without changing what the user actually asked.
3. Route: Find the right source for each question
Once we understand what information is needed, the next step is deciding where to retrieve it.
Not every question should go through a vector database. Some answers live in documents, others in structured databases or real-time APIs.
For Maya’s example, here’s how I would route each request.
An MCP server can expose tools that interact with SQL databases, graph databases, or REST APIs.
However, MCP is a standard for tool integration, not a security layer by itself.
For example, instead of giving an agent unrestricted SQL access, I would expose a specific read-only tool such as get_recognized_revenue(account_id, period, currency).
The application validates the inputs, verifies the user’s identity and permissions, and returns only approved data.
Once the account and metric are clear, the agent can retrieve the RFP and revenue information in parallel.
The key principle: Enforce authorization before retrieved information reaches the AI model or reranker. The agent should only see data the user is permitted to access.
