Agentic Rag Retrieval That An Agent Decides To Use

The Problem: Why ‘Always Retrieve’ Is a Bad Default

You ask your plain RAG system: “What was the revenue in Q3?” It comes back with a page about Q2 marketing spend. The query is vague — “revenue” could mean anything — and the system grabbed whatever was closest in the embedding space. It didn’t stop to think: Do I need to retrieve at all? What exactly should I look for? Is this result good enough?

That’s the core limitation of plain RAG. It retrieves before reasoning. The pipeline is fixed: embed the query → search the vector store → feed top-k chunks to the LLM. There’s no feedback loop. If the retrieved documents are irrelevant, the LLM still has to use them — or ignore them, which wastes tokens and often produces a hallucinated answer.

Think of it this way: a human researcher never works like that. They read the question, think about what information they need, decide where to look, skim the results, and if the first batch isn’t useful, they try a different search. They adapt.

What if an LLM could do the same? What if it could decide whether to retrieve, what to retrieve, and when to stop retrieving — just like a human researcher?

That’s the idea behind Agentic RAG: retrieval that an agent decides to use, not a fixed pipeline step.

Let’s quickly remind ourselves what plain RAG looks like in code. This is the baseline we’re going to improve.

# Block 1: Quick reminder of plain RAG (self-contained)
import numpy as np
from sentence_transformers import SentenceTransformer

# Sample documents
documents = [
    "Q3 revenue was $12.5 million, driven by strong subscription growth.",
    "Q2 marketing spend increased 15% year-over-year to support product launch.",
    "The CFO noted that operating margins improved to 22% in Q3.",
]

# Embed documents
model = SentenceTransformer('all-MiniLM-L6-v2')
doc_embeddings = model.encode(documents)

# User query
query = "What was the revenue in Q3?"
query_embedding = model.encode([query])

# Cosine similarity
similarities = np.dot(doc_embeddings, query_embedding.T).flatten()
top_idx = np.argmax(similarities)

# Retrieve and generate (simulate)
retrieved = documents[top_idx]
print(f"Query: {query}")
print(f"Retrieved: {retrieved}")
print(f"Similarity score: {similarities[top_idx]:.2f}")

When you run this, the top document is the first one about Q3 revenue — that’s good. But change the query to something ambiguous like “Tell me about the company’s performance” and the system might grab the Q2 marketing document because “performance” is a vague term. The system has no way to say “that doesn’t look right, let me try a different search.”

This is the problem that Agentic RAG solves.

What Is Agentic RAG? (Intuition First)

Before we define it formally, let’s build an intuition with an analogy.

Plain RAG is a cafeteria line. You grab whatever is on the tray whether you need it or not. If the tray has broccoli when you wanted pizza, tough luck — you eat it anyway.

Agentic RAG is a chef. The chef reads your order, decides what ingredients to fetch, in what order, and whether to substitute if something is out of stock. The chef doesn’t just dump everything on the counter — they think about what’s needed.

Now here’s the formal definition (after the intuition):

Agentic RAG is a system where an LLM-powered agent reasons step-by-step, decides when to retrieve, formulates its own search queries, evaluates the quality of retrieved results, and may choose to retrieve again or stop.

What this actually means: the LLM doesn’t just answer questions — it runs its own research process. It’s like giving the LLM a library card and saying “go find what you need, and come back when you’re confident.”

The key distinction is that the agent controls the retrieval loop, not the other way around. In plain RAG, retrieval is a fixed step. In agentic RAG, retrieval is a tool the agent can choose to use — or not.

This idea builds on the Self-RAG paper (Asai et al., 2023), which pointed out that “indiscriminately retrieving… regardless of whether retrieval is necessary” is wasteful and often harmful. And it’s grounded in the ReAct pattern (Yao et al., 2022) where reasoning traces interleave with actions.

The Core Loop: Reason → Retrieve → Evaluate → (Stop or Repeat)

Now let’s decompose the agentic RAG loop into four steps. You can visualize it as a flowchart:

1. Reason: The agent examines the user query and its own internal state (memory, previous turns) to decide if retrieval is needed. Sometimes the answer is already in context — no need to search.

2. Retrieve: If retrieval is needed, the agent formulates a search query (which may differ from the user’s original question) and calls a retrieval tool. For example, if the user asks “What’s the latest on climate policy?”, the agent might internally think “I need recent news articles about climate policy” and generate a more specific search query.

3. Evaluate: The agent inspects the retrieved documents for relevance and quality. This is the hardest part — how does an LLM know if it found the right information? The CRAG paper (Yan et al., 2024) introduced a “retrieval evaluator” that returns a confidence degree, triggering different actions. You can do something simpler: ask the LLM to rate relevance on a scale, or use a similarity threshold.

4. Decide: If the retrieved content is sufficient, generate the answer. If not, reformulate the query and loop back to Step 2. If retrieval quality is low, fall back to a different strategy (e.g., web search or ask for clarification).

This loop is essentially the ReAct pattern: reasoning traces interleaved with tool calls. The agent doesn’t just retrieve once — it keeps iterating until it’s satisfied.

Here’s a pseudocode version of the loop:

# Block 2: Pseudocode of the agentic RAG loop (no real API calls)
# This is a conceptual illustration, not runnable as-is

def agentic_rag_loop(user_query, max_retries=3):
    """
    Conceptual loop: Reason -> Retrieve -> Evaluate -> Decide
    """
    context = []
    retrieval_quality = 0.0
    attempt = 0
    
    while retrieval_quality < 0.7 and attempt < max_retries:
        # Step 1: Reason — decide if retrieval is needed
        # (In a real system, the LLM would output a thought)
        if not needs_retrieval(user_query, context):
            break
        
        # Step 2: Retrieve — formulate a search query
        search_query = formulate_query(user_query, context)
        retrieved_docs = call_vector_search(search_query)
        
        # Step 3: Evaluate — check relevance
        retrieval_quality = evaluate_relevance(retrieved_docs, user_query)
        
        # Step 4: Decide — if quality is low, reformulate and retry
        if retrieval_quality < 0.7:
            context.append(f"Previous search for '{search_query}' was not helpful.")
            attempt += 1
        else:
            context.extend(retrieved_docs)
    
    # Generate final answer from accumulated context
    return generate_answer(user_query, context)

The key signal is retrieval_quality. This is the “retrieval evaluator” from CRAG. In practice, you can implement it as a separate LLM call that grades each document on a scale of 1-5, or as a simple similarity score threshold.

The GRASP paper (2025) frames this as a learned policy: deciding to retrieve is not a fixed rule but something the agent learns through experience. For our purposes, we’ll use a simple heuristic.

Building a Minimal Agentic RAG System (Code Walkthrough)

Now let’s build a real agentic RAG system using LangChain. This is the heart of the tutorial. We’ll create an agent that has one tool: a vector store retriever. The agent will decide when to call it, evaluate the results, and retry if needed.

We’ll use OpenAI’s function calling under the hood. The agent will output a “thought” and then either a tool call or a final answer.

# Block 3: Setup — install required packages
# Run in terminal before next block
# pip install langchain langchain-openai langchain-community faiss-cpu

# Block 4: Build a minimal agentic RAG system
# Self-contained: imports, data, agent setup, loop

import os
from langchain_openai import ChatOpenAI
from langchain.agents import AgentExecutor, create_openai_functions_agent
from langchain.tools import Tool
from langchain_community.vectorstores import FAISS
from langchain_community.embeddings import OpenAIEmbeddings
from langchain.schema import SystemMessage, HumanMessage
from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder

# Set your OpenAI API key
os.environ["OPENAI_API_KEY"] = "your-api-key-here"  # Replace with your actual key

# ---- 1. Create a vector store with sample documents ----
documents = [
    "The company's Q3 revenue was $12.5 million, driven by strong subscription growth.",
    "Operating margins improved to 22% in Q3, up from 18% in Q2.",
    "The CFO noted that the subscription business now accounts for 65% of total revenue.",
    "Marketing spend increased 15% year-over-year in Q2 to support the product launch.",
    "Customer acquisition cost decreased by 10% in Q3 due to improved ad targeting.",
]

embeddings = OpenAIEmbeddings()
vectorstore = FAISS.from_texts(documents, embeddings)
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

# ---- 2. Wrap the retriever as a tool ----
def retrieve_docs(query: str) -> str:
    """Search for relevant documents. Input: a search query. Output: retrieved text chunks."""
    docs = retriever.invoke(query)
    # Format as a string with relevance scores (using similarity as proxy)
    formatted = []
    for i, doc in enumerate(docs):
        # We don't have direct similarity scores from FAISS, but we can simulate
        # In production, you'd get scores from the vector store
        formatted.append(f"Document {i+1}: {doc.page_content}")
    return "\n\n".join(formatted)

retrieval_tool = Tool(
    name="vector_search",
    func=retrieve_docs,
    description="Search the company's internal documents. Use this to find information about financial performance, marketing, etc."
)

# ---- 3. Create the agent ----
llm = ChatOpenAI(model="gpt-4", temperature=0)

system_prompt = """You are a helpful assistant with access to a company's internal documents.
You have a tool called 'vector_search' that can retrieve relevant documents.

Follow this process:
1. Think about what information you need.
2. If you need to search, call the vector_search tool with a specific query.
3. Examine the retrieved documents. If they seem relevant, use them to answer.
4. If the documents are not relevant, try a different search query.
5. When you have enough information, provide a final answer.

Be concise but thorough."""

prompt = ChatPromptTemplate.from_messages([
    SystemMessage(content=system_prompt),
    MessagesPlaceholder(variable_name="chat_history"),
    HumanMessage(content="{input}"),
    MessagesPlaceholder(variable_name="agent_scratchpad"),
])

agent = create_openai_functions_agent(llm, [retrieval_tool], prompt)
agent_executor = AgentExecutor(
    agent=agent,
    tools=[retrieval_tool],
    verbose=True,
    max_iterations=5,  # prevent infinite loops
    handle_parsing_errors=True,
)

# ---- 4. Run an example ----
result = agent_executor.invoke({"input": "What was the Q3 revenue and how did margins change?"})
print("\n=== Final Answer ===")
print(result["output"])

When you run this, the agent will output a chain of thought. You’ll see something like:

> Entering new AgentExecutor chain...
Thought: I need to find information about Q3 revenue and margin changes.
Action: vector_search
Action Input: "Q3 revenue and operating margins"
Observation: Document 1: The company's Q3 revenue was $12.5 million...
Document 2: Operating margins improved to 22% in Q3...
Document 3: The CFO noted that the subscription business...

Thought: I have the revenue and margin information. The documents seem relevant.
Final Answer: The Q3 revenue was $12.5 million, and operating margins improved to 22% from 18% in Q2.

What’s actually going on here?

• The agent generated a thought: “I need to find information…”
• It decided to call vector_search with a specific query.
• It received three documents.
• It evaluated them as relevant (implicitly — it didn’t call the tool again).
• It produced the final answer.

Now let’s see what happens when the first retrieval fails. We’ll ask a question that doesn’t match the documents well.

# Block 5: Example where retrieval fails and agent retries
# Self-contained (repeats setup from Block 4)

import os
from langchain_openai import ChatOpenAI
from langchain.agents import AgentExecutor, create_openai_functions_agent
from langchain.tools import Tool
from langchain_community.vectorstores import FAISS
from langchain_community.embeddings import OpenAIEmbeddings
from langchain.prompts import ChatPromptTemplate, SystemMessage, HumanMessage, MessagesPlaceholder

os.environ["OPENAI_API_KEY"] = "your-api-key-here"

# Same documents as before
documents = [
    "The company's Q3 revenue was $12.5 million, driven by strong subscription growth.",
    "Operating margins improved to 22% in Q3, up from 18% in Q2.",
    "The CFO noted that the subscription business now accounts for 65% of total revenue.",
    "Marketing spend increased 15% year-over-year in Q2 to support the product launch.",
    "Customer acquisition cost decreased by 10% in Q3 due to improved ad targeting.",
]

embeddings = OpenAIEmbeddings()
vectorstore = FAISS.from_texts(documents, embeddings)
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

def retrieve_docs(query: str) -> str:
    docs = retriever.invoke(query)
    formatted = [f"Document {i+1}: {doc.page_content}" for i, doc in enumerate(docs)]
    return "\n\n".join(formatted)

retrieval_tool = Tool(
    name="vector_search",
    func=retrieve_docs,
    description="Search the company's internal documents."
)

llm = ChatOpenAI(model="gpt-4", temperature=0)

system_prompt = """You are a helpful assistant with access to a company's internal documents.
You have a tool called 'vector_search' that can retrieve relevant documents.

Follow this process:
1. Think about what information you need.
2. If you need to search, call the vector_search tool with a specific query.
3. Examine the retrieved documents. If they seem relevant, use them to answer.
4. If the documents are not relevant, try a different search query.
5. When you have enough information, provide a final answer.

Be concise but thorough."""

prompt = ChatPromptTemplate.from_messages([
    SystemMessage(content=system_prompt),
    MessagesPlaceholder(variable_name="chat_history"),
    HumanMessage(content="{input}"),
    MessagesPlaceholder(variable_name="agent_scratchpad"),
])

agent = create_openai_functions_agent(llm, [retrieval_tool], prompt)
agent_executor = AgentExecutor(
    agent=agent,
    tools=[retrieval_tool],
    verbose=True,
    max_iterations=5,
    handle_parsing_errors=True,
)

# Ask a question that might not be directly in the docs
result = agent_executor.invoke({"input": "What is the company's strategy for international expansion?"})
print("\n=== Final Answer ===")
print(result["output"])

When you run this, the agent will first search for “international expansion strategy.” Since none of the documents mention it, the retrieved results will be off-topic (maybe about marketing spend). The agent will see that the documents don’t answer the question, and it will try a different query — perhaps “global expansion plans” or “new markets.” After a couple of attempts, if nothing relevant is found, it will say something like “I couldn’t find information about international expansion in the available documents.”

Interpreting the output:

• The agent made multiple retrieval attempts. Each attempt consumed tokens and time.
• The final answer was honest: it admitted it didn’t have the information.
• This is much better than plain RAG, which would have returned irrelevant information and forced the LLM to make something up.

When Should You Use Agentic RAG? (The Honest Trade-off)

Agentic RAG is powerful, but it’s not always the right choice. Let’s look at the trade-offs honestly.

Dimension	Plain RAG	Agentic RAG
Latency	Fast (single retrieval + generation)	Slower (multiple LLM calls, retries)
Cost	Low (few tokens)	Higher (more tokens from reasoning traces)
Accuracy on simple queries	Good	Overkill, may overcomplicate
Accuracy on ambiguous queries	Poor	Much better
Recovery from bad retrieval	None	Can retry or fall back
Complexity	Simple	More complex to build and debug

When to use Agentic RAG:

• Complex questions that require synthesis across multiple documents.
• Questions where the user’s phrasing is vague or ambiguous.
• Applications where retrieval quality is variable (e.g., you have a messy knowledge base).
• When you need the system to be honest about not finding information.

When to skip it:

• Simple factual lookups (e.g., “What’s the CEO’s name?”).
• Latency-sensitive applications (e.g., real-time chatbots).
• Cases where plain RAG already works well and you don’t need the extra complexity.

The “Is Agentic RAG worth it?” paper (2025) found that agentic RAG outperforms plain RAG on complex multi-hop questions but is slower and more expensive. The NVIDIA blog also notes that “traditional RAG is typically faster and less expensive.”

So be honest with yourself: do you really need the agent to think about retrieval? If your use case is straightforward, stick with plain RAG. If you’re hitting the wall of bad retrievals, agentic RAG is your next step.

What We Learned & Where to Go Next

Let’s recap what you learned in this article:

1. Plain RAG retrieves before reasoning — that’s a limitation. It can’t adapt when the first retrieval is bad.
2. Agentic RAG lets the LLM decide when and how to retrieve — it’s like giving the LLM a research process.
3. The core loop is Reason → Retrieve → Evaluate → Decide — the agent iterates until it’s confident.
4. It’s powerful but not always worth the cost — use it for complex, ambiguous queries, not simple lookups.

In the next part of this series, we’ll build agents that use multiple tools — not just retrieval, but also calculators, APIs, and databases. You’ll see how the same agentic loop can orchestrate a whole toolbox.

Check Your Understanding

Remember: What is the key difference between plain RAG and agentic RAG?

Understand: Explain in your own words why the “Evaluate” step is the hardest part of the agentic RAG loop.

Apply: Given a user query “How did our marketing spend change compared to last quarter?”, write a possible chain-of-thought the agent might produce, including at least one tool call.

Analyze: Compare the latency and cost trade-offs between plain RAG and agentic RAG for a simple factual query vs. a complex multi-step query. Which one benefits more from agentic RAG?

Evaluate: In what scenarios would you choose to use a fixed retrieval pipeline over an agentic loop? Justify your answer.

Create: Design a simple experiment to measure whether agentic RAG improves answer quality on a set of ambiguous queries. What metrics would you track?

Related articles

• Part 1: What Makes an “Agent” an Agent? — This article builds on the ReAct loop introduced in Part 1. Understanding that loop is essential for agentic RAG.
• Part 2: Giving an LLM Tools — Agentic RAG treats retrieval as a tool. Part 2 explains how to wrap any function as a tool for an LLM.
• Part 4: Multi-Agent Systems — In Part 4, we built a team of agents. Agentic RAG is a simpler pattern: one agent with one tool, but with a sophisticated loop.