Agentic RAG, When the Retriever Becomes a Tool

Pull-quote: “A fixed pipeline retrieves once and commits. An agent looks at what came back, decides it is not enough, and asks again. That single judgment call is the whole upgrade.”

Why this matters

Classic RAG is a straight line: query in, passages out, answer generated, done. The pipeline has no opinion about whether the retrieval was any good; it commits to whatever came back. Agentic RAG changes the control flow: retrieval becomes a tool the model calls, inspects, and calls again. The retriever stops being a stage and becomes a resource. For a large class of enterprise questions, this is not a refinement; it is the difference between answerable and not.

Where single-pass breaks

Consider “did any supplier flagged in last year’s audits appear in this quarter’s nonconformance reports?” No single embedding of that sentence retrieves the evidence, because the evidence does not exist as one passage. Answering requires: find the flagged suppliers, then search the nonconformance reports for each of them, then join the results. That is a plan, and single-pass pipelines cannot plan.

The loop

                ┌──────────────────────────────────────────┐
                │                                          ▼
   Question ──► Decompose ──► Retrieve(sub-query) ──► Judge evidence
                                                         │
                              sufficient? ── no ──► reformulate /
                                   │                next hop ─────┐
                                  yes                             │
                                   ▼                              │
                            Verify grounding ◄────────────────────┘
                                   │
                            supported? ── no ──► retrieve more, or
                                   │              answer "not found"
                                  yes
                                   ▼
                            Answer + citations

Four capabilities make the loop work:

1. Query decomposition. Split compound questions into sub-queries that can each be retrieved directly.
2. Self-correction. After each retrieval, judge the evidence: relevant, sufficient, on-topic? Weak results trigger reformulation (different terms, different filters, different index) instead of proceeding.
3. Multi-hop retrieval. Use the results of hop one to construct the queries of hop two. This is where joins across documents actually happen.
4. Grounding verification before answering. The final check is not “do I have text” but “does this text support this answer.” Failing that check produces an honest miss report (which searches ran, what came back, what is missing) rather than a confident fabrication.

Systematic literature review is a useful reference point for how far this pattern scales: the methodology requires searching many sources in parallel, deduplicating results with provenance tracking, and documenting every query that ran. “Retrieve once and hope” was never an option there, and the same standard of documented, exhaustive search is what agentic RAG makes practical for ordinary enterprise questions.

The honest costs

The loop is not free, and pretending otherwise is how agentic RAG gets a bad name. Latency multiplies with iterations; token spend multiplies with re-reads of intermediate evidence; and a loop needs a termination budget or a hard question becomes an infinite one. The engineering answer is routing: classify the question first, send simple lookups through the single-pass path, and reserve the loop for questions that need it. Evaluation must also change. You are now testing trajectories, not responses, so the harness needs to score decomposition quality and stopping behavior, not just final answers.

Closing

Agentic RAG is not “RAG plus an agent framework.” It is a control-flow decision: let the system judge its own evidence and act on that judgment. Route simple questions around the loop, budget the iterations, verify grounding before answering, and the retriever becomes what it should have been all along: a tool, called as many times as the question requires, and not once more.