Is Retrieval the Bottleneck in RAG? EBG Reveals the Evidence Gaps—and How to Fix Them

\n

RAG: The Answer Is Right, But Where’s the Evidence?

The AI confidently gave the right answer. But when you checked the citation it provided, the quoted passage didn’t actually support the answer. Or perhaps the answer needed two pieces of evidence, but only one was cited. Can we really trust a system like this for important work?

This is where the limitations of traditional RAG become clear. Until now, many RAG systems have focused on “finding relevant documents and generating an answer.” But reliability in practice isn’t determined by whether a sentence merely sounds like the right answer. What matters more is whether the system has secured sufficient, verifiable evidence for every claim in its answer.

The Answer and the Evidence Are Two Different Problems

A typical RAG pipeline retrieves document passages—or chunks—that are semantically close to the question, then places them in the LLM’s input context. The model then generates an answer based on the search results and its own knowledge.

The problem is that situations like these can easily arise:

• The answer happens to be right, but relies on the model’s prior knowledge rather than the retrieved documents
• The search results contain both pieces of evidence needed, A and B, but the answer cites only A
• A citation is included, but the cited passage says something different from the claim it is supposed to support
• A key piece of evidence needed to answer the question isn’t retrieved, but the model fills in the gap with something plausible

For example, the question “Explain the background to the policy change and when it takes effect” requires at least two kinds of evidence: a document explaining why the policy changed, and the latest announcement specifying when it takes effect.

But simple similarity-based search is optimized to find the one chunk most similar to the question. Finding one relevant piece of information and securing all the information needed to answer a question are entirely different challenges.

The more important question isn’t “Did we retrieve a relevant document?” but “Did we retrieve all the evidence needed for this answer?”

How Evidence-Based Generation Changes the Standard

Evidence-Based Generation, or EBG, doesn’t treat RAG’s output as just an answer. It evaluates the answer together with the verifiable evidence linked to each claim as a single result.

The format is simple:

Answer: Method A is more reliable for multi-document queries. [Evidence 1]
        However, if relationships between documents aren’t addressed explicitly,
        key evidence may be missed. [Evidence 2]

The key isn’t the mere presence of citations. We need to check all four of the following:

1. Is the answer factually correct?
2. Does the cited passage directly support the claim?
3. Is all the evidence needed to answer the question included?
4. Has the model prioritized its own outdated prior knowledge over the retrieved evidence?

This standard is critical in fields like law, medicine, finance, and corporate research, where “the source of the answer” matters as much as—or more than—the answer itself. Users need to be able to check which document, and which passage within it, supports the AI’s conclusion, rather than simply accepting that conclusion at face value.

RAG’s Bottleneck Isn’t Reading—it’s Retrieval Completeness

The message from recent research is clear: many failures happen not because LLMs can’t read long documents, but because the evidence needed to answer the question never made it into the retrieval context in the first place.

The problem becomes more acute with multi-hop questions, which require connecting information scattered across multiple documents. A typical example is a case where you find a clue—such as a person, paper, institution, or date—in the first document, then use that clue to search for a second document.

In these cases, a single top-k retrieval isn’t enough. What’s needed is a process like this:

Analyze the question
→ Break it down into subquestions
→ Retrieve the first piece of evidence
→ Extract entities, dates, and document names
→ Search again for supporting evidence
→ Verify the relationships between the evidence
→ Generate an answer with citations for each claim

Ultimately, trustworthy RAG has to go beyond being a “chatbot that finds documents.” It needs to gather the evidence required to formulate an answer, check for missing evidence, and show what each sentence is based on.

The fact that AI said something right isn’t enough. What matters is whether that answer stands on verifiable evidence.

RAG Document Context Lost Between Retrieved Chunks

“This method improved on previous research.”

If RAG retrieves only this one sentence, can it really answer the question?

At first glance, the sentence seems relevant. But the most important information is missing: What is “this method”? Which studies count as “previous research”? And what improved, and by how much? Splitting a document into smaller pieces can make retrieval more efficient, but it can also strip away the structure and relationships needed to understand the answer.

Chunks are convenient, but they can’t stand in for the whole document

A typical RAG system splits long documents into small chunks, finds the passages that are semantically similar to a query, and passes them to an LLM. This works well for questions that can be answered with a single sentence, such as those about a specific definition or statistic.

But real documents are not just collections of isolated sentences. They contain context such as:

• Document titles and authors
• The structure of sections and subsections
• Terms defined in preceding paragraphs
• Explanations of tables, figures, and footnotes
• Citation relationships within the document
• The logical flow between paragraphs

For example, suppose a retrieved chunk contains only this sentence:

This method performed better than previous studies in a multi-document setting.

It’s hard to trust an answer based on this sentence alone. At a minimum, we need to know:

1. What method does this method refer to?
2. Which paper or model does previous studies refer to?
3. What does performance mean here—accuracy, recall, or processing speed?
4. Which dataset and experimental conditions produced these results?
5. Was the performance difference statistically significant?

In other words, even if a search result appears relevant on its own, it cannot serve as sufficient evidence unless it can be interpreted independently.

The loss of document-level context caused by RAG chunking

This problem can’t be solved simply by making chunks larger. Larger chunks preserve more surrounding context, but can reduce retrieval precision and bring in irrelevant content. Smaller chunks, on the other hand, can make retrieval more precise—but may strip away the referents of pronouns, abbreviations, and other references.

The following expressions are especially prone to losing their meaning when separated into chunks:

• This method, this model, the earlier approach
• This study, previous results, the experiment above
• That paper, the institution in question, the authors
• The results in Table 2, the comparison in Figure 4
• Based on this, however, on the other hand

These expressions only make sense within the structure of the document, not in isolation. So when RAG passes a retrieved fragment straight to an LLM, the model may try to fill in the missing information through inference. That inference may sound plausible, but it may not match the evidence in the source.

Especially weak on “who did what, and in which document?”

The loss of document context is even more pronounced when a question depends on identity—such as a person, author, institution, or paper title.

Consider a question like this:

What problem did a particular researcher identify as a limitation of an earlier paper?

To answer accurately, it’s not enough to find a chunk containing the word “limitation.” The retrieval system needs to preserve at least this chain of relationships:

Researcher → paper in question → earlier work cited → limitation identified

But independent chunks can easily break that chain. One chunk may describe the limitation, another may contain the author information, and yet another may give the title of the earlier work. Similarity search might retrieve all of them, but it cannot guarantee that they are connected within the same document narrative.

The solution isn’t “more chunks”—it’s structure that can be restored

To reduce this problem, a RAG index should store more than the body text. It should also preserve which document and section each chunk belongs to, along with its surrounding context.

In practice, metadata like the following can be useful:

{
  "text": "This method performed better than previous studies in a multi-document setting.",
  "document_title": "Paper Title",
  "section": "Experiments",
  "subsection": "Multi-document Evaluation",
  "authors": ["Author A", "Author B"],
  "position": "page 8, lines 12-18",
  "previous_chunk": "chunk_104",
  "next_chunk": "chunk_106",
  "entities": ["Method A", "Baseline B"]
}

With this structure in place, the system can take follow-up steps after retrieving a result, such as:

• Expanding the retrieved chunk to include the preceding and following paragraphs
• Providing the document title and section heading alongside it
• Recovering what “this method” refers to from the preceding context
• Retrieving related materials, such as tables, figures, and footnotes
• Searching for additional supporting evidence within the same document

Ultimately, a good RAG system isn’t one that simply retrieves the most similar sentence. It should be able to restore the meaning that sentence had in its original document.

The more finely a document is divided, the faster retrieval may become. But answer reliability doesn’t come from similarity scores alone. What matters is not just the fragment, but also the context and relationships within the document it came from.

EBG and RAG: Generation with an Address for Every Piece of Evidence

What if you could trace every claim in an answer back to a specific line in the source?

Traditional RAG focuses on retrieving relevant documents and generating an answer. But even when an answer sounds plausible—even when it appears correct—users are left with an important question:

“So where exactly is the evidence for this answer?”

Evidence-Based Generation (EBG) is an approach designed to answer that question. Rather than simply generating an answer, it requires each claim in the answer to include a line-level citation—an address pointing to the specific lines in the source that support it.

Why Getting the Answer Right Isn’t Enough

A typical RAG response might look like this:

Method A is more reliable than Method B for multi-document queries.

This statement may be correct. But it’s hard to tell which document or experimental result supports it, whether there are any exceptions, or whether the model read the retrieved documents or answered from prior knowledge in its training data.

With EBG, the same answer might look like this:

Method A produced more reliable results than Method B for multi-document queries. [L12-L18]

However, without explicitly tracking relationships across documents,
the system may miss a second key piece of evidence. [L31-L36]

Now users don’t have to take the answer on faith. They can jump to the cited lines and check for themselves whether the source really supports the claim.

Three Conditions EBG Requires

From an EBG perspective, a good RAG answer is more than one that simply gives the right answer. It should meet at least these three conditions:

1. The answer must be accurate.
   The conclusion itself must be consistent with the facts.

2. Every claim must be supported by evidence.
   Each important statement in the answer should be linked to a location in a retrieved document.

3. The evidence must be sufficient.
   If a question requires combining multiple facts, a single citation is not enough. The answer must include all the key evidence needed to support it.

The third condition is especially important for multi-hop questions that connect information across several documents. If reaching a conclusion requires finding both fact A and fact B, citing evidence for only one of them does not make the answer verifiable.

A Citation Doesn’t Automatically Make an Answer Evidence-Based

There’s an important caveat: the presence of a citation does not automatically make an answer trustworthy.

Possible failures include:

• The cited passage is unrelated to the claim.
• The claim requires two pieces of evidence, but only one is cited.
• The citation points to surrounding commentary rather than the document’s conclusion.
• The retrieved documents contain up-to-date information, but the model answers from outdated prior knowledge.
• Information from different documents is incorrectly connected.

EBG therefore asks more than “Is there a citation?” It also evaluates whether the citation directly supports the claim and whether all the necessary evidence is included.

To do this, the paper introduces an Ans+Evid perspective that considers answer accuracy and evidence quality together. Even if an answer is correct, it is not considered high quality if its citations are inaccurate or incomplete.

RAG’s Output Format Needs to Change, Too

With EBG, RAG should organize claims and evidence before producing its final answer.

Analyze the question
→ Break it into subquestions
→ Retrieve evidence
→ Check relationships between pieces of evidence
→ Link citations to individual claims
→ Generate the answer
→ Verify that citations match the claims

In practice, you might consider an output structure like this:

| Component | Role | |---|---| | Answer | The natural-language response the user reads | | Claim | An individual fact or conclusion included in the answer | | Citation | The document, page, and line location supporting the claim | | Evidence text | The actual source text at the cited location | | Verification | The result of checking whether the citation supports the claim |

The key is not to write the answer first and add sources afterward as decoration. EBG is about gathering evidence, checking that it is sufficient, and generating an answer only within the bounds of that evidence.

A Bigger Difference in High-Stakes Fields

This approach is especially useful in work where the cost of a wrong answer is high:

• Law: When exact wording in statutes, case law, or contracts must be traced
• Medicine: When evidence from clinical guidelines, research papers, or approval information must be checked
• Finance: When sources for disclosures, regulations, or risk reports are needed
• Research: When claims and experimental results across multiple papers must be cross-checked
• Enterprise knowledge search: When the currency of policy changes, internal rules, and technical documents must be verified

Ultimately, EBG makes RAG a more honest system. Instead of claiming, “I know the answer,” it requires the model to prove: “This answer came from here.”

Diagnosing RAG Failures: Retrieval vs. Generation

When a model that reads a long document all at once gives a better answer than RAG, we often conclude:

“As expected, the model is better at reading long documents.”

But Evidence-Based Generation (EBG) asks a different question:

Did the generation model really read better?
Or did RAG simply fail to retrieve the evidence needed to answer the question in the first place?

This distinction matters enormously. Misdiagnose the cause, and a company might spend more on larger models or longer context windows while leaving the real problem—a flawed retrieval pipeline—untouched.

What a Long-Context Model’s Advantage Really Means

Long-context models can process documents containing hundreds of thousands or even millions of characters in a single input. This means they may be able to find and connect relevant evidence directly, even when it is scattered across the beginning and end of a document or across multiple sections.

Typical RAG systems, by contrast, first select only a subset of chunks during retrieval. If even one piece of necessary evidence is missing, it becomes difficult for the generation model to produce a complete answer, no matter how capable it is.

Consider the following question:

What did the author of Study A identify as a limitation of Study B?

To answer it, the model needs at least the following:

1. The author or document identifying information for Study A
2. The passage in Study A that mentions Study B
3. The method or claim of Study B
4. The criticism or limitations analysis presented in Study A

A long-context model may find all of these clues across the full set of documents. But if RAG retrieves only chunks that are superficially similar to the question—those containing terms like “limitation,” “method,” or “performance”—it may miss crucial evidence about the author or the citation relationship between the documents.

So if a long-context model gives a better answer, we cannot automatically conclude that it won because of its reading ability. A more precise interpretation is:

Long-context model: It could see all the necessary evidence in context.
RAG: Some of the necessary evidence was missed during retrieval.

In other words, the bottleneck may be retrieval, not generation.

Why RAG Errors Need to Be Separated

RAG failures can be divided into two broad categories.

| Error type | Description | Typical symptoms | |---|---|---| | Retrieval failure | Evidence needed to answer the question never makes it into the context | A key document, table, metadata, or second piece of evidence is missing | | Generation failure | The necessary evidence is present, but the model misinterprets or misconnects it | An inaccurate summary, faulty reasoning, or a conclusion that contradicts the evidence |

Evaluating only the final answer without making this distinction can lead to problems.

For example, when an answer is wrong, a basic evaluation might conclude, “The model couldn’t answer the question.” But the model may not have failed at all—the retriever may simply not have shown it the necessary document.

Conversely, if the search results contained all the evidence but the model ignored or misconnected it, that is a generation or reasoning problem, not a retrieval problem.

To diagnose precisely this distinction, EBG checks not only the answer but also the evidence cited for each sentence.

Question
  ↓
Check retrieved evidence
  ↓
Check each claim in the answer
  ↓
Check citations for each claim
  ↓
Separate retrieval omissions from generation errors

With this approach, instead of the vague conclusion “The answer was wrong,” teams can make a more specific diagnosis:

• Was the necessary document not retrieved?
• Was the document retrieved, but the relevant passage cut off in the context?
• Was there enough evidence, but the model failed to connect two facts?
• Did the model prioritize its existing knowledge over the retrieved evidence?
• Was the answer correct, but a citation for a key claim missing?

You Can’t Evaluate RAG on Correctness Alone

Suppose a model gives the following answer:

Policy X’s eligibility criteria changed in 2026.

The answer itself may be correct. But whether it was grounded in a recently retrieved document or reproduced from something the model remembered in its training data is a separate question.

This distinction can be critical in settings where information changes frequently, such as the latest policies, internal regulations, contract terms, or medical guidelines. Even if the model’s prior knowledge happens to be correct, users still need to verify the basis for trusting the answer.

From an EBG perspective, evaluating RAG means asking all three of the following:

1. Is the answer correct?
2. Is it grounded in the retrieved evidence?
3. Are all of its key claims supported by sufficient citations?

If any one of these is missing, the system may be risky in production. In particular, “the answer is right, but the citations are wrong” can look like a success on the surface, yet amount to a failure in work that requires auditing, review, or reproducibility.

In Practice, You Need Diagnostic Logs

Saving only the final answer isn’t enough to improve a RAG system. At a minimum, you need to log the following:

• The user’s question
• The sub-questions the question was broken into
• The retrieved documents and chunks
• Search rankings and scores
• Evidence determined to be missing
• The answer broken down into individual claims
• Citations for each claim
• Verification results showing whether each citation directly supports its claim

These logs allow operations teams to address problems correctly.

For example, if evidence is missing, the following improvements may be needed:

• Query expansion
• Hybrid search
• Iterative retrieval
• Document-, section-, and metadata-level search
• Entity-based follow-up retrieval
• Restoring context around the chunk

Conversely, if the evidence was sufficient but the answer was wrong, teams should improve the generation prompt, reasoning steps, citation entailment checks, or model selection.

Ultimately, the central point of EBG is simple: when RAG gets something wrong, don’t blame the generation model first. Start by checking whether the evidence needed to answer the question was actually retrieved. Even in moments when a long-context model appears to do better, the real competitive edge may not be “reading better,” but “retrieving all the necessary evidence without missing anything.”

From RAG to Evidence Collection, Verification, and Citation

The old standard for judging RAG’s success—“Did it find relevant documents?”—is no longer enough. Work that requires connecting information across multiple documents calls for a tougher question:

Did it gather all the evidence needed to answer? And did it present each claim in a way users can verify?

From this perspective, RAG is no longer a simple retrieve → generate an answer pipeline. To build a trustworthy system, the entire workflow must be designed to include evidence collection, evidence-completeness checks, claim-level citations, and citation verification.

Why a Correct Answer May Still Be Untrustworthy

Suppose a model answers, “Policy A changed in 2026.” The answer itself may be correct. But it may not be able to answer the following questions:

• Which sentence in which document supports this answer?
• Did it verify both when the change took place and what changed?
• Did it rely on the model’s prior knowledge rather than the latest policy document?
• Does the cited passage directly support the claim?

In conventional RAG, it is easy to count the system as successful if the search results contain a relevant chunk and the final answer sounds plausible. But in real-world work, finding one relevant piece of information is not enough. If an answer consists of multiple claims, each one needs supporting evidence.

A good answer = correctness + faithfulness to evidence + citation completeness

Without this standard, a model may retrieve only some of the information, then fill in the gaps using prior knowledge or inference. The result may sound natural, but users cannot verify it.

How EBG Changes the RAG Workflow

Evidence-Based Generation (EBG) does not treat a single answer as the sole output of RAG. It generates evidence to support the answer and assesses whether that evidence is sufficient.

The conventional workflow is relatively simple:

Question
→ Retrieve similar chunks
→ Feed the results to an LLM
→ Generate an answer

From an EBG perspective, the workflow expands as follows:

Analyze the question
→ Identify sub-questions and the types of evidence needed
→ Search documents, sections, and metadata
→ Search again to fill gaps in the evidence
→ Restore document-level context
→ Check evidence completeness
→ Generate an answer
→ Link citations to individual claims
→ Verify citation entailment

The key difference lies in the goal of retrieval. Conventional RAG focuses on finding text similar to the question. EBG-centered RAG aims to gather the complete set of evidence needed to answer the question.

Look for a “Complete Evidence Chain,” Not Just “Relevant Documents”

For multi-document queries, finding one relevant item in the search results may still not be enough to produce a complete answer.

Consider a question like this:

What limitation did a particular researcher identify in the methodology of an earlier paper?

Answering it may require at least the following evidence:

1. Information identifying the researcher
2. A document or statement written by that researcher
3. The methodology proposed in the earlier paper
4. A passage identifying a limitation of that methodology

Vector search may find the paragraph that is most semantically similar to the question. But it might find only the paragraph analyzing the limitation while missing the passage that explains which methodology it refers to. Or the author information may appear in metadata on the first page and be missed by searches over body-text chunks.

RAG should therefore check more than the number of search results:

• Is there evidence corresponding to every sub-question?
• Do all the pieces of evidence refer to the same subject and context?
• Have details from different documents been incorrectly linked?
• Is there an original passage that can be cited directly for every key claim in the answer?

This shifts the focus from search quality measured by simple Recall@k to evidence completeness.

Citations Are a Verification Interface, Not Decoration

Adding a single source link at the end of an answer is not enough. In settings involving long reports or searches across multiple documents, users need to be able to quickly see which evidence supports each sentence.

The best approach is to cite at the claim level.

Method A produced more stable results on multi-document queries. [Document 1, lines 12–18]

However, unless relationships across documents are explicitly tracked,
the second key piece of evidence may be missed. [Document 2, lines 31–36]

This structure does more than show sources—it gives users a path to verify the answer. They can quickly assess:

• Does the cited text actually support the claim?
• Are any key claims missing citations?
• Are the citations based on the latest documents?
• Is the model saying something that differs from the retrieved evidence?

In fields that require accountability, such as law, medicine, finance, and research analysis, this verifiability can matter more than a fluent answer.

Minimum Checks for RAG in Practice

Applying the EBG perspective requires more than evaluating a model’s response after it has been generated. The evidence should be checked both before and after answer generation.

It is practical to include the following checklist in the pipeline:

[ ] Was the question broken down into answerable sub-questions?
[ ] Were the documents, entities, dates, and table data needed for each sub-question found?
[ ] Were metadata such as document titles, sections, authors, and publication dates preserved?
[ ] If the required evidence spans multiple documents, was retrieval repeated?
[ ] Is every key claim in the answer linked to a citation?
[ ] Does each citation directly support its claim?
[ ] Does the model’s prior knowledge conflict with the retrieved evidence?
[ ] If the evidence is insufficient, is uncertainty indicated rather than presenting a definite answer?

Ultimately, the competitive edge of next-generation RAG lies not just in retrieving more documents faster. It lies in gathering all the necessary evidence, preserving document context, and linking each claim in the answer to a verifiable citation.

Retrieval-augmented generation is only the starting point. A trustworthy RAG system should not merely answer based on retrieved information; it should collect and verify evidence, then present it in an explainable form.