What Is HiRAG? How It Improves Enterprise RAG Search Accuracy by 14.5 Percentage Points and Cuts Latency by 28%

\n

Why Does the RAG Search Bar Return Noise Instead of Answers?

“The question is one line long. Why are there hundreds of search results?”

Imagine asking your company’s knowledge base:

“What are the early termination penalty terms in Customer A’s contract?”

A typical RAG system converts this question into an embedding, then searches the entire document repository for semantically similar documents. That’s where the trouble begins. If contracts, sales proposals, customer support records, legal guides, and outdated policy documents all contain similar terms, the search engine can return a flood of documents that look relevant.

But finding more is not the same as finding the right thing.

How flat search adds noise to RAG

Basic RAG systems often place hundreds of thousands or even millions of documents in one vast vector space and search across it. The implementation is simple, but as an organization’s data grows and becomes more complex, the search space expands with it.

This can lead to problems such as:

• Questions about HR policies return legal documents and announcements alongside the relevant material.
• When searching for an error code, manuals for the wrong product version appear near the top because they have similar meanings.
• An outdated policy may rank higher than the latest one.
• For questions involving exact strings—such as customer names, contract numbers, or product names—semantic search alone may not be enough.
• The more documents are retrieved, the more unnecessary context gets passed to the generative model.

The generative model answers based on the documents it receives. So if the search stage introduces too much noise, even a highly capable LLM may struggle to produce an accurate answer. In other words, a misleading response may be caused not by the model’s reasoning ability, but by the wrong evidence making it into the process in the first place.

RAG’s real bottleneck isn’t “what to search for”—it’s “where to search”

In a large knowledge base, it’s crucial to first understand what the question is asking. For example, a question about “penalty terms” isn’t simply a matter of finding documents that are semantically close to the word “contract.”

First, the system needs to determine:

• Does this question belong to the legal or contracts domain?
• Does it require the terms for a specific customer or product?
• Is the user looking for the latest standard contract, or for an exception clause?
• Should financial figures or customer support history also be consulted?

Flat search explores the entire corpus without making these distinctions. A hierarchical approach, by contrast, uses the intent behind the question to narrow down the data area to search first, then looks for specific documents within it.

It’s like choosing whether to go to the law, technology, finance, or HR section of a library before searching every shelf by book title. A smaller search area means faster results and a lower chance of irrelevant documents slipping in.

Why more search results can lead to worse answers

It’s easy to assume that giving a generative model more search results will make it more accurate. But in real-world RAG operations, the opposite often happens.

When the model receives three relevant documents alongside 20 ambiguous ones, it can run into several problems:

1. The key evidence gets buried.
   Even when the document containing the answer is there, the model may miss the crucial passage among all the surrounding material.

2. Conflicting information gets blended together.
   If an old policy and the latest policy are both included, the model may combine them and produce an answer that doesn’t actually exist.

3. Context costs go up.
   Sending unnecessary documents to the model increases token usage and response time.

4. Plausible-sounding errors creep in.
   The model may connect bits of phrasing from different documents into a fluent response that doesn’t fit the conditions in the actual question.

A good RAG system, then, isn’t simply one that retrieves a lot of documents. It should filter out irrelevant candidates early, then select the most reliable evidence from what remains.

RAG quality isn’t determined by the size of the generative model alone. What you exclude from search also shapes the quality of the answer.

This is where hierarchical approaches like HiRAG are drawing attention: they first use intent to narrow the search area, then refine the results after retrieval. Next, we’ll look at how this approach can shrink the overall search space while doing a better job of finding the evidence that leads to the right answer.

HiRAG RAG’s First Decision: Where to Search

What if a question about a contract clause didn’t send you searching through technical support documents?

In a basic RAG system, an incoming question triggers a similarity search across a vast document repository. It’s simple to implement, but problems arise when contracts, incident reports, HR policies, and product manuals are all mixed together. Irrelevant documents end up in the candidate pool, increasing both search time and the context costs of the generative model.

Before searching any documents, HiRAG first makes one decision:

Which knowledge domain should this question be answered from?

That first decision is handled by a lightweight BERT-based Intent Router. The router analyzes the user’s query and directs it to the appropriate document domain.

| Example user question | Primary search domain | |---|---| | “What are the penalty terms for terminating a contract?” | Legal and contract documents | | “What could be causing error code 504?” | Technical support and incident response documents | | “What is our sales target for the third quarter this year?” | Finance and sales data | | “What are the criteria for applying to work remotely?” | HR and internal policies |

This is more than simply labeling documents. It’s a search-space pruning strategy that narrows down what gets searched in the first place. For example, even if a company has millions of internal documents, a question about a contract can be searched primarily within the subset classified as legal and contract documents.

As a result, instead of indiscriminately scanning every document, the RAG pipeline follows a more efficient sequence:

User question
   ↓
Intent classification (Intent Routing)
   ↓
Select relevant document domain
   ↓
Detailed search within the selected domain

This first step matters because it shapes the quality of every search that follows. Even with the best embedding model or reranker, it’s difficult to find the right evidence if the system starts by searching the wrong document domain. Conversely, accurately choosing the domain that matches the question reduces noise in the candidate pool and makes it more likely that relevant documents will rank among the top results.

Routing does come with a clear risk, however. A question that spans multiple domains—such as “Tell me about the contract terms and who is responsible for system outages”—can be difficult to handle with a single classification. In practice, a fallback strategy is therefore essential: when the router’s confidence is low, search the top two or three domains together, or switch to a search across the entire corpus.

HiRAG starts with a simple idea: instead of searching every document, first decide where the question should go. That small change in sequence is key to reducing search latency and irrelevant results in large-scale RAG environments.

RAG: Capture Both Keywords and Meaning—and Filter with a Small Model

A question looking for error code E1042 and one saying “I keep getting login errors” call for very different search capabilities. The first requires an exact match on a string. The second requires understanding the meaning well enough to find documents that use different terms, such as “authentication failure,” “session expired,” or “unable to connect.”

Can a single search method meet both needs? HiRAG solves this problem by combining Hybrid Retrieval with Small-LLM Reranking.

Find exact terms with Sparse Retrieval

Keyword-based sparse retrieval, such as BM25, scores documents based on words they share with the query. It is especially effective when the exact string matters—for example, a product name, contract clause number, internal system name, or error code.

Suppose a user asks:

What causes error code E1042, and how can I fix it?

If you rely only on embedding-based semantic search, you may miss the right document when E1042 appears infrequently or is not well represented in the training data. BM25, on the other hand, can quickly bring technical documents, incident histories, and FAQs containing the code to the top of the candidate list.

Find the same meaning expressed in different words with Dense Retrieval

Users don’t always ask questions using the official terms found in documents.

• Document: “Authentication token expired”
• User: “I keep getting logged out”
• Document: “Contract termination clause”
• User: “What happens if I cancel before the end of the term?”

This is where dense retrieval—embedding-based semantic search—comes into its own. Even when the query and document use entirely different words, it can find relevant documents by recognizing that their meanings are similar in context.

HiRAG’s RAG pipeline combines sparse and dense retrieval. This helps enterprise document searches capture exact keywords in content rich with proper nouns, numbers, and codes, while also finding semantically relevant documents for natural-language questions with varied wording.

| Search method | Works best for | Example | |---|---|---| | Sparse Retrieval | Exact strings, codes, clause numbers | E1042, product model names, Article 12 of a contract | | Dense Retrieval | Similar expressions, natural-language questions, contextual understanding | “I keep getting logged out,” “I’d like to know the cancellation terms” | | Hybrid Retrieval | Practical queries combining both | “E1042 appears, and I can’t log in” |

Finding plenty of candidates isn’t the same as keeping the right evidence

Hybrid Retrieval is the stage that gathers a broad, dependable set of candidate documents. But finding candidates doesn’t mean they should all be passed to the generation model. Some documents may look relevant without actually answering the question.

For example, a search for “contract renewal terms” might return:

• An automatic renewal clause in a customer contract
• A software license renewal policy
• Rules for converting employees to fixed-term contracts
• A history of past contract changes

All are related to the word “renewal,” but they may not directly support an answer to the user’s question. This is where HiRAG uses Small-LLM Reranking to reassess the candidate documents.

The small language model takes a closer look at how each document relates to the query and reorders them based on one key question: “Does this document actually answer the question?” It helps distinguish degrees of relevance that are difficult to separate using keyword matches or vector similarity alone.

User query
   ↓
Sparse Retrieval: Find candidates with exact keywords and codes
   +
Dense Retrieval: Find candidates with similar meaning
   ↓
Combine candidate documents
   ↓
Small-LLM Reranking: Prioritize documents that directly answer the question
   ↓
Pass only the refined evidence to the generation model

Why use a small model instead of a large one?

Reranking every candidate with a large LLM can improve quality, but costs and response times can rise quickly. In environments that need to handle vast collections of internal documents and simultaneous user requests, this approach can become a bottleneck.

HiRAG assigns reranking to a small model, bringing practical benefits:

• Lower inference costs: Reduces the number of calls to large models and the amount of token usage.
• Faster responses: Quickly refines search results before passing them on to the final answer-generation stage.
• Less wasted context: Removes low-relevance documents and saves input space for the large LLM.
• More consistent answer quality: Reduces the chance that the generation model will be distracted by irrelevant evidence.

Ultimately, HiRAG is about more than simply gathering lots of search results. It combines keyword precision with semantic flexibility, then uses a small model to keep only the evidence that truly matters. This is a practical design for making RAG faster and more reliable across large enterprise knowledge bases.

RAG Performance Metrics: What a 14.5-Percentage-Point Increase in Recall@5 and a 28% Reduction in Latency Mean

The key question for any RAG system is simple: Before the generative model answers, did it find the evidence it actually needs?

In an evaluation on MDE-KB, HiRAG reported a 14.5-percentage-point increase in Recall@5 and a 28% reduction in average query latency compared with Naive RAG. One metric is directly tied to answer quality; the other has a direct impact on user experience and operating costs.

A 14.5-Point Increase in Recall@5: A Better Chance the Answer Is in the Top Five

Recall@5 measures whether the evidence needed to answer a question appears among the top five retrieved documents. No matter how capable a generative model is, it will struggle to produce an accurate answer if the necessary evidence isn’t retrieved.

For example, suppose a user asks:

“What incident response procedure should be followed when a specific error code occurs?”

If the top five results include the relevant technical manual or incident response policy, the generative model can use it to produce a specific, verifiable answer. But if the results contain only irrelevant documents, the model is more likely to offer generalities or generate unsupported claims.

So a 14.5-percentage-point improvement in Recall@5 means a substantially lower chance of missing the necessary documents during the initial retrieval stage. This difference can be especially noticeable in enterprise knowledge bases that bring together documents from many departments, along with technical materials, contracts, and policy documents.

Retrieval quality isn’t simply about “finding similar documents.” It’s about putting the evidence needed to answer the question within a limited set of top results.

A 28% Reduction in Average Latency: Not Just Faster Answers, but More Viable Operations

The reported 28% reduction in average query latency matters, too. In a RAG service, response time reflects the combined cost of retrieval, reranking, prompt construction, and LLM generation. As the document collection grows, searching the entire vector store and reranking a large pool of candidates can quickly become expensive.

HiRAG first classifies the user’s intent, then searches only the areas most likely to be relevant. It combines sparse and dense retrieval, then reranks candidate documents with a small language model. In other words, it reduces the number of documents—and the amount of irrelevant context—that a large model needs to process.

In production, this structure can lead to:

• Less time waiting for an answer
• Reduced strain on retrieval infrastructure, even with many concurrent requests
• Lower costs by reducing the number of tokens sent to large LLMs
• Fewer instances of irrelevant documents taking up space in the context
• Easier deployment in time-sensitive workflows such as customer support, internal search, and technical operations

That said, average latency alone isn’t enough to assess operational quality. Slow requests often cause the most user frustration, so p95 and p99 latency should also be tracked. Even if the average improves, some complex questions may still trigger routing failures or broader searches that significantly delay the response.

Why the Two Numbers Need to Be Considered Together

If retrieval accuracy improves but the system becomes far too slow, it may be impractical for day-to-day use. Conversely, if it’s fast but frequently misses the evidence it needs, RAG becomes less trustworthy.

What makes HiRAG’s results notable is that they target both goals at once.

| Metric | What it means in a real-world service | |---|---| | Recall@5: +14.5 percentage points | A higher likelihood that the evidence needed to answer a question appears in the top search results | | Average latency: -28% | Potential reductions in user wait time, retrieval infrastructure load, and inference costs | | Answer Relevance: 0.92 | A signal that the generated answers were highly relevant to the questions |

However, these figures don’t mean the same results will be reproduced with every company’s data. Outcomes can vary depending on document structure, classification schemes, query types, router accuracy, embedding models, and hardware. HiRAG also carries a risk: if a question is routed to the wrong area early on, the document containing the answer may be excluded from the search.

That’s why real-world deployments should measure not only Recall@5 and latency, but also routing miss rate, nDCG or MRR, faithfulness, and citation accuracy. The key isn’t a single impressive metric—it’s confirming that fast retrieval and reliable evidence go hand in hand.

RAG’s Success Depends Not on Bigger Models, but on Smarter Retrieval Paths

HiRAG’s performance figures naturally draw your attention to a 14.5-percentage-point gain in Recall@5 and a 28% reduction in average latency. But when it comes to real-world deployment, a different question matters more:

If the router takes the wrong path at the first step, can the system still find the right answer?

HiRAG first identifies the intent behind a query, then narrows the search to the relevant domain. For example, a question about contract terms is routed to legal documents, while a query containing an error code is sent to technical support documents. This is far more efficient than conventional RAG, which searches the entire corpus every time.

The catch is that this efficiency also depends heavily on the accuracy of the initial classification. Suppose the router sends the question “What happens to penalty fees and system access if the contract is terminated?” only to the legal domain. If answering it requires information not only from the contract, but also from security policies and account management manuals, documents from those other domains may be excluded from the candidate set from the outset.

Hybrid retrieval and small-LLM reranking can work with great precision within the selected domain. But documents excluded from the initial search space cannot be reranked. If the retrieval path is wrong, the models that follow may not be able to fully overcome that limitation.

Operational Metrics to Track Alongside Performance

When evaluating a hierarchical RAG system like HiRAG, answer relevance and average latency alone are not enough. You should also track the following metrics:

• Routing miss rate: The share of queries for which the router misses the domain containing the answer
• Top-k route recall: The share of queries for which the correct domain appears among the top 2–3 routing candidates
• Fallback rate: The share of queries that switch to full-corpus or expanded search when confidence is low
• Cross-domain query rate: The share of queries that require information from multiple departments or domains at once
• p95 latency: The actual response time for slower user requests, rather than the average
• Groundedness: Whether generated answers faithfully reflect the retrieved source documents

In enterprise settings especially, questions that span multiple business functions are more common than neatly classified, single-intent queries. A request like “Summarize the causes of this quarter’s sales decline and the history of related product outages” requires exploring both financial and technical support data. Locking such a query into a single routing category may speed up retrieval, but it can make the answer less complete.

Design Principles for Safely Deploying HiRAG

When applying HiRAG to a practical RAG architecture, treat the router not as a simple preprocessing step, but as a decision-making model that can make mistakes.

The most practical approach is to:

1. Search multiple top-ranked routes in parallel.
   Instead of letting the router select only the single most likely domain, search across the top 2–3 domains together. This adds some cost, but can significantly reduce critical omissions caused by initial misclassification.

2. Fall back to a broader search when confidence is low.
   If the router’s confidence falls below a threshold, don’t insist on the hierarchical structure. Fall back to searching the entire corpus or a broader set of domains.

3. Break down compound queries.
   When a question combines policy, figures, and technical causes, it can be more effective to split it into several sub-questions and apply a different retrieval path to each.

4. Continuously analyze routing logs.
   Store user queries, selected domains, retrieved documents, documents cited in the final answer, and user feedback together. As this data accumulates, you can identify the domain boundaries where routing errors recur.

5. Update classification and summarization layers as documents change.
   Organizational restructuring, new product launches, and policy changes can quickly make an existing classification system outdated. The index, summary information, and router training data should all be updated as well.

HiRAG’s core value is not simply that “it lowers costs by using smaller models.” More fundamentally, it designs a search path suited to the question, rather than searching every document in the same way.

But a smart search path only works when it points in the right direction. When deploying HiRAG, the first thing to validate should not be impressive average performance figures, but its resilience: whether it can still find the right evidence even when the router gets it wrong.