RAG Arena

machine-learning
rag
llm
langgraph
Implementing and measuring Self-RAG, Corrective RAG, and Adaptive RAG on LangGraph with a shared dataset
Author

Alex de la Puente

Published

August 3, 2026

Let’s build a RAG arena.

A RAG is fairly easy to understand. Also fairly dumb.

Luckily, we have a few papers (there are surely more, but these are the ones on my radar for now) that try to address the limitations of this system and make it more reliable—or at least make it look a little less stupid. I am talking about Self-RAG, Corrective RAG, and Adaptive RAG. All three propose adding some kind of judgment to a retrieval pipeline, and all three report improvements.

The idea is simple: grab a decent dataset and a baseline RAG, then measure the improvement delta—and the cost—of each of these systems.

Limitations of classic RAG

Let’s recall what vanilla RAG does: a question \(x\) arrives, the \(k\) most similar documents are retrieved and pasted into the LLM prompt, and the LLM writes the answer.

The three papers start from the same observation, although each frames it differently:

  • It always retrieves, without checking whether the retrieved material is useful. Self-RAG says this directly: indiscriminately retrieving a fixed number of passages, regardless of whether retrieval was necessary or the passages were relevant, reduces the model’s versatility and can lead to worse answers. The retriever returns \(k\) documents because you asked it for \(k\); whether they are useful is another question, and nobody asks it.
  • The answer is not guaranteed to be consistent with the retrieved material. This also comes from Self-RAG: models are not explicitly trained to follow the facts in the passages they receive. You can put three correct documents in front of the model and still get an answer based on whatever is stored in its weights.
  • If retrieval fails, everything else fails. This is Corrective RAG’s starting point: when the retriever returns irrelevant documents, no generation step can fix it because the system has no mechanism for noticing the problem.
  • Every question costs the same, even when it is not worth it. This is Adaptive RAG’s point: applying the same strategy to a trivial question and to one that requires several hops is wasteful in one case and insufficient in the other.

Put like that, the pattern is clear: the classic pipeline makes no decisions at all. It is a straight pipe. The three papers are, at heart, three different ways of putting decisions into it.

The three decision points

The limitations above correspond to three questions, each living at a different point in the pipeline:

Figure 1: The three points where a decision can be added to a RAG

And here is the interesting part: each paper attacks a different subset of those three points.

A: retrieve? B: useful docs? C: grounded?
Vanilla RAG always never asks never asks
Self-RAG yes, on demand yes, document by document yes, segment by segment
Corrective RAG always yes, and acts when they are not no (it refines before generating)
Adaptive RAG yes, and decides how many times delegated to the chosen strategy no

Self-RAG (Asai et al., 2023) trains the model to emit reflection tokens while it generates: when to retrieve, whether a document is relevant, whether the generated text is supported, and whether it is useful. It does not generate the answer in one go, either, but in segments, using beam search where each candidate’s score comes from those tokens. It touches all three decision points and is the most ambitious of the three.

Corrective RAG (Yan et al., 2024) adds a lightweight retrieval evaluator that classifies the retrieved material and triggers a different corrective action depending on the result, including refining the documents before passing them to the generator. All of its effort is focused on point B.

Adaptive RAG (Jeong et al., 2024) puts a complexity classifier in front of the entire pipeline and routes each question to a different strategy: no retrieval, a single retrieval step, or iterative retrieval. It targets point A, and its interesting metric is not accuracy alone, but accuracy per unit of cost.

The dataset

For this experiment, almighty Claude recommends MuSiQue from the Chaplain0908/RAGRouter dataset, which comes with an already paired corpus and question set.

I asked for a dataset that I could download from Hugging Face, use to measure different types of RAG, and fit on my laptop.

Size
Documents in the corpus 21,100
Questions 3,356

Each question comes with its reference answer, its type, and—this is the important part—its supporting_facts: the corpus documents that were actually needed to answer it. This is quite useful:

Gold document annotations let us measure retrieval separately from generation, without asking any model for an opinion. When a run fails, we can distinguish “it did not find the evidence” from “it found the evidence and answered incorrectly.” Without that, everything is mixed into a single accuracy number and diagnosis becomes impossible.

The questions are also labeled as three variations on the same task:

Type Questions % Supporting docs Answer format
multi_hop 2,590 77.2% 2 to 4 one entity, two or three words
single_hop 398 11.9% always 1 one entity, two or three words
summary 368 11.0% always 5 a paragraph of roughly 100 words

Well, this is something else. Before getting into this, I had never heard the term multi-hop in my life. It turns out that MuSiQue is a multi-hop QA dataset: its questions are deliberately constructed so that you have to chain several documents together, not just find one.

Here is a real example.

Who is the spouse of the Green performer? Answer: Miquette Giraudy

The dataset contains no single piece of information that can be used directly to answer that question correctly. It forces you to make two hops:

  1. One document says that Green is Steve Hillage’s fourth studio album. That does not answer the question, but it establishes the link.
  2. Another document says that Miquette Giraudy is a keyboard player and Steve Hillage’s partner. There is the answer.

Think about what this means for a regular RAG: you can retrieve the document from step 1, which is correct and relevant, and still answer incorrectly because the answer does not exist in that document. You need to retrieve again using what you have just learned.

This clearly favors any agentic RAG system, so we know our results will be slightly biased toward systems that can retrieve again from newly learned information. With a conventional single-stage retriever, results on those multi-hop questions will obviously be worse.

There are some pretty cool things in the dataset, to be honest, including lexical similarity issues: different entities can share the same title. There are six documents titled “Green,” one of them about the symbolism of the color green in ancient Egypt. Lexical similarity to the question can therefore lead you straight to the wrong document, which is exactly what a relevance judgment ought to filter out.

I am not going to spend much time on the dataset’s peculiarities because they would support an analysis of their own, but here are a couple more:

The dataset is highly imbalanced. More than three quarters of the questions are multi-hop, which will favor methods that retrieve several times. It is a bias we need to be aware of. To make the minority groups meaningful, we will have to sample in a balanced way.

The summary questions are a different task. A multi-hop question is answered with two words; a summary takes roughly one hundred and always uses five documents.

The Retrieval Baseline

For the baseline, I will use a conventional RAG mechanism in dense, hybrid, and hybrid plus reranker variants. I designed it as a single node shared by every implementation.

Mode What it does
dense dense vector only, cosine similarity
hybrid dense + sparse, fused with RRF inside Qdrant
hybrid-rerank the previous mode plus a cross-encoder (bge-reranker-v2-m3) that reorders the candidates and keeps the best ones

I gave the reranker a relevance threshold of 0.15 so that it does not return shit.

The MuSiQue corpus is indexed with bge-m3, which returns two representations for every document in a single pass: a 1024-dimensional dense vector (semantics) and sparse weights (lexical importance, like learned BM25). In other words, both lanes of a hybrid retriever come from the same model.

I use Qdrant embedded locally as the vector store, which is more than enough for this. I chose it over Chroma or FAISS because it lets you store the dense and sparse vectors in the same point and fuse both lanes with RRF1 inside the query.

Documents are indexed whole, without chunking. The passages in the corpus are short and do not need it, which simplifies the process.

System evaluation

For evaluation, we will combine deterministic metrics, calculated directly from the dataset and the execution, with LLM-as-a-Judge metrics provided by RAGAS. This separation lets us evaluate information retrieval, answer correctness, and faithfulness to the evidence independently.

Deterministic metrics

The dataset provides the supporting_facts field—the documents that contain the evidence needed to answer each question—which lets us evaluate retrieval directly.

We calculate retrieval recall and retrieval precision over the documents ultimately used to generate the answer:

\[ \text{recall} = \frac{|\text{used} \cap \text{gold}|}{|\text{gold}|}, \qquad \text{precision} = \frac{|\text{used} \cap \text{gold}|}{|\text{used}|} \]

Note

Recall indicates what proportion of the necessary evidence the system managed to find (out of all relevant documents, how many can I retrieve?), while precision measures what proportion of the documents used were actually relevant (out of all retrieved documents, how many are actually relevant?).

This distinction is especially useful when comparing retrieval strategies: one strategy may improve recall by retrieving more evidence, but at the cost of introducing irrelevant context.

LLM-as-a-Judge metrics

The previous metrics are not enough to evaluate an answer semantically. The fact that the reference appears in the text does not guarantee that the answer is correct, nor does it tell us whether the generated claims are supported by the evidence.

For this type of measurement, it is more common to use RAGAS, an evaluation library that implements LLM-as-a-Judge metrics with predefined prompts and evaluation protocols.

We use two metrics: AnswerAccuracy and Faithfulness.

AnswerAccuracy evaluates whether the generated answer is semantically consistent with the reference. It implements NVIDIA’s double-judge scheme: it first evaluates (question, answer, reference) and then repeats the judgment with the answer and reference swapped.

The double-judge system basically measures the answer against the reference twice, swapping the roles of answer and reference:

Evaluation Treated as the “answer” Treated as the “reference” Question to the judge
1 Generated answer Gold/reference To what extent is the answer correct with respect to the reference?
2 Gold/reference Generated answer The same evaluation, with their roles reversed

This matters because an LLM’s judgment is not necessarily symmetric: evaluating A against B can produce a different result from evaluating B against A. The second judgment helps detect and reduce this bias.

The system works like this:

            ┌─ Evaluation 1: Answer → Reference ─► 0 / 2 / 4 ─┐
Question ─► ┤                                                 ├─► Normalize ─► Mean ─► AnswerAccuracy
            └─ Evaluation 2: Reference → Answer ─► 0 / 2 / 4 ─┘                     {0, .25, .5, .75, 1}

The judge determines how correct the answer is with respect to the reference and assigns a score of 0 (incorrect), 2 (partially correct), or 4 (correct). AnswerAccuracy can therefore take only five values: 0, 0.25, 0.5, 0.75, and 1.

For example, if the first judgment considers the answer completely correct (4) but the second considers it only partially correct (2), the resulting AnswerAccuracy is \(\frac{1 + 0.5}{2} = 0.75\). The values 0.25 and 0.75 are particularly informative because they show that the two evaluations reached different conclusions about the equivalence between answer and reference.

Faithfulness, on the other hand, measures whether the answer is supported by the documents used as context. RAGAS first breaks the answer into atomic claims and then evaluates which of them can be inferred from the evidence:

\[ \text{Faithfulness} = \frac{\text{supported claims}} {\text{total claims}} \]

The distinction between the two metrics matters. A run can obtain faithfulness = 1 and answer_accuracy = 0: the answer is fully supported by the documents used, but incorrect with respect to the reference. In that case, the model is not necessarily hallucinating; it may be answering correctly from the wrong evidence.

Deterministic metrics therefore let us measure what evidence we retrieve and whether the expected answer appears in the generation, while RAGAS lets us evaluate whether the answer is semantically correct and grounded in the evidence used.

And finally, we will count the money. It is almost obvious that an iterative system will be able to 1) retrieve better or more useful documents for your query from the database and 2) give better answers based on those documents… but at what cost? It will obviously cost us latency and tokens too. We will try to measure it.

With all that defined, all that remains is to make it work; all the experiment code is in the RAG Arena repository.

References

The three papers:

The dataset and the original MuSiQue paper:

The tools:

And the code for all of this:

Footnotes

  1. Reciprocal Rank Fusion: combine two rankings by adding \(1/(k + \text{rank})\) for each document. It is a fusion by position, not by score, so it does not matter that the scales of the two lanes are completely different.↩︎