RAG Techniques Do Not Add Up: IBM Study Finds +47% Predicted Became +17%

If you have ever stacked a better embedding model, a reranker and a query-rewriting step onto a RAG pipeline and expected the gains to add up, a new study says you will probably be disappointed. In Retrieval Augmented Generation Compositionality, researchers at IBM and the Weizmann Institute (Ido Levy, Segev Shlomov and Leshem Choshen) tested 450 pipeline configurations and found that three retrieval techniques which predicted a combined +47% NDCG@10 gain delivered +17%. The paper is arXiv 2610.29722, and the team released an Apache 2.0 framework, Lasagna, so you can run the same test on your own stack.

The study is a preprint from the authors, tested on one English test bed with proprietary generators, so treat the exact percentages as indicative. The direction of the findings, though, matches what many RAG builders report from experience.

TL;DR

Why the question matters

Modern RAG pipelines have many stages: document processing, chunking, embedding, query enhancement, retrieval, reranking, context compression and agentic verification. Research papers and vendor demos usually test a new technique one at a time, in a stock pipeline. The authors ask whether the gains survive when techniques are combined, whether the choices that maximize retrieval quality also maximize answer faithfulness, and whether conclusions from small corpora hold at large scale. They pre-declared three hypotheses before running experiments, an uncommon practice in this literature.

Finding one: retrieval gains shrink when combined

Retrieval techniques tend to fix overlapping problems. A stronger embedding model already surfaces most of the relevant passages that a reranker would have lifted, so adding the reranker later adds less. The paper quantifies it: three strongest techniques, measured separately, add up to +47% NDCG@10, but together they deliver +17%, a 63% composition shortfall. Across 17 tested combinations the shortfall ranges from 7% to 72%, with a median of 41%. The authors also note that the pre-declared first hypothesis test was not met: the interaction it flagged is real but small, so the headline example is the largest case, not a universal rule.

Finding two: faithfulness behaves differently

Faithfulness, the share of generated claims supported by the retrieved context, behaved differently. Predicted +28%, realized +32%: gains combined nearly additively. The factors that matter also differ. Embedding, enrichment and reranking dominate retrieval quality, while query augmentation and agentic gating dominate faithfulness. The practical consequence: tuning for one metric can hurt the other, so a pipeline optimized for NDCG@10 may produce less grounded answers.

The authors' project page shows main effects and pairwise interactions across eight pipeline factors, with retrieval (167 configurations) in the upper triangle and faithfulness (122 configurations, scored with RAGAS) in the lower triangle.

Finding three: the winner changes with corpus size

The authors scaled the corpus from 5K to 200K documents. The design that led at small scale trailed by 13.7% NDCG@10 at 200K, with the rank flip visible from 25K documents. That matters because many teams tune on a few thousand documents in development and then deploy on millions.

How the experiment was designed

The study unifies eight benchmarks into one multi-domain corpus: five BEIR sets for retrieval (NFCorpus, FiQA, SciFact, ArguAna and HotpotQA) and three question-answering sets for generation (SQuAD v2, HotpotQA and MuSiQue). Each configuration is evaluated on a retrieval track and an end-to-end generation track. Rather than testing every possible combination, which is infeasible, the authors used a D-optimal factorial design over eight dimensions: 167 configurations for retrieval and 122 for faithfulness, with the rest used for operational profiles and scaling runs. Retrieval is scored with NDCG@10, following the BEIR convention of de-duplicating retrieved chunks by parent document before scoring, and faithfulness with RAGAS.

This design is why the results are more informative than a leaderboard: it estimates main effects of each factor and pairwise interactions between them, which is exactly what you need to know whether two techniques overlap.

An intuition for why gains overlap

Imagine your baseline retriever misses a relevant passage in 30 percent of queries. A better embedding model fixes half of those misses. A reranker, tested alone, would also fix about half, by moving the buried passage up. Tested alone, each shows a big gain. But if the embedding model has already fixed the easy half, the reranker has fewer misses left to fix, and its marginal gain is smaller. Query rewriting is the same story: it helps when the first retrieval missed, and a stronger retriever misses less often. That overlap, rather than any flaw in the techniques, is the plausible mechanism behind a 63 percent shortfall. The faithfulness result fits the same logic in reverse: its main levers, query augmentation and agentic gating, act at different stages from the retrieval levers, so their effects do not compete for the same errors. The paper's authors present the numbers, not this explanation, so read the mechanism as our reading of why the data look this way.

What Lasagna is

Lasagna is the evaluation framework behind the study, released under Apache 2.0. According to the project page:

• Eight swappable stages: document processing, chunking, embedding, query enhancement, retrieval, reranking, an agentic loop and answer generation. Each sits behind a fixed interface so swapping one method cannot silently change another.
• 55 built-in components: dense, BM25, SPLADE and hybrid retrievers; cross-encoder, ColBERT, monoT5 and RankGPT rerankers; HyDE, Query2doc, step-back and RAG-Fusion query enhancers; Self-Ask, IRCoT, FLARE, Adaptive-RAG and Self-RAG loops; cited (ALCE) and chain-of-note answers. Each component implementing a published method names its paper.
• Stage isolation checks: each run records what reached each stage, and an arm whose ranked lists never moved is flagged as inert.
• Paired significance tests: a paired sign-flip test with a t interval for each comparison, Holm correction across a sweep, and a test of whether two techniques compound or overlap.
• 31 benchmarks across four metric families, covering retrieval, answer quality with and without an LLM judge, citation scores, and latency, tokens and LLM calls per query.
• Serving: an MCP server for Claude Code, Claude Desktop and Cursor (22 tools by default, none that deletes), a REST API and a Docker stack with six vector backends.

The authors also checked reproducibility: on SciFact and NFCorpus with nothing tuned, dense models land within 0.002 NDCG@10 of MTEB numbers, and BM25 plus a cross-encoder lands within 0.004 of the BEIR paper.

How to use it on your own pipeline

The workflow on the project page is three steps. First, write a technique as a plain Python function, such as a reranker that scores a passage by keyword overlap, and register it with a decorator. Second, name your component next to a built-in in a sweep file, using a speed profile that needs no API key. Third, run the sweep and read the significance table. The goal is a verdict on what your idea adds when the rest of the stack is already in place, not in isolation. There is also a Colab notebook on 100 BEIR SciFact queries that needs no API key.

What this means for RAG builders

1. Do not stack on faith. If you have five improvements that each looked good alone, expect less than the sum. Test the combination.
2. Measure faithfulness too. A change that raises retrieval metrics can lower groundedness; track both.
3. Test at deployment scale. Re-run key comparisons at 10x or 100x your dev corpus before committing.
4. Remove what does not help. If a component's marginal contribution inside the full pipeline is near zero, drop it and save latency and cost.
5. Budget for experiments. The study's cost is a reminder that a proper ablation is hundreds of runs, not three.

Our own earlier guides cover the design choices the paper tests: RAG versus agentic RAG, context injection pipeline design, when to ground versus fine-tune, and prompt engineering versus fine-tuning versus RAG. For components, see multi-vector ColBERT in Sentence Transformers v6, NVIDIA's Nemotron 3.5 reranker, and Perplexity's Q2D-Web agentic RAG benchmark.

Caveats

• Preprint. The paper is a preprint on arXiv, discussed on Hugging Face's papers page, and not independently replicated.
• One English test bed and proprietary generators. The authors say so; results may differ with open generators or other languages.
• One hypothesis not met. Their pre-declared H1 test failed, with a real but small interaction.
• Corpus range. 5K to 200K documents is far below many production systems.
• Vendor stack. The framework includes an MCP server and a Docker stack; as with any new framework, check its maintenance before depending on it.

Related reading

• RAG versus agentic RAG
• RAG context injection pipeline design
• Grounding, RAG and fine-tuning decision guide
• Prompt engineering vs fine-tuning vs RAG
• Sentence Transformers v6 and ColBERT
• Official: arXiv 2610.29722 and Lasagna project page

Details are accurate as of October 22, 2026 and come from the paper and project page. Results are the authors' own.