Perplexity Q2D-Web: 190M-Doc Benchmark for Agentic RAG Retrieval

On September 10, 2026, Perplexity published Q2D-Web (Query2Doc-Web) — a benchmark built to answer a question most retrieval leaderboards still dodge: how good is your first-stage retriever when an agent, not a human, writes the search query? The release extends Perplexity's earlier Q2D embedding benchmark to web scale: 190 million documents, 69,721 agent-reformulated queries, ten languages, and an average of 99.6 positive relevance judgments per query in the combined label set.

If you build agentic RAG pipelines, ship embedding models, or tune hybrid search stacks, Q2D-Web is the first public benchmark that simultaneously hits three constraints production systems actually face — massive corpora, machine-written queries, and dense relevance signal per query. Human-query suites like MS MARCO helped the field for a decade; agentic search is a different distribution.

TL;DR — what Q2D-Web is and why it landed now

The evaluation gap Q2D-Web is trying to close

Most teams running RAG context pipelines know the retrieval step is where answers quietly die — wrong chunks, missed pages, stale embeddings. What gets less airtime is who writes the query at retrieval time.

In classic RAG tutorials, the user types a question and that string goes straight to the vector database. In agentic RAG, the flow looks different: the agent reads conversation history, maybe prior tool outputs, and reformulates a search query before the first retrieval hop. That reformulation is often longer, more specific, stuffed with entity names, or split across sub-queries — nothing like the short keyword strings in MS MARCO click logs.

Perplexity's paper states the problem plainly: large-scale collections usually ship with few evaluation queries, while benchmarks with many queries tend to cap out at millions of documents, not hundreds of millions. And almost all of them evaluate human-written queries. Q2D-Web is positioned as the intersection point — web-scale corpus, agent-scale query count, agent-authored query distribution.

Scale in one comparison table

The label density matters. When each query has only one positive document in the test set, a retriever that ranks a different but equally good page at position two looks like a failure. At ~100 positives per query, the metric tolerates multiple valid answers — closer to how semantic and hybrid search behaves on the open web.

How Perplexity built the query set

The query corpus is not synthetic paraphrasing run in a lab. Perplexity sampled 23,000 PII-free production searches collected over nine months, then expanded that into 69,721 agent-reformulated queries — the kind their agent stack actually emits when grounding answers on the web.

Language mix reflects real product traffic rather than an English-only convenience sample:

• English: 65.8% of queries
• Nine other languages share the remainder — important because multilingual embedding models often look strong on English MTEB rows and weaker everywhere else

Topical coverage spans domains where wrong retrieval is costly: programming, law, health, science, finance, and additional buckets from live usage. Perplexity's initial leaderboard breakdowns show model ordering can diverge substantially by domain and language even when global averages look stable — a reminder that one aggregate Recall@1000 number hides deployment risk.

Three relevance-judgment sets — and why one ground truth is not enough

At 190 million documents, exhaustive human labeling is impossible. Perplexity instead publishes three fixed judgment sets so teams can inspect how retrievers behave under narrower vs broader relevance definitions:

1. Citation judgments

Documents the agent actually cited in its final answer. High precision — if the agent cited it, the page was useful in production — but lower recall, because agents skip relevant pages they never needed to quote.

2. Web Ranking judgments

Labels from Perplexity's production retrieval stack: BM25 plus dense retrieval plus cross-encoder reranking, selecting up to 50 documents per query (averaging about 43.1). This captures what their hybrid pipeline considered worth surfacing even if the answer did not cite every hit.

3. Combined + LLM-judged

The union of Citation and Web Ranking signals, plus LLM-based binary relevance judgments on pooled candidates that still lacked labels — Perplexity uses this to reduce false negatives (relevant pages that never got clicked or cited). The 99.6 average positives per query headline number refers to this combined view.

Across 13 evaluated retrievers (lexical, dense, and late-interaction families), Perplexity reports that relative model ordering stays largely consistent across the three judgment sets, while absolute Recall@1000 scores shift. That is good news for practitioners: you are less likely to pick the wrong embedder because you clicked the wrong leaderboard tab. It is also a reason to look at all three — Citation-heavy views reward precision for answer grounding; Combined views stress recall for research-style agents that read widely before synthesizing.

Early public leaderboard rows (subject to change as submissions arrive) reportedly put pplx-embed-v1-4b near the top on Web Ranking and Combined Recall@1000, with Nemotron-3-Embed-8B leading on the Citation view — illustrating that "best retriever" is not one universal slot.

Subcorpus sampling — making 190M-doc eval tractable

Full-corpus evaluation over 190 million documents is expensive enough that most teams will never run it weekly in CI. The Q2D-Web paper documents a subcorpus sampling shortcut:

• Retain roughly one-third of the corpus
• Select that slice using reciprocal rank fusion (RRF) over pooled retriever runs
• Preserve full-corpus model rankings under combined judgments
• Increase absolute Recall@1000 by only about 3–7 points vs the full corpus

For embedding vendors, that means leaderboard submission can approximate full-scale conclusions without re-indexing the entire web for every ablation. For buyers, it means asking submitters whether scores are full-corpus or subcorpus before comparing numbers across papers.

What this means if you build agent search or RAG

First-stage retrieval is still the bottleneck

Agentic systems that ground via search tools or MCP often run retrieve-then-rerank-then-read. Q2D-Web evaluates the first hop — the embedder or lexical stage that must surface a thousand candidates from hundreds of millions of documents. If that stage misses the right domain entirely, no reranker or long-context window recovers gracefully.

Perplexity's own product arc makes the stakes obvious: they ship Search API infrastructure and embedding models (pplx-embed) in the same ecosystem Q2D-Web measures. The benchmark is partly vendor-neutral science (public leaderboard, arXiv paper) and partly alignment tooling for their retrieval stack — read it with that lens, the same way you would read any lab benchmark tied to a product line.

Human-query benchmarks understate agent-query failure modes

If your eval set is "employees type questions into a Slack bot," you may never see the failure mode where an agent expands a vague follow-up into a six-clause boolean query that drifts lexically from your fine-tuned embedder's training distribution. Q2D-Web is a prompt to log agent reformulations in production and build a private eval slice alongside public scores — the same lesson GBrain's self-graded memory evals teach for long-horizon agents, applied here to retrieval.

Do not confuse retrieval benchmarks with end-to-end answer quality

High Recall@1000 on Q2D-Web does not guarantee your agent writes correct answers. Downstream steps — reranking, snippet extraction, context injection order, citation discipline — still dominate user-visible quality. Use Q2D-Web to shortlist embedders and hybrid configs, then eval end-to-end on your tasks.

When retrieval beats fine-tuning — and when it does not

If your problem is "find the right public web page for an agent-written query," better embeddings and hybrid retrieval are the lever. If your problem is "answer from our private Notion/GitHub corpus with company-specific vocabulary," Q2D-Web is an indirect signal at best — run domain eval or consider whether you need RAG vs fine-tuning vs grounding at all.

What people are asking about Q2D-Web

"Is this just marketing for pplx-embed?"

Partly aligned, not purely marketing. Perplexity publishes the paper, hosts an open Hugging Face leaderboard, and benchmarks competitors' open models alongside their own. Still: they control the corpus snapshot, judgment pipelines, and production ranking labels. Treat top-line rankings as strong priors, not court verdicts — especially for domains underrepresented in their nine-month traffic sample.

"Can I use Q2D-Web to pick an embedder for my internal wiki?"

Only as a coarse filter. Your corpus is smaller, your queries may stay human-authored, and your relevant documents are not web pages about law and finance. Start from Q2D-Web if you deploy open-web agent search; otherwise prioritize in-domain recall@k on your chunks.

"How does this relate to agentic RAG vs PageIndex-style search?"

Q2D-Web measures dense/sparse first-stage retrieval over a fixed web index — the classic embedding/BM25 paradigm at scale. Agentic RAG systems that skip pre-indexing and use grep, filesystem traversal, or structured page trees are not directly scored here. If your architecture avoids a billion-vector index entirely, Q2D-Web informs component choices when you do call a web search API — not whether you need one.

"What metric should I watch?"

The leaderboard emphasizes Recall@1000 across judgment sets — appropriate for first-stage retrieval where rerankers consume a wide candidate pool. If your stack retrieves top-20 only, ask whether leaderboard gaps at k=1000 still matter at k=20 before switching models.

Submission workflow and reproducibility checklist

Perplexity documents a submission path for publicly available Hugging Face models via the leaderboard space. Before you submit or cite a number in a deck:

1. Note the judgment set — Citation, Web Ranking, or Combined — when quoting Recall@1000.
2. Check language and domain slices if your users are not 65.8% English web searchers.
3. Confirm full vs subcorpus eval when comparing against paper numbers.
4. Re-run on your agent reformulations — even 500 logged production queries beat arguing about single global averages.

Official sources: Perplexity Hub announcement, arXiv:2609.08887, Hugging Face leaderboard.

Bottom line

Q2D-Web is the most serious public attempt yet to benchmark first-stage web retrieval under agent-written queries at production scale — 190 million documents, nearly seventy thousand reformulations, and roughly a hundred positive labels per query where older web benchmarks offered one. It will not replace your private eval, and it does not score end-to-end agent correctness. It does give embedding teams, search API buyers, and agent builders a shared ruler for the retrieval hop that everything else depends on.

If you are shipping agentic search in 2026, log your reformulated queries, pick judgment sets that match your precision/recall appetite, and treat Q2D-Web as the web-scale baseline — not the finish line.

Related on explainx.ai

• RAG vs Agentic RAG: why search beats embeddings for code retrieval
• RAG and context injection: designing retrieval pipelines that work
• RAG vs MCP: complete comparison for context-aware agents
• Perplexity Search API index debut and agent wiring patterns
• Grounding vs RAG vs fine-tuning: decision guide
• Semantic vs vector vs hybrid search
• GBrain evals and self-graded benchmarks for agent memory

Benchmark figures, leaderboard rankings, and submission rules are accurate as of September 10, 2026, per Perplexity's Hub post and arXiv:2609.08887. Leaderboard positions change as new models are submitted.