How to Add a Reranker to Your RAG Pipeline: Models, Latency, and Cost

Reranking is the cheapest upgrade many RAG stacks skip. Vector search returns similar embeddings fast, but similarity is not the same as “this passage answers the question.” A cross-encoder reranker scores each query–passage pair jointly and reshuffles your shortlist before the LLM reads context. You keep fast retrieval at scale and pay extra compute only on the top-k candidates.

TL;DR — where reranking sits

Bi-encoder retrieval vs cross-encoder rerank

Sentence Transformers documents the trade-off clearly. A bi-encoder embeds queries and documents independently. You store document vectors once, compare with a dot product, and serve millions of chunks with predictable latency. A cross-encoder runs one neural pass per query–document pair. It is slower, but generally provides superior performance because attention can compare tokens across both sides.

That pattern is why rerankers almost always follow retrieval instead of replacing it. The Sentence Transformers docs describe cross-encoders as often used to re-rank the top-k results from a Sentence Transformer model. Your vector database or search index does the wide net; the reranker does the close read.

If you are still mapping the full pipeline, start with embeddings and vector search, then how chunks land in the prompt. Reranking is the optional quality layer between those two steps.

Hosted rerank APIs (Cohere and peers)

Cohere’s Rerank endpoint is the reference hosted API many teams copy. You POST a query and a list of documents to https://api.cohere.com/v2/rerank. The API returns an ordered list with relevance_score values normalized between zero and one, as described in the Rerank API reference.

Current model tiers include rerank-v5.0-pro for highest quality on long or domain-specific text and rerank-v5.0-fast for high-throughput agent loops. Both support a 32k-token combined query and document context. Cohere’s best practices guide allows up to 10,000 documents per rerank-v5.0 request; the Rerank API reference recommends sending fewer than 1,000 documents in one call for optimal performance.

Practical defaults for builder teams:

• Retrieve 50 chunks with your embedding model from open or closed options.
• Call rerank with top_n set to 10.
• Pass 5 reranked passages into the generator, reserving tokens for citations and instructions.

Cohere scores are query dependent. The docs warn you cannot treat 0.91 as “twice as relevant” as 0.44; use them for ordering, not calibrated business rules, unless you validate on your own queries.

Semi-structured chunks (JSON log lines, ticket exports) work better when you stringify them as YAML in the documents array, per Cohere’s formatting guidance. That detail matters when you prepare web or ticket data for Bedrock knowledge bases.

Azure AI Search semantic ranker (managed rerank)

If your corpus already lives in Azure AI Search, you may not need a separate rerank microservice. Microsoft’s semantic ranking overview describes a secondary ranking step over an initial BM25 or reciprocal-rank-fusion result. Semantic ranker promotes results using language models adapted from Bing.

Important constraints from Microsoft’s documentation:

• Reranking runs on text, even when the initial query was hybrid or vector.
• Only the top 50 results enter semantic ranking, regardless of how many you retrieved earlier.
• Each document summary is trimmed to fit token budgets on title, keyword, and content fields you list in semantic configuration.
• Responses can include captions, highlights, and optional semantic answers when the query looks like a question.

That design aligns with the cross-encoder pattern: expensive scoring on a shortlist, not the whole index. Teams studying for certification or building on Azure should read explainx.ai’s semantic vs vector vs hybrid search guide alongside Microsoft’s docs so you do not confuse “semantic search” vocabulary with embedding retrieval alone.

Semantic ranker is usage billed. Microsoft charges when queryType=semantic is set and the search string is non-empty. Empty search=* browse queries avoid semantic charges even if semantic mode is enabled.

Local and open-weight rerankers

Not every stack can send customer text to a third-party rerank API. Sentence Transformers ships a CrossEncoder class you can run on a laptop GPU or a small cloud instance.

A minimal local loop, adapted from the official usage docs:

from sentence_transformers import CrossEncoder

model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")
query = "What is our refund policy for annual plans?"
passages = [chunk["text"] for chunk in vector_hits[:50]]
ranked = model.rank(query, passages)

top_ids = [r["corpus_id"] for r in ranked[:5]]

Check model.modalities before you assume image–text support. Newer checkpoints such as Qwen3-VL-Reranker accept image URLs and mixed media, which matters for vision RAG releases and multimodal catalogs.

BGE reranker family models on Hugging Face are a common upgrade from MS MARCO MiniLM when you have GPU headroom. NVIDIA Nemotron 3.5 Rerank targets multilingual and vision-heavy pipelines if you are standardizing on NVIDIA’s open model stack.

MS MARCO–trained cross-encoders output logits, not zero-to-one probabilities. Sentence Transformers notes you can pass activation_fn=torch.nn.Sigmoid() when loading if you need bounded scores for logging or UI, without changing sort order.

Sizing top-k, latency, and cost

Vector index retrieval feeds a small shortlist into a RAG reranker before LLM context packing

Reranking cost is roughly linear in the number of candidates × model size. Rule of thumb:

Measure end-to-end p95 latency with and without rerank on your own query log. If rerank adds 400 ms but moves nDCG@5 by ten points on an eval set you trust, keep it. If latency doubles and eval scores barely move, shrink top-k or switch to a fast rerank tier.

Pairwise scoring also multiplies with batch agents. A coding agent that fires twenty retrieval calls per task will amplify rerank bills unless you cache embeddings and results at the session layer.

Do not rerank after you have already stuffed whole documents into the prompt. Summarize or chunk first, then rerank chunk text. Oversized passages get truncated by hosted APIs anyway; Cohere’s v5 models default max_tokens_per_doc to 4096 tokens per document.

When reranking fixes real failures

Chunking quality still limits rerank gains when passages are too large or mis-split

Rerankers help when users ask precise questions over noisy corpora:

• Policy PDFs where the right paragraph uses different words than the query.
• Ticket archives where bi-encoders confuse product names.
• Multi-hop questions where the answer span is short inside a long wiki page.

They help less when:

• Your index has only a few hundred pages and you can pass most of them through a long-context model.
• Queries are mostly unique IDs, serial numbers, or error codes where keyword-heavy hybrid search already wins.
• You have not fixed chunking bugs; reranking garbage chunks still yields garbage context.

Before you add GPU-heavy cross-encoders, run a labeled eval: fifty real user questions, gold passages, and measure recall@k before and after rerank. The LASagna RAG compositionality study is a sobering reminder that retrieval tweaks rarely deliver the gains slide decks promise unless you measure on your data.

Rerank vs agent memory vs MCP retrieval

Reranking does not replace agent memory products or MCP tool routers. It improves which static chunks enter the prompt for a single retrieval step. Agents that rewrite queries, call multiple tools, or store rolling summaries still need harness-level design from guides like Langflow workflows or your own orchestration code.

Think of rerank as a function inside one retrieval hop: retrieve_top_k → rerank → pack_context.

Wiring rerank into common frameworks

Most orchestration libraries expose rerank as an optional step after the retriever returns Document objects. In LlamaIndex, attach a postprocessor such as CohereRerank or SentenceTransformerRerank between the vector retriever and the response synthesizer. In LangChain, ContextualCompressionRetriever wraps a base retriever with a document compressor that calls Cohere or a local cross-encoder.

The integration pattern is the same everywhere: keep your index schema unchanged, call rerank in application code with the query string and page_content fields, then map corpus_id or index back to metadata for citations. Log both the pre-rerank and post-rerank order during eval so you can see which queries benefit. When you upgrade rerank models, rerun that log; vendors ship new -pro and -fast tiers regularly, and explainx.ai tracks major releases in the embedding and retrieval news archive.

Security and data handling

Rerank APIs see raw user queries and document text. Treat them like any other third-party inference endpoint: redact secrets before rerank, log retention policies matter for regulated data, and local cross-encoders help when contracts forbid cloud passage of customer content. That sits alongside broader MCP security practices when agents fetch tools and documents in the same turn.

Checklist before you ship

1. Baseline recall@10 with embeddings only on a frozen eval set.
2. Add rerank on the same set; record latency and dollar cost per thousand queries.
3. Tune top-k until marginal quality gains no longer justify milliseconds.
4. Monitor production with periodic spot checks; embedding drift and corpus updates change rerank value over time.
5. Document the stack so the next engineer knows which model names and API versions you pinned.

Reranking is not magic. It is a disciplined second pass that aligns retrieval scores with answer relevance. For most teams building serious RAG in 2026, that second pass is cheaper than fine-tuning the generator on bad context.

Related reading

• What are embeddings and vector search?
• Semantic vs vector vs hybrid search
• RAG context injection pipeline design
• Grounding RAG vs fine-tuning
• NVIDIA Nemotron 3.5 multimodal reranker
• AI evals for engineers and PMs