NVIDIA Nemotron 3.5 Rerank 8B: An Open Multimodal Reranker for Vision RAG

NVIDIA has published Nemotron-3.5-Rerank-8B-BF16 on Hugging Face, an open reranker for question-answering retrieval that can score documents made of text, page images, or both. According to its model card, it is "a world-class reranker optimized to produce a logit representing the relevance of a document to a text query," built for "classical or agentic vision RAG pipelines." The model has 8.38B parameters and was created on October 15, 2026.

Rerankers are the unglamorous middle of retrieval-augmented generation, and the part that often decides whether an answer is right. This post explains what the model does, how to run it, what the licence allows, and what NVIDIA has not shown.

TL;DR: questions readers will ask

What does a reranker do in a RAG pipeline?

A typical retrieval system works in two stages. A fast embedding model pulls maybe 50 to 200 candidate documents from a large index. A slower, more accurate model then reads the query and each candidate together and re-orders them, and only the top few go to the language model. That second model is the reranker, usually a cross-encoder: it sees the query and the document in the same pass, so it can judge relevance more finely than a vector comparison.

NVIDIA describes exactly that role. The reranker "evaluates these candidate documents in greater detail and reorders them," improving precision and, downstream, the accuracy of generated answers. If the idea of embeddings is new, our explainer on what an embedding is and the guide to semantic, vector and hybrid search cover the first stage.

What is new: documents that are pictures

The notable feature is that a document can be an image. The card says documents may contain "text, page images, or both, including charts, tables, and infographics." That matters for the many real corpora where the answer lives in a scanned PDF page, a slide or a chart, not in clean text. Instead of running OCR and hoping the table survives, a vision RAG pipeline keeps the page as an image and lets the model read it.

This sits alongside a wave of visual-document retrieval releases, including Cohere Embed 5 Pro and Fast on ViDoRe v3 and Perplexity's pplx-embed v2 late multimodal embeddings. Those are first-stage embedders; Nemotron-3.5-Rerank is the second-stage counterpart.

Architecture and limits

Per the card:

• The model has a Pixtral-based vision encoder and a bidirectional Ministral3 language-model tower.
• Training had two stages: first a text-only reranker derived from Ministral-3-8B-Instruct-2512, then a second stage trained on image and combined image-text data.
• Input is one text query plus one document. The document can hold at most one image. Queries must be text.
• The default image longest edge is 1,120 pixels and can go up to 1,540.
• Output is one float per pair, a raw logit, with no built-in normalisation.

Because it is a pairwise scorer, cost scales with the number of candidates you rerank. Scoring 100 candidates means 100 forward passes through an 8B model, each possibly including image tokens. Plan latency accordingly, and rerank only the top candidates the first stage returns.

How to run it

The card gives three paths. The Transformers example loads AutoModelForSequenceClassification with trust_remote_code=True, bfloat16 and Flash Attention 2. It requires a CUDA GPU and transformers between 5.5.0 and 5.17.0; NVIDIA notes the upper bound avoids a Pixtral Flash Attention regression. Training and evaluation used transformers 5.12.1.

The Sentence Transformers route is shorter:

import torch
from sentence_transformers import CrossEncoder

model = CrossEncoder(
    "nvidia/Nemotron-3.5-Rerank-8B-BF16",
    device="cuda:0",
    max_length=32768,
    model_kwargs={"attn_implementation": "flash_attention_2", "dtype": torch.bfloat16},
)
ranking = model.rank("What is NVIDIA NeMo Framework?", [text_doc, image_doc], batch_size=1)

A third path uses vLLM 0.31.0, both offline and as an HTTP server, with the query and document passed through the chat template under the query and document roles.

The card's own example output ranks a document with both image and text first (score -1.80), the image alone second (-1.82), the text alone third (-2.10) and an unrelated biology passage last (-2.96). Note that scores are logits and can be negative, so compare them within one query, never across queries. NVIDIA warns outputs vary with hardware and runtime.

What are the licence and the catches?

NVIDIA says the model is "ready for commercial or non-commercial use" and is governed by the OpenMDW License Agreement 1.1, while the underlying Ministral-3-8B base model is Apache 2.0. Hugging Face tags the licence as "other," so read the licence file before you ship. We have not summarised its terms here.

Other points worth knowing:

• Release date inconsistency. The card's release-date field says 10/20/2026, while the repository was created on October 15, 2026. We treat the repository date as the publication date and the card field as a probable typo, but we cannot confirm which is intended.
• No benchmark table in what we read. "World-class" is NVIDIA's wording. Public reranker leaderboards change quickly, and the card as we saw it did not report scores on a named benchmark. Run your own evaluation.
• GPU-only examples. The shipped examples require CUDA and Flash Attention. Running on CPU or Apple silicon would need your own work.
• Early traction. At the time we fetched the page, it showed 439 downloads and no likes, which is normal for a model hours old and says nothing about quality.
• Custom code. The Transformers path needs trust_remote_code=True. Review the repository code before enabling that in production.

Where should the reranker sit, and how many candidates should it see?

The practical design question is the candidate budget. A cross-encoder cannot scan a whole index, so the first stage sets a ceiling on recall: if the right page is not in the candidate set, no reranker can rescue it. A common starting point is to retrieve a generous set, say 50 to 100 candidates, rerank all of them, and pass the top three to five to the generator. Because this model scores one query and one document per pass, you can batch candidates on one GPU, but each image candidate adds tokens, so image-heavy corpora cost more per query than text-only ones.

Three habits keep the cost manageable:

1. Cache document-side work where you can. Cross-encoders cannot precompute document embeddings, but you can cache final scores for repeated queries and avoid reranking near-identical questions twice.
2. Rerank in tiers. Run a cheap text-only reranker first on text-heavy candidates and use the 8B multimodal model only for candidates that are images or mixed pages.
3. Set an image size on purpose. The default longest edge of 1,120 pixels is cheaper than 1,540. Raise it only if small print in charts is being missed, and measure the change.

How to tell if it is working

Reranker quality shows up in retrieval metrics before it shows up in answers. Track recall at the top three results and mean reciprocal rank on a fixed question set, then check end-to-end answer accuracy separately. If the reranker improves ranking but answers do not improve, the bottleneck is the generator or the chunking, not the reranker. Also inspect failures by document type: charts, dense tables, scanned text and slide decks fail in different ways, and an average can hide a weak spot that matters to your users.

Finally, watch score calibration. The model outputs raw logits, so a threshold such as "drop anything below minus 2" is specific to this model and your data. Pick thresholds from a validation set, not from the single example in the card.

How it fits NVIDIA's Nemotron 3.5 family

The reranker is a small piece of a larger open-model push. Earlier this week we covered the Nemotron 3.5 Super VL 120B multimodal agent model, and the Nemotron 3.5 Bolt model shipped with the RTX Spark launch software. Pairing an open reranker with open generators gives teams a fully self-hostable retrieval stack, which matters for regulated data that cannot leave your infrastructure.

For choices among embedders to put in front of it, see our ranking of open and closed embedding models.

What this means for what you build or pay

If your RAG system already works on clean text, a text-only reranker may be cheaper and enough. This model earns its 8B footprint when your corpus is full of charts, tables, slides and scans, where parsing loses information. An open licence means no per-query reranking fee, but you pay in GPU hours: an 8B model scoring dozens of candidates per query is a real serving cost.

A sensible trial plan: take 100 real questions with known answers, retrieve the top 50 with your current embedder, rerank with this model and with your current reranker, and compare how often the right page lands in the top 3. Test text-only, image-only and mixed documents separately, and check non-English queries if you serve them.

Related reading

• Cohere Embed 5 Pro and Fast on ViDoRe v3
• Perplexity pplx-embed v2 late multimodal embeddings
• Nemotron 3.5 Super VL 120B
• Top open and closed embedding models
• Semantic vs vector vs hybrid search
• What is an embedding?

Details are as shown on the Hugging Face model card on October 15, 2026 and may change.