ReMaRk: Train RL on the Request, Not Each Answer, to Cut GRPO Noise

When a question has several valid answers, which one should a reinforcement-learning reward compare your model against? A new paper from Nanyang Technological University, Many Right Answers, One Training Example, argues the usual recipe makes the wrong choice and adds noise. Its fix, ReMaRk, trains on the request rather than each reply. The authors report reaching the same validation recall in roughly a quarter to a half of the RL steps, and they released code, data and 23 checkpoints at github.com/LiXin97/ReMaRk.

This is a research result with a small, concrete change, which makes it easy to understand and, if you train recommenders or any model with GRPO on multi-answer data, easy to test. Below is the idea in plain language, the numbers, and the limits.

TL;DR

The setup: recommendation requests with many valid replies

The paper studies LLM recommenders trained on Reddit threads. A request such as "mind-bending sci-fi like Arrival?" draws several replies, and each names different titles. After supervised fine-tuning, the model is improved with reinforcement learning: it generates a ranked list, and a reward measures how many recommended titles match what people named.

The state-of-the-art recipe, Rank-GRPO, turns each reply into a separate training example. For one request with five replies there are five rows, and each generated list is rewarded only for the titles named by the single reply in that row. Requests with more replies are also drawn more often.

Why that adds noise

Repliers rarely agree. Most suggested titles are named by only one person. So the same good list can score high against one reply and low against another. The authors measure this: the identity of the drawn reply explains 85% of the variance of Rank-GRPO's advantages, and in more than half of comparisons every list in a group gets the same score, which means no learning signal.

GRPO normalises rewards within a group of sampled outputs, so the signal that matters is how lists differ from each other. If the reward is dominated by an accident of data collection (which reply was paired with this row), the update mostly reflects luck. The paper calls the responder a "nuisance variable."

The fix: one row per request, one reward against everyone

ReMaRk changes only the training unit and the reward:

1. Union reward. A list earns 0/1 credit at each position if any replier of that request named the title.
2. Request sampling. Each request is drawn once, equally, rather than once per reply.
3. Everything else unchanged. The GRPO update, the model and the budget stay the same, with length control held fixed.

The paper's theory note: averaging the reward over repliers is a Rao-Blackwellization, so at a fixed advantage scale the expected update is unchanged and a variance term disappears. The authors keep the 0/1 union reward because GRPO's per-group scaling can amplify one replier's vote when rewards are fractional. They also report that once list lengths are controlled, an averaged reward is level with the union at the 2B size, so the benefit is from pooling the repliers, not from the specific aggregate.

A toy example of the difference

This is our illustration, with made-up titles, not data from the paper. Suppose a request has three replies: reply A names titles 1 and 2, reply B names titles 3 and 4, and reply C names title 5. The model samples two lists: list X contains titles 1 and 2, and list Y contains titles 3 and 5.

Under the per-reply recipe, if reply A is drawn for this training row, list X looks excellent (two hits) and list Y looks poor (zero hits). If reply B is drawn instead, the ranking of the two lists flips in part. The learning signal depends on a coin flip made during data preparation.

Under ReMaRk, both lists are scored against the union of titles 1 to 5. List X gets two hits, list Y gets two hits, and the group sees a consistent view of how good each list is regardless of which reply was "drawn." Noise from the draw is gone, and the comparison between lists reflects the quality of the lists. That is the whole mechanism, and it explains why the authors say the benefit grows with the number of repliers per request.

The results, in numbers

Across seven backbones (Qwen3.5 from 2B to 35B-A3B, Llama-3.2-3B and Llama-3.1-8B) and three datasets (Reddit-v2 movies, Reddit-Books, Reddit-Music), the project page and README report:

• Faster: the validation recall of 2,000 per-reply RL steps is reached in 25 to 53% of the steps in all ten backbone and dataset settings.
• Better at a fixed budget: test recall@20 after 2,000 steps is higher by about 0.8 to 1.6 points than the better per-responder baseline.
• Not worse when trained longer: trained until neither method improves, ReMaRk is never significantly behind Rank-GRPO on test, and in four of seven pairs it reaches Rank-GRPO's best with 18 to 63% of the GPU-hours.

Examples from the README table (test Recall@20 after 2,000 steps, seed 3407):

The gains are modest in absolute points but consistent, with confidence intervals above zero in the settings listed, and the cost saving is the larger story.

Beyond recommendations: ProtoQA

The same union-style reward helps on ProtoQA, a dataset where about 100 people answer each question, with a reported gain of +3.67 R@10 at 250 steps over three seeds. The authors draw a simple rule from this: when RL has several valid human answers, reward them jointly. That generalises to any task where ground truth is a set of acceptable answers: brainstorming, multi-reference summarisation, open-ended QA with several acceptable responses, or tool-use tasks with multiple valid solution paths.

Where it does not help

The authors are direct about one limit: on requests with a single replier, every aggregation equals the per-responder reward, so ReMaRk reduces to Rank-GRPO. Its value grows with the share of requests that have several repliers. The test gain also "follows the responder noise removed," according to their dose-response analysis, so datasets with few multi-reply requests will see little benefit.

Other cautions:

• The evidence is on Reddit recommendation datasets and ProtoQA; whether it carries to other domains is untested here.
• The book and music labels were extracted by an LLM (the README names GPT-5.6) and grounded to Goodreads and MusicBrainz, so label quality is part of the result.
• The checkpoints and results are from the authors' own runs of the Rank-GRPO baseline in the same pipeline, which is sensible for a controlled comparison but not independent replication.
• This is a preprint.

How to try it

The README gives a quick start: clone the repo, create a Python 3.11 environment with the pinned TRL, vLLM and DeepSpeed versions, clone Rank-GRPO's repository for its title parser and catalog, download the processed Reddit-v2 data, fetch a released checkpoint, and run crs_eval.py. The expected test number for the Qwen3.5-4B checkpoint is filter_R@20 of 0.2510. Training is SFT followed by GRPO on four GPUs for 2,000 steps with about 384 lists per step, with separate launch commands for the ReMaRk reward and the Rank-GRPO baseline. Licences: code Apache-2.0; data CC BY-NC 4.0 (non-commercial); Qwen-based checkpoints Apache-2.0; Llama-based checkpoints under Meta's community licences.

If your own data has multiple acceptable answers, the idea can be tested without the whole pipeline: group rows by prompt, compute the reward against the union of acceptable answers, and sample prompts uniformly. Keep rewards 0/1 per item so group scaling stays bounded, as the authors note.

What this means for people training with GRPO

Reward design gets less attention than algorithms, yet this paper shows that a data-construction choice (one row per answer) can cost most of the signal. Practical checks for your own RL runs:

1. Measure reward variance sources. If a large share of advantage variance comes from which reference was paired with the prompt, you are training on noise.
2. Look for groups where all samples get the same score. Those give zero gradient and waste compute; the paper found this in over half of comparisons.
3. Pool references per prompt when several are valid.
4. Report compute, not just final score. The sharpest claim here is about steps and GPU-hours.

For other recent RL-for-LLM efficiency work, see our coverage of asynchronous GRPO with tree packing in TRL, self-distillation with verbal regularisation, and simulation-free RL with Wasserstein tilted flow maps. For the broader view of RL as a path beyond language-model pretraining, read David Silver on Ineffable.

Related reading

• Asynchronous GRPO and tree packing in TRL
• VRSD self-distillation
• WTF: simulation-free RL, 280x less compute
• David Silver and Ineffable
• A $500 GRPO fine-tune that beats frontier models
• FineEnvs: multilingual OpenEnv tasks with GRPO

Details are accurate as of October 15, 2026, from the authors' paper page, project page and README; results are the authors' own.