Apple's Selection-Based Reasoning: Small Search Agents That Pick a Thought Instead of Writing One

Apple researchers have a way to make small search agents faster: stop letting them write their reasoning, and make them pick it. In Selection-Based Structured Reasoning (SSR), a multimodal agent chooses its next high-level reasoning step from a menu of pre-written candidates by scoring how likely each is. Apple reports that per-turn reasoning latency falls by over 90% and total per-question inference latency by 28 to 54%, while average success stays competitive with leading search agents of the same size.

The paper is on arXiv as 2610.01892, from authors at Carnegie Mellon, UIUC and Apple. Apple's page is a summary, and it leaves out several things a reader would want: the benchmark names, the accuracy figures, the baselines and the limitations. This post explains the idea, what is reported, and what to check in the full paper.

Quick reference

Why free-form reasoning is a problem for small agents

A search agent works in turns. At each turn it looks at the question, the images and the results so far, decides what to do next, calls a tool such as a web or image search, and repeats until it can answer. Many recent agents write a block of reasoning before each action.

Apple's summary says the trouble is that small models often generate long reasoning that does not meaningfully guide the next action, and that this text adds substantial inference cost. Every reasoning token is decoded one at a time, so a verbose thought is the slow part of a turn. For a 2B or 4B model that might run on a laptop or phone, that cost is the difference between usable and not. Our post on why small models are suddenly viable covers the economics behind that pressure, and Stanford's prefix-sliding result is another recent attempt to make reasoning cheaper.

How SSR works

A globe with a magnifier over a result card, picturing a multimodal search agent choosing its next step

The page describes three moves.

1. Write the recurring steps down in advance. Common high-level reasoning steps, such as deciding to search with the text, with the image, or to answer, are written as reusable natural-language candidates. The page does not list the candidates, so how they are built is a question for the paper.
2. Select by likelihood. At each turn the model scores every candidate by its likelihood given the current context and picks one. The page notes this needs no auxiliary task head: the language model's own probabilities do the choosing.
3. Score in parallel. Because the candidates are fixed, teacher-forced prefilling can compute all their token likelihoods at once, within and across candidates, sharing a single context KV cache. Instead of decoding a paragraph token by token, the model reads the candidates in one batch.

The speed-up follows from the third point. Prefilling is the fast, parallel phase of inference; decoding is the slow, sequential one. SSR moves the reasoning step from the second phase into the first. If you want background on why the KV cache matters this much, see how DeepSeek cut KV cache memory in V4.1 Flash.

The authors cite two lines of prior work on the page: a July 2026 ICML paper on structured reasoning via tractable trajectory control, and a May 2025 paper on interleaved reasoning via reinforcement learning. SSR sits in the family of methods that impose structure on reasoning rather than letting it sprawl.

What the results say, and what they do not

A bead balanced between a feather and a target, illustrating the latency and accuracy trade-off SSR targets

Apple reports:

• Per-turn reasoning latency: down by over 90%.
• Total per-question model inference latency: down by 28 to 54%.
• Success rate: average competitive with leading search agents of the same scale.
• Training: the 2B and 4B models were trained with multiple reinforcement learning objectives and supervised fine-tuning.

The gap between the two latency figures is worth reading carefully. Reasoning is only part of a turn: the agent still has to call tools, wait on search, and read results. A 90% cut to the reasoning slice gives a 28 to 54% cut overall, which is what you would expect if reasoning is a large but not dominant share of the time.

What the page does not give:

• The names of the seven benchmarks, so we cannot tell how hard they are.
• Any accuracy number, or the baselines it is "competitive" with.
• A limitations section. Our reading of the method, not a claim by Apple, suggests two questions for the paper: how the candidate set is built, and how the agent behaves when a situation falls outside the pre-written steps, since a fixed menu cannot express a thought it does not contain.

Until the benchmark tables are read, treat "competitive" as a claim to verify, not a result. See our other recent Apple research posts for how the company frames similar efficiency gains, including SelfCompact for agent context and Apple Minions for idle-GPU inference.

Why it matters for people building agents

Reasoning can be a design choice, not an emergent text blob. SSR treats reasoning as a controlled vocabulary. That makes agent behavior easier to inspect and log, since each turn leaves a label rather than a paragraph, and easier to evaluate step by step.

It targets the right hardware. Apple's own focus on on-device models makes small, fast search agents a natural interest. Cutting decode work matters more on a phone or Mac than in a datacenter, and Apple's local-AI hardware push is the other half of that story.

It is a template you can borrow. You do not need Apple's code to try the idea in a small agent: define a short list of strategy options, score them with the model's own log-probabilities in one batched forward pass, and take the argmax. Whether it works as well as free-form reasoning on your task is an empirical question, and the paper's reinforcement learning training is part of why it works for Apple.

One turn, step by step (our illustration, not from the paper)

Suppose an agent is shown a photo of a landmark and asked when the building was completed. A free-form agent might write a paragraph: the image shows a tower, I should search for it, then check the completion date. Then it calls a search tool.

An SSR-style agent has a short menu in its head instead, for example: search using the image, search using the text, refine the query with what was just found, or answer now. It does not write any of that. It computes how likely each menu line is given the question, the image and the history, picks the most likely, and moves to the action. Because the menu is fixed, all four likelihoods come from one batched pass over a shared cache of the context. The only text the model decodes is the action itself, such as the search query.

Two consequences follow. First, the cost of reasoning no longer scales with how much the model feels like saying, so latency becomes predictable. Second, a small model cannot ramble itself into a wrong turn, because it can only choose among thoughts a person wrote. That is a strength for reliability and a limit for novelty, which is exactly the trade-off the paper's evaluation needs to quantify.

How this differs from other ways to cut reasoning cost

Most efforts to make reasoning cheaper work by shortening what the model writes: training it to be terse, capping its thinking budget, or switching thinking off for easy questions. SSR takes a different route, removing the writing altogether for the high-level step and keeping the model's judgment as a probability over choices. It is closer in spirit to a classifier than to a chain of thought.

That also marks where it may not transfer. Search is a good fit because the high-level moves repeat: look it up, look at the image, refine, answer. Tasks whose next step is genuinely open-ended, such as debugging unfamiliar code, do not have a short list of recurring moves. If you build agents, a useful test is whether you can list the ten or so strategic moves your agent makes in practice. If you can, a selection layer may cut cost. If you cannot, free-form reasoning is probably earning its keep.

What to check in the paper

• The seven benchmarks and the per-benchmark success rates.
• The baselines and their model sizes.
• How candidates are written: by hand, by an LLM, or mined from traces.
• How much reasoning-free ablations lose, and how often the right step is not on the menu.
• Whether the latency numbers include tool and network time.
• Whether code or checkpoints are released.

Related reading

• Small models have arrived: the economics
• Stanford's prefix sliding: 3x faster reasoning
• Apple SelfCompact: self-compacting agent context
• Apple Minions: distributed inference on idle Macs
• Mixedbread Toast-1 search agent
• Qwen multimodal plugins agent harness
• Apple's companion paper on tool discovery

Details are accurate as of October 15, 2026, based on Apple's research summary page. Read the arXiv paper for numbers.