HybridAgent: Together and Stanford Let Agents Think in Latent Space, Act in Text

Most LLM agents reason in text, one token at a time, because text is also how they call tools. A new paper from Stanford, UIUC and Together AI asks whether an agent can think in compressed latent vectors and still act through text. Their answer, HybridAgent, switches between the two within one generation. The authors report a 10.4 percent average improvement, 35.5 to 54.7 percent fewer tokens and 2.7 to 3.7 times lower cost across 11 benchmarks. The paper is arXiv 2610.18861, submitted October 13, 2026, and the code is on GitHub.

This post explains the idea in plain language, what the numbers mean, what is open, and how it relates to the debate about readable reasoning, which we cover in our pieces on recurrent depth and OpenAI's looped models.

Quick reference

The problem: agents "think out loud" at a price

The paper's introduction frames text as the "lingua franca" of agents: both the reasoning and the interaction with tools happen as generated tokens. Every intermediate step must be decoded and written before the agent can act. That is expensive, since tokens cost money and time, and it is slow.

Latent reasoning is a different line of research. Instead of writing each thought as words, the model feeds its hidden state back into itself and "thinks" in continuous vectors. It is more efficient, but the authors point to two gaps: latent reasoners lack native paths for tool calls across multi-turn agent work, and their vectors are hard to interpret, which makes them difficult to supervise in training and to audit at inference.

HybridAgent tries to close both gaps: think beyond text, but still act through it.

How HybridAgent works

Two modes. The agent generates autoregressively in either text mode (sample a token and embed it) or latent mode (reuse the last-layer hidden state as the next input). It switches dynamically within a single generation.

HybridLink. Standard latent reasoning feeds the hidden state straight back as an input, which causes a mismatch with the pretrained input space and makes transitions between latent states and tokens hard. The paper adds a lightweight residual adapter, HybridLink, that maps last-layer hidden states back into the input embedding space, so latent steps look like valid inputs.

Working memory. For multi-turn work, the agent carries the whole hybrid trajectory (text, latent thoughts and tool observations) in its own continuous working memory. The authors say this losslessly preserves hybrid context across turns, and provide theoretical analysis for it.

Training in two stages.

1. HybridSFT teaches the backbone to coordinate with HybridLink, using activation and self-conditioned training to close the mismatch between training and inference.
2. HybridRL learns when to think in latent space and when to act through text, using hybrid policy factorization and exact rollout replay so policy updates cover complete hybrid trajectories.

The authors say any standard pre-trained LLM can be converted through this post-training pipeline. They instantiate it on Qwen3-Base models.

The reported results

According to the abstract, compared with frontier LLM agents and agentic frameworks spanning text-based and latent methods, HybridAgent "consistently improves performance by 10.4%". Efficiency numbers:

The paper evaluates at three scales (4B, 8B, 14B) across 11 benchmarks: mathematics, science, medicine, knowledge-intensive search and challenging agentic reasoning. It also reports strong test-time scalability and analyses of training dynamics, reasoning auditability and tool-use reliability.

How to read them: these are the authors' numbers from their own evaluation, with baselines they chose. "Frontier LLM agents" in a comparison against 4B to 14B models needs the paper's tables to interpret; we only read the abstract and introduction, not all 45 pages. A mean gain of 10.4 percent hides variation per benchmark. The cost saving follows from fewer tokens and faster decoding, and would depend on your serving setup.

Is the latent thinking readable?

This is the part most relevant to safety. Chain-of-thought monitoring relies on reading a model's written reasoning, and techniques that move thinking into hidden vectors weaken it; see our coverage of Astra's sub-agent text humans can't read and the recurrent looped transformer proposal. The HybridAgent authors say that "surprisingly" the agent can render its latent-mode thoughts as human-readable pseudo-text, making them "transparent and directly inspectable rather than opaque", and the repository includes an interpretable_latent_mode folder. They also report that the agent shifts more internal reasoning into "semantically rich and interpretable latent thoughts", while reserving text for external inspection and tool interaction.

Treat that with care. A readable rendering of a latent vector is a decoding of it, and a decoding may not be a faithful account of what drove the next step. The paper includes an auditability analysis, but independent tests of faithfulness are the right next step before anyone relies on the rendering for oversight.

What people are asking

Is this the same as a "thinking budget" setting? No. A thinking budget limits how many text reasoning tokens a model may write. HybridAgent changes the medium: some of the reasoning happens as vectors that never become tokens, so the saving comes from not decoding those steps at all.

Does it need a new model architecture? The paper's claim is that it does not require training from scratch. A standard transformer is extended with a small adapter and post-trained, so existing backbones can be converted. That is why the experiments start from Qwen3-Base checkpoints.

Why does tool use need text? Tools are called through text: a function name, arguments, a query. The environment returns text observations. Latent vectors cannot be sent to a search API, so the agent has to drop back into tokens at those moments. Learning when to do so is exactly what HybridRL trains.

What does "exact rollout replay" mean for training? Reinforcement learning updates a policy using the trajectories it produced. When some steps are continuous vectors, you must reproduce those vectors exactly when computing the update, or the gradient is wrong. The paper's replay mechanism is how it handles that, according to its introduction.

Will this show up in hosted APIs? Nothing in the paper says a provider is shipping it. It is research code. A hosted model would need the provider to adopt the adapter and the training recipe, so for now the realistic users are teams that run their own open-weight models.

What is in the repository

The GitHub page for togethercomputer/HybridAgent shows four commits at the time we looked and these folders: hybrid_agent, hybrid_inference, hybrid_sft, hybrid_rl, interpretable_latent_mode, examples, scripts and assets, plus a license file and an environment template. That maps cleanly onto the paper: architecture, inference, the two training stages and the interpretable mode. Check the license file and the README for terms and requirements before you build on it.

Why this matters for builders

• Agent cost is dominated by tokens. If your agent spends most of its budget on reasoning text, a method that shrinks that without losing tool access is attractive. Compare it with routing and caching tactics in Microsoft's Token Wars microhack.
• Small models, agent skills. The experiments use 4B to 14B backbones, which is the range people run locally or cheaply. A technique that helps there has a short path to practical use.
• A different route to "reasoning without words". Other labs get there with looped layers; HybridAgent does it with an adapter and mode switching. Related ideas appear in our note on DeepMind's recirculation work on Gemma.
• Overthinking is a cost, too. Our write-up on thinking inertia shows models continuing to reason when asked not to; latent thinking changes what that waste costs.

Caveats

1. We read the abstract, introduction and repository README, not the full results tables.
2. Results are self-reported, with no independent replication yet.
3. Training a HybridAgent needs post-training compute (SFT then RL); the benefit is at inference, not at the cost of getting there.
4. Latent modes complicate debugging and logging. If your tooling depends on reading reasoning text, plan for that.
5. Agents that act with tools still need runtime controls whatever they think in; AgentBeam is the agent security platform from the explainx.ai team that stops AI agents before they take dangerous actions.

How to try it

1. Read the README in the repository and the examples folder.
2. Start with the inference path on a released checkpoint (the README lists HybridAgent-8B-Agentic and HybridAgent-14B-Agentic on Hugging Face) before attempting SFT or RL, which the README says wants 8 to 32 GPUs.
3. Compare token counts and latency on your own tool-use tasks against a text-only baseline with the same backbone.
4. Inspect the interpretable latent mode outputs on a few tasks and judge whether they track what the agent then does.

Related reading

• What is recurrent depth?
• OpenAI's next flagship and looped transformers: claimed vs verified
• Recurrent looped transformer: infinite reasoning depth
• DeepMind recirculation on Gemma 3
• Thinking inertia
• Astra sub-agents and chain-of-thought monitoring
• Official: arXiv 2610.18861 and HybridAgent on GitHub

Details reflect the paper abstract and repository as of October 14, 2026.