Google Research ToolGrad: Answer-First Tool-Use Dataset Generation

Training an LLM to call tools reliably needs a dataset of real tool-use chains — but the standard way to build one, generate a plausible user request and then search for an API path that satisfies it, fails more often than it succeeds. Google Research's answer, published September 10, 2026 and presented at ACL 2026, is to flip the order entirely: verify a working tool chain first, write the prompt to match it second. The result is a 99.8% generation pass rate and a fine-tuned open model that beats gpt-5 on a function-calling benchmark it never trained on.

This matters for anyone building or fine-tuning tool-calling agents: the bottleneck for teaching a model to use tools well has never been model architecture — it's getting enough correct, verified training examples without paying a fortune in wasted search.

TL;DR

Why query-first tool-use data generation fails

Prior frameworks like ToolBench and ToolACE build tool-use datasets in two steps: sample a pool of APIs, ask an LLM to invent a plausible user instruction for them, then run a depth-first search (DFS) agent to hunt for a sequence of API calls that satisfies that instruction. The problem is structural — a hypothetical instruction invented without knowing whether a real solution exists has no guarantee one does. The DFS agent burns a search budget on instructions that were never solvable in the first place, and Google Research's own comparison shows this approach failing to reach even a moderate pass rate on the ToolBench API database.

Google Research's framing is direct: "an expli[cit tool]-use solution provides more unambiguous information than a prompt, making the annotation, from tool usage to the use query, much easier and requiring only one LLM step."

The answer-first fix: build the chain, then write the prompt

ToolGrad inverts the pipeline. It constructs a verified, working API chain first — with actual execution feedback confirming each step succeeds — and only then asks an LLM to write a user query and response that match the finished chain. Writing a prompt to fit a known-good answer is a single, low-ambiguity LLM call; searching blind for an answer to an invented prompt is the expensive, failure-prone part, and ToolGrad removes it entirely.

Textual gradients, borrowed from prompt optimization

The mechanism ToolGrad uses to build that chain iteratively is adapted from TextGrad, a technique that uses an LLM critic to generate plain-language feedback — "textual gradients" — for refining a prompt, the same way numerical gradients refine model weights during standard training. ToolGrad repurposes that idea for building an API workflow instead of a prompt:

1. API Proposer — narrows a sampled pool of APIs down to a few promising candidates that could extend the current workflow
2. API Executors — run the candidate APIs in parallel and produce detailed execution reports
3. API Selector — reviews those reports and picks the single best-performing API call, which functions as the "textual gradient" — directional feedback appended to the workflow
4. LLM Updater — rewrites the synthetic user query and AI response to stay consistent with the newly extended API set

Looping this cycle produces one training sample: a user query, a verified multi-step API workflow, and the AI's final response — all internally consistent because the chain was proven to work before the prompt was ever written.

The numbers: 99.8% pass rate, and beating the teacher model

Tested against ToolBench's database of 16,000+ real-world APIs, ToolGrad reached a 99.8% pass rate generating tool-use data, at a lower total generation cost and fewer optimization steps than the query-first DFS baseline.

To test whether that data actually teaches a model something useful, Google Research built a compact 500-example dataset called ToolGrad-500 and fine-tuned three sizes of Gemma-3 (1B, 4B, and 12B) on it, producing ToolGrad-1B/4B/12B. They evaluated the results on the Berkeley Function Calling Leaderboard (BFCL) — deliberately a different, out-of-distribution tool set from ToolBench, testing whether the models generalized rather than memorized.

A 12B open-weight model fine-tuned on 500 examples landed within a point of Gemini 2.5 Pro and Claude 4.5 Opus, and clearly ahead of gpt-5, on tools it had never seen during training.

The self-evolving result

The most striking detail: ToolGrad-500's data was itself generated by gemini-2.5-flash-lite — and the Gemma-3-12B model fine-tuned on that data ended up outperforming its own teacher model on BFCL. Google Research frames this as a self-evolving property of the pipeline — a smaller student model, trained on a small but verified dataset, surpassing the larger model that produced the data in the first place. ToolGrad-12B also beat other open-source, tool-use-specialized models including ToolACE and Hammer-2.1-7B.

What people are asking

Does this replace fine-tuning frameworks or tool-definition design? No — ToolGrad solves the data problem, not the tool-schema design problem or the fine-tuning mechanics themselves. You still need well-designed tool definitions and a fine-tuning pipeline; ToolGrad just makes the training examples cheaper and more reliable to produce.

Is the code open source? Google Research's announcement links the ACL 2026 paper and a ToolGrad project page but doesn't mention a public code release as of publication — check the official research.google blog post for updates.

Does this only work with Gemma? No — Gemma-3 was the model family Google Research chose to demonstrate the fine-tuning results, but ToolGrad is a dataset-generation method, not a Gemma-specific technique. Any base model that can be fine-tuned on instruction-response pairs with tool calls could in principle use ToolGrad-generated data.

What's the catch? The published results use a small 500-example dataset and a narrow benchmark comparison; Google Research itself flags scaling to "increasingly dynamic and vast API ecosystems" and continuous on-the-fly personalization as future work, not something ToolGrad-500 already demonstrates at scale.

The bottom line

ToolGrad's real contribution isn't a new model — it's a cheaper, higher-yield way to produce the structured, verified data that tool-calling fine-tunes actually need. Flipping "guess a query, search for an answer" into "verify an answer, write the query" turns a search problem into an annotation problem, and Google Research's numbers — a 99.8% pass rate and a 12B open model matching frontier proprietary models on an unseen benchmark — make a strong case that the ordering, not just the model size, was the bottleneck all along.

Benchmark figures, model names, and technical details above reflect Google Research's official announcement as of September 10, 2026 — check the source for any later updates or code release.

Related reading

• Structured output with tool_use and JSON schemas: the definitive guide
• Tool definition and schema design: the context engineering layer most teams get wrong
• What is fine-tuning an LLM? A complete guide
• How to read an AI benchmark and not get fooled
• ReAct prompting: reasoning and acting agents guide
• Gemma 4 updates: flash attention and tool calling
• What is MCP? Model Context Protocol complete guide
• Official: Google Research — ToolGrad announcement