Together AI published its agent evaluation playbook on November 16, 2026, reporting that coding agents completing realistic platform tasks rose from 75% to 97% after the team fixed documentation traps, skills, and API error clarity. The official post frames agent experience (AX) as the new face of developer experience: if Claude Code or Codex cannot reserve a GPU cluster without hallucinating inventory, your API is not production-ready for 2026 workflows.
This is not a model launch. It is a harness story for anyone selling inference, fine-tuning, or cluster APIs to agents instead of only to humans.
TL;DR
| Question | Answer |
|---|---|
| What problem? | Agents using Together APIs hit wrong docs, split data sources, and opaque errors. |
| How measured? | Task prompts run baseline vs MCP vs skills, scored by an LLM judge in sandboxes. |
| What changed? | Targeted doc fixes, skills package, and API errors moved pass rate 75% to 97%. |
| Example failure? | Agents treated CLI examples as live GPU inventory and skipped pricing APIs. |
Why Together built agent evals
Together watched coding agents become the default interface for its developer funnel. Humans still read docs, but agents issue hundreds of CLI calls through tools you never tested in a tutorial.
The post cites GitHub as the gold standard: rich API, CLI, and docs that agents adopted without the author noticing GitHub had a CLI until agents used it constantly. Together wanted the same agent-ready surface for batch jobs, voice agents, autoscaling endpoints, and GPU clusters.
That motivation aligns with HybridAgent latent reasoning research and Together Compiler continuous optimization—including the November 16, 2026 essay on why Together retired megakernels for agent-driven kernel fusion—but agent evals address control-plane correctness, not tokens per second.
The task matrix and configurations
Together built prompts mirroring real tickets, such as listing GPU regions, combining pricing tables, and emitting a valid reserve command without actually reserving.
Each task runs three ways:
- Baseline — prompt only, default web tools, no Together MCP or skills.
- MCP — Together documentation exposed through an MCP server for grounded search.
- Skills — the Together skill package with agent-oriented instructions and examples.
A September 30 matrix in the post scored 192 of 215 runs passing. The GPU cluster pricing task failed every baseline attempt for Sonnet 4.6 early on, while skills runs passed consistently with fewer tool calls.
Automation: sandboxes, proxy, and judge
Referee figure holding a green flag between two evaluation boxes
Manual evals did not scale. Together moved to GitHub Actions spinning agents in Together sandboxes, with a proxy logging API traffic and a results viewer comparing models and configs side by side.
The LLM judge labels runs pass, incomplete, or fail by reading prompts plus session traces. Incomplete is intentional: it flags where humans must investigate whether the bug is agent, API, or missing telemetry.
This pattern mirrors Amazon Bedrock AgentCore evaluation playbook guidance for enterprise agents, but applied to a model host own control plane.
Case study: GPU cluster pricing trap
GPU cluster grid suggesting cloud instance pricing choices
The memorable failure mode: agents read static CLI examples listing sample regions and SKUs and assumed that table was live inventory. They skipped the regions API and pricing page, then printed a plausible but wrong cheapest option.
Together fixed doc structure, strengthened skills to mandate API calls, and clarified error strings so agents could recover without burning tokens. Pass rate for that task went from near zero baseline to reliable skills success.
If you operate any cloud API, assume agents will over-trust fenced code blocks. Our Railway agent incident reporting shows the same class of mistake when agents scan production.
What 75% to 97% does and does not mean
The headline numbers are Together-internal over its task library, not a public leaderboard. Treat them as vendor-reported operational metrics, useful for methodology, not for claiming Together beats other clouds outright.
Still, the delta is the lesson: AX improvements are often documentation and schema work, not a new foundation model. Teams shipping MCP servers and skills without eval loops risk publishing instructions agents misread at scale.
Pair this post with ffmpeg skill for local video agents and Google Stitch skills: skills only help when you measure them against real agent trajectories.
Builder checklist: copy what Together did
- List top 20 agent prompts your users paste, not your marketing feature list.
- Run baseline, MCP, and skills variants in isolated sandboxes with API capture.
- Add an LLM judge with explicit pass criteria per task, including negative constraints like do not charge a card.
- Fix docs that look like live data; separate examples from API responses visually.
- Ship a results viewer so regressions show up when you change error JSON.
- Re-run after every breaking API change; agents do not read changelog footers.
Scoring definitions: pass, incomplete, and fail
Together's judge rubric is simple on paper and hard in production:
- Pass means every requirement in the prompt was met, including negative constraints.
- Incomplete means the agent tried but left steps unverified, such as printing a reserve command without confirming pricing.
- Fail means early abandonment, constraint violation, or objectively wrong output.
Incomplete runs are the valuable signal. They often show recoverable API friction: a 400 error string that sends agents down a web search rabbit hole, or a pagination field renamed without a changelog note. Together says it mines passing runs too, because agents sometimes work around bad errors with extra tool calls, hiding inefficiency until bill shock arrives.
If you operate a model API, copy that discipline: never ship a doc change without rerunning the top ten agent prompts your users paste into Claude Code.
September 30 matrix in more detail
The post embeds a September 30, 2026 evaluation wave: 215 scored runs, 192 passes, with the GPU cluster task failing all baseline attempts while skills passed consistently. Together ran two attempts per task per configuration across three models, producing six trajectories per task for each of baseline, MCP, and skills.
That design isolates harness value. When MCP alone does not fix a task but skills do, the bug is probably procedural (which APIs to call in what order). When both MCP and skills fail, the bug is likely API truth (wrong error codes, stale examples, or broken endpoints).
Developers maintaining MCP servers should note the lesson from our connect fitness MCP guide: grounding alone does not replace skills that encode happy-path workflows.
Agent experience and revenue
Together explicitly ties AX to revenue: if agents cannot provision endpoints, humans bounce to competitors whose docs agents already know. GitHub's 40% agent-opened PRs statistic in the post is illustrative, not a Together metric, but it motivates why API vendors must treat Claude Code as a first-class client.
This is why November also saw OpenAI embedded safety evaluators and California suing over agent escapes: agent reliability is now legal and commercial risk, not only UX polish.
Implementing a minimal eval loop without Together scale
You do not need Together sandbox farm to start:
- Store twenty canonical prompts in a git repo with expected JSON outputs.
- Run Claude Code headless or your CI agent against staging API keys nightly.
- Capture HTTP logs at a proxy; redact secrets before saving artifacts.
- Use a cheap judge model with a rigid rubric before upgrading to human review.
- Block releases when baseline pass rate drops more than five points week over week.
Pair with Pr-Agent and MIT reviewer bots for code paths, but keep control-plane evals separate: merging a PR does not prove agents can create billing resources.
Fairness caveats on the 97% figure
The 97% figure is not a promise about your agent next Tuesday. It reflects Together's task mix after targeted fixes. Your agents may still fail on undocumented edge cases or newer models with different tool biases.
Nor does it benchmark security. Passing a pricing task says nothing about prompt injection against Together APIs when agents can mutate production infrastructure.
How this compares to other eval culture
- OpenAI embedded evaluators (November pilot) focus on safety and scheming inside the lab.
- Together agent evals focus on whether paying customers agents can operate billing APIs.
- SWE-bench style tests measure repo patching, not cloud control planes.
All three stacks belong in a mature agent platform program.
Related reading
- Together AI and Ollama prefix cache guide for long agent sessions
- Nasiko Open Runtime agent token budgets
- Together Compiler inference optimization
- Together post-training B200 pricing
- Longtail-SWE multilingual agent benchmark
- Replica skill and agent skills ecosystem
- Baseten frontier inference guide
- How the Claude Code team does code review
Voice agent and batch tasks in the eval library
Together's published task list also includes voice agent setup and batch inference flows, not only GPU clusters. Those tasks stress different failure modes: audio pipeline env vars, file upload URLs, and asynchronous job polling. When you build your own eval corpus, include at least one async job and one multi-step billing action so you catch agents that succeed on read-only calls but fail when money moves.
Methodology details are accurate as of Together AI post dated November 16, 2026.
