Together Compiler: AI Agents Now Tune Together AI Inference, With 1.3-3x Speedup Claims

Together AI published two posts on November 4, 2026 that describe one idea: stop tuning an inference engine once, and let AI agents keep re-tuning it as the model, the chip and the traffic change. The product is called Together Compiler, or TKCC, and Together says it already runs on many of its most popular serverless endpoints. The company reports 1.3-3x higher tokens per second than strong baselines. Those figures come from Together's own research team, so this post treats them as claims, shows how they were measured, and lists what you can check yourself.

The companion essay, From Static to Polymorphic Inference, argues why. Open models ship faster, use more varied architectures, and meet more varied traffic than the era of one closed model on one GPU type. Together calls the new regime polymorphic inference. TKCC is its first named product.

Quick answers

Why Together says static inference stopped working

Together's research post opens with a list of things that vary. Architectures differ: the company notes that none of the top ten open-source LLMs on OpenRouter share an architecture. Hardware differs: H100, B200, B300, NVL72, Vera Rubin and Groq parts each need different strategies. Workloads differ too, because Together serves many trillions of tokens per day across thousands of customers, each with its own batch size, sequence length, expert imbalance and session length.

The company gives three traffic observations from its own fleet:

• Regional patterns. Working-hours traffic from one country had 1.4x longer prompts than another country: 97k tokens against 70k.
• Model releases. Users first chat with a new model. About two weeks after a release, prompt length rises by 80 percent as people move to agentic workloads.
• Burstiness. Prompt lengths can double suddenly and cache hit rates can fall.

These are observations by Together, not public data. They still make a clear point. A configuration tuned for chat prompts on launch day is not tuned for agent prompts two weeks later. If you want the vocabulary behind this, our explainer on tokens per second covers why throughput and per-user speed trade off.

The essay adds that 80 percent of OpenRouter traffic goes to models released in the last three months, and that Together served more tokens than closed-source model labs on OpenRouter for the first time earlier in October. Both are Together's statements about a third-party ranking and we could not confirm them independently.

How Together Compiler works

TKCC takes three inputs: a traffic shape, a model and a hardware target. Agents then propose optimizations to an inference engine, test them, and keep winners. Together lists three design ideas.

A loop of checked stages standing in for the Together Compiler agent optimization cycle for inference

1. Persistent engines: 9 seconds instead of 467

Most agent-based optimization systems restart vLLM, SGLang or TRT-LLM for every experiment. Together measured engine startup in its table: 467 seconds in total for the status quo, made of process and worker initialization (93 s), weight loading (50 s), KV-cache sizing (21 s), JIT kernel warm-up (93 s), autotuning (193 s) and API server readiness (10 s).

TKCC starts the engine once and hot-reloads any candidate forward pass. Together's table puts its total time to serving at 9.0 seconds: 1.7 s for an RPC call and setup plus 7.3 s for CUDA-graph capture. The company says this gives 252x higher experiment throughput than baseline agentic performance-optimization systems. That ratio is Together's own and depends on what baseline it chose.

Several agents run in parallel and share ideas through a message board, so the persistent engines always have candidates waiting.

2. Two tiers of engines for fidelity

Optimizing a kernel alone, on random inputs, is fast but misleading for modern models. Mixture-of-experts routing and sparse attention are data dependent, so the best strategy changes with real input distributions. Together describes "fast engines" that load only a few layers for quick, low-noise timing, and "slower engines" that load many layers to check that a win holds on real data. This is the same trade-off our readers see in MoE kernel work: a fast micro-benchmark can lie about production.

3. A cost model for parallelism choices

Choosing tensor, expert or other parallelism splits is a search problem. Together built a tiered cost model that uses hardware specs, then microbenchmarks, then measured forward passes, to prune the search space. Its research post shows measured layouts beating the hardware spec's pick, and the best layout shifting with concurrency.

The results, and who measured them

The models named as test subjects include Kimi K3, GLM 5.3, MiniMax M3.1 Flash and DeepSeek V4.1 Flash. For background on those models, see our coverage of Kimi K3 open weights and the DeepInfer engine post for DeepSeek.

What is verified, and what is not

Together says TKCC "sets the state-of-the-art on SemiAnalysis' InferenceX benchmark for low-latency." We did not find a SemiAnalysis post or leaderboard entry confirming that on November 4. The benchmark is a public methodology, but the entries in Together's charts were produced by Together. For the batch-size-1 numbers, the company states that it used its own cost model to choose the parallelism configuration instead of the InferenceX configuration, so the comparison is not like for like. A footnote adds that the Trainium2 run used a synthetic multi-turn dataset, because the implementation does not yet support higher concurrency on limited HBM capacity.

None of this means the numbers are wrong. It means they are single-source until a third party reruns them. Treat 1,205 TPS as "what Together reports on its stack" and not as a ranking.

What this means if you build on open models

Speed on Together may change under you. If the loop continues, an endpoint's latency and throughput can shift without a model version change. Pin your own latency tests and rerun them on a schedule. Together's earlier launch of post-training and B200 pricing shows the company is stacking services around open models; this is the serving side of the same strategy.

Output quality should not change, but check. Together does not describe correctness tests in its posts. Hot-reloaded forward passes that are numerically different from the reference can change outputs slightly. A real regression check compares logits or evals before and after. Our own Cloudflare GLM 5.3 KV-cache leak write-up shows why serving-layer changes deserve tests, not trust.

Ask the vendor three questions. Which baseline produced the speedup? Is the speedup measured at your batch size and prompt length? How does the system roll back a bad optimization? Together mentions canary rollouts in a separate product post, but the research posts do not say how TKCC candidates are gated.

What the agent loop means for the "RSI" label

Together calls the system "the world's first production-scale recursive self-improvement (RSI) inference system." Read the claim narrowly. Agents write and test code that serves models, some of which are themselves agents. No model weights change. The loop closes over engine code, kernels and configuration. That is real automation of performance engineering. It is not a model improving its own intelligence.

It also fits a wider trend. Labs and cloud providers now use coding agents on their own infrastructure. Our posts on inference engines for agentic workloads and on the AI chip landscape give the hardware context for why per-chip tuning matters. For the size of the models involved, see our guide to LLM parameters and model sizes. Cerebras takes the opposite route: it changes the hardware so that weights never move, as covered in how Cerebras runs a trillion-parameter model.

What Together says comes next

The polymorphic inference essay lists four coming items:

1. Self-serve Continuous Compilation. Customers will optimize dedicated endpoints for their own workloads with one click. Limited beta exists today.
2. Shape-shifting speculative decoding. Draft model and lookahead settings that adapt per domain, request, token and load, instead of fixed launch parameters.
3. ThunderAgent Router. Routing requests across heterogeneous deployments. Together released ThunderAgent as an ICML 2026 Spotlight paper.
4. TogetherLink. Workload-aware model routing that ensembles several good open models.

Together also says it will keep working with model labs including Moonshot and MiniMax on post-training for customers' own harnesses.

How to check the claims yourself

1. Read the paper. The research post links a paper; read the evaluation section for batch sizes, sequence lengths and the exact baselines.
2. Look for third-party runs. Search SemiAnalysis' InferenceX results for Together entries over the next weeks.
3. Benchmark your own traffic. Send a fixed set of real prompts to the endpoint at your usual concurrency. Record time to first token, per-user tokens per second and total throughput. Repeat weekly.
4. Compare outputs. Run an eval set before and after any announced optimization to confirm quality stayed flat.
5. Request beta access only if you run dedicated endpoints; the benefit is largest when your traffic shape differs from the average.

Bottom line

Together Compiler is a credible engineering idea with a clear mechanism: skip the 467-second engine restart, test candidates on persistent engines, and keep recompiling as traffic drifts. The performance numbers are large and attractive, but they come from Together alone, with some comparisons that are not like for like. The useful action for builders is to measure your own workloads rather than adopt the headline.

Related reading

• Together Post-Training: B200 pricing and first eight weeks
• TokenSpeed: an inference engine for agentic workloads
• DeepInfer and DeepSeek V4.1 Flash: claimed vs verified
• Kimi K3: 842 tokens per second claim, checked
• What is TPS? Tokens per second explained
• AI chip architectures: GPU, TPU, Trainium, Cerebras, Groq
• Official sources: Together Compiler research post, Polymorphic Inference essay, Moonshot's Kimi K3 post

Figures are accurate as of November 4, 2026, and come from Together AI's own posts. Check the sources for later changes.