TokenSpeed: The Open-Source LLM Inference Engine Built for Coding Agents

TokenSpeed is an MIT-licensed LLM inference engine, built by the non-profit LightSeek Foundation, that is designed around one workload: coding agents with long contexts and dozens of turns. Its own benchmark says it beats NVIDIA's TensorRT-LLM by roughly 9 to 11 percent on Kimi K2.5 with B200 GPUs. That result is the team's own measurement, so treat it as a promising claim, not a settled ranking.

The repo is showing up on GitHub's trending lists this week. At the time of writing, the repository shows about 2.3k stars, 331 forks and more than 1,800 commits. This post explains what TokenSpeed is, how the design differs from other engines, what the published numbers show, and how to decide whether to try it.

TL;DR: questions people ask first

Why build an engine for agents?

The founding post, published May 6, 2026 on the LightSeek blog, makes a straightforward argument. Coding tools such as Claude Code, Codex and Cursor now generate huge volumes of tokens, and at that scale "even small improvements in throughput per GPU" translate into large capacity savings.

The team's point is that agent traffic is not like chat. Contexts routinely exceed 50K tokens and conversations span dozens of turns, so most public benchmarks do not capture it. The team measures against SWE-smith traces, built with the EvalScope team, which they say mirror production coding-agent traffic. The goal they state is to maximize tokens per minute per GPU while keeping a per-user speed floor, typically 70 tokens per second and sometimes 200 or more.

That framing matters for readers choosing a stack. A fast engine for one short prompt is not necessarily the best engine when a single user's session reuses a long prefix over and over. For more on why agent loops stress serving systems, see our coverage of Fireworks Ember and reasoning tokens on Kimi K3 and of Cerebras serving Qwen at 1,500 tokens per second.

How the design differs

The README describes three core components, and the PyTorch Foundation's note, quoted there, calls TokenSpeed "the first to separate the control plane from the execution plane."

• Scheduler: a C++ control plane with a Python execution plane. Request lifecycle, KV cache ownership and overlap timing are encoded as a finite-state machine, and the type system enforces safe KV resource reuse at compile time rather than at runtime. In plain terms: bugs where two requests wrongly share or free the same cache memory are meant to be caught when the code is compiled, not in production.
• Kernels: a pluggable layered system with a portable public API and a central registry. This is how the team supports different accelerators without forking the engine.
• Entrypoint: an AsyncLLM front end integrated with SMG for low-overhead CPU-side request handling.

The modeling layer uses a local SPMD design: developers annotate input and output placement at module boundaries, and a lightweight static compiler generates the collective operations, so model authors do not hand-write communication code.

Why this is interesting: most engines grew from Python research code and added speed afterward. TokenSpeed splits the safety-critical bookkeeping into compiled code while keeping model code in Python for fast iteration. Whether that trade pays off long term depends on how easily outside contributors can add models, which is the thing to watch.

What do the benchmarks show?

The blog's headline chart compares TokenSpeed and TensorRT-LLM on Kimi K2.5 across deployment configurations on NVIDIA B200, without prefill/decode disaggregation. The x-axis is tokens per second per user (latency), and the y-axis is tokens per minute per GPU (throughput).

TokenSpeed inference engine Pareto curve against TensorRT-LLM on Kimi K2.5 with NVIDIA B200 Image: LightSeek Foundation

For coding agents above 70 tokens per second per user, the team says the best configuration is attention TP4 plus MoE TP4, where TokenSpeed "dominates" TensorRT-LLM across the whole Pareto frontier: roughly 9 percent faster at batch size 1 and roughly 11 percent higher throughput around 100 tokens per second per user. Reading the chart, the purple TokenSpeed curve sits above the green TensorRT-LLM curve for that configuration.

Two honest caveats. First, this is a single model and a single GPU type, measured by the engine's own authors. Second, the post states it covers non-disaggregated deployment only; disaggregation was still being cleaned up. A reader running a different model, a different GPU count or a heavily disaggregated cluster should not assume the same gap.

The MLA kernel claim

The second claim is about kernels. Multi-head Latent Attention (MLA) is the attention variant used by DeepSeek-style models, and the team says it built one of the fastest implementations on Blackwell for agentic workloads.

TokenSpeed MLA kernel prefill and decode latency versus TensorRT-LLM MLA Image: LightSeek Foundation

According to the post, the prefill kernel uses NVIDIA-internal knobs to tune the softmax and outperforms TensorRT-LLM's MLA across all five typical prefill workloads with long prefix KV cache. The decode kernel folds the query-sequence axis into the head axis to fill Tensor Core tiles better, and combined with other optimizations the team says it "nearly halves" latency on typical decode workloads with speculative decoding at batch sizes 4, 8 and 16. The post also says TokenSpeed MLA "has been adopted by vLLM," which, if accurate, means the kernel work benefits users who never run TokenSpeed itself.

Where it has been used since May

The README news list is the best evidence of momentum, with the caveat that it is the project's own log:

Day-0 support matters because inference engines live or die on how quickly they run new open-weight models after release. Our coverage of Qwen3.8 Flash and the Qwen4 architecture preview and of a Qwen3.8 Flash Next run on a consumer GPU show how fast that cadence is. The 685 tokens per second per user figure is the project's own and we have not reproduced it.

Why "Switzerland"?

The README argues TokenSpeed is "the Switzerland of open-source LLM inference engines": neutral because it is backed by the non-profit LightSeek Foundation rather than a commercial company, with hardware and engineering support from vendors including NVIDIA and AMD. The founding blog lists compute support from OpenAI, NVIDIA, AMD, Verda and Nebius, and acknowledges TensorRT-LLM, Triton, FluentLLM, vLLM, FlashInfer, SGLang and others for raising the bar. It also says parts of the runtime reflect upstream lineage from FluentLLM.

This is a real differentiator for teams wary of tying their serving layer to a single GPU vendor. It is also worth noting that the contributor list is heavy with vendor employees, so "neutral" describes governance and licensing more than the origin of the engineering effort. The repo includes a GOVERNANCE file; read it if you plan to depend on the project.

How to try it

We did not run TokenSpeed ourselves for this post, so the steps below follow the repository layout, not a tested walkthrough.

1. Open the repo's docs index and read Getting Started, then Launching a Server.
2. Check Model Recipes for your model. Support is recipe-based, so a model not listed should be assumed unsupported until you test it.
3. Use the Docker setups in the docker folder if you want a reproducible environment, and pick the kernel package that matches your hardware (tokenspeed-kernel, -amd or -npu).
4. Benchmark on your own traffic. Replay real agent traces rather than short prompts, and compare against whatever you run today with the same parallelism settings.

A sanity checklist before you move anything to production: pin a version, test long-prefix workloads with your own prompts, confirm tool-calling and structured output behave as in your current engine, and keep the old engine available for rollback. For a lighter-weight comparison on laptops and single-machine setups, see how llama.cpp handles new model support and our post on DwarfStar DS4 local inference.

What is still unproven

• Independent benchmarks. We found none. Until someone outside the project reproduces the Pareto curves, the 9 to 11 percent figures are the authors' claim.
• Breadth. The strongest results are on Blackwell and a few model families. Older GPUs, small models and consumer hardware are not the target.
• Disaggregated serving. The founding post deferred this to a follow-up.
• Operational maturity. The May post said production hardening was still planned; the later README log shows steady day-0 support but does not publish stability data.
• Contribution model. A C++ control plane plus Python plane is powerful but raises the bar for contributors who only know Python.

What this means if you run models

If you serve open-weight models for agents at scale, TokenSpeed is worth a benchmark on your own traffic, especially on Blackwell. If you are a solo developer on a single consumer GPU, it is probably not for you yet; a lightweight runtime fits better. If you simply use hosted APIs, the practical effect shows up indirectly: kernel improvements adopted by vLLM and competition between engines push serving costs down. Hardware makers are moving in the same direction, as in NVIDIA's open Vera Rubin rack designs, where the platform and serving stack are designed together.

Related reading

• Fireworks Ember 1 and Kimi K3 reasoning tokens
• Cerebras serving Qwen at 1,500 tokens per second
• Qwen3.8 Flash Next on a consumer GPU
• Perplexity Lily: an Apple silicon inference engine
• DwarfStar DS4 local inference
• llama.cpp and new model support
• NVIDIA open Vera Rubin designs at OCP

Star counts, benchmark figures and supported models reflect the repository and LightSeek blog as of October 15, 2026, and may change.