DeepSeek DeepInfer: The Open-Source Inference Engine Behind V4.1 Flash, and What Is Still Missing

DeepSeek has open-sourced DeepInfer, the inference framework it describes as the engine behind its large-scale production services, at github.com/deepseek-ai/DeepInfer under the Apache License 2.0. The repository had about 3,300 stars within days of appearing and already has 235 forks. It is real and runnable, but it is a slice of the stack: the README is explicit that the pieces you would need to run a public API, such as the KV cache system and the load balancers, are not in this release.

That gap is the story. The code is genuine production-grade kernel and runtime work, but the headline numbers circulating on social media (155 trillion tokens a day from a single cluster, $38 revenue per card per hour) are not in the repository. This post separates what the repo proves from what DeepSeek and its amplifiers claim, and what it means if you serve models yourself. It follows the October 17 release of DeepSeek-V4.1-Flash-1017, the open-weight model this engine is built to run.

Update (November 1, 2026): Nvidia's "agent swarm" submission and a mixed-up set of numbers

Aggregator posts are circulating a mix of Nvidia and SemiAnalysis numbers (an "agent swarm" submission, figures above 1,000 tokens per second per user, 20.9x and 9.11x cost ratios). We could not find a primary page for most of them, so we do not repeat them here as findings.

The one figure we could check is Nvidia's own: in its technical blog, Nvidia says Vera Rubin NVL72 delivers up to 30x higher throughput per megawatt than GB300 NVL72 at 160 tokens per second per user on the AgentX DeepSeek V4-Pro workload. Nvidia measured this itself, and the post says the results await SemiAnalysis review. See our earlier Rubin NVL72 throughput post.

Practical reading: these are vendor-run numbers on a different DeepSeek model and a different metric (throughput per megawatt) than the V4.1 Flash claims above, so do not compare them to each other or to your own workload.

Update (October 29, 2026): "Nearly 1,000 tokens a second per user" on GB300 is a claim, not a measurement we can check

A new figure is circulating: SemiAnalysis (on X, as relayed by a news summary) says SGLang on an Nvidia GB300 NVL72 system "can get close to 1000 tok/s/user" with DeepSeek-V4.1-Flash, crediting prefill/decode disaggregation plus kernel fusion from SGLang, RadixArk and Nvidia. We could not find a primary SGLang or SemiAnalysis page that states the number with its conditions, so treat it as unverified.

What is documented is more modest and more contested:

• Earlier SGLang claim: a Digg summary of SGLang's posts says it reached 873 tokens per second at batch size 1 on four GB300s, and that its dsv4-flash cookbook advertises 781 tokens per second per user.
• A challenge: per the same summary, a user (@ekzhang1 on X) said the cookbook's reproduction script feeds random token IDs, which degenerate into repeated output, and that real text gives only 200 to 300 tokens per second. We have not reproduced either side.
• Independent data points: one user's patched vLLM on a DGX Station GB300 served 70 tokens per second to a single user and 276 aggregate at 32 users, a different setup (a desk-side box, not a rack).

How to read it: "per user" speed at batch size 1 on a full rack is a best case, and speculative decoding or repetitive output can inflate it. Before you plan around 1,000 tokens per second, ask for the prompt set, batch size, GPU count and whether outputs were checked for quality. For what the engine itself ships, see our DeepInfer analysis above, the SGLang plugin system post and the GB300 workstation guide.

Update (October 31, 2026): SGLang claims 1.09M tokens per second per GPU, with caveats

The SGLang project now says its peak total throughput for DeepSeek-V4.1-Flash on the SemiAnalysis AgentX benchmark rose from 654K to 1.09M tokens per second per GPU over four days, and that it recorded 943 tokens per second per user at concurrency 1 using MegaKernel. Those figures come from a post on the project's official account, @sgl_project, which counts cache, new input and output tokens together. A separate SGLang post from October 29 credits SGLang's kernel fusion on a GB300 NVL72 rack, thanks Nvidia and SemiAnalysis for the benchmarks, and SemiAnalysis also posted about the near-1,000 tokens per second per user result.

How to read the numbers:

We searched for a SemiAnalysis page or SGLang write-up stating these figures with their conditions and did not find one, so treat both numbers as vendor claims we did not reproduce. Two cautions follow from the table. First, the headline mixes cache hits with fresh work: at the 96 percent input cache hit rate DeepSeek reports for its production cluster, most counted tokens are cheap cached input. Second, 943 tokens per second per user at concurrency 1 says little about throughput under load. The useful signal is the direction: two independent open-source engines now report results in the same range on the same benchmark. If you plan capacity, benchmark on your own prompts, batch sizes and GPU count.

Update (October 24, 2026): SGLang and vLLM publish full serving stacks

Both open-source serving projects now say they match or beat DeepInfer on DeepSeek-V4.1-Flash, according to their own X posts as relayed by a secondary summary (we did not independently reproduce any number).

• SGLang reports up to 1.04x DeepInfer's prefill throughput on four GB300 or B300 GPUs and up to 7.4% higher decode throughput across 64 GB300 chips, crediting Mega kernels, SWA bounded replay and Engram optimizations. It says DeepInfer's open release lacks a KV cache system, prefill/decode disaggregation and load balancing, which SGLang supplies through RadixAttention, HiCache and Linker.
• vLLM reports matching DeepInfer's decode curve at batch sizes 8 to 256 with DEP32 rather than EP64DP64 (half the expert parallelism), and 954K tokens per second per GPU on the SemiAnalysis AgentX benchmark on a GB300 NVL72 rack (a combined figure across cached, input and output tokens, not output speed alone).

Caveats: we could not find SGLang's specific 1.04x and 7.4% figures in a primary source, so treat them as unverified. These are vendor-run benchmarks on top-end Blackwell hardware. SGLang's own DeepSeek V4.1 cookbook says V4.1 Flash support has not shipped in a stock release and points to a preview Docker image, so check release notes before relying on either stack in production. This supports the point of this post: the open DeepInfer code is the engine core, and the surrounding serving system is what others are now building.

Update (October 21, 2026): vLLM says it matches DeepInfer within 6%

Hours after the release, the vLLM project posted on X that "vLLM is all you need to reproduce DeepSeek's performance". Its claim: with Wide-EP (wide expert parallelism) and EPLB (expert-parallel load balancing), vLLM matches DeepInfer's decode performance within 6% and generates about $39 per GPU hour in revenue at DeepSeek's off-peak API pricing, at 200 P90 tokens per second on the AgentX workload. vLLM had earlier thanked DeepSeek for releasing an engine "initially built primarily on vLLM".

How to read it:

• Who measured it: vLLM itself, on its own published configurations and recipes. It is a maintainer's benchmark, not an independent test, and we have not reproduced it.
• What it is compared against: DeepSeek's decode setup for V4.1 Flash. The $39 figure is a revenue calculation at DeepSeek's off-peak prices, not a measured profit, and it sits close to the $38 per card per hour that DeepSeek's Zhihu report is said to give for its production cluster.
• Why it matters: if the numbers hold, the missing KV cache and load-balancing pieces matter less than feared, because an open stack already gets within a few percent of DeepSeek's decode throughput.

Treat it as a strong lead to check on your own hardware and prompts, in line with the advice below.

TL;DR: what DeepInfer is and is not

What exactly did DeepSeek release?

The repo has top-level folders for C++ and kernel sources (csrc), a deep_cluster module, the deepinfer Python package, eval, examples, scripts, tests and vendor. It had two commits when we read it, so this is a first public drop, not a long-lived project with history you can audit.

The README lists the supported feature set:

• Optimized kernels, model and sampler runtime, drawing on DeepSeek's own FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and custom TileLang kernels.
• Parallelism: attention data parallelism with balanced context parallelism, and mixture-of-experts expert parallelism with load balancing (EP and EPLB).
• DSpark and confidence-scheduled verification, a speculative-decoding scheme where draft tokens accelerate decoding and confidence estimates schedule when to verify them.
• Sliding-window-attention (SWA) bounded replay, which cuts decoder prefill work.
• Asynchronous scheduling, described as a zero-overhead batch scheduler.
• Batch-invariant inference and deterministic sampling.
• Structured outputs constrained by regex, JSON schema or EBNF grammar.

The only listed model is DeepSeek-V4.1-Flash. Nothing in the README claims support for the older V3 line or for DeepSeek V4 Pro.

What is missing: the part you need for a real service

The README has a section titled "Required for a complete inference service (not included in this release)". It lists four items:

1. A KV cache system.
2. VE/Prefill/Decode disaggregation (the README adds that most of the required engine functionality is already implemented).
3. Load balancing across data-parallel replicas for prefill.
4. Load balancing across data-parallel replicas for decode.

In plain terms, the repo gives you the engine, not the fleet. A production LLM service needs a cache tier that remembers prefixes across requests, routers that spread traffic over many replicas, and the ability to run the compute-heavy prefill stage and the memory-bound decode stage on different machines. The first two are exactly the parts that, in most companies, are tightly coupled to internal infrastructure. Secondary summaries of DeepSeek's X posts say the same thing: those components were held back because they depend on internal systems. That is a reasonable engineering explanation, though it is the summary's wording, not a quote from the README.

If you are weighing this against open serving stacks, our earlier coverage of TokenSpeed and of vLLM's push-based KV transfer for disaggregated inference shows where the open ecosystem already covers those missing pieces.

The technical ideas worth understanding

Asynchronous scheduling

Continuous batching illustrated as a conveyor belt of requests where a new job joins without stopping the line, as in DeepInfer asynchronous scheduling

A naive inference loop has the CPU schedule a batch, hand it to the GPU, wait, and repeat, leaving the accelerator idle during scheduling. DeepInfer's asynchronous scheduling overlaps CPU batch scheduling with device execution, and the README says it is enabled by default. That is the "zero-overhead batch scheduler" the feature list refers to. The gain is largest when batches are small and decode steps are fast, which is the regime a high-throughput Flash-class model lives in.

SWA bounded replay

Prefill is the expensive part of long agent sessions, because every turn resends a long history. With SWA bounded replay, the decoder replays only a bounded suffix instead of the whole prompt. The README says the V4.1 defaults are DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140, and that the retained suffix must cover the SWA window plus the maximum backtracking length. A summary of DeepSeek's posts says this reconstructs attention state from roughly the latest 128 tokens and nearly halves total prefill compute; the 140-token README default is the only number we can confirm directly.

DSpark and confidence-scheduled verification

Speculative decoding lets a cheap draft produce several tokens that the main model verifies in one pass. DeepSeek's variant uses confidence estimates to decide how much to verify. You switch it on with DEEPINFER_ENABLE_MTP_VARLEN=1, and DEEPINFER_USE_NEXTN_DECODE sets the maximum draft length, defaulting to 5.

Batch-invariant inference and deterministic sampling

This one matters more to developers than it sounds. Normally the same prompt can give slightly different logits depending on what else shares the batch, because floating-point reductions change order. With DEEPINFER_BATCH_INVARIANT=1, results stay identical across batch compositions. For fully deterministic sampling you also set DEEPINFER_ENABLE_MTP_VARLEN=0 and pass a fixed seq_seed. Both are NVIDIA-only in this release. If you do reinforcement learning, evals or regression testing against an LLM, reproducible outputs are a real feature.

Structured outputs

You can pass so_args with a schema_type and schema to SamplingParams to constrain generation to a regex, JSON schema or EBNF grammar. The repo ships examples. This is the same capability behind Claude structured outputs on Bedrock and every "strict mode" API, now available for a self-hosted open model.

Verified numbers versus claimed numbers

Open padlock inside a green ring representing the Apache 2.0 open-source release of the DeepInfer inference framework

Here is what we could and could not confirm, by source.

Two details deserve care. First, the README's benchmark scores are described as sanity checks for deployments: "indicative rather than exact", and the eval README notes that on AIME 2026 the rule-based and LLM judges agreed on all 960 rollouts. They show the engine reproduces the model's quality, not that the engine is faster than anyone else's. Second, the production throughput numbers describe DeepSeek's full stack, including the cache tier you cannot download. Even if every figure is accurate, you cannot reproduce it from this repo alone.

For a model that, as covered in our V4.1 Flash cheating and deleted-traces fact check, already has scrutiny on its self-reported results, the same discipline applies here: attribute, then verify.

What developers are saying

The Hacker News thread, item 50163277, had under 50 points and only a handful of comments when we read it, so the discussion is thin. One commenter, schneehertz, called the release a "nuclear bomb" and quoted the 155 trillion tokens a day and $38 per card per hour figures, citing a Zhihu article. Another commenter, ggcr, linked an X post from the LightSeek organization, the group behind the TokenSpeed engine, without further comment. Neither comment is a hands-on test. Treat the figures as the commenter's report of a third-party post, not as verified.

The companion release: deepseek-recipe

Alongside DeepInfer, DeepSeek pushed deepseek-recipe, MIT-licensed, with about 830 stars. It is a set of Rust libraries with Python bindings that does the unglamorous work around a model:

• Converts Messages, Chat Completions and Responses requests into one Conversation format.
• Renders DeepSeek V4 and V4.1 conversations into prompts and encodes them to token IDs with an attached tokenizer.
• Parses model output back into reasoning content, tool calls, JSON output and stop sequences, including streaming.
• Converts structured-output requests (JSON schema or regex) and reports reasoning_tokens usage.
• Preprocesses images for V4.1 with OpenCV (V4 does not accept images, and the library replaces each with the text [Unsupported Image]).

It does not run inference: the caller provides the model, tool execution and HTTP transport. Install is python3 -m pip install deepseek-recipe or cargo add deepseek-recipe@0.2 deepseek-recipe-encoding@0.2 serde_json@1. One security note from its own README: the image fetcher connects to any address by default, so a service that fetches URLs from untrusted requests should install a DestinationFilter, or it can be pointed at internal addresses.

The pairing makes sense. DeepInfer takes token IDs in and gives token IDs out; deepseek-recipe turns an OpenAI-style or Anthropic-style API request into the exact prompt format V4.1 expects. For anyone wrapping the model in their own gateway, the second repo may be the more immediately useful of the two. It also lines up with the DeepSeek-format Anthropic and Responses endpoints described on the V4.1 Flash pricing page.

Why Huawei Ascend 950 support matters

DeepInfer's second hardware target is Huawei's Ascend 950 NPU with CANN 9.2, with a separate requirements-ascend.txt and a DEEPINFER_ACCELERATOR=ascend build switch. That is a public, runnable path for a frontier-class Chinese open model on domestic accelerators. It fits the story in our coverage of DeepSeek and Huawei chip co-design on the Ascend 950DT, where the claims were also stronger than the evidence. Here, at least, the code is in the open: you can read the Ascend path, build it, and check.

It also lands in a politically charged week. The US has ordered DeepSeek removed from federal civilian agencies. Open-sourcing the serving stack changes who controls where the model runs: self-hosted weights plus a self-hosted engine mean no data goes to DeepSeek's API, which is the concern behind many procurement rules. It does not change the model's provenance questions, and it does not make a deployment compliant on its own.

What this means for what you build or pay

The honest benchmark is yours. DeepSeek has not published an apples-to-apples comparison against vLLM, SGLang or TokenSpeed in the README. The Pareto charts show DeepInfer's own prefill and decode curves, and the setup (4 GPUs for prefill, 64 for decode within one scale-up domain) is a rack-scale configuration few teams have.

How to try it

# NVIDIA (use ascend for Huawei NPUs)
export DEEPINFER_ACCELERATOR=nvidia
./build.sh

# runs the default checkpoint, downloaded to the Hugging Face cache on first run
python examples/offline_inference.py

# or point at a local copy
DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017 python examples/offline_inference.py

Create an empty Python 3.12 environment first, as the README instructs. To reproduce the reference evals, the eval folder has driver scripts such as bash eval/aime/run.sh; datasets are not downloaded for you, and HLE and GPQA are gated on Hugging Face. Scoring defaults to an LLM judge service, so set the EVAL_JUDGE_* variables or use the rule-based judge for AIME and GPQA.

Security reminder for a first run: building the repo compiles vendored operators and pulls multi-gigabyte model weights, so use a disposable machine or container, and read build.sh before you run it. If you give coding agents shell access to do this setup, a runtime guard such as AgentBeam, the agent security platform from the explainx.ai team, can block dangerous actions before they run.

Open questions

• Will the KV cache and routing layers ever ship? The README frames them as not included, not as coming. Nothing we found promises a date.
• How does it compare on your workload? No independent benchmark has appeared in the first hours.
• Does it run on non-SM100 NVIDIA parts? The README says SM100 only.
• What about V4 Pro? Only V4.1 Flash is listed.

We will update this post when a hands-on comparison or a DeepSeek technical write-up appears.

Related reading

• DeepSeek V4.1 Flash 1017: open weights, MIT license, benchmarks and pricing
• DeepSeek V4 Flash Vision: the 305B open model
• TokenSpeed: an open-source inference engine for coding agents
• vLLM push-based bidirectional KV transfer for disaggregated inference
• DeepSeek and Huawei Ascend 950DT co-design: claimed vs verified
• DHS orders DeepSeek removed from federal civilian agencies
• DeepSeek V4 Pro: benchmarks, pricing and agent coding

Official sources: DeepInfer repository, DeepInfer eval README, deepseek-recipe repository, DeepSeek-V4.1-Flash-1017 model card.

Repository details, star counts and supported hardware are accurate as of October 20, 2026 and may change as DeepSeek updates the repo.