DeepSeek DeepInfer: The Open-Source Inference Engine Behind V4.1 Flash, and What Is Still Missing

DeepSeek has open-sourced DeepInfer, the inference framework it describes as the engine behind its large-scale production services, at github.com/deepseek-ai/DeepInfer under the Apache License 2.0. The repository had about 3,300 stars within days of appearing and already has 235 forks. It is real and runnable, but it is a slice of the stack: the README is explicit that the pieces you would need to run a public API, such as the KV cache system and the load balancers, are not in this release.

That gap is the story. The code is genuine production-grade kernel and runtime work, but the headline numbers circulating on social media (155 trillion tokens a day from a single cluster, $38 revenue per card per hour) are not in the repository. This post separates what the repo proves from what DeepSeek and its amplifiers claim, and what it means if you serve models yourself. It follows the October 17 release of DeepSeek-V4.1-Flash-1017, the open-weight model this engine is built to run.

TL;DR: what DeepInfer is and is not

What exactly did DeepSeek release?

The repo has top-level folders for C++ and kernel sources (csrc), a deep_cluster module, the deepinfer Python package, eval, examples, scripts, tests and vendor. It had two commits when we read it, so this is a first public drop, not a long-lived project with history you can audit.

The README lists the supported feature set:

• Optimized kernels, model and sampler runtime, drawing on DeepSeek's own FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and custom TileLang kernels.
• Parallelism: attention data parallelism with balanced context parallelism, and mixture-of-experts expert parallelism with load balancing (EP and EPLB).
• DSpark and confidence-scheduled verification, a speculative-decoding scheme where draft tokens accelerate decoding and confidence estimates schedule when to verify them.
• Sliding-window-attention (SWA) bounded replay, which cuts decoder prefill work.
• Asynchronous scheduling, described as a zero-overhead batch scheduler.
• Batch-invariant inference and deterministic sampling.
• Structured outputs constrained by regex, JSON schema or EBNF grammar.

The only listed model is DeepSeek-V4.1-Flash. Nothing in the README claims support for the older V3 line or for DeepSeek V4 Pro.

What is missing: the part you need for a real service

The README has a section titled "Required for a complete inference service (not included in this release)". It lists four items:

1. A KV cache system.
2. VE/Prefill/Decode disaggregation (the README adds that most of the required engine functionality is already implemented).
3. Load balancing across data-parallel replicas for prefill.
4. Load balancing across data-parallel replicas for decode.

In plain terms, the repo gives you the engine, not the fleet. A production LLM service needs a cache tier that remembers prefixes across requests, routers that spread traffic over many replicas, and the ability to run the compute-heavy prefill stage and the memory-bound decode stage on different machines. The first two are exactly the parts that, in most companies, are tightly coupled to internal infrastructure. Secondary summaries of DeepSeek's X posts say the same thing: those components were held back because they depend on internal systems. That is a reasonable engineering explanation, though it is the summary's wording, not a quote from the README.

If you are weighing this against open serving stacks, our earlier coverage of TokenSpeed and of vLLM's push-based KV transfer for disaggregated inference shows where the open ecosystem already covers those missing pieces.

The technical ideas worth understanding

Asynchronous scheduling

Continuous batching illustrated as a conveyor belt of requests where a new job joins without stopping the line, as in DeepInfer asynchronous scheduling

A naive inference loop has the CPU schedule a batch, hand it to the GPU, wait, and repeat, leaving the accelerator idle during scheduling. DeepInfer's asynchronous scheduling overlaps CPU batch scheduling with device execution, and the README says it is enabled by default. That is the "zero-overhead batch scheduler" the feature list refers to. The gain is largest when batches are small and decode steps are fast, which is the regime a high-throughput Flash-class model lives in.

SWA bounded replay

Prefill is the expensive part of long agent sessions, because every turn resends a long history. With SWA bounded replay, the decoder replays only a bounded suffix instead of the whole prompt. The README says the V4.1 defaults are DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140, and that the retained suffix must cover the SWA window plus the maximum backtracking length. A summary of DeepSeek's posts says this reconstructs attention state from roughly the latest 128 tokens and nearly halves total prefill compute; the 140-token README default is the only number we can confirm directly.

DSpark and confidence-scheduled verification

Speculative decoding lets a cheap draft produce several tokens that the main model verifies in one pass. DeepSeek's variant uses confidence estimates to decide how much to verify. You switch it on with DEEPINFER_ENABLE_MTP_VARLEN=1, and DEEPINFER_USE_NEXTN_DECODE sets the maximum draft length, defaulting to 5.

Batch-invariant inference and deterministic sampling

This one matters more to developers than it sounds. Normally the same prompt can give slightly different logits depending on what else shares the batch, because floating-point reductions change order. With DEEPINFER_BATCH_INVARIANT=1, results stay identical across batch compositions. For fully deterministic sampling you also set DEEPINFER_ENABLE_MTP_VARLEN=0 and pass a fixed seq_seed. Both are NVIDIA-only in this release. If you do reinforcement learning, evals or regression testing against an LLM, reproducible outputs are a real feature.

Structured outputs

You can pass so_args with a schema_type and schema to SamplingParams to constrain generation to a regex, JSON schema or EBNF grammar. The repo ships examples. This is the same capability behind Claude structured outputs on Bedrock and every "strict mode" API, now available for a self-hosted open model.

Verified numbers versus claimed numbers

Open padlock inside a green ring representing the Apache 2.0 open-source release of the DeepInfer inference framework

Here is what we could and could not confirm, by source.

Two details deserve care. First, the README's benchmark scores are described as sanity checks for deployments: "indicative rather than exact", and the eval README notes that on AIME 2026 the rule-based and LLM judges agreed on all 960 rollouts. They show the engine reproduces the model's quality, not that the engine is faster than anyone else's. Second, the production throughput numbers describe DeepSeek's full stack, including the cache tier you cannot download. Even if every figure is accurate, you cannot reproduce it from this repo alone.

For a model that, as covered in our V4.1 Flash cheating and deleted-traces fact check, already has scrutiny on its self-reported results, the same discipline applies here: attribute, then verify.

What developers are saying

The Hacker News thread, item 50163277, had under 50 points and only a handful of comments when we read it, so the discussion is thin. One commenter, schneehertz, called the release a "nuclear bomb" and quoted the 155 trillion tokens a day and $38 per card per hour figures, citing a Zhihu article. Another commenter, ggcr, linked an X post from the LightSeek organization, the group behind the TokenSpeed engine, without further comment. Neither comment is a hands-on test. Treat the figures as the commenter's report of a third-party post, not as verified.

The companion release: deepseek-recipe

Alongside DeepInfer, DeepSeek pushed deepseek-recipe, MIT-licensed, with about 830 stars. It is a set of Rust libraries with Python bindings that does the unglamorous work around a model:

• Converts Messages, Chat Completions and Responses requests into one Conversation format.
• Renders DeepSeek V4 and V4.1 conversations into prompts and encodes them to token IDs with an attached tokenizer.
• Parses model output back into reasoning content, tool calls, JSON output and stop sequences, including streaming.
• Converts structured-output requests (JSON schema or regex) and reports reasoning_tokens usage.
• Preprocesses images for V4.1 with OpenCV (V4 does not accept images, and the library replaces each with the text [Unsupported Image]).

It does not run inference: the caller provides the model, tool execution and HTTP transport. Install is python3 -m pip install deepseek-recipe or cargo add deepseek-recipe@0.2 deepseek-recipe-encoding@0.2 serde_json@1. One security note from its own README: the image fetcher connects to any address by default, so a service that fetches URLs from untrusted requests should install a DestinationFilter, or it can be pointed at internal addresses.

The pairing makes sense. DeepInfer takes token IDs in and gives token IDs out; deepseek-recipe turns an OpenAI-style or Anthropic-style API request into the exact prompt format V4.1 expects. For anyone wrapping the model in their own gateway, the second repo may be the more immediately useful of the two. It also lines up with the DeepSeek-format Anthropic and Responses endpoints described on the V4.1 Flash pricing page.

Why Huawei Ascend 950 support matters

DeepInfer's second hardware target is Huawei's Ascend 950 NPU with CANN 9.2, with a separate requirements-ascend.txt and a DEEPINFER_ACCELERATOR=ascend build switch. That is a public, runnable path for a frontier-class Chinese open model on domestic accelerators. It fits the story in our coverage of DeepSeek and Huawei chip co-design on the Ascend 950DT, where the claims were also stronger than the evidence. Here, at least, the code is in the open: you can read the Ascend path, build it, and check.

It also lands in a politically charged week. The US has ordered DeepSeek removed from federal civilian agencies. Open-sourcing the serving stack changes who controls where the model runs: self-hosted weights plus a self-hosted engine mean no data goes to DeepSeek's API, which is the concern behind many procurement rules. It does not change the model's provenance questions, and it does not make a deployment compliant on its own.

What this means for what you build or pay

The honest benchmark is yours. DeepSeek has not published an apples-to-apples comparison against vLLM, SGLang or TokenSpeed in the README. The Pareto charts show DeepInfer's own prefill and decode curves, and the setup (4 GPUs for prefill, 64 for decode within one scale-up domain) is a rack-scale configuration few teams have.

How to try it

# NVIDIA (use ascend for Huawei NPUs)
export DEEPINFER_ACCELERATOR=nvidia
./build.sh

# runs the default checkpoint, downloaded to the Hugging Face cache on first run
python examples/offline_inference.py

# or point at a local copy
DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017 python examples/offline_inference.py

Create an empty Python 3.12 environment first, as the README instructs. To reproduce the reference evals, the eval folder has driver scripts such as bash eval/aime/run.sh; datasets are not downloaded for you, and HLE and GPQA are gated on Hugging Face. Scoring defaults to an LLM judge service, so set the EVAL_JUDGE_* variables or use the rule-based judge for AIME and GPQA.

Security reminder for a first run: building the repo compiles vendored operators and pulls multi-gigabyte model weights, so use a disposable machine or container, and read build.sh before you run it. If you give coding agents shell access to do this setup, a runtime guard such as AgentBeam, the agent security platform from the explainx.ai team, can block dangerous actions before they run.

Open questions

• Will the KV cache and routing layers ever ship? The README frames them as not included, not as coming. Nothing we found promises a date.
• How does it compare on your workload? No independent benchmark has appeared in the first hours.
• Does it run on non-SM100 NVIDIA parts? The README says SM100 only.
• What about V4 Pro? Only V4.1 Flash is listed.

We will update this post when a hands-on comparison or a DeepSeek technical write-up appears.

Related reading

• DeepSeek V4.1 Flash 1017: open weights, MIT license, benchmarks and pricing
• DeepSeek V4 Flash Vision: the 305B open model
• TokenSpeed: an open-source inference engine for coding agents
• vLLM push-based bidirectional KV transfer for disaggregated inference
• DeepSeek and Huawei Ascend 950DT co-design: claimed vs verified
• DHS orders DeepSeek removed from federal civilian agencies
• DeepSeek V4 Pro: benchmarks, pricing and agent coding

Official sources: DeepInfer repository, DeepInfer eval README, deepseek-recipe repository, DeepSeek-V4.1-Flash-1017 model card.

Repository details, star counts and supported hardware are accurate as of October 20, 2026 and may change as DeepSeek updates the repo.