SGLang Adds a Plugin System So NVIDIA and AMD Can Ship Their Own Decode Kernels

SGLang, the open-source LLM serving framework, has added a plugin system that lets GPU vendors ship their own low-latency decode kernels as separate libraries, and two plugins have appeared: sgl-amd-mono for AMD Instinct GPUs and a sibling called sgl-nvidia-mega for NVIDIA. The idea is simple: instead of merging every vendor-specific optimization into SGLang's core, vendors maintain their own kernels, and SGLang routes to them when the model and hardware match. This post explains what is documented in the AMD README and SGLang's own blog, and flags the performance numbers circulating that we could not trace to a primary source.

TL;DR: SGLang plugins

What is a monokernel, and why plugins?

At batch size one, decoding a token in a large mixture-of-experts model is dominated by overhead: launching many small GPU kernels, waiting for collective operations between GPUs, and moving data through memory. A monokernel attacks that by running most of a decoder layer, attention, the MoE block and the tensor-parallel all-reduces, inside a persistent kernel in one or two launches. The sgl-amd-mono README describes exactly this and says it "reduces launch gaps and collective latency during small-batch decoding."

Such kernels are tightly tied to specific chips and model shapes. Maintaining them inside a general framework is costly, which is the argument for a plugin boundary: vendors keep their kernels, SGLang keeps its scheduler and APIs. The AMD library loads through the sglang.srt.plugins entry point and registers each model's spec under the monokernel backend amd. Every step a spec does not serve runs SGLang's standard path.

What does sgl-amd-mono support?

Requirements include SGLang with the monokernel interface, ROCm with FlyDSL and AITER, and an MI350X or MI355X. MiniMax-M3 also depends on two pending SGLang pull requests, and the backend refuses to run until they merge. If the model, GPU or configuration is unsupported, the server refuses to start and lists every reason, which is a good failure mode: you do not silently get a slow path.

The README marks the feature experimental and publishes no performance numbers. For the models themselves, see our coverage of DeepSeek V4.1 Flash 1017, GLM-5.3 Flash and MiniMax M3.1 Flash.

What about NVIDIA's plugin and the speedups?

The NVIDIA sibling, sgl-nvidia-mega, is an Apache-2.0 package of NVIDIA "megakernels" that also loads as an SGLang general plugin. Its README marks it experimental and currently covers one setup: DeepSeek-V4.1-Flash on four GB300 GPUs (SM103) at tensor parallel 4, handling DSpark target verify steps of 1 to 384 rows. Larger verify steps, prefill and extend run SGLang's standard path. You enable it with --enable-nvidia-megakernel; without the flag SGLang runs unchanged, and with it an unsupported setup stops the server at startup, before weights load, and lists the reasons. It needs SGLang's CUDA 13 image and three environment variables for the DeepSeek-V4.1 spec.

The README reports no performance numbers on the page, though it mentions a benchmark/ directory behind its speed figures. Speedup figures circulating elsewhere (for example, 1.38x on GB300 and 1.56x on AMD MI355X) came from secondary write-ups that we could not trace to a primary page, so treat them as claimed, not verified.

What SGLang's own post says, and what it does not

SGLang's blog post, DeepSeek-V4.1 Flash on SGLang: from 35 to 873 tokens/s, is a separate and well-documented optimization story on 4x GB300 at batch size 1. It reports plain decode rising from 35.2 to 203.3 tokens/s, and 873.63 tokens/s in the final speculative-decoding configuration, with the single biggest gain, from 35.2 to 117.8 tokens/s, coming from rearranging FP8 scale layouts so dense GEMMs use the Blackwell MXFP8 kernel. The median gap between ranks arriving at MoE finalize fell from 11.14 to 3.40 microseconds.

Two details matter for reading it next to the plugin news. First, the post does not describe a plugin system or a dedicated low-latency mode. Second, its authors say none of the results use the new kernels DeepSeek released with V4.1, and it uses a fork commit rather than a mainline release, with simulated speculative acceptance and throughput that excludes full prefill. So these 873 tokens/s are not the plugin numbers, and the two stories should not be merged into one speedup claim.

How does a plugin actually get used at runtime?

Reading the AMD README closely, the flow has four parts. First, installation registers the library as a general plugin through a Python entry point, so SGLang discovers it without code changes in the framework. Second, on startup the plugin registers a spec for each supported model under a named backend, here amd. Third, the operator opts in with --monokernel-backend amd, which tells SGLang to route supported decode and speculative-verify steps to the plugin's kernels. Fourth, anything the spec does not cover, such as prefill or an unsupported layer type, runs through SGLang's standard forward path.

That design has a useful property: opting in is explicit, and the failure behavior is strict. If you ask for the backend on a model, GPU or configuration the plugin does not serve, the server refuses to start and prints every reason instead of quietly falling back. For production, that is safer than a silent slow path, because a misconfigured node shows up at deploy time rather than as a latency regression a week later.

The pairing of speculative decoding with monokernels is also worth noting. The README says supported decode and speculative-verify steps are routed to the kernels, which means the plugin is designed for the same agent-era workloads where draft-and-verify schemes matter. SGLang's own kernel post leans on a speculative method called DSpark with a block size of five, which shows why verify-step latency matters so much once speculation is in play.

Who should care?

Three groups. Teams serving open-weight MoE models on AMD Instinct hardware gain the most direct option, since the plugin targets exactly the MI350X and MI355X parts and the models many of them run. Teams on NVIDIA GB300 should watch sgl-nvidia-mega, whose scope is narrow for now (one model, four GPUs). Everyone else, including people on older GPUs, mainly benefits indirectly: a plugin boundary makes it easier for new accelerators to arrive with first-class support, and it may reduce the time between a new model release and a fast serving path on non-NVIDIA hardware.

If you are choosing between engines, the honest summary is that there is no verified head-to-head here. The READMEs publish no benchmark results, SGLang's blog benchmarks a different configuration, and the speedup headlines come from secondary sources. The right move is to test on your own traffic.

Questions we expect readers to ask

Is this production-ready? The AMD README labels the feature experimental, says kernels and the plugin interface may change, and one model's backend is blocked until pending changes merge. Pilot it before depending on it.

Does it work with other engines such as vLLM? The README describes an SGLang plugin only. Nothing in it claims compatibility with other frameworks.

What license applies? sgl-amd-mono is Apache-2.0, so commercial use and modification are allowed with notice requirements.

Can I write my own plugin? The plugin entry point and spec registration suggest so, but the README is the integration documentation and it covers the AMD implementation, not a general authoring guide. Read the code and the SGLang repository before planning your own.

How does this compare with other open inference work?

SGLang is one of several open engines racing on agent-era latency. We have covered DeepSeek's DeepInfer engine, TokenSpeed for agentic workloads and vLLM's push-based KV transfer. The plugin approach is a different bet: rather than one team out-optimizing everyone, let each hardware vendor optimize its own chips behind a stable interface.

Why it matters, and what to be careful about

For operators, the benefit is potentially faster single-user decoding on the newest accelerators without leaving SGLang. For AMD in particular, a vendor-maintained path helps close the software gap with NVIDIA. The risks are the usual ones for experimental kernels: narrow model coverage, pinned software versions, interface changes, and results that depend on batch size one, which is the latency-sensitive case and not the throughput-per-dollar case. If you serve many concurrent users, a monokernel tuned for small batches may help less.

Practical next steps

1. Check whether your model and GPU appear in the support table above.
2. Clone the plugin, pip install -e ., and start SGLang with --monokernel-backend amd.
3. Benchmark against the default path at your real concurrency, not only batch size one.
4. Watch for official benchmark publications from SGLang or AMD before trusting any speedup headline.
5. Pin versions: the README warns that kernels, models and the plugin interface may change.

Details reflect the repositories and posts as read on October 20, 2026.

Related reading

• DeepSeek V4.1 Flash 1017 open weights
• DeepSeek V4.1 Flash KV cache reduction
• DeepInfer open inference engine: claimed vs verified
• TokenSpeed inference engine
• vLLM push-based KV transfer
• NVIDIA Flare local inference server
• MiniMax M3.1 Flash

Official sources: sgl-amd-mono README, sgl-nvidia-mega README, SGLang kernel optimization post and the SGLang repository.