WeMM-Video is a family of four open models from Tencent that watch a video and listen to its soundtrack at the same time. Tencent's WeChat Vision team published the technical report on November 19, 2026, and the weights, a Docker setup and the evaluation pipeline went public on GitHub and Hugging Face. The sizes are 2B, 4B and 9B dense, plus a 35B-A3B mixture-of-experts model. The headline claim is that the flagship sets a new open-source state of the art on omni-modal video tests. That claim is Tencent's own, so this post separates what the paper says from what you can check yourself.
TL;DR: WeMM-Video at a glance
| Question | Answer |
|---|---|
| Who made it? | WeChat Vision at Tencent, with Tsinghua University |
| What does it do? | Takes text, images, audio and video in, writes text out |
| Sizes | 2B, 4B, 9B dense and 35B-A3B mixture-of-experts |
| Languages | Chinese and English |
| Training data | About 1.2 trillion multimodal tokens |
| Modes | Instruct (fast) and thinking (step-by-step) |
| Code license | Apache License 2.0; each checkpoint has its own model card license |
| Formats | BF16, FP8, AWQ W4A16, GGUF Q4_K_M, NVFP4 W4A16 |
| Runtimes | Transformers, patched vLLM, patched SGLang, native GGUF runtime |
| Main claim | New open-source state of the art for omni-modal video (Tencent's evaluation) |
What is an omni-modal video model, and why does the audio matter?
Play button with an eye above it, showing a model that watches video and understands audio in WeMM-Video
Most open video models look at frames and ignore the sound. An omni-modal model also takes the audio track, so it can answer questions such as who is speaking, what the speaker said, or whether the music matches the scene. The paper argues that existing open models "remain brittle" on real video: they miss audio-visual details, fail to line up a sound with the event that caused it, and hallucinate across modalities.
WeMM-Video targets exactly that gap. For a clip with sound, it interleaves visual tokens and audio tokens in time order and adds text timestamps, so the model can tie a spoken phrase to the frame where it happened. That design matters for the use cases Tencent lists: live-stream commerce, short-form drama analysis and interactive video.
How is WeMM-Video built?
The architecture borrows from video-SALMONN 2, according to the report. The dense models sit on Qwen3.5 backbones (2B, 4B and 9B), and the 35B-A3B model sits on Qwen3.6-35B-A3B. The audio encoder starts from Qwen2.5-Omni. So this is a Tencent build on top of Alibaba's open models, which is worth knowing if you track which labs depend on which base models. We covered the Qwen side of that stack in our look at Qwen4 Max and its token plan.
The audio path has two parts. A Whisper-based encoder reads 16 kHz audio in 30-second chunks. A window-level Q-Former then squeezes every 0.1-second window into one token, which gives 10 audio tokens per second of sound. Vision uses each backbone's own image and video encoder unchanged.
Three green segments splitting into three open trays, representing the 2B, 4B and 9B sizes plus the 35B-A3B WeMM-Video family
Training in two phases
Pre-training has three stages. Stage 1 trains only the audio aligner on speech and audio captions while the rest stays frozen. Stage 2 unfreezes everything except the visual encoder and mixes text, image, audio and joint audio-visual data. Stage 3 stretches the context window with long videos, long audio and multi-video inputs.
Joint audio-visual data is the largest slice at 45.4% of the roughly 1.2 trillion tokens, and captioning is the largest task at 37.7%, per the report. Tencent says its caption data includes sports, esports, films, short dramas, interviews, variety shows and live-stream commerce.
Post-training has three steps:
- Multi-teacher on-policy distillation (OPD). Stronger external teachers improve text and vision-language skills, while same-family teachers protect the audio skills.
- Anti-repetition MrDPO. The authors saw endless, repetitive decoding in the base models and trained against mined examples of it, updating LoRA adapters round by round.
- Specialist reinforcement learning. Rule-based rewards for checkable tasks, rubric and teacher-consistency rewards for open-ended ones, with long-context RL at the end.
The repetition fix is a useful detail for anyone who has watched a small multimodal model loop on a long clip. Whether it holds on your own footage is something only a test can show.
What does Tencent claim about performance?
The report evaluates more than 120 benchmarks across video with audio, silent video, audio, image and text. The audio-visual suite alone has 33 benchmarks, including Video-MME with audio, Daily-Omni, WorldSense and OmniVideoBench. The main claims:
| Model | Claim in the report |
|---|---|
| WeMM-Video-35B-A3B (thinking) | Beats Seed2.0-Mini-0428 on 16 of 17 general audio-visual benchmarks and Qwen3.5-Omni-Flash on 16 of 17 |
| WeMM-Video-35B-A3B | "Narrows the gap" to Gemini-3.0-Flash (medium thinking); the report does not claim a win |
| WeMM-Video-9B (thinking) | Beats Qwen3-Omni-30B-A3B-Instruct and MiniCPM-o-4.5-9B on most general audio-visual tasks |
| WeMM-Video-4B (thinking) | Beats the 30B-scale baseline on the vast majority of benchmarks |
| WeMM-Video-2B | Matches or beats the 30B baseline on many perception, hallucination and captioning tests |
| 35B-A3B and 9B on image and text | On par with the Qwen3.6-35B-A3B and Qwen3.5-9B backbones, and ahead of Qwen3.5-Omni-Flash |
Read these with care. The authors chose the baselines, ran many of the comparison numbers themselves ("official results where available and our own evaluations otherwise") and used LLM judges such as GPT-4o and GPT-6.1-Sol for some captioning and interaction scores. The paper also reports that thinking mode adds large gains over instruct mode on most reasoning tasks. That is plausible, but it means the headline numbers come from thinking mode, which costs more tokens and time.
The image and text result is the quiet good news. Adding audio did not damage the backbones: the report says the omni models stay on par with the models they were built from, with better instruction following. If true, you can use one model for documents, screenshots and video instead of two.
How do you run WeMM-Video?
The README gives a concrete path. You need Python 3.12, an NVIDIA GPU with a driver for CUDA 12.9 and FFmpeg. The pinned stack is torch 2.10.0, torchvision 0.25.0, torchcodec 0.10.0 and transformers 5.2.0.
The Transformers route loads tencent/WeMM-Video-2B with trust_remote_code=True. That flag runs custom code from the model repository, so read the code first and pin a commit SHA through the revision argument, as the README itself advises. The example asks one text question, one image question, one audio transcription and two video prompts: one with frames only (video) and one with frames plus the original soundtrack (video_audio).
For serving, Tencent ships Dockerfiles for patched vLLM 0.18.0 and patched SGLang 0.5.16. Both expose an OpenAI-style /v1/chat/completions endpoint on port 8000. You send a video_url item and set use_audio to true to include the soundtrack. A request can carry up to four images, two videos and two audio items within the context length. The README's own example launches with a context length of 8,192, so check how much video that really holds before you plan on long clips.
If you have no big GPU, the GGUF Q4_K_M files run on a native runtime on CPU or CUDA. The model zoo lists BF16, FP8 Block128, AWQ W4A16, GGUF and NVFP4 builds for every size.
Which size should you pick?
| Size | Pick it when |
|---|---|
| 2B | You are testing the pipeline, or need a small on-device model for captions |
| 4B | You want more headroom at low VRAM |
| 9B | You want the best dense option; Tencent says it beats the larger Qwen3-Omni-30B-A3B on most tasks |
| 35B-A3B | You can serve a mixture-of-experts model and want the top reported scores |
We have not run these checkpoints, so treat the table as a starting point from Tencent's claims, not a benchmark of our own.
How does it compare with other open multimodal releases?
Open audio-and-video models are a busy corner right now. Tencent's own Youtu-Parsing Omni 5B focuses on parsing documents, audio and video into structured output, while WeMM-Video is aimed at open-ended questions and reasoning over clips. Liquid AI's LFM3 family and Cloudflare's Clef Omni lean toward small decision models. On the speech side, Haru-4B is an Apache-licensed full-duplex model also built on Qwen3.5. For closed options, Google's Gemini Omni 1.2 Flash generates video rather than understanding it, so it is a different job.
Tencent's open-weight habit is also worth noting. It has shipped a run of open models this autumn, such as HY SimulMT for real-time translation, and Yao Shunyu has commented on open weights and biology risk. WeMM-Video fits that pattern: a product-driven team publishing what it already runs.
Limits and open questions
- Self-reported scores. The baselines, judges and several comparison numbers were chosen or run by the authors. Wait for independent tests on Video-MME, Daily-Omni or your own clips before you rely on the ranking.
- License detail. The code is Apache 2.0, but the README says the model license "covers the WeMM-Video collection as of November 20, 2026" and tells you to check each Hugging Face model card. Read the card for the exact terms before commercial use.
- Custom code and patched servers. You need
trust_remote_codefor Transformers and patched builds of vLLM or SGLang. That is more setup and more trust than a stock model. - Short default context. The README examples run at 8,192 tokens, which limits how much video you can send at once.
- Early signal. The GitHub repo had 21 stars and no forks when we read it, and the Hugging Face paper page showed 27 upvotes. This is a day-old release, not a proven tool.
- Bilingual focus. The models target Chinese and English. Other languages are not claimed.
What does this mean for what you build or pay?
If you build video features, the useful point is cost control. A 9B or 35B-A3B open model you host yourself replaces per-minute API calls to a closed video model, and because it hears audio it can handle talking-head clips, lectures and live-stream replays that frame-only models miss. If you sell to Chinese-speaking users, the bilingual training and the WeChat deployment make it a natural fit.
If you decide on a vendor, do not switch on the paper alone. Run the 2B or 9B on twenty of your own clips, compare against what you use today, and look at hallucinated audio details first, because that is the failure mode the authors say they set out to fix.
What to do next
- Open the Hugging Face collection and read the model card and license of the size you want.
- Run the README's
examples/transformers_video.pyon the 2B model with a clip that has speech. - Compare frames-only against frames-plus-audio on the same clip. The difference is the point of the model.
- Re-run your hardest clip in thinking mode and watch latency and token use.
- Check back for third-party benchmark runs before you commit.
Related reading
- Tencent Youtu-Parsing Omni 5B: parse documents, audio and video
- Tencent HY SimulMT 30B-A3B for real-time translation
- Haru-4B open full-duplex speech model
- Liquid AI LFM3 open-weight family
- Cloudflare Clef Omni open multimodal model
- Gemini Omni 1.2 Flash video model
- Alibaba Qwen4 Max preview and token plan
- Sources: technical report on arXiv, GitHub repository, Hugging Face paper page
Details are accurate as of November 20, 2026. Model licenses, runtime versions and benchmark claims can change; check the repository and model cards before you build on them.
