explainx.ai350K+ learn from us
TrendingAI News TodayExplainx for Business
Pricing
explainx.ai

Upskill in AI — 16 free pathways, live workshops & bootcamps, and 50+ courses from practitioners. Plus the skills, tools, and MCP servers to practice on.

follow us

follow on google

Add explainx.ai as a preferred source

corporate training

support@explainx.ai

get started

Find your pathTake Free Evaluation

community

Join the community

learn

mind: share how you thinkpathways — start freeworkshopsbootcampscoursescompare Explainxcertificationsmock testsexplainx universitycorporate traininglearn skills & mcp

discover

skillsmcp serversexplainx mcpestacktoolsmdx readeragentsllmsdesignsdictionarypeopleagi trackerfelony benchranks

company

aboutvisionmissionteaminstructorsteach on explainxpartnershipscommunityhackathonscareers

content

daily AI newsstate of AI — live resultsblogreleasespromptsgeneratorsresource libraryfor LLMsexplainx.ai kids

solutions

all solutionsdeveloper upskillingmarketing upskillingproduct manager upskillingleadership upskilling

newsletter · weekly

Get AI news, tools, and insights in your inbox.

supportcontactprivacytermsdata rightshow we create contentsubmission guidelines

© 2026 AISOLO Technologies Pvt Ltd

explainx.ai

On this page

  • TL;DR: WeMM-Video at a glance
  • What is an omni-modal video model, and why does the audio matter?
  • How is WeMM-Video built?
  • What does Tencent claim about performance?
  • How do you run WeMM-Video?
  • How does it compare with other open multimodal releases?
  • Limits and open questions
  • What does this mean for what you build or pay?
  • What to do next
  • Related reading
← Back to blog

explainx / blog

WeMM-Video: Tencent Opens Four Omni-Modal Video Models, 2B to 35B-A3B

Tencent, Open Source, Multimodal, Video Understanding, Hugging Face

Part of AI Tools and Apps

Tencent WeChat Vision released WeMM-Video, open audio-visual models from 2B to 35B-A3B. See what the paper claims, how to run it and what to check.

Nov 20, 2026·10 min read·Yash Thakker
add explainx.ai
go deep
WeMM-Video: Tencent Opens Four Omni-Modal Video Models, 2B to 35B-A3B

WeMM-Video is a family of four open models from Tencent that watch a video and listen to its soundtrack at the same time. Tencent's WeChat Vision team published the technical report on November 19, 2026, and the weights, a Docker setup and the evaluation pipeline went public on GitHub and Hugging Face. The sizes are 2B, 4B and 9B dense, plus a 35B-A3B mixture-of-experts model. The headline claim is that the flagship sets a new open-source state of the art on omni-modal video tests. That claim is Tencent's own, so this post separates what the paper says from what you can check yourself.

TL;DR: WeMM-Video at a glance

table · 2 cols
QuestionAnswer
Who made it?WeChat Vision at Tencent, with Tsinghua University
What does it do?Takes text, images, audio and video in, writes text out
Sizes2B, 4B, 9B dense and 35B-A3B mixture-of-experts
LanguagesChinese and English
Training dataAbout 1.2 trillion multimodal tokens
ModesInstruct (fast) and thinking (step-by-step)
Code licenseApache License 2.0; each checkpoint has its own model card license
FormatsBF16, FP8, AWQ W4A16, GGUF Q4_K_M, NVFP4 W4A16
RuntimesTransformers, patched vLLM, patched SGLang, native GGUF runtime
Main claimNew open-source state of the art for omni-modal video (Tencent's evaluation)
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

What is an omni-modal video model, and why does the audio matter?

Play button with an eye above it, showing a model that watches video and understands audio in WeMM-VideoPlay button with an eye above it, showing a model that watches video and understands audio in WeMM-Video

Most open video models look at frames and ignore the sound. An omni-modal model also takes the audio track, so it can answer questions such as who is speaking, what the speaker said, or whether the music matches the scene. The paper argues that existing open models "remain brittle" on real video: they miss audio-visual details, fail to line up a sound with the event that caused it, and hallucinate across modalities.

WeMM-Video targets exactly that gap. For a clip with sound, it interleaves visual tokens and audio tokens in time order and adds text timestamps, so the model can tie a spoken phrase to the frame where it happened. That design matters for the use cases Tencent lists: live-stream commerce, short-form drama analysis and interactive video.

How is WeMM-Video built?

The architecture borrows from video-SALMONN 2, according to the report. The dense models sit on Qwen3.5 backbones (2B, 4B and 9B), and the 35B-A3B model sits on Qwen3.6-35B-A3B. The audio encoder starts from Qwen2.5-Omni. So this is a Tencent build on top of Alibaba's open models, which is worth knowing if you track which labs depend on which base models. We covered the Qwen side of that stack in our look at Qwen4 Max and its token plan.

The audio path has two parts. A Whisper-based encoder reads 16 kHz audio in 30-second chunks. A window-level Q-Former then squeezes every 0.1-second window into one token, which gives 10 audio tokens per second of sound. Vision uses each backbone's own image and video encoder unchanged.

Three green segments splitting into three open trays, representing the 2B, 4B and 9B sizes plus the 35B-A3B WeMM-Video familyThree green segments splitting into three open trays, representing the 2B, 4B and 9B sizes plus the 35B-A3B WeMM-Video family

Training in two phases

Pre-training has three stages. Stage 1 trains only the audio aligner on speech and audio captions while the rest stays frozen. Stage 2 unfreezes everything except the visual encoder and mixes text, image, audio and joint audio-visual data. Stage 3 stretches the context window with long videos, long audio and multi-video inputs.

Joint audio-visual data is the largest slice at 45.4% of the roughly 1.2 trillion tokens, and captioning is the largest task at 37.7%, per the report. Tencent says its caption data includes sports, esports, films, short dramas, interviews, variety shows and live-stream commerce.

Post-training has three steps:

  1. Multi-teacher on-policy distillation (OPD). Stronger external teachers improve text and vision-language skills, while same-family teachers protect the audio skills.
  2. Anti-repetition MrDPO. The authors saw endless, repetitive decoding in the base models and trained against mined examples of it, updating LoRA adapters round by round.
  3. Specialist reinforcement learning. Rule-based rewards for checkable tasks, rubric and teacher-consistency rewards for open-ended ones, with long-context RL at the end.

The repetition fix is a useful detail for anyone who has watched a small multimodal model loop on a long clip. Whether it holds on your own footage is something only a test can show.

What does Tencent claim about performance?

The report evaluates more than 120 benchmarks across video with audio, silent video, audio, image and text. The audio-visual suite alone has 33 benchmarks, including Video-MME with audio, Daily-Omni, WorldSense and OmniVideoBench. The main claims:

table · 2 cols
ModelClaim in the report
WeMM-Video-35B-A3B (thinking)Beats Seed2.0-Mini-0428 on 16 of 17 general audio-visual benchmarks and Qwen3.5-Omni-Flash on 16 of 17
WeMM-Video-35B-A3B"Narrows the gap" to Gemini-3.0-Flash (medium thinking); the report does not claim a win
WeMM-Video-9B (thinking)Beats Qwen3-Omni-30B-A3B-Instruct and MiniCPM-o-4.5-9B on most general audio-visual tasks
WeMM-Video-4B (thinking)Beats the 30B-scale baseline on the vast majority of benchmarks
WeMM-Video-2BMatches or beats the 30B baseline on many perception, hallucination and captioning tests
35B-A3B and 9B on image and textOn par with the Qwen3.6-35B-A3B and Qwen3.5-9B backbones, and ahead of Qwen3.5-Omni-Flash

Read these with care. The authors chose the baselines, ran many of the comparison numbers themselves ("official results where available and our own evaluations otherwise") and used LLM judges such as GPT-4o and GPT-6.1-Sol for some captioning and interaction scores. The paper also reports that thinking mode adds large gains over instruct mode on most reasoning tasks. That is plausible, but it means the headline numbers come from thinking mode, which costs more tokens and time.

The image and text result is the quiet good news. Adding audio did not damage the backbones: the report says the omni models stay on par with the models they were built from, with better instruction following. If true, you can use one model for documents, screenshots and video instead of two.

How do you run WeMM-Video?

The README gives a concrete path. You need Python 3.12, an NVIDIA GPU with a driver for CUDA 12.9 and FFmpeg. The pinned stack is torch 2.10.0, torchvision 0.25.0, torchcodec 0.10.0 and transformers 5.2.0.

The Transformers route loads tencent/WeMM-Video-2B with trust_remote_code=True. That flag runs custom code from the model repository, so read the code first and pin a commit SHA through the revision argument, as the README itself advises. The example asks one text question, one image question, one audio transcription and two video prompts: one with frames only (video) and one with frames plus the original soundtrack (video_audio).

For serving, Tencent ships Dockerfiles for patched vLLM 0.18.0 and patched SGLang 0.5.16. Both expose an OpenAI-style /v1/chat/completions endpoint on port 8000. You send a video_url item and set use_audio to true to include the soundtrack. A request can carry up to four images, two videos and two audio items within the context length. The README's own example launches with a context length of 8,192, so check how much video that really holds before you plan on long clips.

If you have no big GPU, the GGUF Q4_K_M files run on a native runtime on CPU or CUDA. The model zoo lists BF16, FP8 Block128, AWQ W4A16, GGUF and NVFP4 builds for every size.

Which size should you pick?

table · 2 cols
SizePick it when
2BYou are testing the pipeline, or need a small on-device model for captions
4BYou want more headroom at low VRAM
9BYou want the best dense option; Tencent says it beats the larger Qwen3-Omni-30B-A3B on most tasks
35B-A3BYou can serve a mixture-of-experts model and want the top reported scores

We have not run these checkpoints, so treat the table as a starting point from Tencent's claims, not a benchmark of our own.

How does it compare with other open multimodal releases?

Open audio-and-video models are a busy corner right now. Tencent's own Youtu-Parsing Omni 5B focuses on parsing documents, audio and video into structured output, while WeMM-Video is aimed at open-ended questions and reasoning over clips. Liquid AI's LFM3 family and Cloudflare's Clef Omni lean toward small decision models. On the speech side, Haru-4B is an Apache-licensed full-duplex model also built on Qwen3.5. For closed options, Google's Gemini Omni 1.2 Flash generates video rather than understanding it, so it is a different job.

Tencent's open-weight habit is also worth noting. It has shipped a run of open models this autumn, such as HY SimulMT for real-time translation, and Yao Shunyu has commented on open weights and biology risk. WeMM-Video fits that pattern: a product-driven team publishing what it already runs.

Limits and open questions

  • Self-reported scores. The baselines, judges and several comparison numbers were chosen or run by the authors. Wait for independent tests on Video-MME, Daily-Omni or your own clips before you rely on the ranking.
  • License detail. The code is Apache 2.0, but the README says the model license "covers the WeMM-Video collection as of November 20, 2026" and tells you to check each Hugging Face model card. Read the card for the exact terms before commercial use.
  • Custom code and patched servers. You need trust_remote_code for Transformers and patched builds of vLLM or SGLang. That is more setup and more trust than a stock model.
  • Short default context. The README examples run at 8,192 tokens, which limits how much video you can send at once.
  • Early signal. The GitHub repo had 21 stars and no forks when we read it, and the Hugging Face paper page showed 27 upvotes. This is a day-old release, not a proven tool.
  • Bilingual focus. The models target Chinese and English. Other languages are not claimed.

What does this mean for what you build or pay?

If you build video features, the useful point is cost control. A 9B or 35B-A3B open model you host yourself replaces per-minute API calls to a closed video model, and because it hears audio it can handle talking-head clips, lectures and live-stream replays that frame-only models miss. If you sell to Chinese-speaking users, the bilingual training and the WeChat deployment make it a natural fit.

If you decide on a vendor, do not switch on the paper alone. Run the 2B or 9B on twenty of your own clips, compare against what you use today, and look at hallucinated audio details first, because that is the failure mode the authors say they set out to fix.

What to do next

  1. Open the Hugging Face collection and read the model card and license of the size you want.
  2. Run the README's examples/transformers_video.py on the 2B model with a clip that has speech.
  3. Compare frames-only against frames-plus-audio on the same clip. The difference is the point of the model.
  4. Re-run your hardest clip in thinking mode and watch latency and token use.
  5. Check back for third-party benchmark runs before you commit.

Related reading

  • Tencent Youtu-Parsing Omni 5B: parse documents, audio and video
  • Tencent HY SimulMT 30B-A3B for real-time translation
  • Haru-4B open full-duplex speech model
  • Liquid AI LFM3 open-weight family
  • Cloudflare Clef Omni open multimodal model
  • Gemini Omni 1.2 Flash video model
  • Alibaba Qwen4 Max preview and token plan
  • Sources: technical report on arXiv, GitHub repository, Hugging Face paper page

Details are accurate as of November 20, 2026. Model licenses, runtime versions and benchmark claims can change; check the repository and model cards before you build on them.

Spotted something out of date? Let us know.
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

  • ✓ Free
  • ✓ One email a day, no spam
  • ✓ Unsubscribe anytime
Yash Thakker

Written by

Yash Thakker

Yash is an AI expert with over 300K learners. Join his workshops →

View Yash Thakker in People in AI →

Related posts

Oct 28, 2026

Tencent Hy-SimulMT: An Open 30B Model for Real-Time Speech Translation

Tencent Hunyuan released Hy-SimulMT, an Apache-2.0 simultaneous translation model that listens to Chinese or English speech and translates as the speaker talks. Here is what the benchmarks say, what they leave out, and how to run it.

Oct 23, 2026

Tencent Hy-Nomas: An Open Interactive Video World Model You Can Steer by Text and Camera

Tencent Hunyuan published Hy-Nomas on Hugging Face on October 23, 2026: a segment-autoregressive video world model you steer with text and camera paths, plus Hy-Nomas-Bench with 280 test cases. Here is what it does, what it needs and what is still missing.

Oct 15, 2026

Radical Numerics Halcyon: Text-to-Image Without Language Pretraining, 275x Less Compute

Radical Numerics says text-to-image generation does not need a pretrained language model. Its Halcyon family, trained only on 100M image-text pairs, beats two baselines with up to 275x less pretraining compute. Here is what was measured and what the headline hides.