explainx.ai350K+ learn from us
TrendingAI News TodayPathwaysSkills
Pricing
explainx.ai

Upskill in AI — 16 free pathways, live workshops & bootcamps, and 50+ courses from practitioners. Plus the skills, tools, and MCP servers to practice on.

follow us

follow on google

Add explainx.ai as a preferred source

corporate training

support@explainx.ai

get started

Find your pathTake Free Evaluation

community

Join the community

learn

mind: share how you thinkpathways — start freeworkshopsbootcampscoursescompare Explainxcertificationsmock testsexplainx universitycorporate traininglearn skills & mcp

discover

skillsmcp serversexplainx mcpestacktoolsmdx readeragentsllmsdesignsdictionarypeopleagi trackerfelony benchranks

company

aboutvisionmissionteaminstructorsteach on explainxpartnershipscommunityhackathonscareers

content

daily AI newsstate of AI — live resultsblogreleasespromptsgeneratorsresource libraryfor LLMsexplainx.ai kids

solutions

all solutionsdeveloper upskillingmarketing upskillingproduct manager upskillingleadership upskilling

newsletter · weekly

Get AI news, tools, and insights in your inbox.

supportcontactprivacytermsdata rightshow we create contentsubmission guidelines

© 2026 AISOLO Technologies Pvt Ltd

explainx.ai

On this page

  • What problem does LynnReal-Hand solve?
  • Base versus Fast: two operating points
  • Reported benchmarks: HandBench and latency
  • How to try LynnReal-Hand locally
  • How LynnReal-Hand fits next to other world models
  • Training story in plain language
  • Risks, limits, and open questions
  • LynnReal-Hand CLI workflow in practice
  • Paper metrics versus README metrics
  • Integration ideas for agent builders
  • Troubleshooting checklist
  • Who should care?
  • Related reading
← Back to blog

explainx / blog

LynnReal-Hand: Hand-Controlled Video World Models with Streaming at 18 FPS

World Models, Video Generation, Robotics, Open Weights, Computer Vision

Part of Robotics and Physical AI

LynnReal-Hand maps hand motion and camera poses into interactive video with a 33B causal model, Ultra-Fast-VAE, and HandBench scores up to 93 MANO Dice on one H200.

Nov 18, 2026·9 min read·Yash Thakker
add explainx.ai
go deep
LynnReal-Hand: Hand-Controlled Video World Models with Streaming at 18 FPS

LynnReal-Hand is a hand-controlled video world model that maps egocentric hand motion and camera trajectories into coherent object interactions, not just camera fly-throughs. The team published arXiv:2611.21479 on November 17, 2026 and shipped open inference code with Base and Fast variants.

If you build embodied agents or simulators, the control channel matters as much as pixels. Text can say "pick up the cup." A wearable or teleop rig gives you continuous MANO-scale hand poses every frame. LynnReal-Hand is built for that second interface.

Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

What problem does LynnReal-Hand solve?

Interactive video world models already accept text prompts and camera paths. That is enough for exploration, but manipulation needs contact-rich motion: fingers closing on a handle, bimanual coordination, releases.

The abstract states the core difficulty: hand-object dynamics are hard to model, and real-time computation is expensive. Prior work such as hand-conditional generators improved clips, yet full streaming at usable frame rates on one GPU remained scarce.

LynnReal-Hand unifies interaction modeling with efficient streaming generation through four design pieces named in the paper:

Robot hand reaching a small globe, illustrating hand-conditioned embodied video world modelsRobot hand reaching a small globe, illustrating hand-conditioned embodied video world models

table · 2 cols
ComponentRole
100K interaction clipsReal and synthetic environments with hand-aware supervision
Causal flow mapFew-step generation with a three-step Fast variant
Region-aware score distillationCombines video and image priors on contact regions
Ultra-Fast-VAELightweight decoder to cut latency

Base versus Fast: two operating points

Wireframe cube illustrating a 3D video world model control space for LynnReal-Hand hand motionWireframe cube illustrating a 3D video world model control space for LynnReal-Hand hand motion

The GitHub README separates LynnReal-Hand-Base and LynnReal-Hand-Fast:

table · 5 cols
ModelBackboneModeDefault resolutionSampling
LynnReal-Hand-Base33BBidirectional full sequence1344 x 76830 schedule points, CFG 3
LynnReal-Hand-Fast33BCausal streaming960 x 540 (768p config exists)Three flow-map steps, W2 context

Base targets quality-first offline renders. Fast targets online use: fixed first-frame anchor, bounded sliding context, and local temporal-position resets so attention memory does not grow without bound.

The Fast path emits 17 RGB frames per chunk at 540p. The CLI documents separate conditioning caches for Base and Fast; you cannot reuse a Base cache for Fast inference.

Reported benchmarks: HandBench and latency

On HandBench, the authors report:

  • FVD 181.67 and MANO Dice 93.02 for the full-sequence model
  • MANO Dice 92.49 for the three-step causal model

For streaming at 540p on one H200, the README lists 18.03 FPS, 741 ms mean warmed response per chunk, and 34.89 GiB peak allocated GPU memory with sampling-specific AdaLN tables and heavy encoders offloaded.

The repository also publishes a comparison table against JointControlVideo, Generated Reality, and Hand2World variants on latency and memory. Treat those rows as paper measurements until you run lynnreal-hand on your own hardware.

Ultra-Fast-VAE decoding numbers

Decoder-only benchmarks at 1344 x 768 (17 frames, one H200) from the README:

table · 4 cols
DecoderParams (M)Latency (ms)FPS
Native H32423.5831.3020.44
Ultra-Fast-VAE361.246.77363.21

That gap explains why the team treats VAE replacement as a first-class contribution, not an afterthought.

Green leaf lifted from a shaped recess, metaphor for isolating hand contact regions in video generationGreen leaf lifted from a shaped recess, metaphor for isolating hand contact regions in video generation

How to try LynnReal-Hand locally

The maintainers document a Linux + CUDA 12.8 stack with FlashAttention-3. High-level steps from the README:

  1. Clone https://github.com/LynnReal-AI/LynnReal and install the package in a venv.
  2. Run bash scripts/install_fa3.sh and lynnreal-hand doctor.
  3. Download weights: python scripts/download_weights.py --component fast (or base).
  4. Validate packaged example 05: lynnreal-hand validate --config configs/fast.json --example examples/05/example.json.
  5. Prepare cache, then infer with lynnreal-hand prepare and lynnreal-hand infer.

Seven examples ship with aligned images, hand videos, camera npz files, captions, and reference MP4s. Example 05 is the featured pet interaction scene in the README.

For custom data, you supply example.json with hand_video, camera.npz, caption, and frame counts in 124, 158, 243, or 260 frames. Paths must stay consistent with the same field of view.

Add --accelerated on Fast inference after the first compile pass to reuse CUDA graphs in-process.

What you should expect on smaller GPUs

The project openly targets H200. README memory for Base full sequence is 82.59 GiB at 768p in their table. Plan cloud H200 time for serious replication; consumer cards are not the reference platform.

How LynnReal-Hand fits next to other world models

explainx.ai covered MBZUAI PAN 2 as a 29B open world model for simulative agent reasoning up to 20 seconds from text actions. PAN 2 is about plan comparison before acting in latent video space.

LynnReal-Hand is complementary:

  • Control signal: continuous hand articulation plus camera pose, not only language actions.
  • Use case: teleoperation previews, egocentric AR, and manipulation-centric simulators.
  • Architecture emphasis: causal streaming and decoder efficiency for interactive FPS.

If your agent stack already uses ForceHarness or ScaleVLA for robot policies, LynnReal-Hand is closer to the video imagination layer that sits above low-level torque commands.

For policy learned in latent video without hands, see Gemini Robotics 2 whole-body intelligence. Hand-controlled worlds are the interface when the human body remains in the loop.

Training story in plain language

The paper describes Interaction-Preserving Training (IPT) on 100K clips with hand-aware multilevel supervision so small contact regions keep gradient signal. Causal distillation preserves control fidelity when compressing to three steps.

Region-aware score distillation merges complementary video and image priors so fingers and object surfaces stay sharp while global motion stays stable. That matters because hand errors are visually loud: a floating grip breaks immersion faster than a blurry background.

The team also cites related BiWM bidirectional autoregressive world modeling and LynnReal-Omni for multimodal agent workflows in the README citation block.

Risks, limits, and open questions

License: MiniMax H3 Community terms govern the repo. Read NOTICE before product use.

Data release timing: The paper commits to datasets and training code; verify Hugging Face and GitHub release tags when you plan reproducibility studies.

Evaluation generalization: HandBench scores are author-reported. Cross-benchmark comparison to closed labs is not provided in the public README.

Input rigidity: Frame counts are discrete allowed lengths; arbitrary-length phone videos need resampling and camera alignment work described in the data preparation docs.

LynnReal-Hand CLI workflow in practice

The maintainers ship a lynnreal-hand CLI instead of a single opaque script. That matters when you integrate with robot teleop loops where prepare and infer run on different machines.

table · 3 cols
CommandGPU requiredPurpose
lynnreal-hand doctorOptionalCheck FFmpeg, FlashAttention-3, CUDA
lynnreal-hand validateNoVerify example.json paths and frame counts
lynnreal-hand prepareYesBuild conditioning cache from image + hand video + camera
lynnreal-hand inferYesStream MP4 from prepared cache
lynnreal-hand infer --acceleratedYesReuse compiled CUDA graphs in-process

Validation without weights is deliberate: teams can fix camera alignment and caption mismatches before downloading multi-gigabyte Hugging Face packages.

Example 05 pet interaction path

The README spotlights example 05 because it shows bimanual affordances near a pet, a scene where hand motion dominates semantics.

  1. Download Fast weights with python scripts/download_weights.py --component fast.
  2. Run validate on examples/05/example.json with configs/fast.json.
  3. Prepare cache to outputs/cache/fast/example_05.
  4. Infer to outputs/fast/example_05.mp4 with seed 42.
  5. Compare against packaged Fast reference MP4 in the same folder.

If MANO alignment is wrong, output fingers may float. The README stresses that image, hand motion, and camera poses must share field of view and frame order. That is the most common integration bug in hand-conditioned video.

Frame budget constraints

Packaged inputs accept 124, 158, 243, or 260 frames at the source frame rate. Arbitrary phone footage needs resampling documented in data preparation guides on GitHub.

For 768p Fast, switch to configs/fast_768.json and rebuild caches; you cannot hot-swap resolution on an existing 540p cache.

Paper metrics versus README metrics

The arXiv PDF and README occasionally use different resolutions in tables:

  • Paper HandBench highlights MANO Dice and FVD at reported training resolution.
  • README latency table mixes 768p full-sequence Base with 540p Fast chunks.

When you cite numbers externally, specify which row you mean. Reviewers rejected vague "18 FPS world model" claims when the measurement is warmed Fast chunk throughput, not first-frame cold start.

Comparison to Hand2World and JointControlVideo

The README reproduction table positions LynnReal-Hand against Hand2World-Bi, Hand2World-AR, Generated Reality, and JointControlVideo. LynnReal-Hand-Base is slower than Hand2World-Bi on median full-sequence latency in that table, but reports higher MANO Dice on HandBench.

table · 2 cols
DirectionImplication
Accuracy-first offlineConsider Base at 1344 x 768
Teleop streamingFast at 540p with Ultra-Fast-VAE
Smaller backbonesHand2World 5B may fit tighter VRAM

None of those tradeoffs remove the H200-class assumption for official Fast benchmarks.

Integration ideas for agent builders

Simulators: Pipe MANO or MediaPipe hand tracks into hand_video, feed SLAM poses into camera.npz, and let LynnReal-Hand predict visual consequences before executing torques on hardware.

Data flywheel: Use generated failures to label grasp policies; pair with ScaleVLA only after you verify labels on real video.

Safety review: Hand-conditioned worlds can depict dangerous contact. Treat outputs like any synthetic media pipeline with policy filters before user-facing playback.

Troubleshooting checklist

table · 3 cols
SymptomLikely causeFix
OOM on inferBase at 768p on smaller GPUSwitch to Fast 540p or rent H200
Drifting fingersMisaligned camera npzRe-export poses with shared intrinsics
Black frames after prepareMissing hand_video pathRun validate with --check-weights
Slow first inferCUDA graph compileKeep process alive; use --accelerated

Who should care?

table · 2 cols
ReaderTakeaway
Robotics engineerTest whether hand-video conditioning reduces sim-to-real gap for grasp planning demos
AR/VR builderStreaming 540p interaction may pair with passthrough headsets when latency holds on your GPU
Agent researcherCombine with text planners: language picks the task, hands supply the low-level trajectory
EducatorOpen examples make a concrete lab for world models beyond static image diffusion

Related reading

  • MBZUAI PAN 2 open world model
  • ForceHarness and frozen frontier models for robots
  • ScaleVLA 552B vision-language-action backbone
  • External sandboxes for coding agents
  • Mobile AI Factory on-device agents

Primary sources: arXiv:2611.21479 and LynnReal-AI/LynnReal as of November 18, 2026.

Spotted something out of date? Let us know.
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

  • ✓ Free
  • ✓ One email a day, no spam
  • ✓ Unsubscribe anytime
Yash Thakker

Written by

Yash Thakker

Yash is an AI expert with over 300K learners. Join his workshops →

View Yash Thakker in People in AI →

Related posts

Nov 11, 2026

SWAM: Shanghai Jiao Tong World Action Model With Open Weights

The SWAM paper on arXiv 2611.12870 trains world action models on about 47,000 hours of video and demonstrations, reporting 88.10% RoboTwin Easy success and 40% real-world success across three tasks. Authors release code, data, and weights.

Nov 5, 2026

Verne V0.5: An Open-Weight 8B Robot World Action Model, 99.3% on LIBERO, and a Claim Worth Checking

Verne Robotics published V0.5, an open-weight world action model with three checkpoints on Hugging Face. It reports 99.3% on LIBERO and 81.0% clean and 73.4% randomized on RoboTwin 2.0. Its own leaderboard table shows rivals ahead, so we checked the headline claim against the cards.

Sep 4, 2026

Runway's GWM Worlds 2 Doesn't Play a Video. It Keeps a World Running.

Runway published research on GWM Worlds 2, a "General World Model" that generates continuous, interactive 720p video at 24fps with synchronized 48kHz audio — a world you steer with text actions rather than a video you watch. Sessions have no preset length because the world keeps generating from whatever you do next. Here's what actually changed from GWM-1, and where the real limits are.