Can Claude or LLMs Watch a Video? Here's How to Make It Work

Can Claude watch a video? Not the way humans do. Paste a YouTube URL into ChatGPT and you mostly get captions, not frame-by-frame vision. Upload an MP4 to Claude and you often hit unsupported file type. Even Gemini, which accepts video natively, typically samples frames at a fixed rate — fine for a lecture, weak for a 15-second reel with six cuts.

The workable pattern in 2026 is not "give the model the video." It is give the model what matters from the video — speech as text, visuals as deduplicated keyframes, metadata as a manifest — then ask your questions.

TL;DR — what actually works

Bottom line: For Q&A and research, use frames + transcript + manifest. For editing, use video-use. For fastest cloud try, use Gemini video. For privacy and reels, use local prep (crv or DIY).

Why "watching" breaks on most LLM UIs

Multimodal models do not stream 24 fps into context. They receive a bounded set of images and text inside a token budget.

video-use's core insight applies broadly: the LLM should read structure, not raw pixels — but for open-ended video Q&A you need both transcript and selectively chosen frames, not transcript alone.

Solution 1 — Native multimodal APIs (Gemini, GPT-4o)

Gemini video

Google's Gemini models accept video in AI Studio and the Gemini API. Default sampling is often ~1 frame per second — documented behavior teams hit when analyzing long files.

Good for: meeting recordings, lectures, slow-paced demos.
Weak for: TikTok edits, sports highlights, music videos with fast cuts.

See Gemini Omni Flash for Google's video stack direction — generation and understanding are related but not identical products.

ChatGPT + links

Useful when spoken content carries the signal. Weak when the answer is on screen but never spoken — UI walkthroughs, silent demos, visual gags.

Cost: bundled in subscription or API vision pricing; you do not control frame selection.

Solution 2 — Scene-aware local prep (claude-real-video)

claude-real-video (crv, MIT, ~250 GitHub stars as of July 2026) is one open-source answer when you want any LLM — Claude, ChatGPT, Gemini, local models — to reason over what changed visually, without uploading the video to a vendor.

pip install claude-real-video
pip install "claude-real-video[whisper]"   # + audio transcription
brew install ffmpeg                        # macOS; required on all platforms

crv "https://www.youtube.com/watch?v=..."
# → crv-out/frames/*.jpg
# → crv-out/transcript.txt
# → crv-out/MANIFEST.txt

Why scene detection beats 1 fps

How it works (short): yt-dlp or local file → ffmpeg scene-change + fps floor → pixel-diff dedup (not perceptual hash — hashes miss flat-color hue shifts) → subtitle track if present else Whisper → MANIFEST.txt for the model.

Tuning flags that matter:

Optional --keep-audio: saves audio.m4a for models that accept audio (Gemini, GPT-4o audio) when tone and music matter — transcript alone loses both.

Prompt pattern after crv

You are analyzing a video I preprocessed locally.

Read MANIFEST.txt for metadata, transcript.txt for speech, and frames/*.jpg in order.

Questions:
1. What is the main argument in the first 2 minutes?
2. Which frame shows the pricing table?
3. List every scene change topic in chronological order.

Drop the folder into Claude Projects, paste paths in ChatGPT, or attach images in Gemini — same artifacts, any frontier model.

Solution 3 — Transcript-first editing (video-use)

If your goal is produce a new video, not understand an existing one, use video-use:

• ElevenLabs Scribe → word-level takes_packed.md
• LLM reasons over text, emits ffmpeg EDL
• Optional timeline_view composites for cut decisions

The LLM never watches — it reads and edits. Perfect for launch cuts and filler removal (Fable 5 launch pipeline); wrong tool for "summarize this documentary's visual motifs."

Pairing: Run crv for research Q&A; run video-use when you need final.mp4.

Solution 4 — DIY ffmpeg + Whisper

Minimal version without crv:

# Scene frames
ffmpeg -i input.mp4 -vf "select='gt(scene,0.3)',showinfo" -vsync vfr frames/%04d.jpg

# Audio
whisper input.mp4 --model medium --output_format txt

You lose sliding-window dedup, manifest generation, and URL fetch — but keep air-gapped control. Reasonable for one-off internal compliance reviews.

Solution 5 — Video VLMs and structured extraction

For camera feeds and agent loops ("what changed since last frame?"), small video VLMs like Marlin 2B on NemoStation target structured outputs — not consumer YouTube Q&A.

Use when building products, not when a marketer wants a one-off summary of a webinar.

Cost math — why prep pays off

Naive approach: 30 fps × 60 sec × 1,500 tokens ≈ 2.7M tokens/minute of vision — unusable.

crv on a 8-minute talk (illustrative):

You trade minutes of local ffmpeg for orders-of-magnitude fewer vision tokens — the right trade for token budget planning.

Which solution should you pick?

Limitations (honest)

• crv is young — v0.1.x, no formal releases yet; verify on your content types before production pipelines.
• Whisper errors — domain jargon and overlapping speakers still hurt; embedded subtitles beat re-transcription when available.
• LLMs hallucinate on sparse frames — ask for frame filenames as citations in answers.
• Copyright — only process content you have rights to; --cookies is for your authenticated access, not credential sharing.
• No replacement for human review on legal, medical, or safety-critical video.

Related Reading

• Skyroot Vikram-1 Mission Aagaman — summarize the 2+ hour launch livestream
• video-use: Edit Videos With Claude Code
• Marlin 2B — Video VLM for Structured Information
• Gemini Omni Flash — Video Generation
• Token Budget Planning for Agent Loops
• Loop Engineering for Long-Running Agents

Tool versions accurate as of July 3, 2026. claude-real-video: github.com/HUANGCHIHHUNGLeo/claude-real-video. Always verify current Gemini sampling defaults in Google AI documentation.