Mid-Training Language Models on Raw Video
Multimodal large language models learn mostly from paired image-text data or annotated video, and raw web video is rarely used to further train an existing language model.
ProofPaper ↗
Key points
- We study whether raw video, with no captions and no text loss, can serve as mid-training data for a pretrained language model.
- Frames are encoded into continuous visual tokens, and the language model learns to predict the next visual token.
- We mid-train Qwen3-1.7B on raw clips from YT-Temporal-1B and then apply the same image-text instruction tuning to it and to the model without mid-training, so that the two differ only in mid-training.
- Predicting captions fails to outperform next-visual-token prediction, demonstrating that video mid-training can remain purely self-supervised without the computational overhead or labeling noise of automated captioning.
Sources (1)
- [1]Mid-Training Language Models on Raw VideoarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 12:03 AM
Multimodal large language models learn mostly from paired image-text data or annotated video, and raw web video is rarely used to further train an existing language model.
We study whether raw video, with no captions and no text loss, can serve as mid-training data for a pretrained language model.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 8, 2026SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference
- Oct 8, 2026REMORY: Learning Residual Memory for Context Compaction
- Oct 8, 2026OneSearch-VL: Unified Multimodal Deep Research Agent for Image and Video
- Oct 8, 2026VibeEdit: Image Editing with Canvas Instructions
- Oct 7, 2026Iris-3B: Going Beyond the Latent with Pixel-Space Diffusion Training, Conversion and Fine-Tuning
- Oct 6, 2026[AINews] Reflection Beam - 501B-A23B American Open Model