microsoft_Mage-VL

---

license: apache-2.0 library_name: transformers pipeline_tag: image-text-to-text tags:

• multimodal
• vision-language-model
• mage-vl
• video-understanding
• streaming

---

Mage-VL
An Efficient Codec-Native Streaming Multimodal Foundation Model

---

Mage-VL is a codec-native, proactive-streaming multimodal foundation model for image and video understanding, whose visual encoder is trained entirely from scratch at a compact 4B scale. It targets a modern Moravec's paradox of VLMs — strong at complex offline reasoning, yet slow and compute-heavy on simple real-time streaming perception. Instead of decoding video into uniformly-sampled frames and pushing a dense grid of patch tokens through a frozen web-pretrained ViT, Mage-VL follows the structure of modern video codecs: it separates a stream into anchor (I) frames and predicted (P) frames, keeps every anchor patch, and retains only the predicted-frame patches where the codec spends bits — the regions carrying real motion and new detail. This codec-aligned sparsity cuts visual tokens by over 75% while preserving spatio-temporal context, yielding up to 3.5× wall-clock inference speedup over uniform frame sampling.

The system pairs two components:

• Mage-ViT — a from-scratch Codec-ViT visual encoder that allocates tokens by codec-derived spatio-temporal importance, on a shared 16×16 patch grid with 3D rotary position encoding. It is codec-agnostic: the same interface accepts a traditional codec (H.264/AVC, HEVC/H.265) via motion vectors + residual energy, or a neural codec (DCVC-RT) via its learned rate map — no architecture or retraining change.
• Qwen3-4B causal decoder — a Qwen3-4B-Instruct-2507 language backbone (the only pretrained component) that consumes Mage-ViT's variable-length token stream through a lightweight two-layer MLP projector, with a unified interface for images, short/long/ultra-long video, and streaming.

On top of this pair, a System 1 & System 2 dual-process design adds proactive streaming inside a single model: a lightweight cognition gate (System 1) watches each rolling codec window and stays silent on routine content, invoking the full VLM (System 2) only when a response-worthy event completes — no multi-agent pipeline required.

✨ Highlights

• Codec-native & from scratch. The entire visual stack is trained from scratch — no billion-scale image-text ViT initialization. The bio-inspired predictive-patch mechanism (I/P frames at 16×16) cuts visual-token consumption by over 75% (~1/8 or less of dense frame sampling), letting the model train on videos 8× longer under the same budget.
• Codec-native speedup. Codec tokenization sets a superior accuracy–efficiency frontier — up to 3.5× wall-clock inference speedup over uniform frame sampling at matched accuracy, and the fastest of all compared models on most video benchmarks (single 8×B200 node).
• Data-efficient tokenizer. Trained on only ~100M unlabeled images/videos, Mage-ViT matches or beats frontier encoders trained on billions of image-text pairs (SigLIP2 @ 10B, MoonViT @ 2B) — e.g. 99.33% on CIFAR-10 and 85.69% on ImageNet with 256 tokens, showing web-scale pretraining is not essential for a strong VLM front-end.
• Native-resolution scaling. Variable-resolution pretraining lets Mage-ViT improve monotonically with the token budget (peaking >96.1% Food-101 / >86.3% ImageNet at 676 tokens) where fixed-resolution encoders saturate or degrade.
• Matched-LLM video gains. With the 4B Qwen3 backbone held fixed and only the ViT swapped, Mage-VL improves over Qwen3-VL-4B on every reported video and temporal-grounding benchmark — largest on localization-heavy tasks (+22.5 QVHighlight, +17.1 ActivityNet, +11.0 VSI-Bench, +24.5 VideoEval-Pro).
• Strong for its size. On par with Qwen3-VL-4B on static images, and clearly ahead on video understanding and spatial intelligence (+11.0 VSI-Bench, +53.1 CrossPoint, +5.2 EmbSpatial, +22.5 QVHighlight).
• Proactive streaming, single model. A frozen-backbone cognition gate delivers low-latency, event-gated commentary; it tops TimVal / F1 / ROC-AUC / PR-AUC on SoccerNet streaming and generalizes to real 2026 World Cup broadcasts.

📥 Model

A single checkpoint, microsoft/Mage-VL, is one unified model that simultaneously provides image & video understanding and the proactive streaming gate — the same weights answer offline image/video questions and drive event-gated commentary. It covers every Mage-VL capability: image understanding, frame-sampled video, traditional H.264/HEVC codec video, neural DCVC-RT codec video, and event-gated streaming. The repository bundles the codec processor, the neural codec package, and the proactive gate weights — no separate understanding, NVC, or streaming checkpoint is required.

We additionally release microsoft/Mage-ViT — the standalone visual encoder from the two-stage, from-scratch ViT pre-training (cluster-discrimination on ~100M unlabeled image/video frames). This is the ViT-pre-trained checkpoint only: it has not gone through the joint VLM training with the language model. Use it as a data-efficient, codec-native visual encoder or as a drop-in ViT for your own multimodal training.

Model	Task	Backbone	Hugging Face
Mage-VL	image & video understanding + proactive streaming gate	Mage-ViT + Qwen3-4B-Instruct-2507	🤗 microsoft/Mage-VL
Mage-ViT	codec-native visual encoder — ViT pre-training only, no VLM joint training	Codec-ViT (from scratch)	🤗 microsoft/Mage-ViT

🏗️ Architecture

Proactive streaming framework — Mage-ViT incrementally encodes the continuous stream into codec-native visual features shared by the event gate and the causal decoder. The gate scores each rolling window and stays silent on routine content; when it opens, the decoder emits an event-conditioned response.

Mage-ViT — a from-scratch Codec-ViT visual encoder. On a 16×16 patch grid it keeps all anchor (I) frame patches and only the motion-salient predicted (P) frame patches, cutting visual tokens by over 75% while a shared 3D RoPE preserves spatio-temporal positions.

Mage-VL — a unified model where projected visual tokens and text tokens share one causal Qwen3 decoder. Still images become a single spatial block; videos become temporally-ordered codec windows. In streaming mode, a lightweight cognition gate predicts p_speak = g(h_t) per rolling window (over a recurrent streaming memory kept by an event-preserving feature extractor) and triggers generation when p_speak ≥ τ; the response is decoded by the frozen base model from a local sliding window of the most recent codec segments, and a text query can be injected at any time.

Training — a progressive five-stage supervised curriculum (no preference/RL post-training) that produces one unified model:

1. Multimodal alignment via captions — ~350M dense image captions + 4.2M short-video captions.
2. Instruction tuning + short temporal grounding — ~54M image-instruction samples + 3.4M 30–180s video captions.
3. Temporal-horizon expansion — medium/long video (LLaVA-Video, TimeLens, VideoChat-Flash, Molmo2) with retained image SFT.
4. Codec-native long-context adaptation — 350K long videos as rolling codec windows (up to 384/768 frames).
5. Proactive streaming alignment — a cognition gate fine-tuned on ~3.3M streaming samples with the visual encoder and LLM kept frozen (only the gate is trained).

The five stages together produce a single unified model, Mage-VL, that handles image understanding, offline video reasoning, and proactive streaming — no separate variants are shipped.

Two parts of the pipeline apply an AI4AI (AI-for-AI) paradigm: (1) dense recaptioning runs through an agentic closed loop where a GPT-5 rubric scorer grades captions and a Copilot coding agent co-designs the prompt and harness code (e.g. rendering timestamp overlays) under a human validation gate — improving every downstream OCR/doc/chart/perception benchmark and inspiring SkillOpt-Lite; and (2) Stage-3 uses AI-based diagnostics to decide which video categories, resolutions, and frame counts to train on.

📊 Performance

🔬 Key Findings

Beyond the model, the report distills seven empirical findings for efficient multimodal training:

1. Web-scale pretraining is not essential. A from-scratch backbone on ~100M unlabeled frames matches encoders trained on billions of image-text pairs.
2. Variable-resolution pretraining scales monotonically. Quality keeps improving with the visual-token budget instead of saturating/degrading like fixed-resolution encoders.
3. Codec-native tokenization sets a better accuracy–efficiency frontier — up to 3.5× wall-clock inference speedup over uniform frame sampling.
4. Explicit VideoQA SFT is redundant. Dense video captions + standard image SFT are sufficient for strong zero-shot VideoQA.
5. Motion–spatial synergy. Dynamic video training substantially improves static 2D/3D spatial reasoning.
6. AI4AI data pipeline. Agentic closed-loop feedback + prompt/code co-design systematically lift caption quality and downstream scores (inspired SkillOpt-Lite).
7. Zero-Vision SFT for multimodal RL. Bypassing visual SFT in favor of pure-text reasoning SFT unlocks stronger multimodal RL — a compute-efficient path.

🚀 Quick Start

A single checkpoint, microsoft/Mage-VL, covers every capability below.

Capability	Script	How to run
Image understanding	inference.py	--mode offline --image
Frame-sampled video	inference.py	--mode offline --video --video-backend frames
Traditional H.264/HEVC codec video	inference.py	--mode offline --video --video-backend codec --codec-engine traditional
Neural DCVC-RT codec video	inference.py	--mode offline --video --video-backend codec --codec-engine neural
Online image / video (SGLang)	inference.py	--mode online … --base-url <server>
Event-gated streaming commentary	inference_streaming.py	in the GitHub repo

Installation

For offline Transformers inference:

pip install "transformers>=5.7" accelerate pillow torch torchvision \
  opencv-python codec-video-prep

Codec-based video inference also requires ffmpeg and ffprobe on PATH.

Examples

Two sample inputs ship with this repository:

Input	Question	Content
examples/dog.jpg	Describe this image in detail.	Photo of a dog sitting in front of a patterned rug
examples/soccer-broadcast.mp4	Describe this video.	30s, 960×540 football broadcast clip

Offline inference

Download inference.py. Offline mode loads the checkpoint with AutoModelForCausalLM.from_pretrained and supports images, frame sampling, and both codec engines:

# image
python inference.py --mode offline --image examples/dog.jpg \
  --question "Describe this image in detail."

The image depicts a dog sitting on a patterned rug. The dog appears to be a medium-sized breed with a thick, fluffy coat. Its fur is primarily white with patches of black and brown. The dog's ears are perked up, and it has a calm and attentive expression. [...]

# video — uniform frame sampling
python inference.py --mode offline --video examples/soccer-broadcast.mp4 \
  --video-backend frames --num-frames 32 \
  --question "Describe this video."

The video opens with a man in a black polo shirt, sporting a short haircut, standing in a stadium. He is holding a yellow microphone with the BBC Sport logo on it. The background reveals a large crowd of spectators. [...]

# video — traditional codec (HEVC/H.264)
python inference.py --mode offline --video examples/soccer-broadcast.mp4 \
  --video-backend codec --codec-engine traditional --num-frames 32 \
  --question "Describe this video."

The video opens with a BBC Sport broadcast, featuring a presenter in a black shirt holding a yellow microphone. The background reveals a packed stadium, with the scoreboard displaying "ENG 1 ARG 2 FT", indicating the final score of the match. [...]

# video — neural codec (DCVC-RT)
python inference.py --mode offline --video examples/soccer-broadcast.mp4 \
  --video-backend codec --codec-engine neural --num-frames 32 \
  --question "Describe this video."

The video opens with a BBC Sport broadcast, featuring a presenter standing in a stadium filled with spectators. The presenter, dressed in a black shirt, holds a yellow BBC Sport microphone and wears a black earpiece. [...]

Online inference

Online mode talks to an OpenAI-compatible SGLang server. First build and launch the server with the Mage-VL SGLang branch (building it needs protobuf-compiler and a Rust toolchain):

sudo apt-get update && sudo apt-get install -y protobuf-compiler
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs \
  | sh -s -- -y --profile minimal --default-toolchain 1.90.0
source "$HOME/.cargo/env"

git clone -b feat/mage-vl https://github.com/kcz358/sglang
cd sglang
pip install -e 'python[all]'
python -m sglang.launch_server \
  --model-path microsoft/Mage-VL \
  --trust-remote-code

Then send an image or sampled video frames to the running server:

pip install openai

python inference.py --mode online --image examples/dog.jpg \
  --question "Describe this image in detail." \
  --base-url http://localhost:30000/v1

python inference.py --mode online --video examples/soccer-broadcast.mp4 \
  --num-frames 32 \
  --question "Describe this video." \
  --base-url http://localhost:30000/v1

Use --model, --max-new-tokens, and --api-key to override their defaults.

Streaming inference

streammind_gate.safetensors in this repository holds the event gate. Streaming inference splits a video into non-overlapping segments, stays silent on routine content, and generates a caption only when a response-worthy event is detected. Run it with inference_streaming.py from the GitHub repository:

python inference_streaming.py \
  --video examples/soccer-broadcast.mp4 \
  --video_backend codec \
  --segment_sec 8

[t=0.0-8.0s] gate=silence (p=0.19)
[t=8.0-16.0s] gate=response (p=0.55) -> The video features a live sports broadcast from BBC Sport, set in a large stadium filled with spectators. The broadcast focuses on a football match between England and Argentina, with the score displayed as England 1, Argentina 2. [...]
[t=16.0-24.0s] gate=response (p=0.73) -> The video features a sports broadcast set in a large stadium filled with spectators. Four commentators are gathered around a table with a 'BBC Sport' logo, each holding a yellow microphone. [...]
[t=24.0-30.0s] gate=silence (p=0.31)

The gate is trained on codec inputs, so --video_backend codec is the intended setting. Use --video_backend frames for direct frame sampling. Additional controls include --num_frames, --cur_fps, --max_segments, --max_new_tokens, --gate_threshold, and --attn_impl.

📝 Citation

@article{yang2026mage,
  title={Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model},
  author={Yang, Senqiao and Zhang, Kaichen and Jia, Zhaoyang and Guo, Jinghao and Shen, Yifei and Zhang, Xinjie and Zhang, Xiaoyi and Wang, Haoqing and Li, Xiao and Zhang, Peng and others},
  journal={arXiv preprint arXiv:2607.24904},
  year={2026}
}

📄 License

Mage-VL is released under the Apache-2.0 License.