MOSS-VL Technical Report

Pengyu Wang, Chenkun Tan, Shaojun Zhou, Qirui Zhou, Yanxin Chen, Xingyang He, Huazheng Zeng, Jijun Cheng, Chenghao Wang, Xiaomeng Qian, Pengfei Wang, Zhan Huang, Shanqing Gao, Wei Huang, Longjun Cao, Wu Ran, Jie Liu, Changtai Zhu, Hongkai Wang, Yixian Tian, Chenghao Liu, Zhen Ye, Xinghao Wang, Botian Jiang, Guoguo Feng, Zhaoye Fei, Ruixiao Li, Mingshu Chen, Yang Gao, Qinyuan Cheng, Shimin Li, Xipeng Qiu

MOSS-VL is a vision-language model family co-designed for real-time interaction, enabling continuous perception during generation.

How can a vision-language model maintain real-time interaction by perceiving visual input while simultaneously generating text?

Most vision-language models are offline: they wait for a video to finish before processing it, leaving them blind while they generate a response. MOSS-VL treats real-time interaction as a first-class capability by using gated cross-attention to keep the visual stream active while the language decoder generates text. On streaming benchmarks, MOSS-VL-Realtime leads in proactive behavior, correctly timing responses to unfolding events while maintaining a significant latency advantage over standard architectures.

Paper Primer

The core mechanism hinges on gated cross-attention layers that allow the model to ingest new frames without re-encoding the entire history. The model uses a shared timeline for text and vision via XRoPE (cross-attention rotary position embedding) and explicit absolute timestamp tokens, ensuring the decoder remains aware of wall-clock time regardless of frame rate.

The training strategy uses a four-stage curriculum, concentrating real-time capabilities into a final "Realtime-SFT" stage. This stage teaches the model to treat video as a stream of decisions: the model outputs a silence token to keep watching or a response token to speak, with a focal loss re-weighting the rare emission decisions against the frequent silence tokens.

MOSS-VL-Realtime dominates proactive streaming benchmarks.

It achieved a score of 66.0 on the OmniMMI Proactive Alerting subset, compared to 37.5 for the best baseline. A 1.76x improvement in proactive alerting performance.

The architecture significantly reduces time-to-first-token latency as visual context grows.

Compared to the Qwen3-VL-8B backbone, the latency gap widens from 2.8x to 5.1x as visual history accumulates. Nearly double the efficiency gain over standard interleaved architectures in long-context streaming scenarios.

Why is this architecture better for real-time than simply feeding more frames to a standard model?

Standard models re-encode or process the entire visual sequence, causing latency to spike as history grows. MOSS-VL keeps visual tokens outside the decoded sequence and appends only new frames to the cross-attention cache, keeping the decoding state intact.

What is the "L5" capability level mentioned in the paper?

L5 represents the ability to perceive while generating. Unlike L2–L4 models that go "blind" during a reply, an L5 model can revise or interrupt its own output the moment new visual evidence appears.

The model's real-time behavior is "by construction": because visual tokens never enter the decoded sequence, the model's internal state is never interrupted by the arrival of new frames, allowing it to maintain a continuous, reactive stream.

MOSS-VL demonstrates that real-time interaction requires architectural co-design rather than just fine-tuning, shifting the paradigm from "process-then-answer" to "continuous-stream-processing."

Introduction and Motivation

Introduce the need for continuous perception and define real‑time interaction.

Most open vision–language models process video only after the clip ends, treating perception as a batch step. In real‑time settings a live assistant must keep watching while it talks, but current streaming models become blind during each reply, missing the moment when new evidence arrives.

A capability where the model continues to ingest new visual frames while it is generating text, allowing it to revise or interrupt its output as the scene evolves.

How does real‑time interaction differ from the streaming regime used by prior models?

Streaming models (levels L2–L4) ingest video continuously but pause visual input while generating a reply, making them blind to changes that occur mid‑utterance. Real‑time interaction (level L5) removes that pause: the model’s cross‑attention cache is updated with each new frame even as tokens are emitted, so the output can be revised or halted the instant new evidence appears.

**Table 1.** Levels of video understanding, from offline to real-time. Each level lights up one additional capability axis; the dividing line between the streaming regime (L2–L4) and real-time (L5) is whether the model keeps perceiving while it generates—L2–L4 models are blind during a reply, an L5 model is not. MOSS-VL-Realtime achieves L5 behavior, demonstrated through live demos and the released real-time inference code, and is quantitatively validated at levels L2–L4 on four streaming benchmarks; a dedicated benchmark for L5 behavior remains an open problem.

MOSS‑VL‑Realtime attains a 66.0 score versus 37.5 for the strongest baseline on OmniMMI’s Proactive Alerting, and its time‑to‑first‑token advantage grows from 2.8× to 5.1× as visual context increases.

The shift from static video processing to continuous perception unlocks proactive, on‑the‑fly responses.

Model Architecture

How MOSS‑VL stitches vision and language together with gated cross‑attention and a shared 3‑axis position system.

The model couples a 27‑layer vision encoder with a 48‑layer language decoder, but the two only meet inside twelve tanh‑gated cross‑attention layers.

Think of a mail sorter that first checks a gate: if the gate is closed the mail passes untouched, if it opens the sorter routes the mail to a special bin. The gate here lets the language backbone stay unchanged while optionally injecting visual information.

How does this differ from ordinary cross‑attention used in earlier vision‑language models?

Standard cross‑attention always mixes visual tokens into the text stream, forcing the language model to process them at every layer. Here the gate can keep the visual pathway completely dormant, so the model behaves exactly like a pure language model until the gate learns to open.

Compute attention scores between T₁/T₂ and V₁/V₂ (four scores).

Apply softmax to obtain attention weights.

Multiply the weighted sum of visual values by the gate scalar 0 → result is a zero vector.

Add the zero vector to the residual stream; the text tokens remain unchanged.

During training the gate scalar receives a gradient and updates to 0.4.

Re‑run the same step with gate 0.4: the visual contribution now shifts the text representation toward visual semantics.

The gate lets the model start from a pure‑language baseline and only later learn to blend vision when it improves the loss.

Imagine a 3‑D grid where the three axes are time (t), height (h) and width (w). Both text tokens and image patches occupy points in this grid, so they share a common coordinate system.

Why not use the usual 2‑D RoPE for vision and a separate 1‑D RoPE for text?

Separating the encodings would break the shared timeline: a text token could not directly align with a patch that appears later in the video because their positional bases would be incomparable. XRoPE’s unified 3‑axis system guarantees that a token and a patch that occupy the same logical moment have compatible rotations.

Patch 1 gets $(2,2,2)$, Patch 2 $(2,2,3)$, Patch 3 $(2,3,2)$, Patch 4 $(2,3,3)$.

Rotary embedding rotates each patch key by its $(h,w)$ offsets while keeping the $t$ component aligned.

Query “cat” is rotated by $(3,3,3)$, matching the $t$ axis of the patches.

Cross‑attention computes dot‑products; patches with larger $h,w$ offsets receive slightly different phase shifts, allowing the model to distinguish spatial layout.

After the gate opens, the weighted sum of patch values influences the token representation.

XRoPE lets the model treat time and space uniformly, so a single attention head can reason about “when‑and‑where” a visual element appears relative to the text.

**Figure 2** MOSS-VL architecture. Images and video frames are encoded by a 27-layer vision encoder at native dynamic resolution, then pooled 2 × 2 and projected into visual tokens; text is tokenized alongside. The LLM decoder (48 layers, initialized from Qwen3-8B) attends to visual tokens only through 12 tanh-gated cross-attention layers (Gated-XAttention, right inset), one in every four; the other 36 self-attention layers operate on the text sequence alone, so visual tokens never join the decoded sequence. Cross-attention queries carry text positions and keys carry three-axis XRoPE coordinates (t, h, w). Gates are zero-initialized scalars, so training starts from an intact language backbone.

**Figure 3.** XRoPE, the position encoding of the cross-attention channel. Text tokens and visual patches share one three-axis coordinate space $(t, h, w)$, ordered by their logical position in the stream: a text token advances all three axes together, while a frame anchored at coordinate $t$ tiles its patches from $(t, t, t)$ at the top-left to $(t, t+h-1, t+w-1)$ at the bottom-right. The separator token closing each frame on the vision side and that frame's placeholder token in the text stream receive the same coordinate, so both channels advance along a single shared timeline. Rotations are applied to text-side queries and vision-side keys before they meet in cross-attention.

Pre-Training Curriculum

The curriculum stages progressively scale data length and quality to prepare the model for real‑time interaction.

The model must learn to handle both short, dense samples and extremely long contexts, yet a single monolithic pre‑training run cannot give each regime the focus it needs. The solution is to split pre‑training into a curriculum that gradually expands sequence length and refines data quality.

Instead of training on one massive dataset, the model is guided through four increasingly demanding stages, each reshaping the data distribution and the trainable components to match the target capability.

How does this staged curriculum differ from simply scaling up data size in a single pre‑training run?

Scaling up only increases the amount of data but keeps the distribution static; the curriculum deliberately reshapes the distribution—starting with short, dense samples to bootstrap the visual‑language bridge, then progressively lengthening sequences and concentrating on higher‑quality, temporally rich data. This staged shift forces the model to master each capability before moving to the next, which a monolithic run cannot guarantee.

Stage 1: each sample is short, so the model sees $10$ M / $8K \approx 1{,}250$ samples.

Stage 2: longer samples halve the number of samples to $20$ M / $64K \approx 312$.

Stage 3: even longer samples reduce the count to $30$ M / $128K \approx 234$.

Stage 4: the longest samples yield $40$ M / $256K \approx 156$ samples, emphasizing depth over breadth.

As sequence length grows, the model sees dramatically fewer samples, forcing it to extract richer long‑range dependencies from each example rather than relying on sheer quantity.

**Table 4.** The MOSS-VL training curriculum: four pre-training stages followed by SFT and Realtime-SFT (§4). The data of each stage is described in the corresponding subsection. Token counts are the tokens fed to the language model, with vision tokens counted after the 2×2 token compression of the projection module (§2); sequence lengths are in tokens.

The table outlines the training stages, including pre-training (stages 1-4) and post-training (SFT and Realtime-SFT), detailing the number of tokens, samples, maximum sequence length, trainable parameters, and peak learning rate for each.

Post-Training and Realtime-SFT

Post‑training equips MOSS‑VL with real‑time interaction via Realtime‑SFT and a focused loss.

Realtime‑SFT trains the model to decide at each video frame whether to speak or stay silent, and to revise its answer as the visual context changes.

How does Realtime‑SFT differ from standard supervised fine‑tuning?

Standard SFT only learns to generate the next token given a static context. Realtime‑SFT adds two state tokens and trains the model to interleave those tokens with video frames, turning the stream into a sequence of explicit speak‑or‑wait decisions.

Compute class coefficients: $\alpha_s = (n_s + n_r)/(2 n_s) = (2+2)/(2 \cdot 2) = 1$, $\alpha_r = (n_s + n_r)/(2 n_r) = 1$.

For a silence token with predicted probability $p_s = 0.9$, focal weight $w_s = \alpha_s (1 - p_s)^{\gamma} = 1 \cdot (0.1)^2 = 0.01$.

For a response token with predicted probability $p_r = 0.3$, focal weight $w_r = \alpha_r (1 - p_r)^{\gamma} = 1 \cdot (0.7)^2 = 0.49$.

The loss contribution of the response token is therefore 49× larger than that of the silence token, counteracting the class imbalance.

The focal weighting makes the rare response token dominate the gradient, ensuring the model learns to speak when appropriate instead of defaulting to silence.

Collect a mixture of multimodal samples covering images, videos, and text.

Filter, deduplicate, and decontaminate the data against evaluation suites.

Tokenize each sample up to a maximum length of 128 K tokens.

Train with next‑token cross‑entropy over assistant responses using the Megatron‑LM stack.

Scale data, tensor, sequence, and context parallelism to handle 8 K‑ to 256 K‑token sequences.

Evaluation Overview

MOSS‑VL’s evaluation across offline and streaming benchmarks highlights its strengths and gaps.

The offline suite comprises 39 benchmarks spanning five capability domains, while the streaming suite includes four real‑time benchmarks that test levels L2–L4.

MOSS‑VL attains an overall score of 88.1%, surpassing all open‑source baselines.

Table 5 shows MOSS‑VL leading with 88.1% versus 78.0% for Gemma‑4 and 66.3% for LLaVA‑OV‑2.

Streaming evaluation feeds frames as they arrive; each model runs under its native streaming protocol, and we report both serving efficiency (§6.3) and qualitative real‑time session observations (§6.4).

Offline and Streaming Results

Streaming benchmarks show MOSS‑VL‑Realtime leading on three of four tasks.

Recall that the paper treats real‑time interaction as a first‑class capability by decoupling visual perception from the language decoder via gated cross‑attention. This section evaluates how that design fares on streaming benchmarks.

MOSS‑VL‑Realtime achieves the highest average score on three of the four streaming benchmarks.

OVO‑Bench 70.2 vs 65.3 runner‑up, OmniMMI 32.7 vs 25.4, ProactiveVideoQA 47.2 vs 42.7.

**Figure 1.** Results overview. (a) Average scores on four streaming benchmarks against open-source streaming baselines (details in Table 6; the StreamingBench average covers its visual groups, §6.2). (b) MOSS-VL against the best competing model on the proactive streaming subsets—PA (Proactive Alerting, OmniMMI), PO (Proactive Output, StreamingBench), and FAR (Forward Active Responding, OVO-Bench), all testing whether the model speaks up unprompted at the right moment—and on selected offline strengths (details in Tables 5 and 6). Green = MOSS-VL; gray = the labeled competitor.

LLaVA‑OneVision‑2‑8B is an open‑source multimodal model that combines a vision encoder with a language decoder, serving as a strong baseline for visual‑language tasks.

Inference Efficiency and Qualitative Analysis

Qualitative analysis of latency, live behavior, and broader implications.

We first examine inference efficiency by measuring serving latency against a strong baseline that shares the same language backbone.

**Figure 4.** Measured serving latency of MOSS-VL vs. Qwen3-VL-8B — the same Qwen3-8B language backbone, isolating the vision-integration architecture. Both models serve offline with SGLang on a single H200 (TP=1, BF16, identical engine version) with an identical generation-length cap, which every run reaches; every point is the mean of five independent cold starts (error bars: sample std). (a, c) With ViT output matched, the time-to-first-token (TTFT) gap widens from 2.8× to 5.1× as visual context grows, and end-to-end latency from 1.9× to 4.3×. (b, d) On the same video at the same resolution and frame count: Qwen3-VL compresses the temporal axis 2×, whereas MOSS-VL forgoes temporal compression so that each arriving frame can be encoded immediately — carrying about twice the vision tokens on the same input — yet it never falls behind, and its lead grows with stream length. Slower latency growth follows from the append-only cross-attention design: visual tokens never enter the decoded sequence (§2).

Removing temporal compression (i.e., using append‑only cross‑attention) reduces time‑to‑first‑token latency by up to 5.1× compared to the interleaved baseline.

Figure 4 shows the TTFT gap widening from 2.8× to 5.1× as visual tokens increase.

**Figure 5** MOSS-VL-Realtime in the wild: two screen-captured sessions from the released live demo, running on a single H200, shown as frame strips with excerpts of the model's outputs on a shared timeline (long outputs truncated with "..."; t = 0 at the user instruction). (a) Under a standing conditional instruction ("say great! whenever the cat touches the carrot, otherwise stay silent"), the model stays silent throughout and fires exactly at each of the four contacts; the hollow marker samples one of the silent spans, where the per-frame output is the silence token. (b) Under a single instruction ("commentate the video live"), commentary begins within a second and tracks a free-kick sequence — set-up, the referee's whistle, the strike, the celebration, and the updated scoreline — in a professional broadcast register. Outputs are in Chinese, the demo language; English translations of both sessions are given in Appendix A.2, and these and further live cases can be viewed on the official blog (https://openmoss.ai/MOSS-VL/).

Under a standing conditional instruction, the model fires exactly at the four target contacts and remains silent elsewhere.

Figure 5 panel (a) demonstrates this precise behavior.

The overall pattern shows streaming gains where response timing matters, a serving advantage that widens with visual history, and offline strengths on temporal‑reasoning video sets—none of which can be traced to a single component.

Dialogue Template Details

Defines the token stream format that enables real‑time interaction.

All three inference modes use the same ChatML‑style template; offline treats the whole video as one block, while streaming and real‑time prepend the shared system prompt and interleave decision slots with frame placeholders.

A real‑time session consists of alternating <|`im_start`|> tags with role headers and content, where each assistant turn emits a sequence of <|silence|>, <|video|>, and <|response|> tokens.

The warm‑up turn starts with an empty user message, causing the assistant to emit only <|silence|> tokens before any instruction appears.

Each speaking frame carries exactly one <|response|> token, so a reply spanning k frames contains k <|response|> tokens, one ahead of each chunk.

A reply ends with a single closing <|silence|> placed in the same slot as the final <|response|>, serving as the model’s sole end‑of‑reply signal.

During training, about half the samples keep the slot between the user message and the next frame, while the other half drop it, teaching the model to handle both immediate and delayed frame arrivals.

A turn may contain several replies, as shown when the assistant closes one reply, stays silent for a frame, then initiates another without a new user turn.

Each <|video|> placeholder expands to a timestamped vision block containing the frame’s arrival time and the visual tokens derived from the image.

At runtime, the system pushes one frame, waits for the model to emit a fresh <|silence|>, then pushes the next frame, ensuring the stream never stalls behind a long reply.

Live Demo Transcripts

Live demo transcripts illustrate real‑time interaction behavior.

In the “Conditional alert” demo the user asks the model to watch a video and say “Great!” only when the cat touches the carrot; the model emits “Great!” exactly at the four contact frames and remains silent otherwise.

The “Real‑time commentary” demo shows the model narrating a football match as it unfolds, producing a sequence of five sentences that describe the free‑kick, the referee’s whistle, Ronaldo’s run‑up and strike, his celebratory pose, and the final 3–3 scoreline.

**Table 7.** English translations (ours) of the two live-demo sessions in Figure 5. The figure carries the Chinese excerpts; timestamps and the placement of each output on the stream are read off the figure's timeline.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers