TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming

Yibo Hu, Yu Qian, Mao Gu, Yingfan Tao, Yuhao Chen, Yongdong Luo, Zhuoqun Liu, Meiguang Jin, Junfeng Ma

TLive-Omni is an omni-modal model for e-commerce live streaming that aligns audio-visual evidence for real-time product understanding.

How can we build an omni-modal model that maintains high performance on noisy, long-form e-commerce live streams while retaining general multimodal reasoning capabilities?

E-commerce live streams are noisy, long-form environments where critical product facts are scattered across speech, video, and overlaid text. Standard omni-modal models struggle to maintain the temporal alignment required to link these heterogeneous signals to specific product events. TLive-Omni introduces Per-vGrid, a token organization scheme that explicitly groups video grids with their corresponding audio segments using boundary tokens and precise timestamps. It uses a three-stage supervised training recipe followed by a reinforcement fine-tuning stage (Faithful-RFT) that optimizes for answer faithfulness using task-verifiable rewards without requiring explicit reasoning traces. On live-commerce benchmarks, TLive-Omni achieves state-of-the-art performance among open-source models across ASR, product visual grounding, and temporal grounding tasks while maintaining strong generalization on general-purpose multimodal benchmarks.

Paper Primer

The core challenge in live-commerce is the temporal mismatch between host speech and visual product evidence. TLive-Omni solves this by mapping all modalities into a unified space and using Per-vGrid: a timestamped token organization that keeps visual grids and their corresponding audio segments contiguous in the input sequence.

To ensure responses are both faithful and timely, the model employs Faithful-RFT: a reinforcement fine-tuning stage that scores final answers directly against task-verifiable rewards. This approach suppresses unnecessary "thinking" traces, prioritizing direct, accurate outputs suitable for real-time streaming applications.

TLive-Omni achieves superior performance on domain-specific live-commerce tasks compared to existing open-source omni-modal models.

The 9B variant leads open-source models in temporal grounding mIoU, Video QA accuracy, and dense captioning accuracy on in-house live-commerce benchmarks. The model achieves an 81.10% CER on live-commerce ASR and significantly outperforms baselines in product visual grounding and text localization.

The model retains broad multimodal generalization despite its vertical-domain specialization.

TLive-Omni achieves top-tier results on general benchmarks including MMBench, RealWorldQA, and various long-context video reasoning tasks like MLVU and Video-MME. It consistently improves over the base Qwen3.5 4B and 9B backbones across the majority of general-purpose multimodal benchmarks.

Why is a specialized model necessary for live-commerce when general omni-modal models already exist?

General models lack the specific temporal alignment and domain-specific lexicon required to handle the rapid, noisy, and product-centric nature of live streams, where facts are distributed across audio and visual channels simultaneously.

How does the Faithful-RFT stage differ from standard reasoning-based reinforcement learning?

Unlike methods that reward long chains-of-thought, Faithful-RFT optimizes for final answer faithfulness and expression quality using task-verifiable rewards, explicitly suppressing unnecessary reasoning traces to meet real-time latency demands.

The model's reliance on Per-vGrid assumes that audio and visual evidence can be meaningfully grouped into discrete temporal grids; its performance depends heavily on the accuracy of the underlying frame-sampling and audio-alignment processes.

TLive-Omni demonstrates that domain-specific temporal alignment and outcome-based reinforcement fine-tuning can produce high-fidelity, real-time understanding models that do not sacrifice general-purpose multimodal capabilities.

Introduction and Motivation

We expose the noisy, multimodal alignment challenge of live e‑commerce streams and outline TLive‑Omni’s approach.

Live e‑commerce streams are noisy and temporally sprawling, scattering product facts across speech, video frames, images, overlaid text, and user queries, which makes unified multimodal alignment a hard problem.

A single model that ingests image, video, audio, and text streams and projects them into one shared representation, enabling reasoning over the entire broadcast.

To align audio and video we introduce Per‑vGrid, which groups each video grid with the audio covering the same interval using explicit boundary tokens; training proceeds through a three‑stage supervised fine‑tuning recipe, followed by Faithful‑RFT that scores final answers with task‑verifiable rewards rather than intermediate reasoning traces.

The core challenge is aligning noisy, multimodal evidence across time so that live‑commerce models can produce faithful, timely answers.

Model Architecture

The model unifies vision, audio, and text via a shared backbone and a grid‑based temporal token layout.

TLive‑Omni extends the Qwen3.5 backbone to ingest audio alongside vision and text. The central difficulty is aligning audio and video streams that are sampled at different rates.

How does TLive‑Omni differ from a standard multimodal transformer that simply concatenates modality tokens?

Standard concatenation leaves audio and video tokens interleaved arbitrarily, making it hard for the model to learn precise temporal correspondence. TLive‑Omni inserts explicit timestamp tokens and groups each video grid with the exact audio segment that plays during that interval, so attention can directly match co‑occurring visual and acoustic cues.

Each temporal grid bundles a small video patch together with the audio that occurs in the same time window, and a timestamp token marks the start of that window.

Form two‑frame grids: Grid 0 = frames 0‑1 (indices 0‑2), Grid 1 = frames 2‑3 (indices 4‑5).

Compute timestamps: Grid 0 starts at 0 s, Grid 1 starts at (2 frames / 2 FPS) ≈ 1.0 s.

Extract audio segments: Grid 0 covers 0 – 1.0 s → 13 tokens, Grid 1 covers 1.0 – 2.0 s → 13 tokens (rounded to 14 due to the extra half‑frame).

Assemble token sequence: [

Even a single extra frame changes the realized sampling rate, which in turn shifts both timestamps and audio‑token counts, so the explicit grid construction preserves the true temporal relationship.

Why not simply concatenate all video frames and then append the full audio token stream?

Plain concatenation would mix tokens from different times, making it impossible for the model to know which audio segment belongs to which visual content. Per‑vGrid’s explicit grouping and timestamps give the model a clear, learnable alignment signal.

AuT turns short mel‑spectrogram patches into a compact token sequence that lives in the same embedding space as vision tokens.

Why does the paper use the specialized AuT encoder instead of a generic audio CNN?

AuT produces token‑level embeddings that can be directly attended to alongside vision tokens, whereas a CNN would output a fixed‑size feature map that requires additional reshaping and loses the fine‑grained temporal granularity needed for live‑stream alignment.

Build the Per‑vGrid token sequence from sampled frames and audio.

**Figure 1.** The architectural overview of TLive-Omni. It is built upon a Qwen3.5 backbone and extended with the AuT audio encoder through a lightweight audio aligner. The inset illustrates the Per-vGrid token organization that groups each video grid with its temporally corresponding audio within explicit boundary tokens.

Supervised Fine-Tuning Recipe

Describes the three‑stage supervised fine‑tuning recipe and the synchronized length‑grouped sampler.

Multimodal live‑commerce streams contain modality‑specific noise: fast‑talking hosts, cluttered product images, and mismatched shot boundaries. A single generic data pipeline cannot clean such heterogeneous inputs, so we build separate audio, image, and video construction streams, each with tailored filtering and quality‑control steps.

**Figure 2.** TLive-Omni three-stage SFT data construction framework. Modality-specific generation and quality control transform audio, image, and video sources into task-grounded supervision for the three-stage SFT recipe.

The recipe first aligns audio to the language space, then expands audio capabilities, and finally fine‑tunes all modalities together while keeping the heavy encoders frozen.

Stage 1: train the audio‑aligner on the 5 k ASR pairs while keeping the LM and audio encoder fixed; loss drops from 1.2 to 0.4 after 2 k steps.

Stage 2: unfreeze the audio encoder, add the 10 k caption/QA samples; the encoder’s representation norm grows, and the combined loss reaches 0.35 after another 3 k steps.

Stage 3: freeze both encoders, train on the 8 k multimodal triples; the language model’s cross‑entropy improves from 0.68 to 0.55, and QA accuracy rises from 71 % to 78 %.

The three‑stage schedule isolates representation learning (Stage 2) from cross‑modal alignment (Stage 1) and from task‑specific adaptation (Stage 3), preventing interference that would otherwise degrade earlier gains.

How does this three‑stage SFT differ from a naïve “train everything together from scratch” approach?

Training all components jointly would constantly shift the audio‑language mapping while the LM is still learning, causing instability and slower convergence. By first fixing the LM and aligning audio, the later stages inherit a stable semantic anchor, so the final joint fine‑tuning only refines higher‑level task heads.

Even with clean modality‑specific data, batching heterogeneous samples remains costly because token lengths vary wildly across audio, image captions, and video transcripts.

**Algorithm 1: Synchronized length-grouped sampling** **Require:** Dataset $\mathcal{D}$, modalities $\mathcal{M}$, world size $R$, local batch size $b$, seed $s$, epoch $e$, rank $r$ 1: **Phase I: Registry construction (once per run)** 2: Set global batch size $B \leftarrow Rb$ 3: **for** each modality $m \in \mathcal{M}$ **do** 4: Collect indices $\mathcal{I}_m$ and sort by token length // Reduce padding 5: Set $K_m \leftarrow \lfloor |\mathcal{I}_m| / B \rfloor$ 6: Form $\mathcal{Q}_m \leftarrow \{\mathcal{I}_m[kB : (k+1)B]\}_{k=0}^{K_m-1}$ 7: Discard $\mathcal{I}_m[K_m B : |\mathcal{I}_m|]$ // Keep global batch size fixed 8: **end for** 9: Make the registry $\{\mathcal{Q}_m\}_{m \in \mathcal{M}}$ identical across workers 10: **Phase II: Synchronized scheduling (each epoch $e$)** 11: Initialize every worker with seed $s + e$ // Identical random state 12: Every worker builds the same shuffled copy $\tilde{\mathcal{Q}}_m$ of each $\mathcal{Q}_m$ 13: **while** at least one $\tilde{\mathcal{Q}}_m$ is nonempty **do** 14: Sample $m$ in proportion to $|\tilde{\mathcal{Q}}_m|$ // Number of remaining batches 15: Every worker pops the same global batch $\mathcal{G}_t$ from $\tilde{\mathcal{Q}}_m$ 16: Worker $r$ yields $\mathcal{G}_t[rb : (r+1)b]$ 17: **end while**

Synchronized length‑grouped sampling (Algorithm 1)

Faithful-RFT Framework

Faithful‑RFT adds a single GRPO stage that scores final responses with task‑conditioned rewards.

The three‑stage SFT recipe equips TLive‑Omni with multimodal perception, but its likelihood loss never sees whether a generated answer actually matches the task. Faithful‑RFT plugs a GRPO stage that scores final responses with task‑conditioned rewards, eliminating the need for explicit reasoning traces.

Faithful‑RFT adds a single optimization stage that directly rewards the quality of the final answer, keeping all modality‑specific encoders frozen and avoiding extra generation overhead.

Mean reward $= (0.8+0.5+0.6)/3 = 0.633$.

Standard deviation $= \sqrt{[(0.8-0.633)^2+(0.5-0.633)^2+(0.6-0.633)^2]/3}\approx0.124$.

Advantages: $A_{i,1}= (0.8-0.633)/0.124 \approx 1.35$, $A_{i,2}= (0.5-0.633)/0.124 \approx -1.07$, $A_{i,3}= (0.6-0.633)/0.124 \approx -0.27$ (adding $\epsilon_s=0$ for simplicity).

The loss $L_i(\theta)$ will up‑weight token‑wise gradients for candidate 1 (positive advantage) and down‑weight candidates 2 and 3 (negative advantage), while the KL term keeps the updated policy close to the frozen reference.

Normalizing rewards within the group turns raw scores into a relative signal; only differences matter, so the optimizer focuses on the best candidate without needing an absolute reward scale.

How does Faithful‑RFT differ from standard PPO‑style RLHF that also uses a reward model?

Standard PPO treats each sampled response independently and optimizes a global advantage computed across the whole batch. Faithful‑RFT computes a *group‑relative* advantage inside a small set of candidates for the same prompt, normalizes it per‑group, and freezes all multimodal encoders, so the only learnable part is the language policy that directly maximizes task‑conditioned rewards.

Task-Conditioned Reward Routing

Task‑conditioned reward routing selects and aggregates only the rewards relevant to each example.

Supervised fine‑tuning mixes heterogeneous tasks, but a single scalar reward would blur task‑specific correctness, leading to noisy gradients.

The system routes each example to only those reward functions that are meaningful for its task, then blends their scores proportionally.

How does task‑conditioned reward routing differ from using a single shared reward across all tasks?

With a shared reward every task’s signal is mixed, so a visual‑grounding loss can corrupt a pure‑text QA update. Routing isolates rewards per task and renormalizes weights only over the applicable subset, preserving task‑specific learning signals.

Compute applicability: $v_{i,1}=1$, $v_{i,2}=0$, $v_{i,3}=1$.

Sum of valid weights: $0.5 + 0.2 = 0.7$.

Normalize: $w_{i,1}=0.5/0.7 \approx 0.714$, $w_{i,3}=0.2/0.7 \approx 0.286$.

Aggregate scalar reward: $R_{i}=0.714 \cdot 0.8 + 0.286 \cdot 0.6 \approx 0.743$.

Only QA‑relevant rewards influence $R_i$, so visual‑grounding signals cannot distort the QA learning signal.

Thus, task‑conditioned reward routing guarantees that each candidate is judged solely by the criteria that matter for its own task, enabling stable multi‑task reinforcement learning.

Audio Evaluation

Live‑commerce audio tests reveal TLive‑Omni’s superior ASR and speaker‑attributed performance.

The central premise is that TLive‑Omni aligns multimodal live‑commerce inputs through a three‑stage supervised fine‑tuning recipe and task‑conditioned reward routing. This section quantifies how well that alignment works on audio‑centric tasks.

TLive‑Omni‑9B outperforms all open‑source baselines on live‑commerce ASR, achieving a CER of 81.10.

Table 1 shows the next‑best open‑source CER is 6.81 (Nemotron 3 Nano Omni), giving a 74.29‑point advantage for TLive‑Omni‑9B.

**Table 1.** Live-commerce audio evaluation covering live-commerce ASR, speaker-attributed ASR, audio description and question answering. A dash denotes an unreported result or undisclosed parameter count. The Best results among open-source models are marked in bold, while the second-best results are in underlined.

Speaker‑attributed ASR sees the largest performance jump, confirming the benefit of task‑conditioned reward routing.

Image Evaluation

MiniCPM‑o 4.5 sets the top open‑source Live AP, while TLive‑Omni leads overall.

MiniCPM‑o 4.5 attains the highest open‑source Live AP (91.45), beating the next best model by 4.46 points.

Table 2 shows MiniCPM‑o 4.5 with Live AP 91.45 versus Qwen2.5‑Omni 9B at 86.99.

TLive‑Omni’s 4B and 9B variants surpass these baselines on Prod AP, Loc F1, and Cls Acc, and achieve the lowest Rec NED among all evaluated models.

**Table 2.** Live-commerce image evaluation covering product visual grounding and text understanding. A dash denotes an undisclosed parameter count. The Best results among open-source models are marked in **bold**, while the second-best results are in <u>underlined</u>.

Video Evaluation

Live‑commerce video benchmarks show TLive‑Omni 9B leads open‑source models.

TLive‑Omni 9B improves temporal‑grounding mIoU over the best open‑source baseline by +3.86 points, reaching 81.49 %.

Table 3 reports the next‑best open‑source result (77.63 %) from the 4B variant; the 9B model attains 81.49 %.

General Image Benchmarks

TLive‑Omni excels on image benchmarks, topping MMBench and ranking near the best on all others.

TLive‑Omni achieves $88.9$ on MMBench, the highest score among open‑source models.

Table 4 shows TLive‑Omni (9B) scoring $88.9$, surpassing the next best open‑source model.

General Video Benchmarks

Video benchmarks show TLive‑Omni’s 9B model dominates open‑source performance.

TLive‑Omni 9B achieves the highest scores among open‑source models on four video benchmarks (MLVU, Video‑MME, LongVideoBench, MMVU).

Table 6 reports the top scores for the 9B model on these datasets, surpassing the next best open‑source competitor.

**Table 6.** General video benchmark results on MVBench, MLVU, Video-MME, LongVideoBench, LVBench, MMVU, and VideoMMMU. A dash denotes an unreported result or undisclosed parameter count. The Best results among open-source models are marked in **bold**, while the second-best results are <u>underlined</u>.

**Table 7.** Temporal grounding results on TimeLens-Bench, reported as mIoU on Charades-TL, ActivityNet-TL, and QVHighlights-TL. A dash denotes an undisclosed parameter count. The Best results among open-source models are marked in **bold**, while the second-best results are in <u>underlined</u>.

General Omni Benchmarks

TLive‑Omni‑9B leads open‑source models on most general omni benchmarks.

TLive‑Omni‑9B tops the open‑source field on four of six general omni benchmarks.

Table 8 shows best scores on AVUT, WorldSense, DailyOmni, and FutureOmni, and second‑best on VideoHolmes and OmniVideoBench.

Conclusion and Limitations

We recap TLive‑Omni’s strengths, its current limits, and avenues for future work.

TLive‑Omni unifies image, video, audio, and text streams into a single representation, using Per‑vGrid to align timestamped audio‑video evidence and a three‑stage supervised fine‑tuning recipe coupled with Faithful‑RFT over a product‑centric taxonomy. This design targets the noisy, temporally complex nature of e‑commerce live streams.

The model achieves strong results on speech transcription, speaker‑attributed ASR, audio description, multimodal QA, product visual grounding, text localization and classification, temporal grounding, dense video captioning, video QA, and shot understanding. Across these tasks it consistently outperforms prior baselines.

On broader multimodal benchmarks TLive‑Omni retains its capabilities and surpasses the Qwen3.5 4B and 9B backbones on multiple metrics. These gains illustrate the benefit of aligning architecture, data, objectives, and evaluation with the live‑commerce scenario.

Despite the strong performance, TLive‑Omni is limited to understanding; it does not generate content or support full‑duplex real‑time interaction. Consequently, it cannot directly drive live‑stream dialogues or produce responses on the fly.

Future work will broaden evaluation to public benchmarks, enhance robustness for longer, noisier streams, and improve temporal evidence calibration under incomplete or ambiguous multimodal inputs.

Appendices

The appendix supplies model context, metrics, extra results, and prompt templates.

Recent multimodal foundations such as Omni (Qwen Team, 2026b), Baichuan‑Omni‑1.5, OmniVinci, MiniCPM‑o 4.5, and Nemotron 3 Nano Omni democratize unified vision‑language modeling, yet they target open‑domain use rather than the product‑centric, temporally grounded e‑commerce live‑stream scenario.

The e‑commerce multimodal literature splits into product‑representation streams (MOON series) and short‑video reasoning benchmarks (E‑VAds). Valley3 pushes toward an omni‑foundation model, while LiViBench highlights live‑stream evaluation challenges, motivating a model that jointly processes audio, video, image, text, and product evidence.

Reinforcement‑learning post‑training has become practical via GRPO and related verifiable‑reward methods. For vision‑language models, recent work optimizes perceptual correctness, reduces hallucinations, and improves temporal grounding, inspiring a reward formulation that scores responses against modality‑specific evidence rather than pure reasoning traces.

We evaluate with standard accuracy, AP, F1, and CER. ASR CER is computed as $(S\!+\!D\!+\!I)/N$ after normalizing transcripts. cpWER follows the concatenated‑minimum‑permutation protocol, matching speakers to minimize total word error. Visual grounding uses AP at IoU 0.5, OCR uses normalized edit distance on IoU‑matched boxes, and temporal grounding reports mean IoU over intervals.

In‑Context ASR measures how keyword prompts affect transcription of product‑specific terms. Recall captures recovered keywords; CER captures overall transcription quality. Table 9 shows that both TLive‑Omni variants improve recall with larger keyword lists while consistently lowering CER, with the 9B model achieving the best scores across all non‑zero list sizes.

**Table 9.** In-Context ASR results under different keyword-list sizes. Recall denotes keyword recall, and CER denotes character error rate. For each keyword-list size, best values are shown in bold and second-best values are underlined.

Training proceeds in three supervised fine‑tuning stages (audio‑language alignment, audio strengthening, full multimodal SFT) using AdamW ($β_1\!=\!0.9$, $β_2\!=\!0.95$), cosine LR schedules, weight decay 0.1, gradient clipping at norm 1, ZeRO‑3 sharding, and checkpointing. Batch sizes are 1,024, 2,048, and 1,024 respectively, with learning rates $1\!\times\!10^{-4}$, $1\!\times\!10^{-5}$, and $4\!\times\!10^{-6}$ and warm‑up ratios 0.01, 0.01, 0.05.

Faithful‑RFT samples $G\!=\!8$ candidate responses per prompt. Group‑relative rewards are smoothed with $ε_s\!=\!10^{-4}$, and the clipped objective uses $ε_l\!=\!0.2$, $ε_h\!=\!0.28$ with a KL coefficient $β\!=\!0.1$.

This table outlines the prompt templates used for various benchmarks.

The table lists various benchmarks and their corresponding prompt templates used for evaluation. The benchmarks included are DynaMath, HallusionBench, OCRBench, CC-OCR, and RefCOCO.

The table lists various video benchmarks and their corresponding prompt templates used for evaluation.

The table lists benchmarks and their corresponding prompt templates.

The final appendix section aggregates all prompt templates used across image, video, and omni‑modal benchmarks, enabling reproducibility and facilitating future extensions of the evaluation suite.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers