Motion-Omni: End-To-End Joint Speech and Full-Body Motion for Spoken Dialogue

Chengqian Ma, Wei Tao, Haoyu Zhang, Yiwen Guo

Motion-Omni enables LLMs to natively generate speech and full-body motion end-to-end, eliminating the latency of separate motion-generation stages.

How can we unify spoken dialogue generation and full-body motion synthesis into a single, end-to-end autoregressive framework?

Spoken dialogue models and co-speech motion models currently operate as separate systems, forcing a cascade where motion is generated only after the speech audio is finished. This creates a significant latency bottleneck and prevents the motion objective from informing the dialogue generation process. Motion-Omni bridges this gap by conditioning motion generation directly on the Speech Generator’s hidden states, allowing the motion objective to co-adapt the LLM and speech modules. This end-to-end approach matches the motion quality of teacher-based cascades while responding 5.4× faster, achieving sub-real-time latency.

Paper Primer

The core mechanism hinges on a dual-input conditioning interface: the Motion Generator decoders attend to the Speech Generator’s hidden states (keys/values) and consume interpolated speech-token embeddings (queries). This allows the model to generate motion from the same representations that produce speech, effectively bypassing the need for a rendered waveform as an intermediate input.

The training pipeline uses a model-agnostic teacher (LOM) to pseudo-label 1,402 hours of speech-motion pairs. A four-stage curriculum progressively co-adapts the Speech Projector, LLM, Speech Generator, and Motion Generator, ensuring that motion gradients improve speech-motion alignment without degrading the dialogue backbone.

Motion-Omni-Q7 achieves near-teacher motion quality with significantly lower latency.

On the SwDA-500 benchmark, the model matches the same-audio teacher cascade to within 2% on reference-free motion metrics while completing responses in 4.32s (RTF = 0.78). 5.4× faster response time than the equivalent teacher-cascade baseline.

Joint optimization is essential for speech-motion alignment.

Ablation studies show that when the speech pathway is frozen, generated motion remains visibly misaligned with the audio; co-adapting the modules is required to recover synchronization.

Why is this approach superior to simply running a motion model after the speech is generated?

The cascade approach requires a second full inference pass after the audio is complete and prevents motion objectives from updating the dialogue parameters. Motion-Omni removes this second pass and enables reciprocal transfer, where motion supervision preserves timing cues in the speech-generation states.

What is the scope of the motion generated by this model?

The model generates co-speech communicative motion, specifically facial expressions, hand gestures, and upper/lower-body movement. It is not designed for locomotion, dance, or sports-related action generation.

The model is an offline response generator, not a streaming interaction model. It ingests the full user utterance before emitting the first response token, which limits its use in real-time, low-latency streaming dialogue.

By integrating motion as a native output of the LLM, researchers can now optimize for both verbal and non-verbal communication simultaneously, moving away from the rigid, high-latency cascade architectures that currently dominate the field.

Introduction and Motivation

We expose why separate speech and motion models hinder synchronized, expressive dialogue and outline our unified solution.

Separate spoken‑dialogue models and co‑speech motion models operate in isolation, so the avatar’s speech and body movements are generated at different times and can become out of sync, breaking the natural coupling humans rely on for expressive communication.

Building such a unified model faces three core challenges. First, joint optimisation must improve both speech and motion without degrading either, despite their heterogeneous rates (12.5 Hz speech units vs. 30 Hz motion). Second, large‑scale supervision is needed, but existing corpora lack consistent‑voice speech paired with full‑body motion. Third, there is no public benchmark for stochastic open‑ended spoken dialogue that evaluates both modalities together.

We address these challenges with three contributions. (1) Motion‑Omni, an end‑to‑end framework that generates motion directly from the hidden states that produce speech, enabling co‑adaptation of the LLM, Speech Generator, and Motion Generator. (2) A scalable, model‑agnostic teacher pseudo‑labeling pipeline that produces 422,856 quality‑ranked speech‑motion pairs (1,402 hours) for training. (3) SwDA‑500 and a reproducible evaluation protocol for stochastic open‑ended full‑body spoken dialogue, providing matched audio across systems and unified metrics, human evaluation, and latency measurement.

The disconnect between speech and motion models is the primary obstacle to synchronized, expressive spoken dialogue.

Related Work

We situate Motion‑Omni among prior work on co‑speech motion, integrated synthesis, spoken dialogue, and spoken motion.

Co‑speech motion generation maps speech audio to body motion; recent works extend this to full‑body motion using diffusion, discrete, state‑space, or flow‑matching representations.

Audio‑driven facial animation that uses periodic positional encodings to generate face‑only motion.

Extends the Motion Generator via a frozen detokeniser and motion teacher, inheriting facial animation capabilities.

Full‑body co‑speech model that employs diffusion to generate 3‑D motion synchronized with speech.

State‑space model that jointly learns speech‑conditioned motion and denoising.

Employs a flow‑matching representation for co‑speech motion generation.

Uses a Mamba‑style architecture to generate gestures conditioned on speech.

Latent‑space model that learns discrete motion tokens for co‑speech generation.

Four part‑specific VQ‑VAE codebooks (face, hand, upper, lower) for motion tokenisation.

Integrated speech‑and‑gesture synthesis jointly optimises objectives across modalities, enabling shared representations for prescribed text scripts.

Probabilistic diffusion model with parallel speech and gesture heads for joint synthesis.

Conditional‑flow‑matching decoder that models the joint distribution of speech and gesture.

Adds synthetic pre‑training, multi‑speaker support, and prosody control to joint synthesis.

Reuses intermediate TTS timing and prosodic features for efficient full‑body gesture decoding.

Autoregressively interleaves discrete speech and gesture tokens for joint generation.

Spoken Dialogue Models (SDM) equip large language models with speech input and output, but they do not generate body motion.

Introduces discrete audio tokens as a language‑model vocabulary for spoken dialogue.

Adds a speech‑unit decoder and builds on the InstructS2S‑200K dataset.

Provides a discrete 12.5 Hz speech tokenizer for spoken dialogue models.

Flow‑matching decoder for speech generation adopted by Motion‑Omni.

Introduces a Thinker‑Talker architecture for spoken dialogue.

Achieves full‑duplex spoken dialogue without body motion output.

Modality‑aware adaptive post‑training for semantic quality and speech expressiveness.

Adds visual cues for target‑speaker tracking and turn‑taking in noisy multi‑speaker settings.

Spoken motion models aim to generate both speech and body motion in an open‑ended dialogue context.

Jointly predicts speech and body/hand motion tokens with an AnyGPT/LLaMA2‑based autoregressive backbone; facial animation is added post‑hoc.

Generates response text, acoustic tokens, and SMPL‑X pose tokens with a shared autoregressive backbone, lacking native facial expression output.

Combines a frozen GLM‑4‑Voice speech expert with trainable face and body experts; speech model is frozen.

Causally generates full‑body gestures and facial expressions from Moshi’s streaming states, without reporting motion‑loss adaptation.

Unifies spoken dialogue models and co‑speech motion models into a single autoregressive framework, enabling joint generation of speech and full‑body motion.

The Motion-Omni Framework

Describes the unified autoregressive system that jointly generates speech and synchronized full‑body motion.

The Motion‑Omni system ties speech generation and full‑body co‑speech motion into a single autoregressive loop, eliminating the latency and inconsistency of separate pipelines.

The framework treats speech and motion as two parallel token streams produced by the same autoregressive model, guaranteeing perfect temporal alignment.

Project the 4 speech‑unit hidden states $H_s$ with $W_h\in\mathbb{R}^{896\times512}$ to obtain $Z$ of shape (4, 512).

Lookup embeddings $E[u]$ (4 × 512) and project with $W_e$ to get $E[u]W_e$.

Linearly interpolate $E[u]W_e$ from 12.5 Hz to 30 Hz, yielding $Q$ of shape (8, 512).

Each decoder receives the same $Z$ and $Q$, then runs its TQGF layers and self‑attention to output 8 logits per part.

Top‑$1$ logits are mapped to the 256‑entry LOM codebook, producing 8 motion codes per body part.

The interpolation step aligns the slower speech‑unit rate with the faster motion rate, ensuring that every motion frame has a well‑defined speech context.

How does Motion‑Omni differ from the traditional pipeline that first generates audio and then extracts motion?

Traditional pipelines treat speech synthesis and motion generation as two independent stages, often using the final waveform as a proxy for timing. Motion‑Omni skips the waveform entirely for conditioning: the Speech Generator’s hidden states are fed directly to the Motion Generator, so the motion decoder sees the same semantic and temporal cues that the speech decoder uses, guaranteeing exact alignment without an extra audio‑to‑motion conversion.

Four independent decoders, one per body part, read the same speech context but produce part‑specific motion codes, allowing each limb to move according to the same utterance.

Compute $Z = H_s W_h$ → a $1\times4$ vector (e.g., $[0.2, -0.1, 0.5, 0.3]$).

Lookup $E[0]W_e$ → $[0.4, 0.0, -0.2, 0.1]$, interpolate (no change for a single unit) to obtain $Q$.

Face decoder applies its TQGF gate, zero‑ing out the second dimension of $Z$ (irrelevant for facial expression).

Hand decoder’s gate zero‑es the third dimension (irrelevant for hand pose).

Each decoder’s self‑attention sees only the current frame (causal mask), then the MLP head maps to a 256‑code logit vector; the argmax yields a face code 42 and a hand code 7.

The part‑specific gates filter out dimensions that do not affect a given limb, allowing a single shared context to drive diverse motions without interference.

Why don’t the four decoders cross‑condition on each other’s sampled motion outputs?

Cross‑conditioning would introduce a circular dependency at inference time: each decoder would need the others’ predictions before it could produce its own, breaking the autoregressive guarantee of a single forward pass. By sharing only the speech context, the system remains fully parallelizable across body parts while still keeping the motions temporally aligned.

Stage 1: Freeze the LLM and train the speech projector with ASR supervision, learning to map Whisper features into the LLM embedding space.

Stage 2: Freeze the LLM, train the Speech Generator on TTS pairs so it learns to emit discrete speech units from the projected embeddings.

Stage 3: Attach the Motion Generator; run four sub‑stages that gradually increase the quality percentile of the dual‑metric score (12.5 % → 25 % → 50 % → 100 %). The Speech Generator remains frozen at first, then is jointly fine‑tuned to improve audio‑motion alignment.

Stage 4: Unfreeze the LLM, speech projector, Speech Generator, and Motion Generator; train on a mixture of ASR, TTS, Speech‑to‑Speech‑with‑Motion (S2SM), and text‑only (T2T) tasks with a constant‑with‑warmup schedule.

**Figure 1.** Motion-Omni framework with four components and their conditioning topology.

**Table 1.** Training data by stage.

Experimental Results

Key results show Motion‑Omni‑Q7 excels in speech intelligibility, motion quality, and latency.

Motion‑Omni‑Q7 achieves the lowest WER among omni‑modal LLMs at 2.62 %.

Table 2 shows Motion‑Omni‑Q7 at 2.62 % WER, lower than Qwen2.5‑Omni (2.72 %) and Ex‑Omni (2.67 %).

Motion‑Omni‑Q7 attains the lowest teacher‑referenced FGD of 3.03, outperforming all other omni‑modal LLMs.

Table 3 reports Motion‑Omni‑Q7 FGD = 3.03, the smallest value among the listed systems.

A 500‑prompt dialogue‑text benchmark derived from the Switchboard Dialog Act Corpus, covering all 66 topics with realistic conversational turns.

A nine‑subset benchmark that evaluates spoken‑dialogue models on open‑ended dialogue, factual QA, reasoning, instruction‑following, and safety.

A distributional metric that quantifies how close the generated motion feature distribution is to a reference motion distribution.

The figure displays a scatter plot comparing various models based on their Real-time factor (RTF) on the x-axis and Beat correlation (BC) on the y-axis. The x-axis is on a logarithmic scale, with values ranging from 1 to 10. The y-axis ranges from 7.2 to 7.8. The models plotted are "Motion-Omni-Q7" (represented by a red star), "MO-audio + LOM", "Qwen2.5-Omni + LOM", "MO-audio + EMAGE", and "Qwen2.5-Omni + EMAGE" (represented by grey circles).

**Figure.** (a) “Here are five positive character traits: honesty, empathy, kindness, confidence, and curiosity.” Frames at $t = 1.1, 1.9, 2.6, 3.9, 4.8, 6.0, 6.9$ s.

Model Architecture Details

Detailed specifications of the Speech Generator, body VQ‑VAEs, Motion Generator, training regimes, and loss formulation.

This appendix enumerates the concrete architectural components and training settings that underpin Motion‑Omni.

The Speech Generator is a Qwen2‑style transformer that receives a conditioning stream derived from the LLM hidden states and produces discrete speech‑unit tokens.

The TQGF block lets each speech token decide, element‑wise, how much of the LLM‑derived conditioning signal to admit before the token is updated.

The table describes the composition and dimensionality of four body parts: Face, Hand, Upper, and Lower. The columns are "Part", "Dim", and "Composition".

**Table 7.** Motion Generator architectural hyperparameters.

Stage‑wise training proceeds through four stages (Table 8): Stage 1 trains only the Speech Projector, Stage 2 adds the Speech Generator, Stage 3 jointly trains Speech and Motion Generators, and Stage 4 fine‑tunes the entire LLM‑plus‑all modules with a cosine‑plus‑warmup schedule and progressively lower learning rates.

All training runs on four GPUs with DeepSpeed ZeRO‑2, BF16 mixed precision, and gradient checkpointing; the total parameter count is ≈ 8.3 B (≈ 7.6 B LLM, 0.5 B Speech Generator, 150 M Motion Generator, 5 M Speech Projector), consuming roughly 960 GPU‑hours across the four stages.

Data Pipeline and VQ-VAE

Ablation analysis of reconstruction fidelity and speech‑motion alignment metrics.

This section details the quantitative filters used to select high‑quality motion samples and the human study that validates them.

L₍vq₁₎ quantifies how faithfully a frozen VQ‑VAE codebook can reproduce a motion sample, serving as a label‑fidelity filter for curriculum learning.

Why weight the per‑part errors by their feature dimensions instead of simply averaging the four \ell_b values?

Because the parts have different numbers of features; weighting by $D_b$ ensures that each individual feature contributes the same amount to the final score, preventing larger parts from overwhelming the aggregate.

BC measures how tightly the timing of motion beats aligns with speech onsets, acting as a rhythmic‑alignment filter.

Why does the curriculum use only the forward (audio → motion) BC direction instead of the bidirectional version?

Because the model generates motion conditioned on speech; the forward direction directly reflects how well the generated motion follows the audio, which is the relevant training signal. The reverse direction is useful for diagnostics but not for guiding learning.

Extends BC by also measuring how well motion beats predict audio onsets, providing a symmetric view of rhythmic alignment.

Maps each metric to a [0, 1] range by clipping the 5th and 95th percentiles, then inverts the reconstruction error so that higher values always denote better quality.

Produces the final video stimuli by consistently applying a fixed set of assets and rendering settings to all motion outputs.

A paired A/B/tie study quantifies perceived motion quality across four system arms, isolating the contribution of the motion generator.

Together, the reconstruction‑error filter, beat‑correlation metric, robust normalisation, and human study form a comprehensive pipeline for selecting and validating high‑quality co‑speech motion.

Evaluation Benchmarks

Evaluation tables for spoken dialogue quality and speech naturalness.

We evaluate spoken‑dialogue performance on the VoiceBench benchmark and report an automatic speech‑naturalness proxy on SwDA‑500.

**Table 9.** Spoken dialogue quality on VoiceBench (Chen et al., 2026c). Higher is better throughout. GPT-4o-Audio is served through an API and its weights are not released; every other system releases its weights. Dashes denote scores not reported.

Because word‑error‑rate only measures intelligibility, we also report UTMOSv2 as a lightweight proxy for speech naturalness on generated dialogue.

The table compares various TTS models based on their UTMOSv2 en mean scores. The models listed are Qwen3-TTS, CosyVoice 3, GLM-TTS, VoxCPM1.5, F5-TTS, CosyVoice, and Motion-Omni-Q7 (ours).

Motion‑Omni‑Q7 achieves a VoiceBench overall score of 51.00 and a UTMOSv2 of 3.77, outperforming all listed open‑weight TTS baselines on naturalness.

Ablation Studies and Robustness

Ablation studies reveal how motion‑conditioning fusion and curriculum stages affect performance.

The central premise—unifying spoken dialogue and co‑speech motion in a single autoregressive model—has been established earlier; this section quantifies how design choices influence that joint generation.

Two observations emerge: (1) using Speech Generator states as context improves BC over direct LLM context (0.912 vs 0.827), and (2) the gated TQGF operator outperforms plain cross‑attention (0.912 vs 0.905) under the fixed Stage 3a setup.

FGD consistently declines across curriculum stages, reaching its minimum after joint fine‑tuning (Stage 4), confirming progressive improvement of the teacher‑reference motion quality.

Attempting to replace human ratings with a video‑LLM judge (Gemini 3.1 Pro) proved ineffective: even after five prompt iterations, Pearson r never exceeded 0.32 and large‑disagreement counts remained high, leading to the removal of this component from the final evaluation pipeline.

Qualitative visualisations show seven frames sampled for maximal pose diversity and seven evenly spaced frames for cross‑system comparison, all rendered with identical cropping to isolate motion differences.

Conclusion

Summarizes results, acknowledges limits, and details reproducibility and ethical considerations.

We introduced Motion‑Omni, an end‑to‑end framework that lets a conversational LLM jointly generate intelligible speech and synchronised full‑body co‑speech motion, supported by a model‑agnostic pseudo‑labeling pipeline supplying 1,402 hours of teacher‑generated supervision.

Across benchmarks, the reference instance Motion‑Omni‑Q7 achieves the lowest word error rate among omni‑modal systems, the highest beat correlation and diversity on SwDA‑500 without a motion teacher at inference, and superior lip‑sync compared to EMAGE and MambaTalk cascades.

Conditioning motion on the Speech Generator’s hidden states rather than on a decoded waveform keeps reference‑free motion metrics within about 1 % of the same‑audio teacher cascade while eliminating a separate audio‑to‑motion stage, which explains the 5.4× speedup in response time.

All components are swappable; stronger backbones or larger co‑speech motion teachers can be incorporated by re‑running the recipe, extending the current Motion‑Omni‑Q7 instance.

Limitations include dependence on the LOM VQ‑VAE codebook and teacher pseudo‑labels, training only the Motion‑Omni‑Q7 instance, lack of speaker identity or emotion conditioning, English‑only data, and an offline response generation design that precludes low‑latency streaming interactions.

Automatic motion metrics such as FGD and BC are imperfect proxies for perceived naturalness, and the four‑annotator human preference study is exploratory with limited scale.

Reproducibility is ensured by specifying the architecture and conditioning topology in Section 3, detailing four training stages in Section 3.3 with hyperparameters in Appendix A.4, and describing the data pipeline and quality metrics in Section 3.4 and Appendices B.1–B.2.

The evaluation setup, including the SwDA‑500 derivation, cascade baselines, and shared rendering pipeline, is described in Section 4.1 and Appendix C, while the human evaluation protocol and rubric appear in Appendices D and D.1.

Ethics: the human preference study used anonymised system labels, collected no personal data, and evaluated a synthetic speaking avatar with a fixed voice and generic body mesh, avoiding imitation of any specific individual.

AI Use: generative AI tools assisted in polishing text and providing coding help, but were never used to generate or alter experimental results, metrics, or human evaluation data; the authors retain full responsibility for the content.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers