DreamX-Creator 1.0: Democratizing Native Audio-Video Generation at 2K Resolution

Jiashu Zhu, Yanhao Zheng, Ruitian Tian, Rujing Dang, Shen Zhang, Bingze Song, Jiachen Lei, Ruimin Lin, Jiahong Wu, Xiangxiang Chu

A compact 7B native audio-video generator with an autoregressive 1-step 2K refiner.

How can we generate high-fidelity, 2K-resolution video with natively synchronized audio in a single, computationally efficient pipeline?

Most video generators treat audio as an afterthought, leading to poor synchronization and limited reciprocal interaction between visual motion and sound. DreamX-Creator 1.0 uses a 7B joint generator that processes audio and video streams through gated cross-modal attention, allowing the two modalities to influence each other during the denoising process. The system includes an autoregressive 1-step refiner that scales output to 2K resolution without requiring multi-step diffusion sampling, providing a compact, open-weight alternative to larger proprietary models.

Paper Primer

The core mechanism is Gated Cross-Modal Attention: the model maps audio and video to a shared temporal coordinate system and uses token-wise sigmoid gates to modulate cross-modal information flow. This acts like a traffic controller at an intersection, where the gate decides exactly how much influence the audio stream should exert on a specific video token (and vice versa) before the residual update is applied.

To optimize for human perception, the authors use Audio-Video Reinforcement Learning with Modality-Aware Multimodal Feedback. This routes separate reward signals to the video and audio streams while sharing a cross-modal advantage term, ensuring that improvements in one modality do not degrade the synchronization or quality of the other.

DreamX-Creator 1.0 achieves competitive audio-visual synchronization and video quality at a significantly smaller parameter scale than contemporary open-weight baselines.

In human preference studies, the 7B model consistently outperforms smaller research baselines (NAVA, UniAVGen, Ovi) in video quality and audio-visual alignment, while maintaining a lower DeSync score (0.1351) than larger systems. The 7B backbone is the smallest open-weight native joint audio-video model currently available that supports local 2K resolution output.

Why is a "native joint" approach necessary compared to simply generating video and audio separately?

Separate pipelines treat one modality as a fixed condition, which prevents reciprocal interaction. Native generation allows the model to capture dependencies like speech-mouth motion or the impact of physical objects on sound, which are jointly determined in real-world events.

How does the 2K Refiner maintain temporal consistency without multi-step sampling?

The refiner uses an autoregressive 1-step student model distilled from a bidirectional teacher. By refining video in temporal chunks and using previously refined chunks as context, it avoids the computational cost of multi-step diffusion while preserving motion and timing.

DreamX-Creator 1.0 demonstrates that high-fidelity, synchronized audio-video generation is achievable at the 7B scale, providing a reproducible foundation for research that was previously locked behind proprietary, large-scale APIs.

Introduction and Motivation

We expose the limitation of decoupled audio‑video generation and introduce DreamX‑Creator 1.0, a compact native joint generator.

Most existing video generators treat audio as an afterthought, synthesizing it in a separate stage that cannot capture the two‑way relationship between visual motion and sound.

Joint generation is needed because many phenomena—speech‑mouth sync, impact sounds, ambient music—are inherently co‑determined by visual and acoustic cues.

**Figure 1** DreamX-Creator 1.0. A native joint generator couples visual motion, speech, sound effects, and ambience, followed by multimodal post-training and a 2K-oriented video refinement stage.

The key shift is moving from a decoupled audio‑video pipeline to a native joint generation architecture.

Data System and Multimodal Annotation

Transform raw videos into curated, cross‑modal clips ready for joint generation.

Raw videos from diverse sources are noisy, often misaligned, and heavily skewed toward certain audio‑visual patterns, which hampers learning a native joint generator.

The system turns a chaotic collection of videos into a clean, structured set of clips that simultaneously supply high‑quality visual frames, well‑aligned audio, and rich multimodal annotations.

After quality filtering, Clip C is dropped because its visual resolution is below the threshold, leaving A, B, D.

Alignment assessment removes Clip D (fails SyncNet lip‑sync) and keeps A and B, which pass both Synchformer and SyncNet.

Joint annotation produces a multimodal caption for Clip A (“person speaking while walking”) and for Clip B (“car collides, glass shatters”).

Capability inference assigns Clip A to the “Speech and Dialogue” pool and Clip B to the “Human Actions, Object Interactions” pool.

The pipeline discards low‑quality or misaligned clips early, so the final training set consists only of clips that provide reliable cross‑modal signals for the intended generation capability.

**Figure 3.** Overview of the data system for DreamX-Creator 1.0, including heterogeneous data collection, clip construction and filtering, structured multimodal annotation, and capability-oriented data organization.

**Figure 4.** Content distribution of the training dataset.

Joint Audio-Video Architecture

Native Joint Generation jointly denoises video and audio latents using cross‑modal attention and gated output.

Decoupled audio‑video pipelines cannot learn the reciprocal timing between visual motion and sound, leading to mismatched outputs.

Instead of generating video and audio separately, the model denoises both latent streams together and lets them talk to each other through gated cross‑attention, so the motion and sound co‑evolve.

Compute queries from video tokens: $Q = \text{RMSNorm}(W_Q \, \text{LN}(h_v))$.

Stop‑gradient the audio tokens and compute keys/values: $K = \text{RMSNorm}(W_K \, \text{sg}(h_a))$, $V = W_V \, \text{sg}(h_a)$.

Apply RoPE to align time: $eQ = \text{RoPE}_t(Q, \tau_v)$, $eK = \text{RoPE}_t(K, \tau_a)$.

Attention weights: $A = \text{softmax}(eQ eK^\top / \sqrt{d_h})$ (shape $4\times2$).

Cross‑modal output: $C = A V$ (produces a 4‑token update from the 2‑token audio).

Gate each head: compute $g_{i,h}$ from $v_i$, multiply: $\tilde{C}_{i,h}=g_{i,h}\odot C_{i,h}$.

Project and add residual: $v'_i = v_i + W_o(\text{Concat}_h \tilde{C}_{i,h})$.

The gating lets each video token decide how much audio information to absorb, preventing a noisy audio stream from overwhelming the video representation.

Forward pass of the second half of Native Joint Generation (A2V/V2A with gating).

**Figure 2** The DreamX-Creator 1.0 pipeline, comprising native joint audio-video generation, Audio-Video Reinforcement Learning post-training, and Autoregressive 1-Step 2K Refinement.

**Figure 5.** Architecture of the joint audio-video generator. The two streams are processed independently in the first half and interact through gated A2V and V2A cross-attention in the latter half.

Modality-Aware Reinforcement Learning

Post‑training RL refines joint generation using modality‑aware rewards and targeted gradients.

The flow‑matching pretraining captures the joint distribution but does not directly optimize the perceptual criteria that users care about—sharp video, clear audio, and tight audio‑visual timing.

We treat the joint generator as an RL agent that receives separate but coordinated rewards for video quality, audio fidelity, and their synchronization, letting the model improve each stream while keeping them in lockstep. Like a coach giving feedback to a dancer and a singer together, focusing on both their individual performance and their duet timing.

How does this differ from a standard single‑reward RL setup for video generation?

Standard RL would collapse all quality dimensions into one scalar reward, letting a strong video score drown out a weak audio signal. Here we keep the three advantages separate, add the shared cross‑modal term to each stream, and weight tokens by synchronization relevance, so improvements in one modality cannot hide degradations in the other.

Normalize each score: $(s_i - \mu)/\sigma$ yields $[-1.0,\,1.0,\,0.0]$.

Add $\epsilon$: $[-0.99,\,1.01,\,0.01]$.

Apply sigmoid: $\sigma([-0.99,\,1.01,\,0.01]) \approx [0.27,\,0.73,\,0.50]$.

Scale by $\alpha_e$ and add 1: $w^{(e)} = 1 + 0.5 \times [0.27,\,0.73,\,0.50] \approx [1.14,\,1.37,\,1.25]$.

The three tokens receive weights $1.14$, $1.37$, and $1.25$, so the middle token (the most relevant mouth region) influences the loss most strongly.

Weighting concentrates gradient updates on the few tokens that drive audio‑visual timing, while still allowing the other tokens to receive a baseline signal.

Initialize the policy $\pi_{\theta}$ from the pretrained DreamX‑Creator 1.0 checkpoint.

For each first‑frame / prompt pair $(I,c)$, sample $G$ joint candidates $\{(v^{(j)},a^{(j)})\}_{j=1}^{G}$ from the current policy.

Evaluate each candidate on video quality, audio quality, prompt consistency, and synchronization to obtain raw rewards.

Normalize rewards within the $G$ candidates to compute advantages $A_v$, $A_a$, and $A_{av}$.

Route $e_{A_v}=A_v + A_{av}$ to the video stream and $e_{A_a}=A_a + A_{av}$ to the audio stream.

Compute token weights $w_i^{(e)}$ (Eq 9) and apply depth‑dependent gradient scaling to the audio‑to‑video pathway.

Perform a policy gradient update using the composed loss $L_{\text{RL}}$ (Eq 11).

Periodically replace the rollout policy $\pi_{\theta_{\text{old}}}$ with the updated $\pi_{\theta}$ to close the online loop.

**Figure 6** Modality-aware reinforcement learning for joint audio-video generation. Video- and audio-specific rewards are routed to their corresponding streams, while the shared audio-visual synchronization reward jointly optimizes both streams and their cross-modal interactions.

Autoregressive 2K Refinement

2K Refiner upgrades low‑resolution joint outputs to 2K video via a single‑step autoregressive refinement.

Generating 2K video directly with a joint audio‑visual model overwhelms memory and compute, yet low‑resolution outputs miss the crisp textures and sharp edges needed for realistic playback.

Think of the low‑resolution video as a rough sketch and 2K Refiner as a skilled artist who adds fine brush strokes in a single pass, preserving the sketch’s composition while instantly delivering high‑detail artwork.

Chunk A: the model receives the LR latents for frames 0‑1 and a zero‑filled HR buffer for previous chunks.

It performs a single denoising step, producing a 4×4 HR patch for frames 0‑1.

Chunk B: the HR patches from Chunk A are fed back as temporal context while the LR latents for frames 2‑3 are supplied.

The model again denoises once, yielding HR patches for frames 2‑3 that align seamlessly with the earlier patches.

This one‑step per‑chunk workflow yields full‑resolution video without the quadratic cost of diffusion across the entire frame sequence.

Train a bidirectional multi‑step diffusion teacher on HR latents corrupted with flow‑matching noise, conditioned on LR video.

Introduce a degradation curriculum that gradually adds blur, noise, compression, and temporally correlated jitter to the HR inputs.

Convert the teacher to an autoregressive multi‑step refiner via teacher‑forcing: each chunk is denoised using ground‑truth past HR chunks.

Distill the autoregressive refiner into a 1‑step student with Distribution Matching Distillation (DMD), matching the student’s output distribution to the teacher’s refinement distribution.

Train the student under self‑rollout: generate full videos with its own predictions, apply DMD loss, and add pixel‑space supervision (DISTS + ℓ₂) against HR ground truth.

**Figure 7.** Training pipeline of 2K Refiner. A bidirectional multi-step refinement teacher is adapted into an autoregressive multi-step refiner and then distilled into an autoregressive 1-step 2K student. The final model performs one denoising evaluation per temporal chunk refinement.

How does 2K Refiner differ from a naïve multi‑step diffusion run at 2K resolution?

Naïve diffusion would apply dozens of denoising passes to the entire high‑resolution video, incurring quadratic compute in both spatial and temporal dimensions. 2K Refiner collapses those passes into a single autoregressive step per chunk, leveraging a teacher‑trained prior and DMD distillation to retain quality while scaling linearly with video length.

Empirical Results and Benchmarking

Quantitative and human‑rated results demonstrate DreamX‑Creator 1.0’s advantages.

The paper’s premise is that modeling audio and video jointly—rather than in separate stages—lets the generator learn their reciprocal relationship.

In side‑by‑side user studies DreamX‑Creator 1.0 wins 64.2 % of video‑quality comparisons against the strongest baseline (Ovi).

Figure 8 reports a 64.2 % win rate for DreamX‑Creator 1.0 versus Ovi on the video‑quality dimension.

Synchformer measures temporal misalignment between audio and video streams, producing a desynchronization score.

How does Synchformer differ from a simple cross‑correlation of audio and video embeddings?

Cross‑correlation treats each modality as a flat vector and ignores local temporal structure. Synchformer explicitly evaluates short‑window similarity across time, yielding a fine‑grained offset estimate rather than a single global similarity score.

SyncNet is a lip‑audio synchronizer that checks whether spoken words match visible mouth movements.

Why can’t SyncNet replace Synchformer for general audio‑video alignment?

SyncNet is trained on visible‑speech data and expects clear lip movements; it fails on non‑speech events (e.g., explosions, music) where no mouth cues exist, whereas Synchformer works on any audio‑visual pair.

LTX‑2.3 is a large open‑weight audio‑video generator (22 B parameters) that serves as a strong external baseline.

What aspect of LTX‑2.3’s architecture makes it harder to achieve low $DeSync$?

Because audio and video are decoded independently and only merged late, temporal cues cannot influence each other during generation, leading to larger synchronization errors.

MiniMax‑H3 is an open‑source 33 B model that excels at audio‑video synthesis, especially in lip‑sync quality.

Why does MiniMax‑H3 still trail DreamX‑Creator 1.0 on $VQ$ despite its sophisticated hierarchy?

Its hierarchy focuses on temporal alignment; visual fidelity depends on separate image‑generation pathways that are not as heavily regularized, leading to slightly lower perceptual video quality.

**Figure 8.** Human preference study comparing DreamX-Creator 1.0 with Ovi, UniAVGen, NAVA, and DaVinci (top), and with Wan2.7, Kling v3, and MiniMax-H3 (bottom). Each horizontal stacked bar reports Win/Tie/Lose percentages from the perspective of DreamX-Creator 1.0 under blind side-by-side comparison.

Related Work

We situate DreamX‑Creator within the landscape of audiovisual generation and related refinement techniques.

Early audiovisual synthesis treats one modality as a fixed timeline, limiting reciprocal generation. Joint diffusion models and diffusion transformers instead sample audio and video together, using coupled denoisers and shared representations. DreamX‑Creator retains modality‑specific streams but enables bidirectional, context‑dependent exchange via gated attention.

Preference‑based post‑training aligns generators to human judgments, with methods such as DDPO, Diffusion‑DPO, and D3PO optimizing image‑level rewards. Extending this to video introduces temporal coherence and multimodal criteria, leading to approaches like VADER, VideoDPO, DanceGRPO, Flow‑GRPO, and DiffusionNFT. DreamX‑Creator addresses joint audio‑video alignment by decomposing feedback into audio, video, and cross‑modal components within a first‑frame‑conditioned RL framework.

Video restoration methods like BasicVSR++, RealBasicVSR, and VRT propagate information across frames to improve detail and consistency. Diffusion‑based refiners (StableVSR, Upscale‑A‑Video, VEnhancer) and large‑scale systems (VideoGigaGAN, SeedVR) further enhance realism, while recent pipelines (LPM, SparkVSR, UltraFlash) target real‑time high‑resolution generation. Few‑step distillation techniques such as DUO‑VSR, FlashVSR, and LiteVSR reduce sampling cost, and our refiner leverages full‑clip attention to distill under an offline refinement contract.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers