Kandinsky 6.0 Video: Foundation Models for Synchronized Video and Audio Generation
Team Kandinsky, Julia Agafonova, Bulat Akhmatov, Mikhail Aksyutin, Grigorii Alekseenko, Anastasia Aliaskina, Olga Androsova, Vladimir Arkhipkin, Anna Averchenkova, Alexander Belykh, Serafima Bocharova, Sofiya Bogakovskaya, Anton Bukashkin, Mark Bulygin, Kirill Buzygin, Irina Cheremnykh, Kirill Chernyshev, Mikhail Chernyshov, Vladimir Chernyy, David Chikovani, Georgy Daniltsev, Denis Dimitrov, Anna Dmitrienko, Vladimir Dokholyan, Sergey Emelyanov, Dmitry Ermilov, Georgii Fedorov, Polina Gavrilova, Nikolai Gerasimenko, Aleksandr Gordeev, Andrey Inozemtsev, Andrei Ivaniuta, Alexander Ivanov, Mikhail Karaev, Anastasiia Kargapoltseva, Ivan Kirillov, Nikita Kiselev, Valeria Kobenko, Yury Kolabushin, Denis Koposov, Anatoly Korobov, Vladimir Korviakov, Kirill Kozlov, Denis Krzhivokolskiy, Konstantin Kuklev, Alexander Kunitsyn, Sergey Kuzin, Vladislav Lakhtionov, Alexey Letunovskiy, Maxim Litvinov, Alexander Lyulkov, Georgy Makarov, Kirill Malakhov, Egor Malykh, Mikhail Mamaev, Dmitrii Mikhailov, Polina Mikhailova, Ivan Mikheev, Elizaveta Muromtseva, Nikolai Nazarkin
Kandinsky 6.0 Video is a dual-stream diffusion model family for synchronized text-to-audio-video generation.
How can we build a foundation model capable of generating high-fidelity, synchronized video and audio from text or image prompts?
Generating video with synchronized audio is difficult because the system must maintain semantic, temporal, and emotional consistency across two distinct modalities simultaneously. The authors use a dual-stream CrossDiT architecture that connects separate video and audio streams via bidirectional cross-attention, trained through a multi-stage pipeline of continuous pretraining, supervised fine-tuning, and reinforcement learning. Kandinsky 6.0 Video Pro outperforms its predecessor and remains competitive with proprietary models, achieving a 47% relative reduction in word error rate for speech after reinforcement learning.
Paper Primer
The core mechanism is a dual-stream CrossDiT architecture: it processes video and audio in parallel through asymmetric transformer backbones, using bidirectional cross-attention layers to align the modalities temporally and semantically. This is like a duet where two musicians play separate instruments but listen to each other's cues to stay in rhythm.
Reinforcement learning post-training significantly improves speech intelligibility.
Word error rate (WER) on a held-out speech benchmark decreased from 0.235 to 0.124 for the Pro model.
The training pipeline employs a continuous strategy: audio streams are pretrained from scratch on large-scale audio corpora, then connected to pretrained video streams for joint training on paired audio-video data. This is followed by a two-stage supervised fine-tuning process and reinforcement learning adapted from OmniNFT to optimize for synchronization and lip-sync.
Why use a dual-stream architecture instead of a single unified transformer?
The dual-stream approach allows the model to leverage pretrained video capabilities while training the audio stream from scratch, facilitating better temporal and semantic alignment through cross-attention without losing unimodal fidelity.
What is the role of the super-resolution model in this system?
The base generator produces clips in SD resolution; the built-in super-resolution model, consisting of a Latent Upscaler and an SR-DiT, restores detail and upscales the output to Full-HD (1920×1080) without requiring additional text prompts.
Kandinsky 6.0 Video provides an open-source foundation for high-quality, synchronized audio-visual generation, demonstrating that reinforcement learning can effectively bridge the gap between raw synthesis and precise lip-sync performance.
Introduction
Introducing the core challenge of synchronized audio‑visual generation.
Generating video together with perfectly aligned audio remains a bottleneck: existing open‑source systems either drop synchronization, limit clip length, or sacrifice resolution, while the most capable commercial tools keep their code closed. This mismatch leaves researchers without a reproducible, high‑fidelity baseline for audio‑visual synthesis.
We need a unified foundation model that can generate both video and 44 kHz audio in lockstep, because downstream applications (e.g., virtual avatars, immersive storytelling) require the two modalities to share timing, semantics, and emotional tone.
**Figure 1.** A collage of various AI-generated video and image samples showcasing the capabilities of Kandinsky 6.0 Video, including scenes of a drummer, a giant stone figure, an octopus on a building, a person in a car, a person running in a field, a violinist, an astronaut skateboarding on a planet, and a car underwater.
The need for synchronized audio‑visual foundation models is paramount for advancing reproducible, high‑quality multimodal generation.
Data Processing Pipeline
How the authors built a multi‑modal data pipeline to enable synchronized audio‑visual generation.
The audio‑visual model required far more data than the previous image‑only version, and naïvely scaling the old pipeline would have caused severe synchronization mismatches and storage bottlenecks.
The pipeline first learns each modality in isolation, then gradually fuses them while tightening synchronization constraints, ending with a fine‑tuned, distilled model that can generate coherent video, audio, and captions.
How does this pipeline differ from simply concatenating audio and video streams before training?
Simple concatenation would expose the model to mismatched timing from the start, forcing it to learn synchronization from noisy data. By pretraining modalities separately and only fusing them after they have learned stable representations, the pipeline gives the model a clean signal at each step, dramatically reducing the risk of desynchronised outputs.
Stage 1 trains the image encoder on 100 k images, producing a stable visual embedding space.
Stage 2 trains the audio encoder on 100 k audio clips, yielding a compatible acoustic embedding space.
Stage 3 merges the two encoders, feeding paired (I, A) samples; the model now learns to map I‑embeddings to A‑embeddings and vice‑versa.
After joint training, a small RL loop (10 k rollouts) fine‑tunes the combined model for lip‑sync accuracy.
Finally, knowledge distillation compresses the 12‑layer transformer into a 6‑layer student, preserving the learned sync.
Staging the training isolates the hardest problem—audio‑visual alignment—until the unimodal representations are already reliable, which is why the pipeline scales to billions of samples without collapsing.
**Figure 5.** Training pipeline of Kandinsky 6.0 Video. The video stream (initialized from the pre-trained Kandinsky 5.0 Video model) and the audio stream are first pre-trained separately, then fused and trained jointly in T2AV mode and in mixed T2AV/I2AV mode. Joint pre-training is followed by supervised fine-tuning with model soup, RL-based post-training, and distillation.
Data Processing Infrastructure was upgraded with table partitioning, load‑balancing for thousands of processors, and extended vector indexes to cluster audio‑visual embeddings.
The T2I dataset supplies 50 million high‑quality images, re‑annotated with captions generated by Qwen3‑235B‑A22B and rewritten into four detail levels.
The T2V dataset contributes 20 million filtered video clips, each with stable frame rates and no internal scene cuts, captioned by the same pipeline as T2I.
The T2A dataset provides 40 million audio tracks (5–20 s) extracted from the video collection, each described by Qwen3‑30B‑A3B.
The T2AV dataset contains 7 million audio‑video segments that pass strict audio quality, lip‑sync (LSA ≥ 3), and desync (≤ 0.1) filters, ensuring reliable synchronization for speech scenes.
The I2AV dataset offers 116 k image‑conditioned samples, each paired with a multi‑modal caption generated via a two‑stage pipeline (audio description → unified caption).
Table 1 shows the domain breakdown of the SFT datasets; the “people action” domain dominates both visual (1,410) and sync (1,460) stages, while “tech” is the sparsest.
Supervised Fine‑Tuning (SFT) uses the same 11‑domain taxonomy to first improve visual quality (Stage 1) and then enhance audio‑video sync and lip‑sync (Stage 2).
The RL dataset supplies 2,984 curated clips with bilingual T2AV captions, enabling online reinforcement learning with multi‑objective rewards.
Super‑Resolution (SR‑DiT) data aggregates 150 k high‑resolution clips from a 4K video collection, a filtered 5‑0 T2V subset, and 15 k synthetic camera‑motion clips, all passed through quality filters before paired HQ/LQ training.
The Russian Cultural Code (RCC) dataset expands to 1.2 M images and 530 k video scenes, annotated in both Russian and English to guide culturally aware generation during pre‑training and joint T2AV training.
Model Architecture
The architecture unifies video and audio streams via a dual‑stream CrossDiT with bidirectional cross‑attention.
The CrossDiT core processes video and audio in parallel, letting each modality attend to the other through blockwise cross‑attention.
Two parallel transformer streams—one for video, one for audio—exchange information via bidirectional cross‑attention, while each stream retains its own asymmetric DiT backbone.
How does the bidirectional cross‑attention differ from the usual cross‑attention used in encoder‑decoder models?
Standard cross‑attention passes queries from one modality to keys/values of another in a single direction (e.g., text‑to‑image). Here each block performs two cross‑attention steps: video‑to‑audio (V2A) and audio‑to‑video (A2V), allowing both streams to update each other within the same layer.
Step 1: Compute RoPE = (Nframe · fps) / 24 = (12 · 30) / 24 = 15.
Step 2: The RoPE factor of 15 is applied to the query/key rotations for that frame.
Step 3: Total parameter budget for the Lite model is 2 + 0.6 + 0.4 = 3 B parameters.
Step 4: The cross‑attention accounts for 0.4 / 3 ≈ 13 % of the total parameters, illustrating its modest size relative to the streams.
The RoPE formula normalizes any frame rate to the 24 fps baseline, so a 30 fps clip yields a factor of 15, ensuring consistent positional encoding across videos of different speeds.
**Figure 1.** Simplified scheme of a single Kandinsky 6.0 Video transformer block. The video (top) and audio (bottom) streams each apply self-attention, cross-attention to the text embeddings (T2V and T2A), cross-modal attention, and a feed-forward layer. In the V2A block, video queries attend to audio keys and values; in the A2V block, audio queries attend to video keys and values.
**Figure 2.** Detailed scheme of the Kandinsky 6.0 Video transformer block. Each stream has its own text encoder block and timestep embedding; the timestep embedding modulates every sub-layer through scale, shift, and gate parameters. In the cross-modal attention layers, video tokens use 3D RoPE and audio tokens use 1D RoPE.
Forward pass through one CrossDiT block (bidirectional cross‑attention).
Training Hyperparameters
Training hyperparameters and data sampling settings for Kandinsky 6.0 Video Lite and Pro.
This section enumerates the hyperparameters used during the continuous pre‑training stages of Kandinsky 6.0 Video Lite and Pro. The values are presented side‑by‑side to highlight the differences between the two model sizes.
Reinforcement Learning Post-training
RL post‑training refines audio‑video sync by routing advantages and surgically shaping gradients.
The paper’s unified transformer already produces synchronized audio‑video output, but fine‑grained lip‑sync still lags behind human perception.
After supervised fine‑tuning, the model is further refined by rewarding generations that align video, audio, and lip movements, while penalising mismatches.
Mean $\mu_r = (1.2+0.8)/2 = 1.0$, standard deviation $\sigma_r = \sqrt{[(1.2-1)^2+(0.8-1)^2]/2}=0.2$.
Advantages: $\text{adv}_1 = (1.2-1)/\max(0.2,0.1)=1.0$, $\text{adv}_2 = (0.8-1)/0.2 = -1.0$.
Clamp to $[-5,5]$ (no change) and rescale: $r_{\text{token},1}=((1/5)+1)/2=0.6$, $r_{\text{token},2}=((-1/5)+1)/2=0.4$.
Assume $\text{pos}_1=0.05$, $\text{neg}_1=0.12$, $\text{pos}_2=0.07$, $\text{neg}_2=0.10$, and equal attention weights $\text{attn}=1$.
Policy contribution for rollout 1: $0.6\times0.05 + 0.4\times0.12 = 0.084$; for rollout 2: $0.4\times0.07 + 0.6\times0.10 = 0.088$.
Aggregated policy loss = $(0.084+0.088)/2 = 0.086$ (before KL).
Advantage centering prevents a uniformly high‑scoring reward from dominating; the clamping‑to‑$[0,1]$ step guarantees stable gradients regardless of reward magnitude.
How does this RL step differ from standard policy‑gradient RL used in text generation?
Instead of sampling full sequences and computing a scalar return, the method computes per‑token rewards from a multi‑branch advantage stack, broadcasts them onto the latent token grid, and blends a DPO‑style positive/negative loss with a KL anchor. The advantage‑clamping and region‑wise attention weighting are also unique to the audio‑visual setting.
**Figure.** Reward Phase
**Figure 8.** Training phase of RL post-training. The rollout latent is re-noised and denoised by the current LoRA, the old LoRA, and the frozen base (SFT) model. Predictions of the current and old LoRA are combined into positive and negative predictions; their per-token MSE losses to the rollout latent are interpolated with the rewards $r_{\text{token}}$ from the reward phase and aggregated with the region-wise attention weights $\text{attn\_w}$ derived from A2V cross-attention. A KL term with respect to the base-model prediction regularizes training; the old LoRA is an exponential moving average of the current one.
Distillation Process
Distillation splits into trajectory compression and adversarial refinement to retain quality.
Distillation follows the two‑stage recipe of Kandinsky 5.0: first compress the teacher’s sampling trajectory, then recover lost perceptual detail with an adversarial objective.
RL post‑training reduces the Pro model’s word error rate from 0.235 to 0.124.
Measured on 64 prompts with three random seeds; paired Wilcoxon test p < 10⁻³.
RL post‑training reduces the Lite model’s word error rate from 0.179 to 0.127.
Same 64‑prompt benchmark; paired Wilcoxon test p < 10⁻³.
On the public Harvard sentences, Pro WER drops from 0.251 to 0.180.
Statistical test p = 0.005 on 100 prompts.
$\pi$‑Flow trains the student on‑policy, using the states the student actually visits, which preserves the teacher’s trajectory without the mode‑collapse pitfalls of distribution‑matching methods.
How does $\pi$‑Flow avoid the mode collapse observed with DMD‑style distribution‑matching methods?
$\pi$‑Flow’s loss is computed on the student’s own rollout (on‑policy), so it never forces the student to match a full teacher distribution that may contain low‑probability modes. This local alignment prevents the optimizer from collapsing onto a few dominant modes, a problem that arises when a global KL or moment matching objective is used.
Sim‑LADD refines the trajectory‑compressed student by adversarially training on states taken from the student’s backward denoising rollout, thereby narrowing the train‑inference gap.
Why does generating training states from the student’s backward rollout improve realism compared to the forward‑noising states used in standard LADD?
Backward rollout states lie on the same distribution the student will traverse at test time. Training the discriminator on these states ensures it learns to distinguish realistic versus generated samples under the exact conditions the student will face, eliminating the train‑inference gap that arises when forward‑noising states differ from the inference trajectory.
The final distillation pipeline first compresses the teacher into a 10‑NFE student via $\pi$‑Flow, then refines the student with Sim‑LADD, back‑propagating adversarial gradients through the last five sampling steps.
Super-Resolution Data
Details the adversarial refinement and joint super‑resolution training pipeline.
Temporal misalignment and loss of high‑frequency detail plague video generation; without a dedicated refinement step the model cannot recover fine motion cues.
Instead of a single global loss, the model penalizes mismatches between real and generated latent features at several scales, forcing the generator to restore fine‑grained structure while the discriminator learns to spot subtle artifacts.
Compute the distance: |0.8 − 0.3| = 0.5.
Subtract the distance from the margin: 1 − 0.5 = 0.5.
Apply ReLU: max(0, 0.5) = 0.5 → hinge loss = 0.5.
If the distance were ≥ 1, the loss would be 0, indicating the discriminator is already confident.
The loss only penalizes cases where real and fake features are too close, driving the generator to amplify differences that matter for high‑frequency fidelity.
Training one model to upscale both by 2 and by 4 forces it to learn scale‑invariant representations, while equal‑probability sampling prevents bias toward either factor.
Extract video crops of size 512 × 768, 512 × 512, or 768 × 512 using center, random, optical‑flow, or high‑detail selection.
Discard crops with dark borders, banding, or compression artifacts based on spatial and temporal quality metrics.
Apply one of two Real‑ESRGAN‑style degradation pipelines to produce the low‑resolution counterpart.
Bilinearly upsample the low‑resolution clip before K‑VAE encoding so that LQ and HQ latents share spatial dimensions.
Train the SR‑DiT model on paired latents with equal sampling of ×2 and ×4 scales.
Train each Latent Upscaler independently to map degraded latents to clean high‑resolution latents.
Run the Sim‑LADD adversarial refinement stage using the hinge loss and fused Transformer Decoder discriminators.
**Figure 9.** Rollout strategies for adversarial training. Top: standard LADD, where the intermediate state $x_t$ is obtained by forward noising of the clean sample $x_0$ and denoised in a single step with gradients. Middle: backward student rollout with gradients propagated through all denoising steps. Bottom: backward student rollout with gradients propagated only through the final steps.
SR-DiT Objective
Defines the SR‑DiT loss and training objective for restoring high‑quality latents.
SR‑DiT must learn to invert the degradation pipeline and recover high‑quality latent representations.
We train the model to predict a velocity field that, when applied to a noisy low‑resolution latent, flows it back to the original high‑resolution latent.
Compute noisy latent: \tilde{z}_{LQ} = (1‑0.7²)·1.0 + 0.7·0.5 = 0.51 + 0.35 = 0.86.
Transform u to time: t = 5·0.2 / (1 + 4·0.2) = 1 / 1.8 ≈ 0.56.
Form mixed state: $x_t$ = (1‑0.56)·2.0 + 0.56·0.86 ≈ 0.88 + 0.48 = 1.36.
Assume the model predicts velocity v_$\theta$ = 0.3 (conditioned on anchor a and mask m).
Estimate clean latent: \hat{z}_{HQ} = 1.36 – 0.56·0.3 ≈ 1.36 – 0.17 = 1.19.
Flow loss: `L_flow` = (0.3 – (0.86 – 2.0))² = (0.3 – (‑1.14))² = (1.44)² ≈ 2.07.
The example shows how a single scalar $t$ controls the blend between clean and noisy latents and how the predicted velocity directly corrects the noisy latent toward the clean target.
How does SR‑DiT’s flow‑matching loss differ from the denoising loss used in standard diffusion models?
Standard diffusion models predict the noise added to a latent at a given timestep, whereas SR‑DiT predicts a velocity that directly transports the noisy low‑resolution latent back to the high‑resolution latent. The flow loss therefore measures the error between predicted velocity and the true latent difference, not between predicted and actual noise.
**Figure 3.** SR-DiT architecture. The noisy video latent, first-frame HQ anchor, and binary anchor mask are concatenated along the channel dimension and processed by timestep-modulated DiT blocks. The output head predicts the flow-matching velocity. The model operates without text conditioning.
SR-DiT Training Stages
We train SR‑DiT in three stages, introducing anchor masking and sparse NABLA attention to improve temporal consistency.
Temporal consistency is hard because the model could over‑rely on a fixed reference frame, leading to artifacts when that frame is missing at inference time.
During training we sometimes hide the first‑frame reference (the “anchor”) so the model learns to reconstruct video without depending on that cue.
Step 1: Concatenate the anchor (a 4‑dim vector) and mask (a one‑hot 4‑dim vector) with each $x_t$, yielding a channel‑augmented tensor of shape (4 × 6).
Step 2: With probability 0.15 we replace the anchor and mask entries by zeros, so the tensor becomes all zeros at the anchor channel for all frames.
Step 3: The model processes the tensor; because the anchor channel carries no signal, attention must rely on temporal context from frames t=1…3.
Step 4: After training, when the anchor is omitted at inference, the model produces plausible reconstructions for frame t=0 using learned temporal patterns.
Zeroing the anchor forces the network to learn a true generative prior over the first frame rather than copying it, which mitigates grid and checkerboard artifacts.
Stage 1 (dense attention): train on 8 × H100 GPUs using AdamW ($\beta$ = (0.9, 0.95), weight decay 10⁻³), gradient clipping 1.0, LQ noise s = 0.7, learning rate 10⁻⁵ with 500 warm‑up steps, EMA decay 0.999, scheduler scale 5.0; run for 7,250 steps (~4.5 days).
Stage 2 (sparse NABLA attention): replace dense attention with sparse NABLA, apply top‑CDF block selection with $P=0.8$ annealed from 0.95 over the first 200 steps, use sliding‑tile attention window 11 × 7 × 7 (time × height × width) with each spatial block containing 8 × 8 latent tokens; train for 6,000 steps with the same optimizer settings and 200 warm‑up steps (~3 days).
Stage 3 (gradient accumulation): continue training on the same hardware, but accumulate gradients over two micro‑batches before each optimizer step to increase effective batch size.
All stages initialize the model from the text‑free Kandinsky 5.0 checkpoint.
Latent Upscaler Training
Trains a latent upscaler to enlarge low‑resolution latent video while preserving degraded content.
The latent upscaler (LU) bridges the resolution gap left by SR‑DiT, turning a low‑resolution latent video into a high‑resolution latent while keeping the degraded content consistent.
LU learns to expand a coarse latent grid into a finer one, letting SR‑DiT focus on restoring clean high‑quality content later.
Input grid cells: (0,0), (0,1), (1,0), (1,1).
Each cell is expanded into a 4 × 4 block, producing 16 cells per input cell.
After expansion, the full grid contains 8 × 8 = 64 cells.
The Charbonnier loss compares each predicted cell value $p_{ij}$ to the target $t_{ij}$ via $\sqrt{(p_{ij}-t_{ij})^2+\epsilon^2}$ with $\epsilon=10^{-3}$.
Pixel‑space L1 loss is computed on the decoded 256 × 256 crop of the upscaled latent, encouraging correct color and structure.
Even though the upscaler only sees degraded inputs, the combination of latent and pixel losses forces it to generate high‑frequency details that SR‑DiT can later refine.
Initialize all modules from the frozen K‑VAE decoder; randomly initialize output heads.
For the first 100 steps, train only the output heads while keeping other weights frozen.
From step 101 onward, unfreeze all layers and continue training with the full loss.
Compute Charbonnier loss on the latent prediction and L1 loss on the decoded pixel crop every second step.
Apply latent temporal L1 loss (weight 0.1) initially; after 50 k steps increase its weight to 0.5 and add decoded pixel temporal L1 loss (weight = 1.0).
Run three learning‑rate phases: 50 k steps at $2.2\times10^{-4}$, 40 k steps at $1.1\times10^{-4}$, then 10 k steps at $1.1\times10^{-5}$.
How does the Latent Upscaler differ from the SR‑DiT model itself?
SR‑DiT operates on full‑resolution latents to restore clean high‑quality video, while LU only learns to enlarge a degraded latent representation. LU never sees clean HQ data; its job is purely spatial upsampling, leaving the restoration task to SR‑DiT.
Super-Resolution Validation
We evaluate SR‑DiT and the Latent Upscaler using no‑reference metrics and PSNR on held‑out clips.
Validation focuses on no‑reference image/video quality scores and on PSNR measured against decoded latent targets, because the generated videos lack paired high‑quality ground truth.
Latent Upscaler (LU) achieves a mean PSNR gain of +1.30 dB over pixel‑space bilinear upsampling across the 94‑clip validation set.
Table 5 reports LU 41.70 dB versus bilinear 39.26 dB, yielding a +1.30 dB improvement.
The validation suite also reports MUSIQ, DOVER, CLIP‑IQA, Laplacian artifact, and warp‑error scores; higher values indicate better image quality for MUSIQ, DOVER, and CLIP‑IQA, while lower values are better for Laplacian artifacts and warp error.
The table presents performance metrics across different scales ($\times 4$ and $\times 2$) and stages (Stage 1, Stage 2, Stage 3). The columns include MUSIQ, DOVER, CLIP-IQA (all higher is better), and Lapl. artifacts and Warp error (both lower is better).
**Table 5.** As Table 5 shows, the $\times 4$ LU achieves higher mean PSNR than pixel-space bilinear upsampling in all four groups. At $\times 2$, it improves PSNR on the image-derived camera-motion clips and the 4K collection synthetic-degradation group, while pixel-space bilinear upsampling scores higher on the two weak-degradation video groups. The pixel-space baseline decodes the low-resolution latent and then bilinearly upsamples the decoded video. By contrast, bilinear interpolation directly in latent space, followed by decoding, yields only 21.44 dB at $\times 2$ and 19.76 dB at $\times 4$, showing that this simple latent-space baseline reconstructs the target poorly.
Experimental Results
Kandinsky 6.0 Video uses a unified transformer and multi‑stage training to sync audio‑visual generation.
Kandinsky 6.0 Video Pro outperforms all baselines on the majority of VABench metrics.