SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation

Junsong Chen, Jincheng Yu, Yitong Li, Shuchen Xue, Haozhe Liu, Jingyu Xin, Yuyang Zhao, Tian Ye, Zhangjie Wu, Zian Wang, Daquan Zhou, Ping Luo, Song Han, Enze Xie

SANA-Video 2.0 uses a hybrid linear-softmax backbone and cross-depth residual routing to generate high-resolution video efficiently.

How can we generate high-resolution video efficiently by replacing standard quadratic attention with a hybrid architecture that uses sparse attention and learned residual connections?

Full-softmax video diffusion transformers incur quadratic costs that become prohibitive as video resolution and duration increase, forcing a trade-off between sequence length and computational feasibility. SANA-Video 2.0 replaces the uniform softmax stack with a hybrid architecture: it uses gated linear attention for the majority of layers and inserts periodic softmax anchors at a 3:1 ratio, while Block Attention Residuals route completed block summaries across depth to maintain representation rank. This design achieves competitive VBench quality (84.30) while remaining 3.2× faster than full-softmax baselines at 720p, with efficiency gains that widen as video duration increases.

Paper Primer

The core mechanism hinges on a "hybrid-plus-routing" strategy: linear attention provides $O(N)$ global mixing, while sparse softmax anchors periodically inject exact token interactions. To prevent the information loss typical of linear states, Block Attention Residuals (AttnRes) act like a memory buffer, allowing later layers to pull completed feature summaries from earlier blocks rather than re-deriving them from scratch.

Hybrid attention recovers softmax-level expressiveness at a fraction of the latency.

At 720p/60s, the SANA-Video 2.0 DiT forward pass is 3.2× faster than a matched full-softmax baseline.

AttnRes improves deep-layer representation quality.

Enabling AttnRes increases the mean effective rank of deeper layers by ~12% in controlled same-checkpoint probes.

Why use a hybrid approach instead of just optimizing the existing full-softmax attention?

Full-softmax attention scales quadratically ($O(N^2)$), making it punishing for video where a single 1080p clip spans tens of thousands of latent tokens. The hybrid design keeps the majority of computation in $O(N)$ linear layers, which scales gracefully with duration.

Is the 25% softmax anchor ratio a universal constant for this architecture?

No, the authors identified 25% as the optimal quality-efficiency "knee" for the video regime through reduced-resolution proxy studies, rather than assuming the ratio used in language models.

Researchers can now scale video generation to longer durations and higher resolutions by swapping quadratic attention for a hybrid backbone that preserves rank through cross-depth residual routing.

Introduction

Introducing SANA‑Video 2.0: hybrid attention cuts latency while preserving video quality.

Video diffusion models have become the workhorse for high‑fidelity generation, but their quadratic attention makes long, high‑resolution clips prohibitively expensive.

The core problem is that dense $O(N^{2})$ attention dominates compute for video, yet pure linear attention loses the rich token‑to‑token interactions needed for fine spatiotemporal detail.

Building on this motivation, SANA‑Video 2.0 combines gated linear attention with periodic gated‑softmax anchors and augments the stack with Block Attention Residuals (AttnRes) to reuse refreshed representations.

**Figure 1.** **SANA-Video 2.0 at a glance.** (a) Text-to-video examples, including embodied and physics-following scenarios. (b) One-H100 720p/5s latency, including the final Sol-Engine 5B and 40-layer 14B results; VBench marks the 5B and Wan 2.2 quality points. (c) 5B DiT-forward speedup over duration; 14B scaling is in Appendix G.2.

SANA‑Video 2.0 delivers softmax‑level quality while cutting latency by orders of magnitude.

Preliminaries and Model Overview

Video diffusion transformers face prohibitive quadratic attention costs at high resolution.

Video diffusion transformers must process tens of thousands of latent tokens per 1080p clip, making the softmax attention matrix quadratic in sequence length and thus computationally prohibitive. This quadratic cost motivates the need for more efficient attention mechanisms.

DiTs are transformer models that generate video by iteratively denoising a latent representation of each frame, treating the video as a sequence of latent tokens.

Flow matching trains the model to predict the instantaneous velocity that moves a noisy latent toward the clean video latent across continuous time.

Instead of computing a full $N\times N$ attention matrix, gated linear attention compresses token interactions into a fixed‑size state that is updated by gated bilinear products, yielding linear $O(Nd^2)$ complexity.

Attention Residuals replace the standard additive skip connection with a learned weighted aggregation of earlier block outputs, allowing later layers to directly reuse completed representations.

The Hybrid Attention Mechanism

Hybrid layers expose completed block summaries so sparse attention retains full‑rank representations.

Dense quadratic attention dominates compute in video diffusion transformers; the hybrid layer mitigates this by applying full‑rank mixing only at sparse depths and using Attention Residuals to recycle completed block summaries.

Instead of forcing every layer to mix all tokens, the model performs full‑rank attention only at a few “sparse” depths, letting the rest of the network rely on cheaper operations.

How does Hybrid Attention differ from simply using a single sparse‑attention layer?

Hybrid Attention interleaves dense and linear attention repeatedly; a single sparse layer would give the model only one high‑rank view, forcing all subsequent processing to rely on low‑rank approximations and degrading quality.

Each block emits a compact summary of its completed updates; later layers read these summaries as additional sources, so the network can recover information that would otherwise be lost in the sparse depths.

Layer 1 (attention) produces an update u₁; p₁ ← u₁.

Layer 2 (FFN) produces update u₂; p₂ ← p₁ + u₂.

End of Block 1: freeze b₁ ← p₂, reset p₁ = 0 for Block 2.

Layer 3 (attention) produces u₃; p₁ ← u₃.

Layer 4 (FFN) produces u₄; p₂ ← p₁ + u₄.

End of Block 2: freeze b₂ ← p₂.

At layer 3 the router’s source set 𝓥₃ = {b₀, b₁, p₁}; the weighted sum combines the original embedding, the completed summary b₁, and the current partial sum p₁.

The partial sum lets each block accumulate its own contribution without storing every intermediate layer, and the router can still draw on that fresh information together with older block summaries.

Why does AttnRes use a shared query instead of a per‑layer query like earlier language‑model variants?

Per‑layer queries multiply parameters by the number of depths. Sharing a single query across all depths removes that overhead while still producing depth‑varying weights because each source is RMS‑normalized per token, so the softmax sees different values at each depth.

Group every S = 8 consecutive transformer layers into one block.

Within a block, each sublayer’s output is added to a running partial sum p_ℓ.

The router reads p_ℓ as an additional source in the set 𝓥_ℓ.

When the block ends, freeze the partial sum as the block summary $b_k$ and reset p_ℓ = 0 for the next block.

Later layers route over the source set 𝓥_ℓ = {b₀, b₁,…, p_ℓ}, combining the initial embedding, all completed summaries, and the current partial sum.

Because only one summary per finished block is stored, memory shrinks from O(N L d) to O(N ⌈L/S⌉ d), roughly an S‑fold reduction.

Training Recipe

How the video diffusion model is trained through a staged curriculum.

The training recipe stitches together data curation, a resolution‑duration curriculum, and preference‑based fine‑tuning, each stage sharpening the model’s ability to generate high‑quality video.

We filter the raw video pool along two independent axes—visual quality and motion richness—so that early stages keep many easy clips while later stages focus on the hardest, most informative examples.

After cleanup, 8 clips remain (those with at least one positive axis).

Pre‑training threshold: Q≥0, M≥0 → all 8 clips enter.

Continual threshold: Q≥1, M≥0 → 5 clips (Q=1) survive.

SFT threshold: Q≥1, M≥1 → 2 clips (both axes high) are kept.

The funnel gradually concentrates on clips that are simultaneously high‑quality and high‑motion, ensuring the final model sees the most challenging examples.

Why keep the quality and motion axes separate instead of merging them into a single score?

Because a single aggregate would let static but clean clips dominate early training, starving the model of motion cues. Separate axes guarantee that motion‑rich clips survive even if their visual quality is lower.

We reshape the logit‑normal distribution so that the extreme clean‑noise and pure‑noise tails receive extra probability mass, ensuring the model sees enough very low and very high noise examples.

How does tail‑flooring differ from simply increasing the variance of the logit‑normal?

Increasing variance spreads the mass but still leaves the extreme tails thin; tail‑flooring explicitly injects a fixed probability into the tails, guaranteeing their presence regardless of variance.

Instead of a fixed shift per resolution, we compute a shift that grows logarithmically with the number of latent tokens, so longer clips automatically receive more high‑noise probability.

Compute interpolation weight w = (ln 8,580 − ln 4,290) / (ln 23,000 − ln 4,290) ≈ 0.31.

Log‑shift = (1 − w)·log 3 + w·log 6 ≈ log 3.9.

Shift ≈ exp(log 3.9) ≈ 3.9, so the 720p clip uses a shift of ~3.9 instead of the fixed 6.

Even though the 720p clip is twice as long, the shift grows only modestly, preserving enough low‑noise samples for texture learning.

Why not keep a single fixed shift per resolution as in earlier pipelines?

A fixed shift creates a jump in the noise distribution when the resolution changes, causing some token counts to receive too much or too little high‑noise mass. The token‑count‑aware formula smooths this transition.

TQD routes each clip to the noise regime where its strengths are most useful, letting motion‑rich but noisy clips dominate high‑noise training and clean static clips dominate low‑noise training.

How does TQD avoid sacrificing either motion or texture quality?

By sending motion‑rich clips to high‑noise timesteps, the model learns global motion patterns while the noise masks texture errors; conversely, clean static clips are trained at low noise where fine detail can be recovered.

Self‑Flow distills features within a clip by conditioning on two timesteps, encouraging the model to retain information across the diffusion trajectory.

Why is Self‑Flow turned off during the SFT stage?

Because SFT focuses on aligning the model with human preferences; the auxiliary distillation would constrain the model’s ability to adapt to the new objective.

We maintain a smoothed copy of the model parameters throughout training, which serves as a stable teacher for both pre‑training and post‑training stages.

What would happen if the EMA were disabled?

The model would lose a stable reference during DPO, leading to higher variance in the preference loss and potentially destabilizing fine‑tuning.

Pre‑train on ~30 M clips at 480p, 5 s duration, using flow‑matching with TQD weighting and tail‑floored timesteps.

Continual stage on ~10 M clips, gradually increasing resolution (480 → 720p) and duration (5 → 8 s); switch to token‑count‑aware flow‑shift and drop TQD.

SFT on ~10⁴ high‑quality, high‑motion clips at 720p, 8 s; keep EMA active, turn off Self‑Flow, and add the standard flow‑matching objective.

Post‑training: apply Diffusion‑DPO with a frozen EMA reference, then refine with Reward Feedback Learning (ReFL) using HPSv3++, DeQA‑Score, and UniPercept rewards.

**Figure 3.** Training and data pipeline for the 5B checkpoint. Curated in-house clips feed pre-training, continual training, SFT, and preference-based post-training (DPO + ReFL), with stage-specific data selection, video shapes, and training objectives per stage.

We fine‑tune the model to prefer videos that rank higher on human‑aligned criteria, using a pairwise preference loss while keeping a frozen reference model as a stable baseline.

Why keep the reference model frozen during Diffusion‑DPO?

A frozen reference provides a stable target; otherwise both policy and reference would drift together, erasing the preference signal.

ReFL optimizes the model against frozen image‑reward networks by evaluating a few sampled frames, propagating gradients through the decoder but not through the full diffusion sampling trajectory.

How does ReFL avoid backpropagation through the entire sampling trajectory?

It stops the diffusion process at a single intermediate timestep, decodes the latent, evaluates rewards on a few frames, and backpropagates only through that deterministic slice; the earlier stochastic steps are treated as fixed inputs.

The curriculum progresses from a broad, low‑quality pool to a narrow, high‑quality set, while the training objectives evolve from TQD‑weighted flow‑matching to token‑count‑aware shifts and finally to preference‑driven fine‑tuning.

Experimental Results

SANA‑Video 2.0 delivers top VBench scores while slashing inference cost.

SANA‑Video 2.0 attains VBench Total 84.30 and the highest Quality 85.61, surpassing all prior video generators.

Table 2

Implementation Details: SANA‑Video 2.0 offers 5 B and 14 B hybrids. The 5 B model uses 32 layers at width 2,560; the 14 B model uses 40 layers at width 4,096 (≈14.25 B parameters) and trains at 384×B200 scale. Both share LTX‑VAE 2.3 latents, Gemma text features, flow matching, 25 % softmax anchors, and Block AttnRes.

Design Space Exploration: All proxy studies fix depth = 28, width = 3,072 and train at 256p/81 frames. The softmax‑layer placement is varied while all other hyper‑parameters stay constant, and validation loss is measured per diffusion timestep because loss is highly non‑uniform across the noise axis.

Softmax Ratio: Five hybrids (0 %–100 %) were tested. Pure linear (0 %) yields validation loss 0.955, pure softmax (100 %) 0.945, while the 25 % hybrid (3:1 ratio) attains 0.905 loss with only a modest latency increase over the 14.3 % hybrid. The 25 % setting is chosen as the Pareto knee for all later experiments.

Residual Attention: With the 3:1 hybrid fixed, we probe AttnRes designs. Removing the explicit timestep input from the router improves loss from 0.962 to 0.920. Sharing query projections across layers matches per‑layer loss (0.495 vs 0.496) while cutting memory overhead 4×. Block span S = 8 is kept as the default because it balances memory (≈+2 %) and provides two 3:1 attention cycles per aggregation block.

Efficiency Scaling: Under a compiled no‑AttnRes protocol, the hybrid backbone gains 1.16×–2.01× speedup as width grows, and 1.41×–1.45× speedup as depth grows. Hardware scaling shows a 1.98× speedup on GB200 versus H100 for the 14 B model at 736×1280×121. The hybrid advantage grows with longer video sequences where attention dominates compute.

**Table 2.** VBench quality comparison. Green intensity ranks the top three results per metric. Superscripts after method names denote score sources and are defined in Appendix D. Only the 81-frame SANA-Video 2.0 result is shown here; the appendix also provides the full protocols, extended baselines, and our 121- and 193-frame operating points.

**Figure 4.** Reading loss by timestep, then the softmax-ratio proxy. (a) On a production checkpoint, validation loss spans a ~3x range across the noise axis, so the scalar random-t mean (dashed) hides structure. We therefore read every proxy comparison per timestep rather than as one scalar. (b,c) Fixed depth-28, width-3,072 architecture-search backbones (256p/81f): (b) 500-step mean training loss and (c) held-out normalized-latent velocity MSE at 10K (200 videos; pointwise 95% CIs) for five ratios.

**Table 3.** AttnRes training and design probes. (a) Late-stage continuation; (b–d) early short-run probes. Wins counts improved noise buckets; green shading marks the selected designs.

Mechanistic Analysis

Ablations quantify how Attention Residuals recover rank and enable cross‑depth reuse.

Enabling Attention Residuals raises the mean effective rank of deeper linear layers by 11.7 %.

Mean effective rank increased by 11.7 % at maximum noise; rank did not decrease in 22 of 24 layers.

Removing completed‑source inputs at block entries reduces effective rank by up to 91 %.

Effective rank drops 82–91 % when completed sources are omitted at block entry points, while mid‑block removal has negligible effect.

**Figure 6.** **Evidence for the selected AttnRes design.** (a) A shared query remains feature-adaptive across timesteps. (b) Enabling depth aggregation recovers effective rank in deeper linear layers. (c) Learned routing increasingly reuses completed blocks.

Deployment and Optimization

Sol‑Engine delivers multi‑stage latency reductions and low‑precision efficiency.

Sol‑Engine accelerates SANA‑Video 2.0 by 3.58× on NVIDIA B200 GPUs.

Table 4 shows latency dropping from 62.65 s to 17.52 s, a 3.58× speedup.

These gains are orthogonal to modeling contributions: the backbone uses only standard softmax and conv‑free SwiGLU primitives, so it maps directly onto production‑ready kernels. Consequently, the stack integrates seamlessly with existing deployment pipelines.

Sol‑Engine is a three‑stage deployment stack that first fuses kernels, then reuses residuals to skip many denoising steps, and finally applies sparse attention on the softmax anchors, collectively slashing latency.

How does Sol‑Engine differ from a typical inference engine that only applies kernel fusion?

Beyond kernel fusion, Sol‑Engine adds diffusion‑cache residual reuse (skipping 17 denoising steps) and sparse attention on the softmax anchors, so it reduces both compute and memory, not just kernel‑level overhead.

**Figure 8 | Qualitative outputs before and after Sol-Engine acceleration.** Two matched prompts compare the baseline with the directly measured 3.58× pipeline; each row samples matched progress through the clip.

**Figure 9.** Qualitative comparison with Physical AI models. In the presented comparison, SANA-Video 2.0 outperforms the similar-sized Cosmos3-Edge [1] (4B) and delivers competitive results against much larger models, including Cosmos3-Nano [1] (16B) and Lingbot-Video [27] (30B).

AttnRes Routing Analysis

Ablation studies quantify how AttnRes routing components affect video diffusion quality.

F.1 probes whether the router degrades performance after useful video representations have formed by continuing training from a mature 4.5 B hybrid checkpoint.

AttnRes does not hurt reconstruction quality after continued training.

Mean MSE 0.4851 with AttnRes versus 0.4855 without it on 100 held‑out clips.

F.2 defines a routing‑selectivity metric based on normalized Shannon entropy of the aggregation weights $\alpha_l$.

F.3 measures how the shared router’s entropy varies across denoising timesteps, revealing strong timestep‑dependent behavior.

The shared router exhibits 2.15× greater entropy variation across timesteps than a per‑layer alternative.

Adaptivity$_s$ = $\sigma_t[\hat{H}_s]$ is 2.15× larger for the shared query.

F.4 investigates whether the explicit router‑timestep projection learns timestep‑specific information.

Explicit router‑timestep projection yields a negligible performance gain.

Maximum validation‑loss difference among Normal, Mean, and Shuffle conditions is 0.0002.

Router‑offset vectors are nearly constant across timesteps.

Mean cosine similarity of offsets across 21 timesteps is 0.979; the varying component is only 13 % of the constant norm.

F.5 examines how much routing mass is assigned to completed block summaries in deeper layers.

F.6 zeroes routing weights to completed block sums and measures the impact on the effective rank of the hidden state $\mathbf{h}_l$.

Removing completed‑block routing drastically reduces representation rank at early block entries.

Effective rank drops by 82–91 % at layers 9, 17, 25 when routing to completed sums is disabled.

F.7 toggles depth aggregation on a production checkpoint and compares the effective rank $r_{\text{eff}}(S)$ of the gated RoPE‑rotated linear state.

Scaling and Deployment Measurements

Appendix G details scaling and deployment measurements for the hybrid video diffusion model.

Hybrid 14 B model attains up to 3.07× speedup over the full‑softmax baseline on GB200 for long 45 s video sequences.

Full/Hybrid speedup rises from 1.58× at 5 s to 3.07× at 45 s (GB200), while H100 shows a rise from 1.40× to 2.73×.

G.1 fixes a 720p/10 s sequence while jointly increasing width and depth; the hybrid’s absolute latency savings grow from 128 ms to 948 ms, and the Full/Hybrid speedup contracts from 1.88× to 1.45× because wider feed‑forward layers dominate compute.

G.2 compares the same 40‑layer, width‑4096, 14 B hybrid on H100 versus GB200; GB200 cuts forward latency by 1.69×–2.02× (median 1.91×) while memory usage follows the same upward trend.

G.3 evaluates parameter‑matched hybrid and full‑softmax backbones; as video duration expands from 5 s to 45 s, the hybrid’s speedup improves from 1.40× to 2.73× on H100 and from 1.58× to 3.07× on GB200, confirming a mostly linear advantage at larger scales.

G.4 isolates the cost of SANA‑Video’s temporal‑convolution FFN path; holding depth and width constant, the convolution adds a 20 %–29 % overhead that grows with duration, and the 60 s outlier is excluded from analysis.

G.5 reports module‑level composition timings, confirming that the selected attention‑ratio and removal of the temporal‑convolution path together preserve the scaling benefits observed in earlier subsections.

Related Work and Conclusion

We recap related video diffusion work, summarize SANA‑Video 2.0 results, and outline future directions.

Most open video generators use diffusion transformers with quadratic softmax token mixing, including Wan 2.1/2.2, HunyuanVideo, CogVideoX, and Open‑Sora. Alternatives such as MAGI‑1 and LTX‑Video reduce sequence cost via temporal chunking or stronger latent compression. Efficient token mixers replace the N × N map with a fixed‑size state through linear attention or state‑space models, though compressed states can limit representation rank.

In visual generation, DiG applies gated linear attention to image DiTs, SANA‑Video introduces a pure‑linear video DiT, and SANA‑WM/SANA‑Streaming extend these backbones to world modeling and streaming editing. Hybrid language models retain periodic softmax layers: Qwen3‑Next and Kimi‑Linear use 3:1 layouts, Kimi K3 adds cross‑layer Attention Residuals, and output gating stabilizes softmax. Sparse‑VideoGen, Radial Attention, SpargeAttn, PISA, and Native Sparse Attention further reduce cost within each remaining softmax operator.

SANA‑Video 2.0 builds a mostly‑linear hybrid‑attention backbone at 5 B and 14 B scales, using 75 % gated linear attention and 25 % periodic softmax anchors, with Block AttnRes propagating refreshed representations across depth. On a 40‑layer, width‑4096 model trained at 384×B200 scale, the 5 B checkpoint reaches VBench Total 84.30 in 13.2 s on a single H100, and its DiT forward is 3.2× faster than full softmax at 720p/60 s tensor shape; integrated with Sol‑Engine it yields a 3.58× end‑to‑end speedup and AttnRes raises deep‑layer effective rank by ≈12 %.

Future work will extend the O(N)‑dominated backbone beyond the current 8 s horizon toward minute‑scale video, testing whether anchors can preserve global consistency over much longer contexts. The bidirectional operator can be adapted to causal generation by pairing a causal Gated DeltaNet with the hybrid‑plus‑AttnRes layout, enabling streaming and action‑conditioned rollouts for robotics and world models. Additional avenues include anchor‑specific sparse kernels, calibrated low‑precision (NVFP4) formats, few‑step distillation, and learned content‑adaptive anchor placement and AttnRes routing.

Extended Related Work

Provides additional related‑work context and full model‑training specifications.

Modern video generators rely on diffusion transformers that apply a dense, quadratic $N \times N$ softmax attention across all spatio‑temporal tokens. Systems such as Wan 2.1/2.2, HunyuanVideo, CogVideoX, and Open‑Sora scale this design by enlarging backbones or improving training recipes, but the $O(N^2)$ cost grows sharply with resolution and duration.

Some works shorten the sequence: MAGI‑1 generates video in temporal chunks, while LTX‑Video compresses latent representations. Efficient sequence models attack the backbone directly; linear attention replaces the explicit $N \times N$ map with a fixed‑size state, and state‑space models use recurrent or selective updates. In vision, DiG adds gated linear attention to image diffusion transformers, and the SANA‑Video line builds pure‑linear video DiTs with a temporal‑convolution FFN.

Hybrid language models interleave efficient state‑based layers with occasional exact softmax layers (e.g., Qwen3‑Next, Kimi‑Linear). Sparse methods such as Sparse‑VideoGen, Radial Attention, SpargeAttn, PISA, and Native Sparse Attention reduce work inside each softmax layer, whereas our hybrid design reduces how many layers invoke softmax at all.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers