MC-Sparse: Deconstructing and Closing the Dense–Sparse Attention Gap in Diffusion Transformers

Jiarui Chen, Zeqiang Lai, Jiangshan Wang, Ziheng Ouyang, Ye Huang, Xiangyu Yue, Cewu Lu, Chunchao Guo

MC-Sparse accelerates diffusion transformers by reusing exact token-level attention selections and residuals across denoising steps.

How can we make sparse attention in diffusion transformers as accurate as dense attention while maintaining high computational efficiency?

Diffusion transformers struggle with the quadratic cost of attention as sequence lengths grow, but existing sparse-attention methods sacrifice generation fidelity to save compute. MC-Sparse is a training-free framework that selects individual key-value tokens for query groups and caches these selections—along with output residuals—to reuse across denoising steps. This approach achieves higher fidelity to dense-attention outputs and larger speedups than existing sparse baselines, delivering up to a 2.32× denoising speedup on 3D generation tasks.

Paper Primer

Sparse attention methods often rely on block-level selection, which forces queries to share coarse, suboptimal key-value (KV) sets. MC-Sparse deconstructs this bottleneck by isolating three sources of error: structural binding, selection inaccuracy, and discarded attention mass.

MC-Sparse is a training-free framework: it groups queries into tile-aligned clusters using a fast principal-direction divisive partitioning (PDDP) algorithm, selects individual KV tokens based on exact attention mass, and compensates for discarded tokens using cached residuals. The core move is temporal reuse: it performs these expensive operations only at anchor steps and applies the cached metadata and residuals to subsequent denoising steps.

MC-Sparse achieves significant denoising speedups with negligible quality loss compared to dense attention.

Evaluations on video and 3D generation models (Minimax-H3-Base and HY3D-Internal). 1.80× speedup on Minimax-H3-Base and 2.32× speedup on 3D asset generation at 15% attention density.

Token-level selection and residual compensation outperform block-sparse baselines in fidelity.

Cumulative ablation on Wan2.1-1.3B-T2V shows consistent improvements in PSNR and LPIPS as each component (token granularity, grouping, and residual compensation) is added. Residual compensation alone provides up to 3.54 dB PSNR gain in specific configurations.

Why does this approach use temporal reuse instead of just optimizing the selection process at every step?

Exact token-level selection is computationally expensive and causes irregular memory access. Reusing selections from earlier steps amortizes these costs while exploiting the stability of attention patterns across the denoising trajectory.

How does this method avoid the "structural binding" issue found in block-sparse attention?

By moving to token-level KV selection, the method removes the constraint that entire blocks of keys must be retained together, allowing for finer-grained, query-specific attention.

The Dense-Sparse Attention Gap

We expose why sparse attention harms fidelity in long‑sequence Diffusion Transformers and motivate a framework that bridges the dense‑sparse gap.

Diffusion Transformers ($DiT$) achieve high‑fidelity video and 3D generation, but their quadratic attention cost explodes with longer sequences. Existing sparse schemes, notably Block Sparse Attention ($BSA$), reduce compute by discarding whole $KV$ blocks, which introduces a dense‑sparse gap: structural binding error from coarse blocks, selection error from approximate scores, and discarded tail error from omitted tokens.

**Figure 1.** Quality and efficiency of MC-Sparse on Minimax-H3-Base at 768p. Left: quality metrics and DiT denoising latency. At 15% and 25% attention density, MC-Sparse achieves 1.80× and 1.61× speedup over dense attention (FA3), respectively. Right: frames generated at 15% density closely match the dense-attention reference across the video.

**Figure 2.** Oracle comparisons of the dense-to-sparse gap on Wan2.1-1.3B-T2V at 480p. Experimental settings are provided in Appendix A.

The core trade‑off is preserving generation fidelity while cutting compute for long‑sequence $DiT$s.

Background and Baselines

Survey of long‑sequence diffusion transformers, efficient attention, and the baseline block‑sparse attention.

This section situates our work among prior long‑sequence diffusion transformers and the spectrum of efficient attention mechanisms, then formalizes the baseline block‑sparse attention that our method refines.

Transformer‑based models that generate long sequences (video, 3‑D assets) by iteratively denoising latent tokens. Their quadratic attention cost becomes a bottleneck as token counts grow to tens or hundreds of thousands.

Methods that learn a reduced‑cost attention pattern. Linear‑attention variants (SANA, SANA‑Video, MHLA) replace the softmax with kernel approximations; sparse‑attention variants (VMoBA, VSA, SpargeAttention2) select a subset of keys per query; hybrid variants (SLA, SLA2) combine both ideas.

Approaches that approximate full attention at inference time without retraining. They partition tokens into blocks (e.g., SVG, XAttention, SpargeAttention) and estimate contributions of unselected blocks (SVG‑EAR, PISA). Some exploit cross‑step redundancy (AdaSpa, DiTFastAttn, QuantSparse) to reuse attention information across diffusion steps.

Block‑Sparse Attention groups queries and keys into tiles, then each query block keeps the highest‑scoring key‑value blocks according to a mean‑pooled similarity score.

The MC-Sparse Architecture

How MC‑Sparse restores dense‑attention fidelity while keeping sparse cost.

Dense attention gives the best generation quality but scales quadratically, so vanilla block‑sparse attention (BSA) sacrifices fidelity to stay tractable.

An oracle picks the entries that carry the most probability mass under a fixed attention density, providing an upper bound on what any practical selector could achieve.

Instead of keeping whole blocks, MC‑Sparse selects individual KV tokens for each query group, eliminating the “KV binding” constraint of block‑sparse attention.

Rank the tokens by score: token 1 (0.25), token 4 (0.20), token 5 (0.15), token 2 (0.10), …

Select the top‑3 tokens → indices {1, 4, 5}.

Gather their key/value vectors and feed them to the sparse attention kernel.

The remaining 5 tokens are ignored, removing the irrelevant neighbors that block‑sparse would have kept.

Token‑level selection preserves the high‑mass region of the attention distribution while discarding low‑mass neighbors that cause the structural binding error.

Queries are clustered into equal‑size groups that match the GPU tile size, so each group can share a single KV selection without breaking parallelism.

Compute the principal direction of all four queries (approximately (0.55, 0.45)).

Project each query onto this direction: p₁≈0.55, p₂≈0.55, p₃≈0.55, p₄≈0.55 (ordered by projection).

Split at the median → first two queries (q₁, q₂) form Group A, last two (q₃, q₄) form Group B.

Each group now contains exactly C = 2 queries, ready for tile‑aligned processing.

Median‑split PDDP guarantees balanced groups while keeping queries that point in similar directions together, which is essential for efficient GPU tiling.

Exact KV selection is computed only at “anchor” steps and then reused for several subsequent “reuse” steps, amortizing its cost.

At anchor steps we record the difference between dense and sparse outputs and add this residual back at reuse steps, correcting the “discarded tail” error.

MC‑Sparse alternates between an anchor phase that computes dense attention and caches metadata, and a reuse phase that applies sparse attention with the cached metadata plus residual correction.

For each diffusion step t, compute $Q_t$, $K_t$, $V_t$.

If t is an anchor (ISANCHOR(t, L) true):

• Group queries with Fast PDDP → $G_t$.

• Perform exact KV selection (Equation 1) → $S_t$.

• Run dense attention → $O^{full}_t$.

• Run sparse attention with ($G_t$, $S_t$) → $O^{sparse}_t$.

• Compute residual $R_t$ = $O^{full}_t$ − $O^{sparse}_t$ and cache ($G_t$, $S_t$, $R_t$).

• Output $O_t$ = $O^{sparse}_t$ + $R_t$.

Else (reuse step):

• Retrieve cached (G, S, R).

• Run sparse attention with (G, S) → $O^{sparse}_t$.

• Output $O_t$ = $O^{sparse}_t$ + R.

**Figure 3.** Three sources of the gap between dense and sparse attention.

**Figure 5.** Absolute temporal change of attention outputs and dense–sparse residuals under token-level selection. Lower values indicate smaller absolute variation across $\Delta$ denoising steps.

**Figure 6.** Pipeline overview of MC-Sparse.

**Algorithm 1** MC-Sparse **Require:** Diffusion steps $T$, reuse interval $L$, KV budget $K$ **Ensure:** Attention outputs $\{\hat{\mathbf{O}}_t\}_{t=1}^T$ 1: **for** $t = 1, \dots, T$ **do** 2: $\quad$ Compute $\mathbf{Q}_t, \mathbf{K}_t$, and $\mathbf{V}_t$ 3: $\quad$ **if** $\text{ISANCHOR}(t, L)$ **then** 4: $\quad \quad (\mathbf{O}_t^{\text{full}}, \mathbf{l}_t) \leftarrow \text{FLASHATTN}(\mathbf{Q}_t, \mathbf{K}_t, \mathbf{V}_t)$ 5: $\quad \quad \mathcal{G}_t \leftarrow \text{GROUPQUERIES}(\mathbf{Q}_t)$ 6: $\quad \quad \mathcal{S}_t \leftarrow \text{EXACTKVSELECTION}(\mathbf{Q}_t, \mathbf{K}_t, \mathbf{l}_t, \mathcal{G}_t, K)$ 7: $\quad \quad \mathbf{O}_t^{\text{sparse}} \leftarrow \text{SPARSEATTN}(\mathbf{Q}_t, \mathbf{K}_t, \mathbf{V}_t; \mathcal{G}_t, \mathcal{S}_t)$ 8: $\quad \quad \mathbf{R}_t \leftarrow \mathbf{O}_t^{\text{full}} - \mathbf{O}_t^{\text{sparse}}$ 9: $\quad \quad \mathcal{C} \leftarrow (\mathcal{G}_t, \mathcal{S}_t, \mathbf{R}_t)$ 10: $\quad \quad \hat{\mathbf{O}}_t \leftarrow \mathbf{O}_t^{\text{full}}$ 11: $\quad$ **else** 12: $\quad \quad (\mathcal{G}, \mathcal{S}, \mathbf{R}) \leftarrow \mathcal{C}$ 13: $\quad \quad \mathbf{O}_t^{\text{sparse}} \leftarrow \text{SPARSEATTN}(\mathbf{Q}_t, \mathbf{K}_t, \mathbf{V}_t; \mathcal{G}, \mathcal{S})$ 14: $\quad \quad \hat{\mathbf{O}}_t \leftarrow \mathbf{O}_t^{\text{sparse}} + \mathbf{R}$ 15: $\quad$ **end if** 16: **end for**

Empirical Performance

Experimental results showing MC‑Sparse’s fidelity gains and speedups.

MC‑Sparse delivers 28.44 dB PSNR on Minimax‑H3‑Base while retaining only 25 % of query–key interactions.

Table 1 reports 28.44 dB PSNR at 25 % density for MC‑Sparse versus 23.66 dB at 32.4 % density for Sol‑Attn.

On the HunyuanVideo‑13B and Wan2.1‑14B models, MC‑Sparse consistently yields the highest PSNR and SSIM and the lowest LPIPS among all sparse baselines, while achieving 1.52×–1.82× speedup over dense attention.

Component Contributions

We quantify how each MC‑Sparse component affects image quality across densities.

We ablate each MC‑Sparse component, starting from vanilla BSA, to see how it changes PSNR, SSIM, and LPIPS at several attention densities.

Adding exact KV selection with cross‑step reuse raises PSNR by +2.55 dB at density $s\!=\!0.2$.

Table 3 shows PSNR 20.96 → 23.51 dB when the exact KV step is added.

Introducing token‑level KV granularity further improves PSNR by +1.03 dB at density $s\!=\!0.25$.

Table 3 reports PSNR 21.60 → 22.63 dB after adding token granularity.

Query grouping contributes an additional +0.96 dB PSNR gain at density $s\!=\!0.2$.

Table 3 shows PSNR 23.51 → 24.47 dB when query grouping is enabled.

Residual compensation adds +1.25 dB PSNR for MC‑Sparse at $s\!=\!0.20$.

Table 5 compares MC‑Sparse w/o residual (20.96 dB) to MC‑Sparse + residual (22.21 dB).

Applying the same residual compensation to vanilla BSA yields a +1.45 dB PSNR boost at $s\!=\!0.20$.

Table 5 shows vanilla BSA (21.67 dB) versus BSA + residual (23.12 dB).

**Figure 4.** Temporal stability and accuracy of reused exact selection.

Kernel Efficiency

Quantifies MC‑Sparse’s speed and latency gains versus competing sparse attention kernels.

MC‑Sparse attains near‑ideal speedup across sparsity levels, reaching 96 %–99 % of the theoretical $1/s$ bound.

Table 6 shows MC‑Sparse’s speedup $ρ$ between 4.79× and 4.96× with efficiency $η \approx 0.96$–$0.99$, outperforming SVG2 ($η \approx 0.80$–$0.85$) and PISA ($η \approx 0.80$–$0.87$).

Compared with SVG2 and PISA, MC‑Sparse consistently delivers higher efficiency $η$ and larger speedup $ρ$ across all sparsity settings, while keeping the query‑grouping cost low.

**Table 6.** Token-sparse attention kernel. Table 6 compares attention kernels at matched KV-token densities on Wan2.1-14B-T2V at 720p, excluding selection and grouping. SVG2 uses $Q_c = 300$ and $K_c = 1000$. MC-Sparse achieves 96%–99% of the ideal $1/s$ speedup, versus 80%–81% for SVG2 and 82%–87% for PISA, showing that token-level KV gathering preserves kernel efficiency.

**Table 7.** Query-grouping latency on Wan2.1-T2V (ms per layer). Amortized over 40 steps: 2 Fast PDDP calls or Flash-KMeans with a 50-iteration initialization and 39 two-iteration updates.

**Table 8.** Per-layer cost of offloading cached output residuals and selected KV indices to CPU memory. $V$ is the transfer volume and $C$ the sparse-attention FLOPs; $C/V$ gives the break-even compute-to-PCIe-bandwidth ratio for overlap. Check marks indicate transfers hidden by prefetching one layer ahead in the measured setting.

Experimental Details

Experimental setup, implementation specifics, and efficiency measurements for the evaluated models.

We evaluate video generation using the Penguin Benchmark prompts from SVG‑EAR and the VBench image‑prompt pairs, sampling 50 examples per task. For Minimax‑H3‑Base we apply the official prompt‑enhancement API and generate 345‑frame videos (~14.4 s long).

Analysis and ablation experiments run on the Wan2.1‑1.3B‑T2V model at 480p. Metrics are computed by uniformly sampling diffusion steps and transformer layers outside the dense warm‑up regions, using the same evaluation set as the corresponding generation task.

Warm‑up configuration details are listed in Table 9; each entry shows the fraction of warm‑up steps or layers relative to the total. For Hunyuan the fractions 1/20 and 1/40 refer to dual‑stream and single‑stream layers respectively, and MC‑Sparse inherits the same dense warm‑up schedule as its baseline.

Baseline configurations follow SVG‑EAR, with key SVG2 and SVG‑EAR parameters summarized in Table 10. The table reports query and key cluster counts ($Q_c$, $K_c$) and the top‑$P$ probabilities for both settings.

Efficiency is measured as speedup relative to dense attention based on DiT denoising latency, including the dense warm‑up computation. SpargeAttn runtimes are omitted because they rely on INT8/FP8 kernels, while all other methods use BF16, making direct latency comparison unfair.

All runs use optimized operators such as RoPE and normalization; dense attention during warm‑up employs FlashAttention‑3 (FA3). Minimax‑H3‑Base leverages the Diffusers library, also using FA3 for both dense baseline and warm‑up steps.

Wan 2.1, HunyuanVideo, and HY3D‑Internal execute on a single Hopper GPU, including condition encoder and VAE when needed. Minimax‑H3‑Base distributes work across eight Hopper GPUs, with one GPU handling encoding/decoding and the remaining seven running the denoiser via Ulysses sequence parallelism.

FlashAttention cannot expose the full normalized attention matrix, so we perform a two‑pass exact selection: the first pass yields dense outputs and row‑wise log‑sum‑exp values; the second recomputes $QK$ scores, recovers probabilities, aggregates group‑wise, and applies top‑$K$ to the attention mass without materializing the dense matrix.

The token‑sparse attention kernel is implemented in CuT‑eDSL with dedicated producer and consumer warps. Producer warps asynchronously gather selected key/value rows into shared memory, while consumer warpgroups apply WGMMA operations and online softmax, overlapping the next gather with the current computation.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers