GRACE: Generation-Aware Latent Compression for Efficient Video Generation

Jiyoung Kim, Paul Hyunbin Cho, Jisu Nam, Donghoon Lee, Hyunsung Go, Yeonkyeong Lee, Hansaem Kim, Seungryong Kim

GRACE compresses video latents by splitting them into a frozen base and a learned residual, aligned to the DiT.

How can we compress video latents to accelerate diffusion models without losing the high-fidelity generation capabilities of pre-trained models?

Video diffusion models are computationally expensive because they process a massive number of latent tokens. Simply increasing autoencoder compression degrades generation quality, and retraining the diffusion transformer (DiT) to handle new latent distributions is prohibitively costly. GRACE addresses this by keeping a frozen base latent from a pretrained autoencoder and learning a residual latent to capture missing details. It aligns these latents within the frozen DiT’s feature space and uses asymmetric denoising to ensure the residual builds on a stable base. This approach reduces token counts by nearly 8× and latency by 11.1× at 480×832 resolution, while matching the generation quality of the original, uncompressed pipeline.

Paper Primer

The core mechanism is a dual-latent representation: the base latent anchors the generation in the pretrained space, while the residual latent captures high-frequency details and motion. By aligning these latents to the frozen DiT's intermediate features, the system ensures the compressed representation remains compatible with the generator's learned distribution.

Asymmetric denoising acts as a stabilizer: the base latent is denoised at an earlier timestep than the residual, allowing the residual to recover missing information relative to a settled, reliable foundation. This requires no extra sampling passes, as both latents are processed in a single forward pass.

GRACE achieves near-identical generation quality to the uncompressed Wan2.1-14B pipeline while significantly reducing computational overhead.

VBench-I2V and T2V scores at 480×832×81 resolution. 11.1× reduction in latency and nearly 8× fewer tokens.

The speedup scales with resolution, providing greater efficiency for higher-fidelity generation.

Performance metrics at 736×1280×81 resolution.

Why is reconstruction fidelity alone insufficient for high-compression video generation?

Optimizing an autoencoder solely for reconstruction shifts the latent distribution away from what the pretrained DiT learned, causing generation artifacts. GRACE explicitly aligns the compressed latent with the DiT's internal feature space to maintain compatibility.

How does this method avoid the high cost of retraining the diffusion model from scratch?

GRACE reuses the frozen pretrained DiT and adapts it using lightweight LoRA fine-tuning and extended input projections, leveraging the existing generative knowledge rather than learning the latent space from zero.

Introduction

Video diffusion models stall because their token count explodes with resolution.

Video diffusion models achieve impressive quality but their computational cost scales quadratically with the number of tokens processed by the Diffusion Transformer (DiT). Because the token count follows the spatio‑temporal resolution of the latent produced by the video autoencoder, higher‑resolution videos explode the token budget.

The core obstacle is that the DiT must handle an ever‑growing token sequence, making both training and inference prohibitively expensive.

**Figure 1. Teaser.** We present GRACE, a novel latent compression technique that fine-tunes pre-trained video generation models, such as Wan2.1-14B (Wan et al., 2025), to generate from substantially fewer latent tokens (e.g., nearly 8x fewer). (Top) Text-to-video (T2V) samples at 736p from Wan2.1-14B before compression and from GRACE, using the same prompt. (Bottom left) The same comparison at 480p. (Bottom right) On VBench-T2V (Huang et al., 2023; 2024), GRACE preserves the generation quality of Wan2.1-14B before compression and achieves higher scores than existing high-compression autoencoders (HaCohen et al., 2024; Zheng et al., 2026; He et al., 2026), while generating 11.1x faster at 480p and 15.5x faster at 736p. All models in the plot are evaluated at matched resolutions with 50 sampling steps. The plot reports text-to-video, and the speedups above are measured on image-to-video.

The primary bottleneck in video diffusion is the sheer number of tokens the DiT must process.

Background and Related Work

Prior video‑diffusion acceleration ideas and the core autoencoder‑DiT components are reviewed.

Video diffusion models are costly because each denoising step processes a large token set; the literature therefore explores two orthogonal routes: reducing the number of steps or shrinking the per‑step token count.

Methods that modify the autoencoder architecture to increase the spatial compression factor beyond the standard $f=8$, thereby lowering the number of spatial tokens.

Approaches that progressively increase the temporal compression factor $t$, often by aggregating frames or using temporal pooling.

Techniques that allocate more channels per token to retain detail at high compression ratios, often imposing a structured layout on the channel dimension.

Methods that regularize the video latent using a pretrained vision model so that diffusion learns a more tractable representation, typically training both autoencoder and generator from scratch.

Approaches that keep the Diffusion Transformer (DiT) frozen and adapt the autoencoder to match its expected latent distribution, either by constraining reconstructability or aligning patch embeddings after divergence.

Wan2.1‑14B is a 14‑billion‑parameter video diffusion model that serves as the high‑capacity backbone for our experiments.

A video autoencoder compresses a raw video tensor into a compact latent $z$ and can reconstruct the video from $z$, enabling diffusion to operate on a much smaller representation.

DiT is a transformer that denoises latent tokens over a diffusion schedule, conditioned on time embeddings and optional text prompts.

Dual-Latent Autoencoder Design

Stage 1 trains a dual‑latent autoencoder and aligns its compressed latent to the pretrained DiT.

Reusing the pretrained autoencoder preserves its latent space, but naïvely adding compression blocks harms generation because the latent drifts away from what the Diffusion Transformer (DiT) expects.

The compressed latent is split into a frozen base part that stays in the original DiT space and a learnable residual part that carries the missing detail.

Down‑sample the video spatially by $r_s=2$, yielding $x_{\text{low}}$ of size $2 \times 2$ per frame.

Encoder $E$ produces $z_{\text{base}}$ of shape $2 \times (1+2) \times 2 \times 2$ (channels × time × height × width).

Residual encoder $E_{\text{res}}$ processes the full video and outputs $z_{\text{res}}$ of shape $1 \times (1+2) \times 2 \times 2$.

Concatenation yields $z$ of shape $3 \times 3 \times 2 \times 2$, ready for the diffusion backbone.

The base part preserves the global structure while the residual channel injects the fine‑grained details lost during down‑sampling.

How does this dual‑latent design differ from simply increasing the channel width of a single encoder?

Increasing channels would still force the encoder to compress the full‑resolution video into a single latent, which quickly saturates the reconstruction capacity. By separating a frozen base (which stays in the pretrained DiT space) from a learnable residual, we keep the base anchored while allowing the residual to carry only the missing high‑frequency information.

The loss forces the compressed latent to occupy the same region of the frozen DiT’s feature space as the original pretrained latent, making it easier for the diffusion model to learn.

Compute $h^{\text{pre}}_3$ from $z_{\text{pre}}$ (shape $2 \times 3 \times 2 \times 2$).

Compute $h^{\text{cmp}}_3$ from $z$ (shape $3 \times 3 \times 2 \times 2$), upsample to match $h^{\text{pre}}_3$, and add residual projection $P_3$.

Calculate cosine similarity between the two feature tensors, obtaining $0.78$.

Alignment loss contribution for layer 3 is $1 - 0.78 = 0.22$.

The alignment loss penalizes mismatched feature directions, encouraging the residual channels to produce representations that the frozen DiT can already interpret.

Why not simply train the entire autoencoder from scratch instead of keeping $E$ frozen?

Freezing $E$ guarantees that $z_{\text{base}}$ lives in the exact latent space the pretrained DiT was trained on, which dramatically reduces the adaptation burden in Stage 2. Training $E$ would let $z_{\text{base}}$ drift, re‑introducing the same mismatch the paper aims to eliminate.

**Figure 2.** Overall architecture. Stage 1 trains the autoencoder. The frozen encoder $\mathcal{E}$ maps the downsampled input to $z_{base}$, the residual encoder $\mathcal{E}_{res}$ maps the full-resolution input to $z_{res}$, and $\hat{\mathcal{D}}$ decodes both, while $\mathcal{L}_{align}$ matches the compressed latent to the pretrained latent inside the frozen DiT. The dashed box on the right of Stage 1 shows a simplified view of the blocks added to $\mathcal{E}_{res}$ and $\hat{\mathcal{D}}$. Stage 2 adapts the DiT to the compressed latent with LoRA, denoising $z_{base}$ at a lower noise level than $z_{res}$ at every step, so the base is denoised first.

**Figure 3.** Effect of the dual latent and $\mathcal{L}_{align}$. (a) single latent, (b) dual-latent representation without $\mathcal{L}_{align}$, and (c) ours, all generated at $480 \times 832 \times 81$. (Left) latent distribution, projected with t-SNE and colored by kernel density, with its uniformity metrics (Yao et al., 2025) and the total VBench (Huang et al., 2023; 2024) score below, followed by T2V samples generated after Stage 2. (Right) spatio-temporal structure of the latent, shown as its principal components mapped to RGB under the input frames. More samples are in Figs. I.11 and I.12.

Adapting the Diffusion Transformer

We adapt the pretrained DiT to compressed latents using asymmetric denoising and lightweight projection tuning.

Video diffusion models suffer from a token explosion: the latent representation is too large for efficient generation. To keep the pretrained Diffusion Transformer (DiT) usable, we must adapt it to the compressed dual‑latent space without retraining the whole autoencoder.

We give the DiT a thin “bridge” to the compressed latent by widening its I/O projections and by injecting a lightweight LoRA adapter, then we denoise the base and residual parts on slightly different schedules so the residual always builds on a clean foundation.

How does this adaptation differ from simply enlarging the DiT’s hidden dimension?

Enlarging the hidden dimension adds many new parameters and discards the pretrained weights, whereas our approach only adds a small projection layer and LoRA adapters that keep the original weights intact and cost only a few extra parameters.

Compute `u_base` = max(0.8 − 0.15, 0) = 0.65 and `u_res` = min(0.8, 1) = 0.8.

From Eq. 1 we obtain noise levels $\tau_{b}$ase ≈ 0.3 and $\tau_{r}$es ≈ 0.5.

Inject noise: `z_base` = (1 − 0.3) `z_base` + 0.3 $\epsilon_{b}$ase, `z_res` = (1 − 0.5) `z_res` + 0.5 $\epsilon_{r}$es.

Form modulation m = `W_mod` $\phi$(0.3) + `W_res`^{mod} $\phi$(0.5) (the second term starts at zero).

DiT predicts velocities \hat{v}_base and \hat{v}_res conditioned on m, and the flow‑matching loss is evaluated on both.

Because `u_base` < `u_res`, the base latent is always denoised earlier, giving the residual a cleaner context to refine.

Widen the DiT I/O projections to accommodate C + C' channels.

Insert LoRA adapters into every transformer block and freeze the original weights.

For each video, sample $\delta$ ∼ U($\delta_{m}$in, $\delta_{m}$ax) and \hat{u} ∼ U(0, 1 + $\delta$).

Derive `u_base` and `u_res` from \hat{u} and $\delta$, then compute the corresponding noise levels $\tau_{b}$ase, $\tau_{r}$es.

Noisify the base and residual latents separately using the two $\tau$ values.

Compute the modulation vector $m$ by mixing the pretrained `W_mod` with the zero‑initialized `W_res`^{mod}.

Run the adapted DiT to predict velocities for both latents and back‑propagate the average flow‑matching loss.

**Figure 4.** **Denoising order.** Each row shows the sampling steps of $z_{base}$ and $z_{res}$ (Left) and the generated frames (Right), with $\tau = 1$ pure noise and $\tau = 0$ clean. (Top) Both parts are denoised at the same noise level. (Bottom) $z_{base}$ stays $\delta$ ahead of $z_{res}$ in schedule time $u$ (Eq. 11) at every step.

Reconstruction Performance

GRACE matches the pretrained pipeline’s quality while using far fewer tokens and running much faster.

The central premise is that video diffusion models are limited by the sheer number of latent tokens; GRACE splits the latent into a compact base latent for structure and a residual latent for detail, letting the Diffusion Transformer work on a much smaller representation while preserving fidelity.

GRACE attains generation quality within 0.02 of the pretrained pipeline while using eight times fewer tokens and running eleven‑point‑one times faster.

Table 2 shows GRACE’s VBench‑T2V total score 85.85 versus the baseline’s 85.85, token count reduced from f8t4c16p2 to f16t8c32p2 (≈ 8×), and latency 138.4 s versus 851.5 s.

**Figure 5.** Qualitative comparison at $480 \times 832 \times 81$. VBench (Huang et al., 2023; 2024) samples from Wan2.1-14B (Wan et al., 2025) before compression and GRACE (Ours), generated from the same prompt, for text-to-video (top) and image-to-video (bottom). Best viewed when zoomed in.

Video Generation Results

GRACE speeds video generation up to 15.5× faster while preserving quality.

GRACE achieves up to 15.5× faster video generation at 736×1280×81 while staying within 0.04 of the pretrained pipeline on I2V.

Table 3 reports the latency reduction and comparable VBench scores; Table 2 shows an 11.1× speedup at lower resolution.

Quantitative comparison shows GRACE outperforms DC‑Gen on every VBench total despite using fewer tokens, and surpasses LTX‑Video on T2V while matching its I2V total.

**Figure 6.** Qualitative comparison at $736 \times 1280 \times 81$. VBench (Huang et al., 2023; 2024) samples from Wan2.1-14B (Wan et al., 2025) before compression and GRACE (Ours), generated from the same prompt, for text-to-video (top) and image-to-video (bottom). Best viewed when zoomed in.

Ablation Studies

We quantify how each design choice impacts video generation quality and training efficiency.

Adapting the Diffusion Transformer (DiT) to the compressed latent dominates training cost. Reusing a pretrained DiT cuts this cost dramatically, yet alone it cannot close the quality gap.

The dual latent component raises the T2V total score by +1.43.

Table 4, rows II–III show the improvement when the dual latent is added.

The offset component raises the I2V total score by +0.39.

Table 4, rows II–III report the gain from adding the offset.

Using pretrained DiT features as the alignment target increases the semantic score by +3.9.

Table 5 compares alignment targets; the pretrained DiT row achieves 82.55 versus 78.65 for “None”.

$L_{align}$ reduces the residual latent loss to 0.15, a 0.45 drop compared with no alignment.

Figure 7 shows the residual loss curve with and without $L_{align}$.

The denoising order $\tau_{base}<\tau_{res}$ (our design) achieves the highest total of 95.48.

Table 6 reports the total metric for each denoising schedule.

**Figure 7.** Convergence during DiT adaptation. With and without $\mathcal{L}_{align}$; solid lines show EMA.

Aesthetic Quality Analysis

We examine where GRACE’s dual‑latent design falls short across metrics and resolutions.

DC‑Gen outperforms GRACE on the aesthetic‑quality dimension in text‑to‑video, yet it lags behind on every aggregated total score. The LAION aesthetic predictor underlying this dimension aligns with photographic taste rather than fidelity to the prompt or conditioning image, so higher aesthetic scores reflect a shift in appearance that other dimensions penalize.

In text‑to‑video, GRACE achieves superior background consistency and lower temporal flickering at both 480×832 and 736×1280 resolutions, but it exhibits more motion than the pretrained pipeline (dynamic degree 62.50 vs 56.94 at 480×832). Higher motion generally harms these metrics, explaining GRACE’s advantage despite its greater movement.

For image‑to‑video at 480×832, LTX‑Video leads on background consistency, motion smoothness, and temporal flickering, aided by an extra guidance branch for temporal consistency. It also moves the least of all models (dynamic degree 24.80 vs 38.62 for GRACE), and lower motion correlates with higher scores on these dimensions.

GRACE preserves the pretrained pipeline’s color and style when generating text‑to‑video, whereas DC‑Gen often departs from that appearance. In image‑to‑video the conditioning image anchors the visual style, so the divergence between GRACE and DC‑Gen is less pronounced.

Tables D.2a and D.2b (resolution 736×1280×81) echo the earlier observations: DC‑Gen again leads aesthetic quality in text‑to‑video, while LTX‑Video attains the highest subject‑consistency score in image‑to‑video. Table D.1 provides the full per‑dimension VBench breakdown at 480×832×81, with the Single‑latent Baseline defined and the best scores highlighted.

**Figure A.1:** Detailed autoencoder architecture. $\mathcal{E}_{res}$ adds a downsampling stage between its middle blocks and head, consisting of residual blocks and a strided causal convolution, paired with a parameter-free shortcut that folds space and time into channels.

Latent Noise Analysis

Quantitative ablation results showing the impact of each latent component.

The following table aggregates the numeric outcomes of the ablation study; each row corresponds to a model variant and each column to a measured metric.

Per-Latent Reconstruction Analysis

Ablation and cost analyses reveal the base latent dominates reconstruction while residual latent drives adaptation.

We inject controlled noise into either the Base Latent or the Residual Latent while keeping the other clean, sweeping the interpolation factor $\tau$ from 0 (no change) to 1 (pure noise) and measuring PSNR and LPIPS on the Panda‑70M dataset.

Tracking the flow‑matching loss on the residual channels during DiT adaptation shows that the base channels converge identically with or without the alignment loss, while the residual channels converge faster and to a lower loss when alignment is applied.

**Table D.1.** Per-dimension VBench (Huang et al., 2023; 2024) scores at $480 \times 832 \times 81$. *Single-latent Baseline* is our baseline without the dual latent or the alignment loss. The best score in each row is in bold. $^\dagger$Reported for reference only, as the official VBench-I2V quality score does not include this dimension.

**Table 1.** Comparison of different video generation models across various dimensions, including Latent tokens, Latency, I2V metrics, and Quality metrics. The best results for each row are highlighted in bold.

**Table D.2.** Per-dimension VBench (Huang et al., 2023; 2024) scores at 736x1280x81, following Tab. D.1. The best score in each row is in bold. ‡Measured with spatial tiling in the VAE encoder to avoid running out of memory when encoding the conditioning image at this resolution.

**Table 1.** Comparison of different video generation models across various dimensions, including latent tokens, latency, I2V metrics, and quality metrics.

**Table D.3.** Results of the blind user study comparing our method with Wan2.1-14B and DC-Gen across T2V and I2V tasks, evaluated on visual quality, temporal consistency, and text alignment.

Limitations include loss of fine details for small objects under compression, a 1.13 dB PSNR gap to the single‑latent baseline (Table 1), and the need for an extra Stage 1 autoencoder training run (≈8.5 H200 GPU days) when adapting GRACE to a new pretrained pipeline.

Autoencoder Architecture Details

Appendix A details the GRACE autoencoder architecture and its training setup.

The GRACE autoencoder splits video information into a Base Latent and a Residual Latent, each processed by a dedicated encoder and combined in a decoder that extends the original video encoder.

Base encoding obtains $z_{base}$ from the frozen encoder $E$, which first downsamples frames by a spatial factor $r_s$ and a temporal factor $r_t$ using bilinear interpolation.

Residual encoding uses a second encoder $E_{res}$ with the same architecture as $E$, adds an extra downsample block between its middle blocks and head, and includes a non‑parametric shortcut that compresses both space and time into the channel axis.

The decoder $\hat{D}$ expands the input convolution from $C$ to $C + C'$ channels, adds a symmetric upsample block, and incorporates a gated cross‑attention that injects the first video frame encoded by $E_{res}$.

The single‑latent baseline removes the Base/Residual split, using a single encoder without any reduction of the input, while keeping the same compression block, shortcut, and first‑frame cross‑attention unchanged.

Training proceeds in two phases: Phase 1 updates $E_{res}$ and $\hat{D}$ at $256\times256\times81$ resolution, and Phase 2 raises the resolution to $512\times512\times81$ while freezing $E_{res}$.

Table A.1 lists the hyperparameters for both phases, including learning rates of $8\mathrm{e}{-5}$ and $4\mathrm{e}{-5}$, batch sizes of 32 and 64, and the same optimizer settings (AdamW, betas $(0.9,0.999)$, weight decay $1\mathrm{e}{-4}$).

The loss combines $L_{1}$, $L_{LPIPS}$, and a tiny $L_{KL}$ term, with $L_{align}$ applied only in Phase 1 to align the latent to the pretrained Diffusion Transformer (DiT).

The baseline trains solely with $L_{recon}$, omitting $L_{align}$ and keeping all other hyperparameters identical to the dual‑latent setup.

Diffusion Model Details

Detailed architectural and implementation specifics for the GRACE diffusion model.

We reuse the Wan2.1 DiT and only modify components that must accommodate the compressed latent. The input and output projections are extended along the channel axis; the mask width follows the temporal compression factor, doubling with the latent, and newly added channels are zero‑initialized while copying pretrained weights into the original channels.

Because compression spreads tokens farther apart than the DiT expects, we rescale the rotary positional encoding (RoPE) indices by $r_s$ in space and $r_t$ in time, restoring the token spacing the pretrained model was trained on.

The base and residual latents are denoised at different noise levels, so the DiT receives two timesteps; we add a second modulation projection $W_{\text{res}}^{\text{mod}}$ for the residual timestep and sum the two modulations, while each output head is conditioned separately on its own timestep. The Base Latent stores coarse structural information, whereas the Residual Latent carries fine‑grained detail. The single‑latent baseline extends the projections similarly but does not split the latent, using only one timestep and omitting $W_{\text{res}}^{\text{mod}}$.

During Stage 2 we keep the autoencoder frozen and standardize each latent part with statistics measured on the training set, following the pretrained pipeline. We fully fine‑tune the input projection, the output heads, and $W_{\text{res}}^{\text{mod}}$, together with the modulation and normalization parameters of every block. LoRA is applied to all linear layers in the attention and feed‑forward blocks, as well as to the key and value projections of the image cross‑attention for image‑to‑video.

We train on the Panda‑70M dataset at a resolution of 480×832×81 using AdamW, with hyperparameters summarized in Table B.1 (e.g., pretrained DiT input dim 72/32, hidden dim 5120, 40 blocks, 40 heads, patch size (16, 16), learning rate 1e‑4, effective batch size 128, offset $\delta$ ∈ [0, 0.6], shift $s$ ∈ [2, 5]). Text conditioning is dropped with probability 0.1 for classifier‑free guidance.

Training Hyperparameters

Implementation details, evaluation protocols, and extra results for the appendix.

Initialization. We keep the pretrained weights for the base channels, and zero‑initialize the weights for the residual channels along with $W_{res}$ mod.

Denoising schedule. We sample the offset per video rather than fixing it, so a single model covers a range of offsets at inference. The shift $s$ of Eq. 1 is sampled the same way, and we find that a lower shift, which spends more sampling steps near the clean end, recovers detail better on the compressed latent.

The values are listed in Table B.1. At inference, we run the same 50 sampling steps as the synchronous schedule over $\hat{u} \in [0, 1 + \delta]$, and $\delta = 0$ recovers the synchronous schedule exactly. In the first few steps, only the base is denoised while the residual stays at pure noise, and in the last few, only the residual is denoised while the base is kept fixed until the final step.

The schedule uses the following parameters: 50 steps, guidance scale 5.0, offset $\delta$ = 0.15, shift $s$ = 3.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers