LongE2V: Long-Horizon Event-Based Video Reconstruction, Prediction, and Frame Interpolation with Video Diffusion Models

Cheng-De Fan, Chun-Wei Tuan Mu, Chen-Wei Chang, Chin-Yang Lin, Kun-Ru Wu, Yu-Chee Tseng, Yu-Lun Liu

LongE2V uses video diffusion priors to unify event-based video reconstruction, prediction, and frame interpolation.

How can we leverage pre-trained video diffusion models to reconstruct high-fidelity, long-horizon video from sparse event streams while avoiding the blurriness of traditional regression methods?

Event cameras capture high-speed dynamics as sparse, intensity-free streams, making high-fidelity video recovery an ill-posed problem prone to blur or temporal drift. LongE2V fine-tunes a pre-trained video diffusion model to treat these tasks as conditional generation, using event voxels as explicit motion guidance. The method outperforms state-of-the-art baselines across reconstruction, prediction, and interpolation, maintaining temporal coherence even in long-horizon sequences.

Paper Primer

The core challenge is that traditional regression-based methods for event-to-video conversion suffer from "regression-to-the-mean," which blurs textures, while existing generative models struggle with long-term stability and error accumulation. LongE2V addresses this by leveraging a pre-trained video diffusion model (CogVideoX) and introducing mechanisms to manage temporal dependencies and alignment.

LongE2V is a unified framework: it uses Autoregressive Unrolling and Adaptive Context Switching to prevent drift during long-horizon generation, and Reencoding Alignment with Cross Residual Correction to ensure bidirectional consistency during frame interpolation. The model essentially treats event streams as a dense motion guide, injecting them into the diffusion transformer's latent space to synthesize frames.

LongE2V achieves superior perceptual quality and temporal stability compared to regression-based and existing generative baselines.

Quantitative results on ECD, MVSEC, and HQF datasets show consistent improvements in PSNR, SSIM, and LPIPS metrics across reconstruction and prediction tasks. In zero-shot frame interpolation, the method significantly outperforms dedicated supervised baselines like CBMNet-Large and TLXNet+, preserving high-frequency details where others produce ghosting or blur.

Why use a diffusion-based approach instead of the standard recurrent architectures like E2VID?

Regression-based methods like E2VID are prone to blurring textures due to "regression-to-the-mean." Diffusion models allow for generative synthesis that preserves high-frequency details and handles complex dynamics more effectively.

How does the model handle the "drift" that typically plagues long-term video generation?

The model uses Adaptive Context Switching, which monitors attention weights between current and context tokens. If relevance drops below a threshold, the model discards the current generation and updates the context to the immediate predecessor, preventing the accumulation of errors.

LongE2V demonstrates that pre-trained video diffusion priors can be efficiently adapted for neuromorphic sensing, providing a flexible, unified solution for multiple event-based vision tasks without requiring task-specific architectures.

Motivation and Problem Framing

We frame the key challenges of event‑based video generation and motivate LongE2V.

Event cameras capture asynchronous brightness changes with microsecond resolution, avoiding motion blur but producing sparse, intensity‑free streams that are ill‑posed for standard vision pipelines. Existing regression‑based reconstructions blur textures, while diffusion‑based generators accumulate errors over long horizons, leading to temporal drift and ghosting artifacts in interpolation.

We need a unified approach that can reconstruct high‑fidelity video, predict long sequences without drift, and interpolate frames zero‑shot, all from sparse event data.

When a regression model averages over many plausible reconstructions, it collapses to a bland middle ground, erasing fine textures and causing blurry outputs.

**Fig. 1. Event-based video generation.** We leverage pre-trained video diffusion priors to address three distinct inverse problems within a single architecture. Depending on the input condition, our model performs: (a) **Video Reconstruction**, recovering high-fidelity textures from sparse event streams, (b) **Video Prediction**, generating long-term sequences from a single start frame with minimal drift via our autoregressive unrolling strategy, and (c) **Video Frame Interpolation**, achieving zero-shot adaptation to synthesize intermediate frames by leveraging event dynamics as temporal guidance.

**Fig. 2.** Challenges in event-based video generation. We highlight failure cases in state-of-the-art methods: (a) Reconstruction: Regression-based methods (e.g., E2VID+[Stoffregen et al. 2020]) suffer from “regression-to-the-mean,” causing **blurry textures** and detail loss. (b) Prediction: Direct video diffusion (e.g., VDM-EVFI[Chen et al. 2025a]) on long sequences suffers from error accumulation, leading to severe **color temporal drift**. (c) Interpolation: Existing networks (e.g., CBMNet-Large [Kim et al. 2023]) fail to capture complex intermediate motion, producing significant **ghosting artifacts**. Our method leverages video diffusion priors to ensure high-fidelity and stable generation across all tasks.

Regression fails for long‑horizon reconstruction because averaging over many plausible futures erodes texture detail and amplifies drift.

Prior Approaches and Baselines

We survey event‑based video reconstruction, interpolation, and long‑term generation methods, highlighting their evolution and limitations.

Event‑based cameras produce streams of brightness changes, making intensity reconstruction ill‑posed. Early optimization pipelines gave way to deep learning, starting with recurrent U‑Net baselines such as E2VID.

E2VID+ extends the original recurrent U‑Net by adding a lightweight attention module that refines temporal features while keeping inference cost low.

TLXNet+ builds on the transformer‑based TLXNet by incorporating a hierarchical token‑mixing scheme that better captures the sparse spatio‑temporal structure of event data.

Event‑based Video Frame Interpolation (EVFI) exploits the microsecond temporal resolution of event streams to synthesize intermediate frames. Recent works fall into warping‑synthesis hybrids, flow‑based cycle‑consistency, adaptive computation, and diffusion‑driven direct synthesis.

Recurrent U‑Net trained on synthetic event data; established the first deep baseline for intensity reconstruction from events.

Introduces lightweight gating mechanisms and channel pruning to reduce inference latency without sacrificing reconstruction fidelity.

Bridges the domain gap between synthetic training data and real‑world event streams via adversarial feature alignment.

Replaces recurrent processing with multi‑head self‑attention, allowing each event to attend globally across time.

Leverages contrastive objectives on temporally shuffled event patches to learn robust representations without labeled video.

Maps event spikes directly to spiking neurons, preserving the asynchronous nature of the sensor.

Uses a generative adversarial framework to produce sharper intensity maps, mitigating blur introduced by pixel‑wise losses.

Conditions diffusion generation on textual descriptions, enabling semantic control over reconstructed frames.

Adapts video diffusion models to the sparse event domain, learning to denoise event‑derived frames iteratively.

Predicts residual frames between consecutive event‑derived reconstructions, reducing the diffusion model’s workload.

Combines optical flow warping of existing frames with a synthesis network that fills occluded regions.

Enforces forward and backward flow consistency between synthesized intermediate frames, improving temporal coherence.

Allocates more processing resources to regions with dense event activity, using a learned importance map.

Guides hierarchical feature alignment with fine‑grained event cues, yielding sharper interpolated frames.

Fine‑tunes a video diffusion model on event‑derived frames to generate intermediate frames directly, bypassing explicit flow.

Long‑term video generation faces error accumulation over many frames. Prior works address this via memory‑augmented or streaming architectures, training‑free noise rescheduling, and techniques that narrow the train‑inference gap.

Introduce external memory modules that store latent representations, allowing the generator to retrieve past context without back‑propagating through all frames.

Alters the diffusion noise schedule at inference time to reduce accumulated sampling error without retraining.

Aligns training objectives with inference-time sampling strategies, reducing distribution shift that causes drift in long sequences.

Fine‑tunes a diffusion model by unrolling multiple future steps during training, teaching the model to anticipate its own predictions.

Dynamically updates the conditioning context based on a learned confidence signal, preventing the model from drifting when predictions become uncertain.

The LongE2V Architecture

We describe the core autoregressive unrolling mechanism that aligns events with video diffusion.

Naïve long video generation reuses the last generated frame as the seed for the next chunk, which quickly compounds errors and produces drifting artifacts.

CogVideoX I2V is a diffusion‑based video generator that first compresses raw video into a latent space with a 3‑D VAE, then denoises those latents with a Diffusion Transformer (DiT).

Think of the model as a student who first learns from solved examples (ground‑truth frames) and then practices by solving its own problems, using its own answers as the next inputs.

Step 1: Train with $C_0, C_1$ as conditioning; the model learns to denoise latent $Z_t$ to match $Z_0$.

Step 2: Run an inference pass to produce a prediction $\hat{F}_2$ from the learned model.

Step 3: Replace $C_2$ with $\hat{F}_2$ and continue training; the model now sees its own output as context.

Step 4: Generate $\hat{F}_3$ using $\hat{F}_2$ as part of the conditioning, completing the 4‑frame sequence.

The model learns to correct its own drift because each prediction is immediately used as context, exposing it to the distribution of its own errors.

Iterative training loop implementing Autoregressive Unrolling.

After decoding latents to pixel space, we flip the frames temporally, then re‑encode them so that forward and backward branches share a common temporal reference.

**Fig. 3. Autoregressive Unrolling.** To bridge the domain gap between training and inference, we employ an iterative training strategy. Initially the model is trained with Ground Truth (GT) context frames for convergence (left). Subsequently, we activate the unrolling mechanism by performing an inference pass to generate predictions, which then replace the GT context frames for fine-tuning (right). This iterative feedback loop forces the model to adapt to its own generation errors, mitigating error accumulation during long video generation.

**Fig. 5. Reencoding Alignment and Cross Residual Correction.** To address temporal misalignment caused by the discrepancy between latent-space and pixel-space flipping, we propose Reencoding Alignment. The denoised latents, $\hat{Z}_0^{fwd}$ and $\hat{Z}_0^{bwd}$, are decoded into pixel space, flipped temporally ($Flip_{pix}$), and then re-encoded via the 3D VAE to yield the aligned latents $\tilde{Z}_0^{fwd}$ and $\tilde{Z}_0^{bwd}$. To mitigate information loss inherent in this re-encoding process, we employ Cross Residual Correction. We calculate the residual difference between the original and the re-encoded latents (e.g., the subtraction node $\hat{Z}_0^{fwd} - \tilde{Z}_0^{fwd}$) and inject this detail information into the opposite branch. Specifically, the forward residual is added to the backward aligned latents $\tilde{Z}_0^{bwd}$ to produce the final corrected latents $Z_0'^{bwd}$, and symmetrically, the backward residual is injected into $\tilde{Z}_0^{fwd}$ to obtain $Z_0'^{fwd}$. This symmetric Cross Injection mechanism promotes temporal consensus between branches while preserving fine-grained details. Light-colored boxes represent information loss.

Training and Augmentation Strategy

The pipeline augments event voxels, aligns multimodal latents, and fine‑tunes a diffusion transformer with LoRA.

Sensor variation can corrupt raw event voxels, causing unstable training. Our pipeline first augments voxel density, then aligns multimodal latents before fine‑tuning the diffusion transformer with LoRA.

We artificially increase the density of sparse event voxels to mimic sensor noise and improve model robustness.

Duplicate the event to its immediate neighbor $(0,0,1)$, yielding two events.

Apply Gaussian blur: each of the two voxels now has value 0.6, and their eight surrounding voxels receive 0.1.

Clamp values: all voxels remain below the 255‑count ceiling.

Higher voxel density gives the model more signal to learn from, reducing collapse to blank frames.

How does this differ from naïve up‑sampling of events?

Simple up‑sampling merely repeats events, preserving the original sparsity pattern. Our augmentation adds stochastic blur and local duplication, creating richer, more varied patterns that better simulate real‑world sensor noise.

Apply Event Voxel Density Augmentation to each input voxel stream.

Resize and crop the first frame, context frames, and current video frames to a common resolution.

Encode all streams with a frozen 3D VAE, producing latent tensors.

Concatenate latents along the frame dimension (F) then along the channel dimension (C).

Expand the First Projection Layer to accept the additional event channel.

Fully fine‑tune the expanded projection layer while applying LoRA to the DiT blocks.

Compute loss on $Z_t$; apply 5 % dropout to $Z_{x0}$; inject text prompts with 20 % probability.

**Fig. 4. Overview of our training pipeline.** To enhance robustness against sensor variations, input event voxels undergo Event Voxel Density Augmentation, and the first frame, context frames, and current video frames are synchronously resized and cropped to ensure spatial alignment. All inputs are encoded into latents via a frozen 3D VAE. These latents are aligned and fused through frame dimension concatenation and channel dimension concatenation. Finally, we expand and fully fine-tune the First Projection Layer to accommodate the additional event channels, while the DiT backbone is efficiently fine-tuned using LoRA.

The model decides whether to keep the current context or replace it based on how much attention current tokens pay to context tokens.

Why not simply use a fixed context length instead of this adaptive switch?

A fixed length cannot react to varying motion dynamics; the adaptive metric measures actual relevance, discarding stale context only when attention indicates it is no longer useful.

Why can’t we just flip the latent tensor without decoding?

Because the VAE’s temporal compression discards ordering information; decoding restores the true pixel chronology before flipping, ensuring the re‑encoded latent aligns with the opposite temporal direction.

We inject the residual from the opposite temporal branch to compensate for information lost during the decode‑encode loop.

What would happen if we omitted the residual injection?

The decode‑encode loop would introduce a reconstruction loss, leading to blurred or missing high‑frequency details and reduced temporal consistency between forward and backward branches.

Empirical Evaluation

LongE2V adapts a diffusion transformer to turn event streams into high‑fidelity, temporally stable video, avoiding regression blur.

We first recap the core premise: regression‑based event reconstruction blurs textures and fails over long horizons, while LongE2V leverages a pretrained diffusion transformer to produce sharp, temporally consistent video from sparse event streams.

LongE2V attains the highest SSIM across all three datasets, improving over the best regression baseline by +0.023 SSIM.

Table 1 shows baseline SSIM = 0.685 (E2VID) versus LongE2V SSIM = 0.708.

**Table 1.** Quantitative results on ECD [Mueggler et al. 2017], MVSEC [Zhu et al. 2018], and HQF [Stoffregen et al. 2020]. Comparing our method against SOTA baselines (Red: best; blue: second). In Reconstruction, we consistently achieve the best LPIPS scores, indicating superior perceptual quality compared to regression-based methods. In Prediction, our method significantly outperforms VDM-EVFI across all metrics and datasets, validating our robust long-term generation capabilities.

**Fig. 6.** Qualitative comparisons on ECD [Mueggler et al. 2017], MVSEC [Zhu et al. 2018], and HQF [Stoffregen et al. 2020] datasets. Our LongE2V recovers high-frequency textures where regression baselines (E2VID+, HyperE2VID) suffer from blurring (Row 1). In prediction tasks, we avoid the severe noise accumulation and ghosting artifacts (red arrow) seen in VDM-EVFI (Rows 2–3), maintaining superior structural fidelity and temporal stability.

**Fig. 7. Zero-shot interpolation on BS-ERGB and HQF.** Baselines (TLXNet+, CBMNet-Large) suffer from structural collapse or blur under large motion (Top), whereas our LongE2V captures accurate dynamics. On fine text (Bottom), our Reencoding Alignment eliminates the ghosting seen in baselines, ensuring legibility.

**Fig. 8. Qualitative ablation on reconstruction.** Top: Ablated variants; Bottom: Full Method. w/o Pretrained prior yields noise; w/o Context causes structural ambiguity; w/o Autoregressive unrolling leads to point artifacts (drift); and w/o Adaptive context switch causes grid artifacts. Our method ensures stable, high-fidelity results.

**Fig. 9. Visual ablation on interpolation.** w/o Reencoding Alignment causes ghosting due to latents misalignment; w/o Cross Residual Correction blurs fine details due to VAE loss; and w/o Event Voxel Density Augmentation yields artifacts from density mismatch. Our Full Method restores sharp, coherent details comparable to Ground Truth.

Ablations and Limitations

Ablation results quantify each component’s impact on video reconstruction quality.

This section isolates the contribution of each design element by removing it and measuring the resulting drop in quality.

Our zero‑shot EVFI method outperforms supervised baselines on perceptual quality.

LPIPS improves from 0.170 (CBMNet‑Large) and 0.166 (TLXNet+) to 0.124.

Removing Reencoding Alignment degrades perceptual fidelity.

LPIPS rises from 0.124 to 0.202 when the component is omitted.

Removing Cross Residual Correction harms perceptual quality.

LPIPS increases from 0.124 to 0.161 without this correction.

Omitting Event Voxel Density Augmentation slightly reduces perceptual quality.

LPIPS grows from 0.124 to 0.129 when the augmentation is absent.

**Fig. 13. Limitations.** (a) Failure cases under sparse or low-quality event streams. (b) Sensitivity to noise where "hot pixel" is preserved or amplified in the reconstructed frames.

**Fig. 10. Text-Guided Event Video Colorization.** *Left to Right*: Input events; standard reconstruction (geometry baseline); text-stylized frames; and a temporal XT-slice (at red line). The results demonstrate our model effectively decouples event-driven motion from text-defined appearance, while the XT-slice confirms the generated textures are temporally coherent.

**Fig. 12.** Additional zero-shot interpolation results on BS-ERGB [Tulyakov et al. 2021] and HQF [Stoffregen et al. 2020] datasets.

Supplementary Details

Supplementary material, tables, and figures that further illustrate the paper’s results.

The appendix gathers extra quantitative tables and qualitative figures that complement the main experiments, highlighting the superiority of our reconstruction and prediction pipelines across multiple metrics and datasets.

**Table.** Subject Consistency comparison.

**Table 5.** Ablation of Different Backbones on HQF dataset.

**Fig. 11.** Additional qualitative comparisons on ECD [Mueggler et al. 2017], MVSEC [Zhu et al. 2018], and HQF datasets [Stoffregen et al. 2020]

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers