FlowMimic: Mask-Free Visual Editing and Generation with Pixel-Pair Warped Flow Field for Online Video Editing Data Generation and Modality Mimicry

Dingyun Zhang, Lixue Gong, Wei Liu

FlowMimic generates video editing training data in real-time by warping image-editing pairs into temporally coherent pixel-flow sequences.

How can we train a single model to perform both video generation and mask-free video editing by aligning video and image modalities through a pixel-pair warped flow field?

Video editing models currently rely on labor-intensive, curated datasets that require complex mask annotations or error-prone synthetic video generation. This limits task scalability and forces models to depend on external aids like auxiliary mask inputs during inference. FlowMimic bypasses this by treating images as single-frame videos and applying a shared, time-varying deformation field to both source and target images. This "pixel-pair temporal warped flow field" forces the model to learn consistent pixel-level editing correspondences across time without needing pre-curated video pairs. The model achieves effective, region-aware video editing across diverse tasks using only image-editing samples, while aligning the output distributions of image and video modalities through mutual imitation losses.

Paper Primer

The core mechanism hinges on a 4D temporal warped flow field: the system applies identical, randomized geometric transformations (pans, zooms, elastic distortions) to both the source and target images of an editing pair. By forcing these pixel-pairs to "walk" synchronously through time, the model learns to maintain editing effects across frames as a natural consequence of spatial correspondence.

FlowMimic achieves effective video editing across multiple categories using only image-editing samples and online-generated video counterparts.

Post-training on a pretrained Text-to-Image-Video (T2IV) model (Wan2.1-T2V-1.3B) using this data paradigm. The model exhibits responsiveness to video editing tasks within only a few thousand training steps.

To internalize region-aware editing without external masks, the authors introduce "sense-related" tasks—such as referring-expression segmentation—paired with latent-level and attention-level losses. This allows the model to localize textual instructions to specific visual regions and modify only those areas during inference.

Why does the model not require real-world coherent video pairs for training?

The authors posit that temporal consistency of pixel-level editing correspondences is sufficient for learning. Even if objects deform unnaturally in synthetic training sequences, the model learns to maintain frame-consistent, pixel-aligned editing relationships that generalize to real-world video at inference.

How does this approach differ from existing training-based video editing methods?

Most existing methods rely on auxiliary mask sequences or fine-tuned MLLM connectors to localize editing regions. FlowMimic internalizes this capability by training on sense-related tasks, enabling the model to perform region-aware editing without explicit mask inputs during inference.

Researchers can now scale video editing training by leveraging existing, massive image-editing datasets, effectively treating video editing as a temporal extension of pixel-aligned image editing.

Introduction and Motivation

We expose the video editing gap and outline FlowMimic’s mask‑free solution.

Current video‑editing pipelines rely on costly mask annotation, error‑prone synthesis, and multi‑stage filtering, which limits task diversity. This raises two questions: can we obtain video‑editing data without such pipelines, and must that data be human‑intelligible temporally?

Training video‑editing models requires massive, task‑specific video pairs, but existing collection methods are labor‑intensive and poorly scalable, leaving video editing far behind image editing in diversity and capability.

Our naive motivation is to let each spatially corresponding pixel‑pair “travel” synchronously through time while preserving its spatial correspondence, thereby exposing the model to clusters of point flows that encode the temporal editing effect.

**Figure 1.** Overview of the video editing pair generation paradigm with pixel-pair temporal warped flow field. For any given image editing training sample of a task, we can obtain video editing samples in real time via this field that enable the model to learn the video editing capability for this task. Such video editing samples align with the model’s cognitive patterns, eliminating the need to adapt them to conform to human perception. In this paper, all video editing tasks of interest are trained jointly only with our online generated video samples; the bottom-right corner illustrates an editing result for object insertion. Please refer to Sec. 4.3 for more results.

Our contributions are threefold: (1) a scalable pixel‑pair temporal warped flow field that creates video editing pairs in real time from image edits; (2) modality‑mimic generation and editing losses that align image and video output distributions; (3) sense‑related tasks and region‑aware losses that endow the model with mask‑free, instruction‑driven editing capabilities.

The shift from mask‑dependent pipelines to mask‑free, online video‑editing data generation enables unified image‑video editing without extra annotation.

Pixel-Pair Temporal Warped Flow Field

We introduce a pixel‑pair temporal warped flow field that generates coherent video edits from single‑image edits.

Generating video‑editing training data on‑the‑fly is expensive because each video must preserve the exact pixel‑wise edit relationship across time. The naive approach of applying independent random transforms to each frame quickly breaks that correspondence, leading to noisy supervision. Our solution is to warp both source and target images with a shared, time‑varying deformation that guarantees the edit mapping stays aligned.

We treat the source‑target image pair as two sets of matching pixels and apply the *same* spatial deformation to both over time, so every frame inherits the exact edit mapping established in the first frame.

Frame 0: $G_0 = G_{\text{base}}$ (no displacement).

Frame 1: $\tau = 1/(F-1)=0.5$, so $G_1 = G_{\text{base}} + 0.5\begin{bmatrix}1\\0\end{bmatrix}$ shifts every coordinate right by 0.5.

Frame 2: $\tau = 1$, so $G_2 = G_{\text{base}} + \begin{bmatrix}1\\0\end{bmatrix}$ shifts right by 1 pixel.

Apply the same three grids to both source and target images via $\operatorname{Sample}$, producing three aligned frames where the edited region slides right uniformly.

Because the same grid sequence is used for source and target, the pixel‑wise edit relationship is preserved even though the whole scene moves.

How does the Pixel‑Pair Temporal Warped Flow Field differ from conventional optical flow?

Optical flow estimates a per‑pixel motion field *between* two existing frames, often noisy and asymmetric. Our flow field is *synthetically generated*: we prescribe a single deformation sequence and apply it identically to both source and target images, guaranteeing perfect forward‑backward correspondence and eliminating the need for any learned motion estimation.

Wan2.1 is a Text‑to‑Image‑Video (T2IV) model that builds on Diffusion Transformers, encoding video latents as spatio‑temporal tokens and denoising them with cross‑attention to a text caption.

Sample an image‑editing pair $(I_{\text{ref}1}, I_{\text{tar}})$ from the curated dataset.

Build base grids $G_{\text{ref}1}$ and $G_{\text{tar}}$ from the normalized coordinate space of the pair.

Select a motion type $M$ (e.g., pan‑right, zoom‑in) and draw a random seed $S$.

For each frame $t\in\{0,\dots,F\!-\!1\}$ compute $G_t = f_\alpha(G_{\text{base}}, M, t, S)$.

Apply the same $G_t$ to both source and target via $\operatorname{Sample}$, yielding $I_{\text{ref}1}^{\,t}$ and $I_{\text{tar}}^{\,t}$.

Collect the $F$ frames into a video‑editing sample ready for flow‑matching training.

Motion Types and Data Generation

Defines motion primitives that warp pixel grids to generate diverse video editing training data.

Generating video editing data requires realistic, controllable motion; without diverse motion primitives the model would only see trivial frame‑to‑frame changes and fail to learn robust editing.

Rotates every point around the image origin, mimicking a camera pan that keeps the scene geometry intact.

At $t=0$, $\theta(0)=0^\circ$, so $R_{\theta(0)}$ is identity; $G_0$ equals $G_{\text{base}}$.

At $t=0.5$, $\theta=45^\circ$; rotate each point to $(\tfrac{-y+x}{\sqrt2},\tfrac{x+y}{\sqrt2})$.

At $t=1$, $\theta=90^\circ$; points become $(-y, x)$, yielding $G_1 = \{(0,0),(-1,0),(0,1),(-1,1)\}$.

Rotation preserves pairwise distances, so the model sees a pure camera‑orbit without stretching the content.

How does rotation differ from a simple translation that also moves pixels?

Translation adds a constant offset to every coordinate, leaving the orientation unchanged; rotation multiplies coordinates by a matrix, causing points to follow circular arcs and preserving the shape of objects while changing their orientation.

Locally expands or contracts regions around sampled centers, emulating object‑level deformations such as stretching a rubber band.

Compute $r_1$ for each corner: $\sqrt{0.5^2+0.5^2}=0.707$.

Attenuation $\gamma_1=1-0.707/1=0.293$.

Displacement $\mathbf{D}_1 = 0.4 \times 0.293 \times (\mathbf{p}-\mathbf{c}_1)$, moving corners outward by ≈0.12.

Resulting grid points shift slightly away from the center, creating a gentle local stretch.

Only points inside the radius feel the deformation; the effect fades smoothly to zero at the boundary.

Why not use a global scaling factor instead of these localized deformations?

Global scaling uniformly stretches the whole image, which cannot capture region‑specific shape changes; local stretch allows independent manipulation of separate objects while leaving the rest of the scene untouched.

Applies piecewise‑linear anisotropic scaling to the entire image, reproducing smooth zoom‑in/out or aspect‑ratio changes over time.

Frame 1 ($\tau=0$ of segment 1): stretch $(1,1)$ → identity.

Frame 2 ($\tau=1$ of segment 1): stretch $(1.2,1.0)$ → horizontal zoom.

Frame 3 ($\tau=0$ of segment 2): stretch $(1.2,1.0)$ → carries over.

Frame 4 ($\tau=1$ of segment 2): stretch $(1.0,0.8)$ → vertical compression.

Linear interpolation guarantees no sudden jumps, so the model sees a temporally smooth zoom trajectory.

How does anisotropic scaling differ from isotropic scaling, and why is the distinction important?

Isotropic scaling uses the same factor for $x$ and $y$, preserving aspect ratio; anisotropic scaling allows independent $x$ and $y$ factors, enabling effects like horizontal pan‑zoom or vertical squash that better mimic real camera motions.

Generates smooth, fluid‑like random displacements that evolve over time, resembling natural deformations such as water ripples.

Convolve each channel with $G_{\sigma}$ → smoothed field $D(0) \approx [[-0.2,0.0],[0.0,-0.2]]$.

Scale by $\alpha = 2 / 2 = 1$ and $\tau=0$ → no displacement at first frame.

At $t=1$, $\tau=0.5$, repeat sampling → $D(1)$ yields non‑zero displacement added to $G_{\text{base}}$.

Resulting grid shows a gentle shear‑like distortion that varies smoothly between frames.

Because the same random seed is used for source and target, the displacement fields are identical, preserving pixel‑wise correspondence.

Why not simply add raw noise without Gaussian smoothing?

Raw noise creates high‑frequency jitter that breaks spatial coherence, making the motion look unrealistic; Gaussian smoothing ensures neighboring pixels move together, yielding fluid‑like deformations.

Chains two or more basic motions sequentially, allowing the model to observe combined effects such as a pan plus a local stretch.

Apply local stretch → points near the center move outward by ≈0.06.

Add pan displacement $\tau\,\alpha_p\,\mathbf{e}_x$ (horizontal shift) for each frame.

Resulting grid shows both a gentle outward bulge and a steady rightward drift.

Because both components are continuous, the composite motion remains smooth without abrupt jumps.

Could the order of applying $W_1$ then $W_2$ change the final result?

Yes; many motions are not commutative (e.g., stretching then rotating yields a different shape than rotating then stretching). The paper fixes an order to keep the transformation deterministic.

Each generated video frame inherits a textual caption that appends a short description of the motion, e.g., “The camera pans left.” This motion caption is concatenated with the original editing caption during training.

Motion‑strength hyperparameters control how aggressive the deformation is: low strength yields smooth, high‑frame‑rate‑like transitions, while high strength produces pronounced, low‑frame‑rate‑like changes that accelerate convergence but favor rigid edits.

When video pixels are flattened into visual tokens, the diffusion model receives paired source‑target videos that maintain exact pixel‑wise correspondence across time, exposing a regular pattern that the model can learn to track.

Modality Mimicry

Modality Mimicry aligns video and image editing through a unified loss on the first frame, enabling mask‑free video editing.

Training video‑editing models requires massive paired video data, which is costly to collect. The authors therefore generate training pairs online by warping image‑editing pairs across time, but the naive approach still leaves a gap between image‑only and video‑aware learning. The trick is to treat the first video frame as an image and enforce that the model mimics image‑editing behavior on that frame, while still learning temporal dynamics.

**Figure 2. Overview of the pretrained T2IV model [18] and our modality mimic generation loss.** The area to the left of the grey dashed line provides a schematic overview of our adopted pretrained T2IV model, while the area to the right introduces the modality mimic generation loss used in our generation training step. The DiT blocks are shared components across both sides of the dashed line. In the schematic illustrating the modality mimic generation loss, we present the newly created T2I sample as the $(M + 1)$-th sample in the current training batch. In our implementation, this new sample replaces the first T2I sample in the current training batch. The flame symbol denotes that the parameters in those parts are optimized during training, whereas the ice denotes that the parameters in those parts remain frozen. Please refer to Sec. 3.1 and Sec. 3.3 for details, respectively.

We make the model treat the first video frame exactly like an image‑editing sample, forcing it to reproduce the same pixel‑wise edit that an image‑to‑image model would produce.

How does Modality Mimicry differ from ordinary image‑to‑image translation?

Image‑to‑image translation only learns a static mapping between two images. Modality Mimicry embeds that static mapping as a constraint on the first frame of a video, while the rest of the frames are still driven by the temporal flow field. Thus the model learns both a per‑frame edit and a coherent motion trajectory.

Base grid $G_{\text{base}}$ contains normalized coordinates $\{(0,0),(0,1),(1,0),(1,1)\}$.

For $t=0$, $\tau=0$, so $G_{0}=G_{\text{base}}$; sampling yields the original source frame $I_{\text{src}}^{0}=I_{\text{src}}$.

For $t=1$, $\tau=0.5$, the pan grid is $G_{1}=G_{\text{base}}+\tau\begin{bmatrix}\alpha\\0\end{bmatrix}=G_{\text{base}}+\begin{bmatrix}0.25\\0\end{bmatrix}$; sampling produces a frame shifted half a pixel right.

For $t=2$, $\tau=1$, the pan grid is $G_{2}=G_{\text{base}}+\begin{bmatrix}0.5\\0\end{bmatrix}$; sampling yields the source image shifted one pixel right (with border reflection).

The same grids are applied to $I_{\text{tgt}}$, so each frame of the target video preserves the exact pixel‑wise edit established between $I_{\text{src}}$ and $I_{\text{tgt}}$.

Because the deformation sequence is identical for source and target, the pixel‑wise edit learned on the first frame automatically propagates to every subsequent frame, eliminating the need for per‑frame masks.

To make the model aware of which region the instruction refers to, a referring‑expression segmentation head is added; its loss operates both on latent features (latent‑level) and on attention maps (attention‑level), encouraging the model to focus on the correct region during editing.

Minor architectural tweaks—such as adding a lightweight cross‑attention bridge between the image‑editing branch and the video branch—allow the shared DiT blocks to exchange region cues without increasing parameter count.

Finally, a stylized T2I module can be plugged in to inject artistic styles into the generated video, reusing the same mimic framework while keeping the editing region consistent.

Architecture and Sense Losses

We introduce sense‑related tasks and losses that enforce pixel‑level editing consistency across video frames.

Video editing demands that the same pixel‑wise edit applied to the first frame persist unchanged throughout the temporal dimension, yet most diffusion‑based pipelines treat each frame independently, causing drift and inconsistent edits.

We give the model explicit “senses” of where an edit should happen and how it should be reflected across frames, turning vague caption guidance into concrete region‑level supervision.

Model predicts an attention mask M̂ = [[0.9, 0.8, 0.1, 0.0], [0.85, 0.78, 0.05, 0.02], [0.12, 0.07, 0.03, 0.01], [0.0, 0.02, 0.01, 0.00]].

Compute contour loss: sum of squared differences over all 16 pixels yields `L_contour` ≈ 0.018.

Cross‑attention map A (text ↔ visual) is a 4×4 matrix; cosine similarity with M (flattened) gives `L_cross` ≈ 0.12.

During back‑propagation the gradients push the high‑attention values toward 1 and the low‑attention values toward 0, tightening the mask around the edited region.

The loss pair forces the model to treat the edited region as a distinct “sense” that must be consistently highlighted across all frames, preventing drift.

How does the Sense Contour Loss differ from a standard segmentation loss used in image‑to‑image translation?

Standard segmentation losses compare a predicted mask to a ground‑truth mask but ignore the textual instruction that generated the mask. Sense Contour Loss is coupled with the cross‑attention map loss, so the model must simultaneously align visual attention and language cues, which a plain segmentation loss does not require.

**Figure 4.** Overview of the modality mimic editing loss in editing training step and our model architecture adaptations to editing task, including reference-inject self-attention mask, separate 3D rope with frame correspondence, and element identity encoding. In the schematic illustrating the modality mimic editing loss, for better clarity, we present the newly created I2I sample as the $(M + 1)$-th sample in the current training batch. In our implementation, this new sample replaces the first I2I sample in the current training batch. Please refer to Sec. 3.3 and Sec. 3.5 for details, respectively.

**Figure 6.** Overview of sense-related tasks and losses. Additionally, the reference position select task is introduced in the bottom-right corner. Please refer to Sec. 3.4 and Sec. 3.5 for details, respectively.

These additions integrate seamlessly with the existing DiT backbone, preserving the diffusion‑based generation while endowing the model with explicit region awareness.

Experimental Results

Online video‑editing data generation eliminates the need for pre‑collected video pairs.

Our editing pipeline generates video training pairs on‑the‑fly, requiring zero pre‑collected video editing samples.

Pixel‑pair Temporal Warped Flow Field synthesizes temporally coherent video pairs from single‑image edits (Sec 3.2).

FiVE‑Bench evaluates non‑rigid element replacement in videos, testing whether motion and physical effects stay consistent after editing.

UNIC‑Bench focuses on object removal from videos, requiring the model to delete the target and any associated physical effects.

The warped flow field can apply a family of parametric motions—pan, zoom, rotation, stretch‑local, and stretch‑global—to both source and target images, guaranteeing synchronized deformation across the entire video.

**Figure 8.** Single-reference non-rigid element replacement results on FiVE-Bench [52]. Our method yields natural replacements while generating corresponding motion and physical effects, e.g., the walking motion of the lion in the second example and the dinosaur-tail shadow in the fourth. The sixth example demonstrates multi-target editing.

**Figure 12.** Object removal results on UNIC-Bench. Unlike previous mainstream approaches [21, 64] that rely on object mask sequences to help the model locate regions for removal, we employ sense-related tasks and losses, enabling the model to delete specified objects through natural language. The fifth example demonstrates that our model can perceive and remove the associated physical effects, such as shadows, corresponding to the deleted object, whereas our object removal image editing training data lacks this property.

Related Work

We survey recent image and video editing datasets, curation pipelines, paradigms, and unified models.

Extensive image‑editing datasets have grown in scale and diversity. MagicBrush provides 10 K manually annotated source‑instruction‑target triplets. Pico‑Banana‑400K offers 400 K pairs generated by Nano Banana2 with MLLM‑based quality scoring. GPT‑IMAGE‑EDIT‑1.5M unifies three prior collections into 1.5 M high‑quality triplets using GPT‑4o3. ImgEdit contributes 1 M curated pairs covering novel, complex edits.

Video‑editing data curation has followed several pipelines. InsV2V synthesizes 16‑frame videos by adapting Prompt‑to‑Prompt on a diffusion model and filters them with CLIP metrics. Señorita‑2M collects online videos and applies a specialist pipeline (stylizers, remover, inpainter) before CLIP‑based filtering. Ditto leverages edited key‑frames and depth maps. InsViE‑1M uses a two‑stage edit‑then‑filter process on real videos. OpenVE‑3M orchestrates VLMs (e.g., GPT‑4o) and I2V models (Wan2.1‑I2V) in a multi‑stage generation‑filter workflow. An internship project by Dingyun Zhang built camera‑controllable body‑animation data with a 3D engine and Mixamo, later reused for video relighting and recamera tasks.

Video‑editing paradigms split into training‑free and training‑based families. Training‑free methods rely on inversion of a pretrained video model’s latent space; AnyV2V edits the first frame, inverts the video, and injects spatial‑temporal features during DDIM sampling. Videoshop improves reconstruction via noise extrapolation. ContextFlow adopts rectified‑flow inversion with off‑the‑shelf image editors. Training‑based approaches fine‑tune model parameters or add LoRA modules: I2VEdit and GenProp fine‑tune I2V models; VACE trains a plug‑and‑play context adapter; VideoPainter introduces an ID‑Adapter for temporal identity; UNIC fine‑tunes a DiT‑based model with task‑dependent RoPE and condition bias, also training a mask predictor for region localization.

Task‑specific editing models target particular visual problems. Light‑A‑Video is a training‑free relighting system with a consistent light‑attention module and progressive light fusion. ANYPORTAL handles relighting and background replacement through a three‑stage pipeline (background generation, light harmonization, consistency enhancement). VFace swaps faces using target‑structure guidance and frequency‑spectrum attention interpolation. KeyTailor injects high‑definition details driven by keyframes for virtual try‑on. Rig‑Your‑Portrait combines explicit 3D guidance (personalized mesh, normal maps) with an identity‑expression‑illumination decoupling strategy to enable controllable portrait animation.

Unified image‑video (IV) editing and generation methods aim to serve multiple modalities with a single model. EditVerse employs a full self‑attention transformer that interleaves tokenized text and visual inputs. InstructX encodes instructions and source visuals with a VLM, replaces text embeddings in a DiT, and feeds the result to the diffusion model. UniVideo unifies video understanding, generation, and editing via a multimodal DiT (MMDiT) conditioned on VLM‑derived tokens. VINO follows the same pattern, integrating image and video generation/editing in one framework. Omni‑video 2 also relies on a VLM to translate instructions and source video into conditioning tokens for a video diffusion model.

Extended Video Editing Results

We demonstrate flexible, mask‑free video editing across diverse tasks, including trajectory and normal‑map edits.

Our model learns trajectory‑customized video editing from only ≈3 K reversed RefCOCO samples.

Figure 10 (bottom rows) shows the model inserting a specified object along a colored trajectory while preserving shadows and motion.

The model generalizes from <2 K global normal‑map samples to body‑normal map animation unseen during training.

Figure 10 second row demonstrates realistic body animation using normal maps despite no body‑specific training data.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers