Beyond Spatio-Temporal Priors: A Generalizable Approach for Dense Correspondence Matching
Luping Liu, Bingyi Kang, Yifan Wang, Dong Xu
FreeMatching extends dense correspondence to identity-preserving image editing by combining foundation models with teacher-guided refinement.
How can we build a dense correspondence model that works for both classical motion tasks and complex identity-preserving image editing?
Dense correspondence models rely on rigid geometric or smooth motion priors, which fail when an object’s appearance or pose changes drastically during image editing or reference-guided generation. FreeMatching addresses this by combining generative and semantic foundation representations with heterogeneous supervision and teacher-guided iterative refinement. This approach achieves competitive performance on classical benchmarks while significantly improving correspondence quality on challenging identity-preserving tasks.
Paper Primer
FreeMatching integrates generative priors from FLUX and semantic features from DINOv3 into a unified transformer architecture. The core mechanism is a two-stage training paradigm: supervised pre-training on diverse tracked video and synthetic data, followed by teacher-guided iterative refinement that uses a frozen model to correct local misalignments without requiring dense ground-truth annotations.
FreeMatching provides superior identity-preserving correspondence in image editing and reference-guided generation (IEG) tasks.
On the IEG-Bench evaluation, FreeMatching outperforms existing generalist models like UFM and RoMa across multiple reconstruction metrics, including Ref-MSE, LPIPS, and feature-based similarity. In 10k-adaptation comparisons, FreeMatching achieves a Full-MSE advantage of 2.48 over UFM and 5.57 over RoMa.
The framework maintains competitive performance on classical geometric and motion-based correspondence tasks.
The model achieves state-of-the-art or near-SOTA results on standard benchmarks like ETH3D, ScanNet-1500, and Sintel, demonstrating that the extension to IEG does not degrade classical matching capabilities. The model achieves lower End-Point Error (EPE) than adapted UFM and RoMa variants on ScanNet and ETH3D benchmarks.
Why is this approach necessary for image editing compared to standard optical flow?
Standard optical flow assumes smooth, continuous motion, which breaks down when an object's identity is preserved across drastic changes in pose, appearance, or scene composition. FreeMatching explicitly models these identity-preserving correspondences rather than physical continuity.
How does the model evaluate identity preservation without ground-truth correspondence?
The authors use a reconstruction-based protocol: they warp the source image to the target using the predicted correspondence map and measure the reconstruction error within the covisible region. Low error indicates high-quality, identity-preserving alignment.
FreeMatching demonstrates that correspondence models can move beyond rigid geometric priors by leveraging foundation-model representations, enabling both robust matching in generative tasks and a new quantitative metric for evaluating identity preservation.
Beyond Spatio-Temporal Priors
We expose the limits of classic motion priors and motivate a new framework for identity-preserving correspondence.
Dense correspondence matching has long depended on spatio‑temporal priors that assume smooth motion and rigid geometry, which works for classic tasks but collapses when transformations preserve visual identity while breaking physical continuity.
These are the built‑in assumptions that motion is smooth over time and objects retain rigid shapes, enabling simple models but limiting flexibility when appearance or pose changes dramatically.
**Figure 1.** Extending Dense Correspondence to IEG. FreeMatching matches across optical flow, geometric matching, and IEG. Each block shows input pairs above and our bidirectional warps below. The classical examples vary in temporal separation (top) and geometric complexity (bottom).
The limitation of classical motion priors hampers modern editing tasks that require identity preservation.
Architecture and Overview
We introduce FreeMatching, a unified transformer that removes spatio‑temporal priors by jointly attending image tokens and foundation features.
Dense correspondence has progressed from separate optical‑flow and geometric‑matching pipelines to unified deep models such as UniMatch and UFM. Recent work adds semantic awareness and foundation‑model features, yet most approaches still rely on rigid spatio‑temporal priors.
FreeMatching removes the need for explicit spatio‑temporal priors by feeding two images into a single Diffusion Transformer (DiT) that jointly attends their latent tokens together with projected DINOv3 features, then decodes a dense correspondence map and a covisibility mask in one pass.
Form token sequence $T = [t_{a,1},\dots,t_{a,4},t_{b,1},\dots,t_{b,4}]$ of length $8$.
Run one DiT self‑attention layer: each token attends to all $8$ tokens, mixing visual and semantic cues.
Decoder reads the attended representation and predicts correspondence $C$ for each target pixel, e.g. $C(0,0) = (1,1)$.
Decoder also predicts covisibility mask $M$, e.g. $M(0,1)=0$ for a newly revealed region.
Joint attention lets the model directly compare any source‑target pair, so it can recover matches even when traditional flow assumptions (smooth motion, rigid geometry) are violated.
How does FreeMatching differ from earlier unified models like UniMatch?
UniMatch still builds a cost volume and refines matches iteratively, preserving an implicit geometric bias. FreeMatching replaces the cost‑volume pipeline with a single global attention step, eliminating the need for any spatio‑temporal prior and allowing the model to learn identity‑preserving correspondences directly from data.
**Figure 2.** FreeMatching Architecture. Our transformer is initialized from FLUX.2-klein-base-4B. Each image is encoded into latent tokens and augmented with projected DINOv3 features. The two token sequences are concatenated for joint attention, and a task-specific decoder predicts the dense correspondence map and covisibility mask in a single forward pass.
Training Data and Strategy
We freeze generative encoders and train a transformer to predict correspondence and visibility from combined latent and semantic tokens.
Supervised pre‑training must handle diverse motions and appearances while keeping the generative prior stable. We therefore freeze the FLUX VAE and DINOv3 feature projectors, and only train the transformer‑based correspondence head.
We treat each image as a sequence of latent tokens, enrich them with semantic patches, and let a transformer attend across the two concatenated sequences to output a correspondence map and a visibility mask.
Image A tokens: $t_{A,1},t_{A,2},t_{A,3},t_{A,4}$; Image B tokens: $t_{B,1},t_{B,2},t_{B,3},t_{B,4}$.
Project DINOv3 vectors $d_{A,i}$ and $d_{B,i}$ linearly and add to the corresponding tokens: $e_{A,i}=t_{A,i}+W d_{A,i}$, $e_{B,i}=t_{B,i}+W d_{B,i}$.
Concatenate the enriched embeddings: $E=[e_{A,1},\dots,e_{A,4},e_{B,1},\dots,e_{B,4}]$.
Add position encodings that encode both spatial coordinates and a binary image‑id flag, yielding $E'$.
Feed $E'$ to the transformer; the output head produces a $4\times4$ correspondence matrix $C$ and a $4\times4$ visibility mask $M$.
Concatenating the two sequences lets the transformer attend jointly, so each token can directly query any token from the other image, enabling dense correspondence without a separate matching module.
The table lists various datasets and their corresponding ratios used in the study, including ScanNet++, BlendedMVS, FlyingThings, DynamicReplica, Spring, PointOdyssey, FlyingChairs, and custom Video and Blender datasets.
How does this supervised pre‑training differ from ordinary fine‑tuning of a pretrained transformer?
In ordinary fine‑tuning all layers are updated on the downstream task, risking loss of the generative prior. Here the FLUX VAE and DINOv3 projectors are frozen, so only the transformer that learns cross‑image attention and the task‑specific decoder receive gradients, preserving the strong image prior while adapting to dense correspondence.
Teacher-Guided Iterative Refinement
We assess correspondence quality and data consistency using a reconstruction‑based protocol.
The reconstruction‑based protocol measures how well the predicted correspondence $C$ can warp the source image $I_a$ into the target $\hat I_b$, using several complementary metrics.
An EMA copy of the student predicts an initial warp; a frozen teacher supplies a corrective offset; composing them yields a refined warp that becomes the training target.
EMA update: the student’s current warp $0.2$ is blended with the previous EMA value $0.0$, yielding $0.9\cdot0 + 0.1\cdot0.2 = 0.02$.
Compose correction: add the teacher’s offset $0.1$ to the EMA warp $0.02$, producing a refined warp of $0.12$.
Use $0.12$ as the target for the next training iteration; the EMA will now incorporate this refined value.
The EMA provides a low‑pass filter on the student’s predictions, while the teacher’s fixed correction drives the warp toward the true alignment without destabilizing training.
**Figure 3.** Teacher-Guided Iterative Refinement. An exponential moving average (EMA) copy of the student predicts the correspondence used to produce the Initial Warp. A frozen teacher estimates a local correction; composing it with the initial correspondence produces the Refined Warp and a correspondence target for training the student. The EMA copy evolves with the student while the teacher remains fixed.
Warp the source image $I_a$ with the predicted correspondence $C$ to obtain $\hat I_b$.
Compute reconstruction metrics (e.g., $MSE$, $LPIPS$, feature similarity) between $\hat I_b$ and the target $I_b$ within the covisible region $M$.
Interpret low reconstruction error as high matching accuracy on benchmark pairs.
For novel pairs, treat unusually high error in $M$ as an indicator of identity‑preserving inconsistency in the data.
For pose estimation, FreeMatching also attains the highest AUC scores on both ETH3D and ScanNet‑1500, confirming that the refined correspondences translate into superior downstream geometry.
Empirical Validation
FreeMatching drops rigid priors with a generative transformer and iterative refinement.
FreeMatching builds on the premise that dense correspondence can be improved by removing spatio‑temporal priors and using a generative transformer with iterative refinement.
FreeMatching attains 1.05 EPE on the Sintel Final optical‑flow benchmark, comparable to leading methods.
Table 3 reports an endpoint‑error of 1.05 for FreeMatching, versus 0.40 for UniMatch and 0.43 for SEA‑RAFT.
Beyond optical flow, FreeMatching also lowers endpoint error on ScanNet and ETH3D compared with the adapted UFM and RoMa baselines after the same 10 k refinement steps.
**Table 1.** Quantitative comparison of optical flow methods on Sintel and KITTI-2015 benchmarks.
**Figure 4.** Visual comparisons on IEG-Bench using UNO examples. The references in Image B are composed into Image A ($B arrow A$). Each method warps Image A toward Image B to evaluate correspondence quality.
FreeMatching achieves competitive performance on classical benchmarks while excelling on identity‑preserving correspondence tasks.
Identity-Preserving Correspondence Benchmarking
FreeMatching sets new IEG‑Bench records, confirming superior identity‑preserving correspondence.
Recall that dense correspondence models traditionally rely on rigid spatio‑temporal priors, whereas FreeMatching removes those constraints via a generative transformer and iterative refinement to preserve object identity.
FreeMatching outperforms UFM on IEG‑Bench Ref‑MSE by $2.33$ points.
Table 4 shows Ref‑MSE $10.41$ for FreeMatching versus $8.08$ for UFM.
After $10\,k$ adaptation updates, FreeMatching still leads across all four metrics, beating UFM by $2.48$ in Full‑MSE and RoMa by $5.57$.
**Table 5.** Comparison after weak adaptation. We additionally compare FreeMatching, UFM, and RoMa after 10k adaptation updates using UNO+OmniGen2 pairs. Table 5 reports reference-mask and full-image MSE, together with DINOv2 and OpenCLIP similarities as complementary feature metrics. FreeMatching achieves the best scores across all four metrics. Together with the MSE, LPIPS, SigLIP 2, and DINOv3 results in Table 4, these results show consistent advantages across metrics under both evaluation settings. Its paired Full-MSE advantages over UFM and RoMa are 2.48 (95%
The table compares three methods: FreeMatching (10k-IEG), UFM (10k-IEG) [7], and RoMa (10k-IEG) [5] across four metrics: Ref-MSE, Full-MSE, Ref-DINOv2, and OpenCLIP.
**Figure 5.** Fine-Grained Identity Assessment. Nano Banana edits Image A into Image B ($A arrow B$). DINOv3 similarity heatmaps compare the warped source with the target within covisible regions, revealing local appearance inconsistencies. Red boxes highlight affected regions, including the altered balloon text.
Component Analysis
We ablate each component to quantify its impact on dense correspondence performance.
To understand which parts of FreeMatching drive its performance, we incrementally add components and measure the DINOv3 similarity on the IEG‑Bench.
**Figure 7.** Comparison with Semantic Matching. In this multi-object example, the displayed SD-DINO [47] reconstructions, obtained with the prompts “book” and “vase”, exhibit incomplete shapes and mixed object appearance. FreeMatching preserves more complete object shapes and local details without text prompts.
The monotonic increase in DINOv3 scores confirms that each added module—FLUX initialization, semantic DINOv3 features, heterogeneous video & Blender supervision, and teacher‑guided refinement—contributes positively, with the full stack achieving a 12.70‑point gain over the bare baseline.
Architectural Specifications
Detailed architecture and dataset specifications for the correspondence model.
Appendix A details the full architecture, while Appendix B enumerates the datasets used for pre‑training and evaluation.
DINOv3 is a frozen Vision Transformer that provides high‑level semantic patch embeddings, which we inject into the generative transformer to give it scene‑level context.
FLUX.2-klein-base-4B is a large diffusion‑trained transformer that serves as the generative backbone, providing a powerful prior for dense correspondence prediction.
We also employ a Diffusion Transformer (DiT) as the core generative component, inheriting the spatial‑position encodings that allow attention across the concatenated source‑target token sequence.
In the input stage, images are encoded by a frozen VAE into a latent grid $z \in \mathbb{R}^{h \times w \times 32}$, where $h = H/8$ and $w = W/8$, then packed into $128$‑dimensional tokens.
Semantic features from DINOv3 are projected and added to these tokens, after which the source and target sequences are concatenated (1152 tokens at $288 \times 512$ resolution) and processed jointly by the transformer.
The transformer outputs a token sequence that is unpacked, de‑normalized, and passed through a convolutional decoder to produce a three‑channel image: two channels for dense correspondence $C \in \mathbb{R}^{H \times W \times 2}$ and one for covisibility mask $M \in \mathbb{R}^{H \times W \times 1}$.
Appendix B describes the data used for training and evaluation, spanning standard benchmarks, a large real‑world video collection, a synthetic Blender set, and the IEG dataset.
Standard datasets include BlendedMVS, ScanNet++, FlyingThings, Spring, FlyingChairs, PointOdyssey, and DynamicReplica, sampled in proportion to cover diverse geometry and motion.
The real‑world video dataset draws 520 k clips from ViPE, with pseudo‑ground‑truth generated by AllTracker and processed using a sliding‑window of length $L = 16$ and $N_{\text{iters}} = 4$ refinement steps.
The synthetic Blender dataset contains 360 k image pairs rendered from Objaverse assets, filtered by scale, aspect‑ratio, and silhouette occupancy to ensure high‑quality geometric supervision.
The IEG dataset builds on UNO, extracting shared object masks via Qwen3‑VL‑8B and SAM 3, then merging instance masks into unified binary masks for correspondence training.
Inference Profiling and Costs
Provides cost tables, evaluation metrics, human study results, and a diffusion‑based extension.
Baseline adaptation cost experiments use 16 H2O GPUs, a global batch size of 32, and 288 × 512 input resolution.
**Table 8.** Inference cost on one H20 GPU at external resolution 288 × 512 and batch size 1. Latency is averaged over 50 calls after warm-up; memory is peak allocated GPU memory.
**Table 8.** Inference profiling. Table 8 reports a separate inference profile on a single H20 GPU at external input resolution 288 × 512 and batch size 1. We use 10 warm-up calls followed by 50 timed calls to each model's inference wrapper, with CUDA synchronization around the timed loop. We report mean latency per image pair and peak allocated GPU memory.
D Objective Evaluation details the reconstruction‑based protocol used to assess warped images against ground‑truth targets.
D.1 defines the masked Mean Squared Error (MSE) that measures pixel‑level fidelity within the covisible region.
D.2 introduces the masked LPIPS metric, which captures perceptual similarity using an AlexNet backbone.
D.3 measures semantic and identity alignment with DINOv3 and SigLIP 2 cosine similarities.
D.4 lists implementation details: masking thresholds, normalization conventions, and the subset of 100 image pairs used for IEG‑Bench.
D.5 describes complementary metrics (Full‑MSE, Ref‑DINOv2, OpenCLIP) and how paired statistics are bootstrapped.
D.6 reports weak adaptation controls, comparing flow‑only and IEG‑only refinement data sources.
The table presents performance metrics for different FreeMatching checkpoints across various datasets (Ref-MSE, Sintel Clean, Sintel Final, KITTI, ScanNet, and ETH3D). The rows correspond to FreeMatching-S1, FreeMatching (10k-Flow), and FreeMatching (10k-IEG). Bold values indicate the best performance for each metric.
**Table 10.** Effect of the refinement data source. We report IEG-Bench Ref-MSE ($\times 100$) and EPE on classical benchmarks. Lower is better; best results within this comparison are bold.
**Table 1.** Comparison of different models across Human, LPIPS, and DINOv3 metrics.
F outlines a diffusion‑based extension that models a distribution $p(C \mid I_a, I_b)$, enabling multiple plausible correspondences.
G mentions SD‑DINO, a recent zero‑shot semantic correspondence method that combines Stable Diffusion and DINOv2 features.