Self-OPD: On-Policy Distillation for Flow Matching Models without Teacher

Shiyi Zhang, Mushui Liu, Yunze Tong, Wanggui He, Siyu Zou, Jinlong Liu, Yunlong Yu, Jian Song, Hao Jiang, Pipei Huang, Bo Zheng

Self-OPD replaces external teachers with reward-weighted self-exploration to align flow matching models across multiple objectives.

How can we align flow matching models to human preferences without needing a pre-trained teacher model or unstable trajectory-level policy gradients?

Aligning flow matching models with downstream objectives typically requires training task-specific teachers, which is computationally expensive and leads to conflicting gradient updates when multiple objectives are combined. Self-OPD eliminates the teacher by branching the student's own trajectory into stochastic candidates, evaluating them against a self-reference baseline, and using the resulting advantages to pull the velocity field toward high-reward directions. This teacher-free approach outperforms prior reinforcement learning and teacher-based distillation methods, achieving superior performance on text rendering, compositional accuracy, and aesthetic preference simultaneously.

Paper Primer

The core mechanism is an all-branch pull-push objective: the model treats the student's local neighborhood as a set of competing trajectories, pulling the velocity field toward branches that outperform a deterministic self-reference baseline and pushing it away from those that underperform. By fusing multiple rewards into a single composite score before ranking these branches, the model avoids the gradient interference that plagues parameter-space fusion.

Self-OPD achieves state-of-the-art alignment across multiple tasks without requiring task-specific teachers.

In mixed-reward benchmarks, Self-OPD achieves a 0.95 GenEval score and 96.0% OCR accuracy, while simultaneously outperforming teacher-based DiffusionOPD on aesthetic preference metrics (PickScore 23.87 vs 22.72). The method achieves higher final performance than teacher-based baselines while reducing total wall-clock training time by approximately 2x.

Why is reward-level fusion superior to the standard approach of blending gradients?

Standard field-level fusion blends gradients in parameter space, which causes destructive interference when objectives conflict. Self-OPD fuses rewards at the trajectory level, ensuring the model optimizes for a single sample that satisfies all criteria simultaneously.

What is the role of the "self-reference baseline" in this framework?

The baseline is a deterministic ODE rollout from the parent state; it provides a variance-reducing reference point that allows the model to convert terminal rewards into normalized advantages for each stochastic branch.

Introduction and Motivation

Self‑OPD replaces teacher‑based supervision with self‑exploration to provide dense step‑wise guidance.

On‑policy distillation (OPD) uses a pretrained teacher to supply dense step‑wise supervision, but training a separate teacher for each new objective is costly and mismatches between teacher and student distributions cause error accumulation. Flow‑matching (FM) models learn a continuous velocity field $v_\\theta$ that transports noise to data, yet aligning them with downstream objectives remains difficult because existing reinforcement‑learning approaches rely on sparse terminal rewards, leading to high‑variance gradients and fragile multi‑objective optimization.

OPD supplies the student with dense per‑step targets by regressing its velocity toward a teacher’s prediction, stabilizing training and improving sample efficiency.

Flow‑GRPO treats the denoising process as a sequential decision problem and applies trajectory‑level policy‑gradient updates using rewards evaluated at the end of each trajectory.

**Figure 1.** Overview of Self-OPD. (a) Comparison of alignment paradigms. (b) Single- and mixed-reward results. (c) Generated samples from a single Self-OPD model.

**Figure 2. Comparison of alignment paradigms.** (a) **Flow-GRPO** relies on trajectory-level policy gradients from terminal rewards. (b) **Flow-OPD** requires a pretrained teacher for step-wise MSE supervision. (c) **Self-OPD (Ours)** turns local stochastic branches and a self-reference baseline into dense pull-push supervision, without requiring any teacher.

The key shift is replacing teacher‑based supervision with self‑exploration‑driven dense step‑wise signals.

Flow Matching Foundations

How the paper replaces an external teacher with a self‑generated advantage signal.

Flow‑matching models need dense, step‑wise supervision, which traditionally forces the use of a pre‑trained teacher that adds cost and complexity.

FM learns a continuous velocity field that pushes a sample of pure noise along a straight line until it becomes a data point.

How does Flow Matching differ from score‑matching based diffusion models?

Score‑matching learns the gradient of the log‑density (the score) and then integrates a stochastic diffusion process, whereas Flow Matching directly learns the velocity that transports noise to data along a deterministic straight‑line path, eliminating the need for a diffusion schedule.

Self‑OPD lets the model become its own teacher: it perturbs its current on‑policy trajectory, measures the resulting advantage of each branch, and then pulls the policy toward better branches while pushing it away from worse ones.

Deterministic prediction: $x_{t_{1}}^{\theta}= x_{t_{0}} + (1-t_{0})\,v_{\theta}(x_{t_{0}},t_{0}) = 0 + 0\cdot v_{\theta}=0$.

Stochastic perturbation: draw $z\sim\mathcal{N}(0,1)$, e.g. $z=0.8$, then $x_{t_{1}} = x_{t_{1}}^{\theta} + \sigma_{t_{0}}|\Delta t_{0}|z = 0 + 1.0 \times 0.5 \times 0.8 = 0.4$.

Self‑reference baseline (no noise) remains at $0$. Advantage = $-\|0.4\|^{2} + \|0\|^{2} = -0.16$ (negative, so this branch is penalized).

Pull‑push update scales $v_{\theta}$ by the advantage weight, reducing the velocity in this direction.

The stochastic branch that drifts away from the baseline receives a negative advantage, causing the pull‑push objective to push the policy back toward the deterministic prediction.

Why does Self‑OPD not need an external teacher like conventional OPD?

Self‑OPD generates its own dense supervision by locally exploring the student’s on‑policy trajectory; the advantage signal derived from these self‑generated branches replaces the teacher’s role, removing the need for a separate pretrained model.

By turning the model into its own teacher, Self‑OPD provides step‑wise, dense feedback without any extra parameters, enabling efficient on‑policy learning.

All-Branch Pull-Push Distillation

All‑branch updates pull toward high‑advantage trajectories and push away from low‑advantage ones, with direction‑aware attenuation.

When only the best sampled branch is used, the target can flip dramatically between steps, discarding useful information from other plausible trajectories.

Instead of regressing to a single “best” branch, the method aggregates the whole exploration neighborhood: positive‑advantage branches pull the deterministic prediction toward better directions, while negative‑advantage branches push it away, with a direction‑aware attenuation that softens repulsion when a negative branch points almost the same way as the best.

How does Pull‑Push Distillation differ from the standard Best‑of‑K update?

Best‑of‑K discards all but the single highest‑advantage branch, so the target can jump abruptly and useful gradient information from other branches is lost. Pull‑Push keeps every branch: positive branches attract the prediction, negative branches repel it, and the attenuation $d_k$ ensures that repulsion is softened when a negative branch points almost the same way as the best, preserving smooth updates.

Branch 2: cosine similarity $=\frac{\langle(1,0.2),(2,0)\rangle}{\|(1,0.2)\|\|(2,0)\|}= \frac{2}{\sqrt{1.04}\cdot2}=0.98$, so $d_2 = 1-0.98 = 0.02$ (strong attenuation).

Branch 3: cosine similarity $=\frac{\langle(1.9,0.1),(2,0)\rangle}{\|(1.9,0.1)\|\|(2,0)\|}= \frac{3.8}{\sqrt{3.62}\cdot2}=0.99$, so $d_3 = 1-0.99 = 0.01$ (even stronger attenuation).

Positive branch (index 1) uses $d_1=1$ (pull). The update to $v_\theta$ therefore pulls toward $\delta_{\text{best}}$ and only a tiny repulsive component from branches 2 and 3, because their $d_k$ are near 0.

Direction‑aware attenuation turns almost‑aligned negative branches into negligible repellers, preventing them from cancelling the beneficial pull of the best branch.

Theoretical KL Interpretation

Shows how the pull‑push loss equals a per‑step KL minimization and how Self‑OPD implements it.

The pull‑push objective collapses under the normalization into a KL divergence, exposing why the loss directly drives the student mean toward the reward‑tilted target.

For each diffusion timestep the optimal update moves the student mean exactly toward the reward‑tilted target distribution, scaled by the transition precision.

Compute the precision factor $1/(\sigma_t^{2}|\Delta t_j|)=1/(0.04\times0.5)=50$.

Form the Monte‑Carlo estimate of the expectation: $\frac{1}{K}\sum_{k=1}^{K}(x_{t_{j+1},\theta}-x_{k}) = ( \mu - 1.2 + \mu - 0.8 + \mu - 1.0)/3$ where $\mu$ is the current mean.

Assume the current mean $\mu=0.9$. The average difference is $(0.9-1.2)+(0.9-0.8)+(0.9-1.0) = -0.3+0.1-0.1 = -0.3$, divided by 3 gives $-0.1$.

Multiply by the precision: $50 \times (-0.1) = -5.0$. The gradient update moves the mean by $-5.0$, i.e. $\mu_{\text{new}} = 0.9 - 5.0 = -4.1$ (illustrating the strong pull when variance is small).

Even with only three branches, the KL gradient can produce a large correction because the step size is governed by the transition precision, not by the number of samples.

Self‑OPD replaces the intractable expectation in the KL gradient with a Monte‑Carlo average over the $K$ sampled SDE branches, weighting each branch by its signed advantage $A_k$ and the direction gate $d_k$.

How does this per‑step KL update differ from the standard KL minimization used in variational inference?

Standard variational KL treats the whole trajectory as a single distribution and optimizes a global ELBO, whereas here the KL is computed and minimized at each diffusion step, yielding a local, analytically tractable gradient that directly informs the mean update.

The pull‑push objective is mathematically equivalent to a per‑step KL minimization, so Self‑OPD’s Monte‑Carlo update faithfully follows the KL gradient.

Multi-Objective Reward Fusion

Reward-level fusion merges multiple objectives into a single trajectory score, avoiding gradient conflicts.

Multi‑objective alignment often collapses into a “seesaw” when gradients for different metrics point in opposite directions, forcing a compromise that harms all metrics.

Instead of mixing gradients in the parameter space, the method ranks whole sampled trajectories by a single composite reward, letting the best trajectory satisfy all objectives at once.

How does Reward‑Level Fusion differ from the usual weighted‑loss blending?

Weighted‑loss blending adds the losses $\lambda_{m}L_{m}$ in parameter space, so the optimizer sees a sum of gradients that can cancel each other. Reward‑Level Fusion instead computes a single scalar $r^{(k)}$ per sampled trajectory, ranks the trajectories, and updates the model using only the winning trajectory’s advantage. No gradients are summed, so there is no cancellation.

Normalize each score: $\tilde{r}^{(k)}_{1}= (0.8-0.5)/0.1 = 3.0$, $\tilde{r}^{(k)}_{2}= (0.5-0.4)/0.2 = 0.5$.

Compute the composite reward: $r^{(k)} = 0.6 \times 3.0 + 0.4 \times 0.5 = 1.8 + 0.2 = 2.0$.

Suppose a competing branch $k'$ yields $r^{(k')} = 1.7$; branch $k$ wins the ranking and its advantage $A_{k}$ drives the velocity update.

The weighted sum of normalized scores preserves the relative ordering of branches, so the branch that looks best under each individual objective also looks best under the composite reward.

**Figure 4.** Field-level fusion couples preference quality to the prompt family; reward-level fusion does not. For each approach, we contrast the preference-reward distribution of images generated on aesthetic/preference prompts (orange) against that of images generated on the GenEval/OCR task prompts (blue). The distribution peaks are placed at the measured means from Tab. 2 (the “separate test set” and “same test images” columns, respectively); the spread is illustrative and only serves to visualize the shift $\Delta$ between the two prompt families. For DiffusionOPD, the preference distribution shifts sharply when moving to the task prompts, whereas for Self-OPD, the two distributions nearly coincide.

Experimental Setup

Self‑OPD sets new teacher‑free performance peaks on mixed‑reward training.

Self‑OPD (Ours) attains the highest PickScore (0.9691) among all teacher‑free methods.

Table 2 shows Self‑OPD reaching 0.9691, surpassing the next best teacher‑free entry (0.9597).

Implementation details: we fine‑tune SD3.5‑Medium [6] at 512×512 resolution, applying LoRA [14] to the transformer. Training uses $K=8$ branches, noise level $\eta=0.7$, and two SDE steps per update, optimized with AdamW [16] at a learning rate of $3\times10^{-4}$.

**Table.** **Tasks and Rewards.** We evaluate three tasks with corresponding reward models: (1) text rendering, scored by OCR character-level accuracy between the generated and target text; (2) compositional generation, scored by GenEval [11]; and (3) aesthetic and human-preference alignment, scored by PickScore [17] and HPSv2 [31]. We also train mixed-reward variants fusing each task scorer.

Per‑task reward fusion follows Eq. 18: for OCR we set $\lambda_{\text{OCR}}:\lambda_{\text{PickScore}}:\lambda_{\text{HPSv2}}=3:1:1$, and analogously for the GenEval task, keeping PickScore and HPSv2 as shared quality guards.

Performance Evaluation

Self‑OPD sets new state‑of‑the‑art across all mixed‑reward benchmarks.

Self‑OPD achieves the highest scores across all mixed‑reward metrics, outperforming both teacher‑based and other teacher‑free baselines.

Table 2 shows Self‑OPD reaching 0.9536 on GenEval strict, 0.9676 on GenEval continuous, 0.9745 on OCR, 24.47 on PickScore, and 0.4099 on HPSv2.

The quantitative gains translate into visibly superior images, as shown in the qualitative figures that follow.

**Figure 8.** Strict vs. continuous scoring on “a photo of a white handbag and a red giraffe”. (a) base; (b) Self-OPD.

**Figure 9.** Aesthetic quality on GenEval prompts. Each method uses its mixed-reward model if available, or its GenEval-trained model otherwise (Flow-GRPO, GRPO-Guard). Self-OPD satisfies the compositional requirements (counting, spatial relations, attributes) while generating images with higher aesthetic quality, consistent with its higher PickScore/HPSv2 in Tab. 2.

**Figure 10.** Aesthetic quality on OCR prompts. Each method uses its mixed-reward model if available, or its OCR-trained model otherwise (Flow-GRPO, GRPO-Guard). Self-OPD renders the target text accurately while producing more visually appealing scenes with richer backgrounds and realistic details.

**Figure 5. Qualitative comparison.** Each method uses its mixed-reward model if available, or the prompt-specific reward model (e.g., OCR for text prompts). Self-OPD achieves superior performance in both accuracy and aesthetics.

Ablation and Efficiency Analysis

Self‑OPD replaces a teacher with a self‑referenced advantage signal, improving training efficiency.

Flow matching needs dense supervision; Self‑OPD supplies it by turning the model’s own exploratory branches into a per‑step advantage signal.

**Figure 6.** Ablation of core designs. (a) Branch handling: Best-of-$K$ (red), all-branch pull-push distillation (green), and Self-OPD with $|\Delta t|$-aligned KL (blue). (b) Gradient gating: Bounded $d_k$ (blue) vs. unbounded $d_k$ (red).

Full Self‑OPD with $|\Delta t|$‑aligned KL (blue) converges fastest and attains the highest performance among the three branch‑selection strategies.

Figure 6 (a) shows the blue line dominating the green and red lines throughout the 1200‑step training horizon.

Bounding the repulsion coefficient to $d_k\in[0,1]$ prevents the catastrophic collapse observed with an unbounded $d_k\in[0,2]$.

Figure 6 (b) demonstrates a sharp performance drop for the unbounded (red) curve around step 600, whereas the bounded (blue) curve remains stable.

Weighting early timesteps accelerates convergence compared with uniform weighting, while oversampling early steps (TIS) harms final performance.

The proposed weighting concentrates loss on early timesteps, leading to quicker error reduction; the TIS variant degrades below the base model.

**Figure 7** Training efficiency: OCR (left) and GenEval (right) vs. wall-clock time. Blue: Self-OPD mix from scratch. Green: Self-OPD mix warm-start. Red: DiffusionOPD student phase. The green shaded region indicates the time spent training warm-start checkpoints (37 h). The red shaded region indicates DiffusionOPD's teacher training phase (85.75 h), during which the student model is idle. The dotted horizontal line marks DiffusionOPD's final performance level. All training times for DiffusionOPD are taken from the original report.

Self‑OPD from scratch reaches DiffusionOPD‑level OCR (0.946) in ≈62 h and GenEval (0.915) in ≈90 h, beating the 97 h total of DiffusionOPD.

Figure 7 shows the blue curve attaining the dotted horizontal benchmark well before the red curve’s teacher‑training window ends.

Self‑OPD warm‑start achieves OCR (0.946) in ≈48 h and GenEval (0.915) in ≈44 h, roughly twice as fast as DiffusionOPD while delivering higher final scores.

Figure 7’s green line reaches the dotted benchmark within the shaded 37 h warm‑start interval, then continues to improve.

Self‑OPD’s branch‑wide advantage weighting and bounded repulsion together yield faster, more stable training than any teacher‑based alternative.

Self-Referenced Exploration

Self-OPD generates local advantage signals by branching stochastic futures and comparing them to a deterministic baseline.

Instead of relying on an external teacher, the model asks “what if I perturb my own prediction?” and measures how much better or worse each perturbed future scores compared to the unperturbed trajectory.

Compute the standard deviation of the three rewards: $\operatorname{std}\{5.0,3.0,6.0\}\approx1.25$.

Branch 1 advantage: $A_1 = (5.0 - 4.0) / (1.25 + \epsilon) \approx 0.80$.

Branch 2 advantage: $A_2 = (3.0 - 4.0) / (1.25 + \epsilon) \approx -0.80$.

Branch 3 advantage: $A_3 = (6.0 - 4.0) / (1.25 + \epsilon) \approx 1.60$.

The normalized advantages highlight that branch 3 points toward a clearly better direction, while branch 2 signals a detrimental direction, all without any external teacher.

How does the Self‑Referenced Advantage differ from the usual advantage estimator in reinforcement learning?

In standard RL the advantage compares a sampled return to a learned value baseline, requiring a separate value network. Here the baseline is the deterministic ODE rollout of the same model, and the “advantage” is a purely local, reward‑based difference that does not involve any extra network or value function.

**Figure 3** Self-OPD pipeline. *Top:* The student branches its prediction into $K$ SDE paths (blue), generates images via ODE rollouts (purple), and scores them ($r^+, r^{\text{ref}}, r^-$). *Bottom-left:* Rewards define a self-referenced advantage $A_k$ that pulls toward high-reward velocities $v_+$ and pushes away from low-reward velocities $v_-$. *Bottom-right:* A direction-aware coefficient $d_k$ gates the push to prevent it from counteracting the pull, resulting in the multi-branch loss $\mathcal{L}_{\text{Self-OPD}}$.

Supplementary Analysis

Supplementary details on evaluation metrics, protocols, and qualitative results.

Appendix A expands the main paper with full evaluation details, a theoretical proof, extended analysis, and extra qualitative examples.

A.1 introduces the GenEval benchmark, which measures compositional image generation by checking a set of sub‑requirements such as object count, spatial relation, and color.

Strict scoring is binary per image: a score of 1 is given only when **all** sub‑requirements are satisfied, otherwise 0; the final metric is the average over all prompts.

Continuous scoring assigns a fractional credit equal to the proportion of satisfied sub‑requirements (range 0–1) for each image, and then averages across prompts.

A.2 describes two preference‑evaluation protocols used with PickScore and HPSv2: one that reuses the same generated images as GenEval, and another that evaluates on a held‑out aesthetic prompt set.

In the “same test images” protocol, each model generates images for both the GenEval and OCR test suites, and the same images are scored by PickScore and HPSv2; the final preference score is the unweighted average of the GenEval‑prompt and OCR‑prompt scores.

The “separate test set” protocol evaluates PickScore and HPSv2 on a disjoint collection of aesthetic‑oriented prompts (DrawBench), measuring how well a model generalizes to prompts it has not seen during training or GenEval evaluation.

Because the “same test images” protocol keeps all metrics on the identical set of generations, it isolates model quality from distribution shift and is therefore the primary comparison for preference metrics.

B provides additional qualitative results, highlighting the aesthetic advantage of Self‑OPD on task‑specific prompts.

Under the “same test images” protocol, Self‑OPD attains the highest PickScore and HPSv2 among all baselines—including Flow‑GRPO, GRPO‑Guard, DiffusionNFT, Flow‑OPD, and DiffusionOPD—while also satisfying the compositional requirements of GenEval prompts.

Qualitatively, Self‑OPD renders text accurately and produces richer lighting, more natural textures, and coherent backgrounds, whereas competing methods often yield flatter scenes, over‑saturation, or loss of detail.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers