UNIEVO-VL: an on-Policy Self-Distillation Training Recipe for Multimodal Model Self-Improvement

Fang Wu, Da Xing, Yanjie Huang, Junxi Wang, Ji Wang, Hejia Geng, Guancheng Wan, Bowen Zuo, Xiaomin Li, Shixiang Tang, Xinyu Xiang, Zehong Wang, Shiyi Du, Peng Xia, Shuangjia Zheng, Yining Hong, Li Erran Li, Jure Leskovec, Yejin Choi

UniEvo-VL uses critique-conditioned on-policy self-distillation to improve multimodal generation without external rewards.

How can multimodal models use their own visual understanding to iteratively improve their image generation capabilities through self-distillation?

Multimodal models can identify their own generation errors, but they lack a direct way to internalize this feedback into their base generation policy. UniEvo-VL treats self-critiques as privileged information: it conditions a teacher model on a revised prompt while the student learns to match the teacher's denoising trajectory using only the original prompt. This on-policy self-distillation improves compositional generation performance on benchmarks like GenEval and GenEval2 without requiring external reward models or corrected-image targets.

Paper Primer

The core mechanism hinges on converting visual critique into a revised prompt that acts as a "privileged" teacher signal. The student model, which only sees the original prompt, is trained to minimize the divergence between its denoising transitions and those of the teacher, effectively distilling the corrective logic into the model's weights.

UniEvo-VL significantly improves direct generation performance on compositional benchmarks.

Performance on GenEval increased from 0.747 to 0.808, and GenEval2 Soft-TIFA scores rose from 32.97 to 35.53. A ~8% relative gain on GenEval and ~7.7% on GenEval2.

The authors distinguish between improvements retained in the model and those gained via inference-time reflection. While training improves direct generation, reflection remains useful, suggesting that internalizing corrective experience and using feedback at inference time are complementary strategies.

Why use a revised prompt as privileged information instead of just training on the corrected image?

Training on corrected images can be brittle; by distilling the corrective conditioning into the denoising policy, the model learns how to adjust its generation process to satisfy constraints rather than simply memorizing a specific target output.

Does this approach require an external, more powerful model to provide the critiques?

No, the framework is designed for self-evolution; the model acts as both teacher and student. However, experiments show that using more powerful external critics (like GPT-5.6-Luna) can raise the performance ceiling of the self-evolving model.

UniEvo-VL demonstrates that multimodal models can effectively self-improve by treating their own critiques as a distillation signal, shifting the burden of correction from inference-time prompting to the model's core generation policy.

Introduction and Motivation

UniEvo‑VL lets a multimodal model improve itself by learning from its own visual critiques.

Modern multimodal models can both generate images and understand them, opening the possibility for the model to critique its own outputs and use that feedback to improve future generations. Leveraging this unified capacity, we propose UniEvo‑VL, a framework that turns visual critiques into privileged supervision via On‑Policy Self‑Distillation (OPSD), allowing a single model to act as both teacher and student.

**Figure 1.** **From understanding to self-evolution.** Generated drafts become training experience through visual critique and generator updates. The conceptual loop motivates both self-critique and an optional extension with a stronger external critic.

It treats the model’s own visual critique as a privileged signal, feeding that signal back as a revised prompt so the generator can learn to correct its mistakes without external teachers.

The shift from static training to a self‑evolving generation loop enables multimodal models to continuously improve from their own feedback.

The UniEvo-VL Training Recipe

We detail the self‑evolution loop that turns critiques into improved generation.

The model repeatedly generates an image, critiques it, and uses the critique to revise the prompt, forming a loop that gradually improves the output.

Generation: $I = M_{\text{gen}}^{\theta}(p,\epsilon)$ yields three cups.

Assessment: $M_{\text{und}}^{\theta}(p,I)$ returns $c=$ “need exactly two cups”, $a=0$.

Revision: $p' = M_{\text{und}}^{\theta}(p,c)$ becomes “two red cups, exactly two”.

Verification: generate $I' = M_{\text{gen}}^{\theta}(p',\epsilon)$; assessment now yields $a=1$.

Update: EMA teacher $\bar{\theta}$ receives $p'$ as privileged input; student $\theta$ continues learning from $p$.

The loop converts a concrete visual error into a semantic prompt correction, guaranteeing monotonic improvement without external supervision.

How does Self‑Evolution Iteration differ from simply re‑running the generator with the same prompt?

In naive re‑generation the prompt never changes, so the model can only hope stochastic noise will fix the error. Here the model’s own critique $c$ rewrites the prompt, providing a targeted semantic correction that the student learns to anticipate.

After a revised prompt is created, the model checks that the new image truly satisfies the prompt before accepting the revision.

Generation: $I' = M_{\text{gen}}^{\theta}(p',\epsilon)$ now contains exactly two red cups.

Assessment: $M_{\text{und}}^{\theta}(p',I')$ returns $a'=1$, confirming the correction.

Verification passes, so $(p',\epsilon)$ is added to the experience buffer.

The verification step filters out revisions that would otherwise propagate a new error, acting as a safety net for the self‑distillation loop.

Why is a second assessment needed if the first critique already identified the discrepancy?

The first critique only tells the model *what* is wrong; the second assessment confirms that the *proposed* fix actually resolves the problem, preventing the loop from learning from a still‑flawed prompt.

The critic’s feedback becomes a privileged prompt that guides a fixed teacher model, which then distills its knowledge into the student that only sees the original prompt.

Teacher forward: $\text{gen}_{\bar{\theta}}(s_j,p')$ produces a latent prediction aligned with the corrected image.

Student forward: $\text{gen}_{\theta}(s_j,p)$ produces a latent prediction based only on the original prompt.

Distillation loss $L_{\text{distill}}$ measures the KL divergence between the two predictions at each step $s_j$.

Gradient updates adjust $\theta$ to reduce this divergence, while $\bar{\theta}$ remains fixed.

The student gradually learns to generate images that already satisfy the correction, removing the need for the privileged prompt at inference time.

Why is the teacher kept fixed while the student is updated?

The teacher provides a stable target distribution; if both models moved simultaneously they could chase each other, preventing convergence. Fixing the teacher isolates the learning signal.

Algorithm 1: UniEvo‑VL (Critique‑conditioned on‑policy self‑distillation)

**Figure 2.** The UniEvo-VL training loop. A generated draft is inspected along semantic, text, and visual-quality axes. The resulting feedback is synthesized into a revised prompt, which acts as the privileged condition for the EMA teacher. Transition matching updates the student's generator LoRA; the critic is fixed. At deployment, the evolved model uses the original prompt directly.

Empirical Performance

UniEvo‑VL consistently outperforms the Base model across all generation tasks.

UniEvo‑VL (GPT‑5.6‑Luna) achieves the highest native GenEval score of 0.882.

Table 1 lists a native score of 0.882 for the GPT‑5.6‑Luna configuration, exceeding the Base model on every metric.

A collection of image‑generation tests that jointly assess semantic correctness, compositional fidelity, and visual quality.

How does the GenEval Benchmark differ from standard image‑generation benchmarks?

Standard benchmarks usually report a single aggregate score, whereas GenEval splits evaluation into three orthogonal axes (semantic, compositional, and OCR) and provides multiple metrics per axis, enabling fine‑grained analysis of where a model improves or regresses.

**Figure 3.** UniEvo-VL with revision verification across training updates. Solid curves show direct generation; dashed curves allow one inference-time critique. Lines connect evaluated checkpoints across runs; n counts plotted points.

**Figure 4.** Improvement and preservation on historical 500-prompt subsets, where scale is normalized to 0–100. The percentage-point changes are subset-specific, not full-test-set gains.

**Table 1.** Performance of UniEvo-VL. Parentheses identify the critic; “+ verification” denotes revision-prompt filtering. Base is averaged over different random seeds of the identical pretrained model. Bold and underlined values indicate the best and second-best for each dataset and metric.

**Table 2.** **Direct generation and reflection before and after training.** (a) Generation scores, where bold marks the highest score in each row. (b) Training gain is the UniEvo-VL score minus the base score under direct generation, while reflection gain is reflected minus direct within each model. Gains use unrounded scores. Native scores use 0–1 for GenEval/OCR and 0–100 for GenEval2. Atomic uses percent (gains in percentage points), and HumanPref uses 0–10.

**Table 3.** Compositional generation under different critics. All entries are Gemini compositional-fidelity ratings (0–10). (a) GenEval uses 553 prompts; Base denotes the Qwen-VL experiment's baseline. (b) GenEval2 uses 800 prompts; $\Delta$ is Evolved minus Base. Bold indicates the highest score in each column of (a), and the higher Evolved score and larger $\Delta$ within each row of (b), including ties.

UniEvo‑VL consistently outperforms base models across generation tasks.

Ablations and Qualitative Analysis

We probe how each component affects performance and document extra analyses.

D.5 examines how difficulty varies across fixed 500‑prompt subsets for each task, separating “Easy” (baseline scores 5–10) from “Hard” (0–4) before training.

Normalized difficulty gains are reported as percentage‑point changes on a 0–100 scale; the denominator is the full score range, not the baseline mean, and values are rounded to the nearest integer.

**Table 8.** Full-set learning curves without post-revision verification, rounded to three decimals. Each checkpoint uses 553 GenEval, 800 GenEval2, and 1,018 OCR prompts; task and HumanPref scores retain their 0–10 scale.

D.6 reports native GenEval2 Soft‑TIFA scores across update steps, using 800 prompts and the released Qwen3‑VL‑8B‑Instruct VQA evaluator.

**Figure 6.** Complete GenEval2 Soft-TIFA trajectory for GPT-5.6-Luna critique, updates 0–300. The Qwen-critic baseline and update-200 endpoints are shown as separate markers. GM and AM use each run's own baseline.

D.7 notes that GenEval’s reliance on object detection, color classification, and geometric rules can penalize images that otherwise satisfy the prompt, explaining why high human‑assessed correctness does not always translate to high GenEval scores.

D.8 describes ten instances where the judge API returned no response (`block_reason` = OTHER), meaning the system could not parse a judgment rather than indicating a prompt violation.

D.9 observes a stylistic bias toward muted colors introduced by the feedback‑prompt design, which improves human‑preference scores but may accumulate unwanted aesthetic effects.

**Figure 7.** Qualitative examples of UniEvo-VL on GenEval2. Examples cover object counts, colors, materials, and patterns. Rows show the base model, UniEvo-VL direct generation, and UniEvo-VL with up to one critique. Each column shares the original prompt and seed; both UniEvo-VL rows use checkpoint 160 of the Qwen critic-confirmed variant. A KEEP decision retains the direct image.

**Figure 8.** Qualitative examples of UniEvo-VL on OCR. Examples illustrate text rendering across varied scenes; headings summarize the scene and quote the requested text. Rows show the base model, UniEvo-VL direct generation, and UniEvo-VL with up to one critique. Each column shares the original prompt and seed; both UniEvo-VL rows use checkpoint 130 of the Qwen critic-confirmed variant. A KEEP decision retains the direct image.

Instead of learning from on‑policy critiques, SFT tries to teach the generator to reproduce an accepted refined image directly from the original prompt.

How does SFT differ from the on‑policy self‑distillation (OPSD) used in the main method?

OPSD trains the student on the same on‑policy rollout states that the teacher generates, preserving the temporal dynamics of the generation process. SFT instead creates static targets from accepted images and interpolates latents, discarding the on‑policy rollout and therefore missing the curriculum‑like signal that OPSD provides.

Related Work and Conclusion

We situate UniEvo‑VL among prior self‑distillation and feedback‑driven image generation works.

Recent work leverages visual understanding to guide image generation, from Han et al. (2025) scoring drafts with an understanding branch to SRUM (Jin et al., 2025) combining image‑level and object‑level self‑rewards. UniCorn (Han et al., 2026) introduces proposer, solver, and judge roles within a single model, while ReflectionFlow (Zhuo et al., 2025) refines images via flawed‑image, reflection, and improved‑image triplets. UniReason (Wang et al., 2026) adds world‑knowledge reasoning before generation and visual refinement after, and UniEvo‑VL extends this line by transferring corrective feedback into direct generation, measuring the added benefit of reflection after training.

Feedback‑driven optimization appears in Meta‑TTRL (Tan et al., 2026) aggregating rubric assessments, Flow‑GRPO (Liu et al., 2025) and DiffusionNFT (Zheng et al., 2025) updating diffusion generators via rewards, and DiffusionOPSD (Zhou et al., 2026) turning image‑level gradients into clean‑output targets. On‑policy distillation with privileged conditioning is explored by OPD (Agarwal et al., 2024), SDPO (Hübotter et al., 2026), DiffusionOPD (Li et al., 2026), D‑OPSD (Jiang et al., 2026), and Dreaming in Flow (Hao et al., 2026), each matching teacher and student trajectories without paired target images. UniEvo‑VL similarly derives privileged conditioning from its own visual critiques, updating only the generator while keeping a fixed critic, thereby avoiding gradient flow through the critic.

We introduced UniEvo‑VL, an on‑policy self‑distillation approach that turns visual critique into supervision for generation. Evaluations on GenEval, GenEval2, and OCR show training improves native scores both with and without reflection, with the combined setup achieving the highest metrics. However, gains vary across configurations and regressions on easier prompts reveal limits to preservation, highlighting challenges for self‑evolution.

Implementation and Configuration

Implementation choices for the generation‑understanding pipeline and training recipes.

We pair Qwen‑Image‑2512 with Qwen‑VL because the former already embeds a Qwen2.5‑VL encoder, making the latter a natural partner for visual‑understanding‑driven critique.

The flow‑based objective predicts a local velocity $\theta(s_j, p)$ for each latent step $j$ and penalizes deviation from a reference prediction using the loss $L_{\text{flow}}(\theta)$.

During acquisition we keep the base generator weights, text encoder, VAE, and critique modules frozen; only LoRA parameters of the image‑generation branch receive gradient updates.

**Table (b) GenEval2**

Two training recipes are provided: one without post‑revision verification (Table 4) and one with verification (Table 5); they differ in feedback backend, learning rate, and gradient‑accumulation settings.

Critique Interface Details

Details the training setup, critique workflow, and evaluation pipelines for UniEvo‑VL.

Inference can run in a direct mode or with a single critique iteration; the latter keeps an accepted draft and only regenerates when needed.

**Table 4.** Training configuration for acquisition without post-revision verification.

A.4 explains that optimizer updates do not reflect a fixed compute budget because each update follows generation, critique, and acquisition filtering, with extra regeneration when verification is required.

With post‑revision verification, only about 10–21 % of acquisition attempts are accepted, meaning five to ten attempts are needed per accepted training pair.

A.5 reports that the main runs use an EMA teacher decay of 0.999; slower updates (e.g., 0.995) improve performance, while a decay of 0.99 becomes unstable.

**Figure 5.** EMA teacher decay on GenEval. Task and HumanPref scores across three EMA decays. Both metrics favor 0.995 and 0.999 over 0.99, with 0.995 achieving the highest scores in this sweep.

B Critique Interface operates on the original prompt and generated image, splitting feedback into semantic, text, and quality axes before synthesis.

The semantic axis checks object presence, counts, attributes, and spatial relations; the text axis verifies exact wording; the quality axis flags anatomical or rendering defects.

Synthesis combines actionable corrections into a standalone generation prompt that preserves the user’s intent.

If no visible problem is found, the system emits KEEP, leaving the draft unchanged.

Prompt synthesis matters because raw criticism is negative prose, whereas the generator needs a positive, concrete specification to produce the corrected image.

Matching on student states ensures the teacher’s guidance targets the student’s current noisy representation, providing local supervision.

C Datasets include GenEval, GenEval2, and an OCR collection, each providing textual specifications for image generation without paired targets.

GenEval evaluates compositional fidelity across six categories (single, two, count, color, position, attribute binding) using 553 prompts.

GenEval2 expands the challenge with 800 prompts and higher atomicity, testing the ability to satisfy many simultaneous constraints.

The OCR task measures how accurately generated images render quoted text strings within varied scene contexts.

D defines three scoring protocols—Gemini atom checks, holistic Gemini ratings, and native benchmark metrics—and clarifies that evaluation does not feed back into training.

D.1 details the Gemini evaluation pipeline, including the check accuracy `S_check` and strict success `S_strict` formulas.

**Table 5.** Training configuration for acquisition with post-revision verification on all three tasks. Gradient accumulation operates over transition-loss evaluations.

The scoring formulas are `S_check` = (∑ `z_ij`)/(∑ $K_i$) and `S_strict` = (1/N) ∑ 1(∑ `z_ij` = $K_i$), measuring average pass rate and all‑check success per image.

D.2 presents native benchmark metrics: the official GenEval detector‑based score, the Soft‑TIFA probabilistic evaluator for GenEval2, and the Flow‑GRPO OCR edit‑distance metric.

The official GenEval metric averages six category accuracies, weighting each category equally.

GenEval2’s Soft‑TIFA computes both geometric and arithmetic means of answer probabilities, scaling results by 100.

Flow‑GRPO OCR compares recognized text to the target string using normalized Levenshtein distance, awarding a score between 0 and 1.

D.3 compares UniEvo‑VL against prior methods; Table 7 reports per‑category and overall scores for direct and one‑critique settings.

**Table 6.** Full-cohort Gemini evaluation for the verified recipe. Image counts are per checkpoint and inference condition. HumanPref is evaluated separately on each corresponding image pool.

D.4 analyzes the full judge‑less learning curves, noting that post‑revision verification stabilizes training especially on harder tasks.

The table presents a performance comparison of various methods across different categories: Single, Two, Count, Color, Position, Attr., and Overall.

Critic Prompt Templates

Details the critic interfaces, lenses, and example prompts used for image evaluation and revision.

This appendix records the exact critic and prompt‑synthesis interfaces used in the Qwen‑based acquisition‑with‑verification experiments, employing Qwen3‑VL‑8B‑Thinking for image critique and Qwen3‑VL‑8B‑Instruct for generating revised prompts.

**Algorithm 2** SFT on verified revised images (one-revision recipe) **Require:** Model interfaces $\mathcal{M}_{\theta}^{\text{gen}}$ and fixed $\mathcal{M}_{\theta}^{\text{und}}$, prompt batches $\mathcal{P}_k$, schedule $\{\sigma_j\}_{j=0}^T$, selected indices $\mathcal{J}$ 1: **for** $k = 0, 1, \dots$ **do** 2: $\quad \mathcal{A}_k^{\text{SFT}} \leftarrow \varnothing$ *Verified experience acquisition (no gradients)* 3: $\quad$ **for** each $p \in \mathcal{P}_k$ **do** 4: $\quad\quad$ Choose seed $r$; draw $\epsilon \sim \mathcal{N}(0, \mathbf{I})$ using $r$ 5: $\quad\quad I \leftarrow \mathcal{M}_{\theta}^{\text{gen}}(p, \epsilon)$ 6: $\quad\quad (c, a) \leftarrow \mathcal{M}_{\theta}^{\text{und}}(p, I)$ 7: $\quad\quad$ **if** assessment is invalid or $a = 1$ **then** 8: $\quad\quad\quad$ **continue** 9: $\quad\quad$ **end if** 10: $\quad\quad \tilde{p} \leftarrow \mathcal{M}_{\theta}^{\text{und}}(p, c)$; skip if invalid 11: $\quad\quad I' \leftarrow \mathcal{M}_{\theta}^{\text{gen}}(\tilde{p}, \epsilon)$ 12: $\quad\quad (c', a') \leftarrow \mathcal{M}_{\theta}^{\text{und}}(p, I')$ 13: $\quad\quad$ **if** assessment is valid and $a' = 1$ **then** 14: $\quad\quad\quad \mathcal{A}_k^{\text{SFT}} \leftarrow \mathcal{A}_k^{\text{SFT}} \cup \{(p, \text{sg}[I'], r)\}$ 15: $\quad\quad$ **end if** 16: $\quad$ **end for** *Accepted-image supervision* 17: $\quad$ **for** each optimizer accumulation window $\mathcal{C} \subseteq \mathcal{A}_k^{\text{SFT}}$ **do** 18: $\quad\quad \ell_c \leftarrow 0$ 19: $\quad\quad$ **for** each $(p, I', r) \in \mathcal{C}$ **do** 20: $\quad\quad\quad z^* \leftarrow \text{sg}[\text{Enc}(I'; r)]$ 21: $\quad\quad\quad$ Recreate Gaussian $\xi$ using $r$; $u \leftarrow \text{sg}[\xi - z^*]$ 22: $\quad\quad\quad$ **for** each $j \in \mathcal{J}$ **do** 23: $\quad\quad\quad\quad z_j^{\text{SFT}} \leftarrow (1 - \sigma_j)z^* + \sigma_j \text{sg}[\xi]$ 24: $\quad\quad\quad\quad \ell_c \leftarrow \ell_c + \frac{\|\mathcal{M}_{\theta}^{\text{gen}}(z_j^{\text{SFT}}, p) - u\|_2^2}{|\mathcal{C}||\mathcal{J}|d}$ 25: $\quad\quad\quad$ **end for** 26: $\quad\quad$ **end for** 27: $\quad\quad$ Accumulate $\nabla_{\theta} \ell_c$ over LoRA parameters 28: $\quad\quad$ Clip gradients, apply one AdamW update, and clear gradients 29: $\quad$ **end for** 30: **end for** 31: **return** $\mathcal{M}_{\theta}^{\text{gen}}$

For each generated image $I$ from prompt $p$, the critic receives the image and the current evaluation prompt and queries three complementary lenses: semantic fidelity, rendered text, and visual quality.

Each lens returns either a single actionable correction or the exact response “No issue on this lens.” Only the final textual verdict is passed downstream; internal reasoning is discarded.

The prompt‑synthesis module combines the original prompt with any non‑empty, axis‑labeled critic notes to produce a self‑contained text‑to‑image prompt; if all lenses report no issue, the pipeline emits “KEEP” and bypasses synthesis.

After revision, the regenerated image $I'$ is evaluated against the original request $p$; the experience is retained only when this second assessment is valid and finds no remaining actionable discrepancy, while the revised text serves as the privileged teacher during distillation.

F.2 introduces the exact semantic critic prompt, which uses the fidelity lens to enforce object identity, exact counts, attributes, and explicit spatial relations.

The fidelity rules prioritize the head noun, ignore prompt‑silent extras, and require a complete end‑state description rather than a “change X to Y” phrasing.

Example 1: a purple banana prompt – the image shows a yellow banana; **Fix**: a single banana whose whole skin is solid purple, stem to tip, natural banana shape, on the wooden table.

Example 2: a jazz musician prompt – the image displays only icons; **Fix**: a portrait of one jazz musician as a person in a studio setting, with no icons or text.

Example 3: a green pear prompt – the image matches the request, so the verdict is “No issue on this lens.”

F.3 defines the exact visual‑quality critic prompt, which applies the aesthetic lens to assess lighting, exposure, structural artifacts, composition, and detail without considering prompt compliance.

The aesthetic scope includes natural lighting, correct white balance, recoverable highlights and shadows, and realistic contrast, while never flagging AI‑generated appearance or style preferences.

Example 1: a white ceramic mug scene – the image suffers from an implausible color cast, clipped highlights, and muddy shadows; **Fix**: render with natural window light, accurate white balance, recovered highlight and shadow detail, realistic ceramic and wood texture, and moderate natural contrast.

Example 2: a lighthouse scene – the lighthouse is cropped at the top with excessive foreground; **Fix**: reframe to include the whole lighthouse and tighten the foreground.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers