Read it Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation

Runhui Huang, Qihui Zhang, Zhe Liu, Yu Gao, Jie Wu, Hengshuang Zhao

SpectraReward uses image-conditioned prompt likelihood as a training-free reward signal for text-to-image RL.

How can we use pretrained MLLMs as effective, training-free reward models for text-to-image reinforcement learning?

Reinforcement learning for image generation typically relies on expensive human preference labels or complex, error-prone question-decomposition pipelines to guide model training. SpectraReward bypasses these by using a frozen Multimodal Large Language Model (MLLM) to measure how well a generated image can be "read back" into the original prompt. It calculates the mean image-conditioned prompt log-likelihood, treating this semantic spectrum as a deterministic reward signal. This approach consistently improves generation performance across multiple benchmarks, with Self-SpectraReward—where a model uses its own understanding branch to score its generation—matching or outperforming much larger external reward models.

Paper Primer

SpectraReward treats the MLLM as an inverse translator: it conditions the model on the generated image and computes the likelihood of the original prompt tokens. This "semantic spectrum" provides a dense, token-level signal that reflects how accurately the image captures the prompt's requirements without requiring any fine-tuning or preference data.

Self-SpectraReward extends this to unified multimodal models by using the model's internal understanding branch to score its own generation branch. This creates a closed-loop, self-improving system that benefits from the natural distributional alignment between the model's visual comprehension and its generation process.

SpectraReward significantly improves compositional fidelity and instruction following compared to existing MLLM-derived reward methods.

On the TIIF-Bench benchmark, SpectraReward improves the BAGEL baseline by +10.0 on short prompts and +6.2 on long prompts at 512 resolution. Outperforms the strong AlphaGRPO baseline by 6.3 and 2.1 gains on TIIF-Bench and GenEval, respectively.

Reward-policy alignment is more critical than raw reward model scale.

Self-SpectraReward matches or surpasses external reward models that are 30× larger, suggesting that internal alignment between understanding and generation branches is a key factor for effective RL. Achieves +1.2 on GenEval and +2.1 on GenEval2 compared to the best external MLLM backbone.

Why is this approach more efficient than existing MLLM-based reward methods?

Unlike methods that require expensive human preference annotations or complex question-decomposition pipelines, SpectraReward is training-free and off-the-shelf, reusing the MLLM's existing image-text alignment knowledge via a single forward pass.

Does scaling the reward MLLM always lead to better performance?

No; the authors observe that scaling is not monotonic. Larger MLLMs may prioritize advanced reasoning tasks that are less relevant to the fundamental image-conditioned captioning required for this reward signal, with 30B-scale models often proving sufficient.

For researchers building unified multimodal models, this paper demonstrates that internal self-rewarding is a highly effective, resource-efficient alternative to external reward models, provided the understanding and generation branches are well-aligned.

Motivation and Core Concept

We expose why existing reward models are brittle and introduce SpectraReward as a training‑free alternative.

Current training‑free reward functions for image generation suffer from calibration issues: they either rely on noisy scalar judgments or on multi‑step verification pipelines that are brittle and complex. These shortcomings limit the reliability of reinforcement‑learning fine‑tuning for text‑to‑image models.

SpectraReward treats a frozen pretrained MLLM as a teacher that reads a generated image and asks how likely the original prompt could be reconstructed; the average token log‑likelihood becomes a deterministic reward.

Scalar scoring asks an MLLM to emit a single discrete alignment score for an image‑text pair, treating the model as a judge.

VQA decomposition rewrites the prompt into atomic verification questions, lets the MLLM answer each, and aggregates the confidences into a reward.

**Figure 1.** **Overview of SpectraReward.** (a) Pretrained MLLMs naturally induce a semantic spectrum that measures how well a generated image aligns with the prompt. SpectraReward aggregates this into a reward for T2I RL. (b) During RL training, SpectraReward steadily increases together with visible improvements in complex scene generation. (c) We study nine reward MLLM backbones from four MLLM families, with external reward MLLMs spanning three families and 4B to 235B parameters. Scaling the reward MLLM backbones brings non-monotonic gains. Qwen3-VL-30B-A3B achieves the best performance among external MLLMs, while Self-SpectraReward, using BAGEL's own understanding branch as the reward model, outperforms all external MLLMs. (d) Both SpectraReward and Self-SpectraReward bring significant and consistent improvements across all six downstream benchmarks compared to the baselines.

**Figure 2.** Comparison of MLLM-based reward functions. (a) Scalar scoring directly asks an MLLM to assign a discrete image-text alignment score, making the reward sensitive to judge calibration and scoring noise. (b) VQA decomposition converts the prompt into atomic questions and aggregates verifier confidence, but introduces a two-stage pipeline and depends on the quality of question decomposition. (c) SpectraReward computes the image-conditioned prompt likelihood through a single teacher-forced forward pass. The resulting token-level likelihoods form a semantic spectrum, whose average is used as the scalar reward. (d) Self-SpectraReward instantiates the same prompt-likelihood reward within a unified multimodal model by using the policy's own understanding branch, removing the need for an external reward MLLM and improving reward-policy alignment.

Existing reward models are brittle because they depend on noisy scalar judgments or complex verification pipelines, making RL fine‑tuning unstable.

The SpectraReward Mechanism

Describes the self‑reward mechanism that lets a unified model evaluate its own generations.

Training‑free reward functions for image generation are notoriously poorly calibrated, which makes reinforcement learning unstable. The core trick is to obtain a reward directly from a pretrained multimodal model’s own image‑conditioned language capability, sidestepping any external scoring pipeline.

The model scores its own generated image by asking its internal understanding branch how likely the original prompt is when conditioned on that image.

Sum the two token log‑likelihoods: $-0.20 + (-0.50) = -0.70$.

Divide by $T-1 = 2$ to obtain the average: $-0.70 / 2 = -0.35$.

The resulting reward $R(x,y) = -0.35$; a higher (less negative) value would indicate a better alignment between image and prompt.

This toy calculation shows that the reward directly reflects how well each token of the prompt is predicted given the image, making mismatches immediately penalized.

How does Self‑SpectraReward differ from using an external pretrained MLLM as a reward?

Self‑SpectraReward reuses the model’s own understanding branch, so the reward is computed without launching a separate large‑scale MLLM service. This removes extra latency and memory overhead and guarantees that the reward distribution is calibrated to the policy’s own generation space, whereas an external MLLM may be mismatched and costly.

Language‑prior cancellation: the raw reward contains the MLLM’s inherent language prior, but because every image in a prompt group shares the same prompt, the prior term is identical and therefore cancels out when computing group‑relative advantages.

Token‑level semantic sensitivity: when an image violates a specific attribute (e.g., “two cats” vs. five cats), the likelihood drop concentrates on the mismatched token (“two”). Similarly, swapping “guitar” for “chair” depresses the likelihood of the word “guitar” while leaving other tokens largely unchanged.

Reward ranking reliability: across multiple rollouts of the same prompt, SpectraReward consistently assigns higher scores to images that faithfully realize the prompt and lower scores to those with missing objects, wrong attributes, or incorrect spatial relations, providing a trustworthy ordering signal for RL.

Main Results and RL Performance

Self‑SpectraReward beats prior rewards on multiple benchmarks.

Self‑SpectraReward attains 89.8 on GenEval, a 5.5‑point gain over the BAGEL baseline.

Table 1 reports 89.8 for Self‑SpectraReward versus 84.3 for BAGEL at 512‑resolution inference.

The improvements extend across the full suite of text‑to‑image benchmarks, as shown in the comparative table.

AWM scales each policy gradient by how well the generated image aligns with the reward, smoothing updates and preventing destabilizing spikes.

How does AWM differ from standard PPO?

Standard PPO applies a fixed clipping ratio to the policy ratio, whereas AWM multiplies the gradient by a data‑driven alignment weight, directly modulating update magnitude based on reward quality.

**Figure 4.** The visual interpretation of SpectraReward. The reward ranking is consistent with the visual quality ranking.

**Figure 5.** Qualitative comparison.

SpectraReward consistently outperforms prior reward models across all evaluated benchmarks.

Backbone Sensitivity Analysis

Ablation of reward backbones reveals scaling limits and the strength of Self‑SpectraReward.

We evaluate SpectraReward with several MLLM backbones to see how backbone choice and scale affect reward quality. The experiments span two diffusion generators (SD3.5‑M and BAGEL) and include both external and self‑backbones.

**Figure 3.** **Token-level semantic sensitivity of SpectraReward.** Positive and negative images are evaluated using the same positive prompt. For attribute mismatch, instantiated as a counting error, the negative image mainly lowers the likelihood of “Two”; for object identity mismatch, replacing the guitar with a chair sharply lowers the likelihood of “guitar”. Bars show image-conditioned prompt-token log-likelihoods with error bars calculated over four pairs, and dashed lines show the resulting sequence-level reward value, i.e., SpectraReward.

Scaling Qwen3‑VL beyond 30B degrades GenEval performance.

GenEval drops from 85.2 at 30B to 73.2 at 235B (−12.2 points).

Ablation and Component Analysis

We isolate each component to see how it affects reinforcement‑learning performance.

We first compare three RL algorithms—FlowGRPO, AWM, and DiffusionNFT—under the Self‑SpectraReward regime; AWM delivers the strongest downstream results, so we adopt it for the main experiments.

Scalar scoring degrades performance sharply; GenEval falls by 6.3 points relative to the baseline Prompt Likelihood.

VQA‑Score yields modest improvements on the TIIF‑Bench metrics but does not raise GenEval, indicating that a binary yes/no query captures only limited alignment information.

In contrast, the image‑conditioned prompt likelihood used by SpectraReward consistently outperforms both alternatives across all three benchmarks, confirming that a likelihood‑based reward is the most effective way to harness MLLMs for image‑generation RL.

We also examine reward granularity. The default sequence‑level advantage averages token log‑likelihoods before computing group‑relative advantages, whereas the token‑level variant treats each token separately and then averages. Three normalization scopes are tested: global std, group std, and per‑token std.

Across all configurations, token‑level advantage provides no clear benefit over the sequence‑level approach; the latter remains the preferred default for stable training.

Extended Qualitative Analysis

Additional ablations reveal how EOS masking and VAE inputs impact reward quality.

Masking the EOS token during SpectraReward computation raises GenEval by 3.0 points.

GenEval improves from 85.1 to 88.5 when EOS is excluded (Table 6).

While GenEval benefits from EOS masking, TIIF‑Short slightly drops (89.5→85.1) and TIIF‑Long stays unchanged, suggesting the gain is specific to overall semantic alignment.

Including the VAE feature in Self‑SpectraReward reduces GenEval by 2.7 points.

GenEval drops from 87.8 to 85.1 when VAE input is used (Table 7).

The VAE input, while enriching visual detail, appears to distract the reward model from pure semantic alignment, leading to lower scores across all three metrics.

**Table 6.** The effect of including the EOS token in the reward calculation

**Figure 6.** Additional reward-ranking examples. For each prompt, samples are ordered from lower to higher SpectraReward reward.

**Figure 7.** Additional qualitative comparison. Red text highlights the prompt constraints that are especially sensitive to spatial relation, counting, attribute binding, or long-prompt composition. Self-SpectraReward more consistently satisfies these highlighted constraints than the BAGEL baseline.

Detailed Benchmark Results

Detailed benchmark tables reveal consistent gains of SpectraReward and Self‑SpectraReward across tasks and resolutions.

Self‑SpectraReward raises the TIIF‑Bench short‑prompt overall score from 90.4 (BAGEL) to 95.7, a +5.3 point gain.

Table 9 shows the short‑prompt overall scores for BAGEL and Self‑SpectraReward at 512‑resolution inference.

**Table 9.** Performance of proprietary models and state-of-the-art open-source models on TIIF-Bench testmini subset. Evaluated systems are grouped into (i) diffusion-based open-source models, (ii) autoregressive open-source models, and (iii) proprietary models. The results of SpectraReward and BAGEL are evaluated by GPT-4.1. “Inf. SRR” indicates executing the inference-time self-reflective refinement.

The improvements are not confined to a single metric; they appear across compositional generation, fine‑grained instruction following, and knowledge‑grounded generation, and they survive resolution up‑scaling.

Extended Benchmark Data

DPG‑Bench and WISE results show SpectraReward variants achieve top scores across resolutions.

SpectraReward achieves the highest 512‑resolution overall score on DPG‑Bench.

Table 10 reports a 93.33 overall score, surpassing BAGEL (86.11) and AlphaGRPO (84.33).

Self‑SpectraReward attains the top overall score at 1024 resolution.

Table 10 shows a 89.8 overall score, higher than BAGEL (86.67) and AlphaGRPO (84.33).

Self‑SpectraReward + Self‑CoT reaches the highest overall WISE score.

Table 11 reports an overall of 0.76, exceeding BAGEL (0.68) and AlphaGRPO (0.71).

SpectraReward also improves the global, relation, and “other” categories on DPG‑Bench, while Self‑SpectraReward yields the strongest attribute scores. On WISE, the Self‑CoT augmentation boosts performance in space, biology, physics, and chemistry domains.

DPG-Bench Evaluation

SpectraReward leads DPG‑Bench performance, with detailed comparisons across models.

SpectraReward attains the highest global DPG‑Bench score of 93.80 at 512 px resolution, outpacing all prior models.

Table 10 reports a global score of 93.80 for SpectraReward; the next best is 90.97 for DALLE3.

World Knowledge Reasoning

Emu3 tops the WISE benchmark with an overall score of 0.84.

Emu3 attains the highest overall WISE score of 0.84, outpacing the next‑best model by 0.08.

Table 11 shows Emu3 at 0.84 overall, while the runner‑up SD3.5‑large scores 0.76.

Conclusion and Summary

We recap SpectraReward’s training‑free reward and its self‑reward extension, then outline appendix contents.

Training‑free reward functions for image generation often mis‑calibrate; SpectraReward sidesteps this by measuring image‑conditioned prompt likelihood directly from pretrained MLLMs, yielding a stable RL signal without any preference data.

We also introduced Self‑SpectraReward, letting a model’s own understanding branch serve as its reward for the generation branch, creating a closed‑loop self‑improving system that rivals much larger external reward models.

Across diverse generators, RL algorithms, reward‑MLLM backbones, and out‑of‑distribution benchmarks, SpectraReward consistently boosts text‑to‑image quality and outperforms prior MLLM‑derived reward methods; notably, larger reward MLLMs do not guarantee better performance, while a well‑aligned self‑reward can match or exceed them.

The appendix details (A) the current limitations of prompt‑likelihood rewards and future research directions, (B) extra implementation specifics such as reward prompt formatting, EOS masking, and VAE feature usage in Self‑SpectraReward on the BAGEL dataset, and (C) supplementary experimental findings on EOS token removal and VAE‑feature effects.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers