Bridging Interleaved Multi-Modal Reasoning as a Unified Decision Process

Zican Hu, Xuyang Hu, Yiming Liu, Zuwei Long, Wei Liu, Yunzhuo Hao, Jiawei Gu, Linjie Li, Yu Cheng, Zhenhong Sun, Weibo Gu, Xing Sun, Zhi Wang

BRAID unifies text and image generation into a single reinforcement learning loop for interleaved multi-modal reasoning.

How can we unify text and image generation into a single reinforcement learning framework to improve interleaved multi-modal reasoning?

Unified multi-modal models often struggle to optimize interleaved text-image reasoning because reinforcement learning (RL) is typically applied only to text, leaving image generation to supervised surrogates that cannot adapt to task-specific rewards. BRAID casts the entire interleaved reasoning trajectory as a unified Markov decision process, allowing the model to propagate a shared advantage signal into both autoregressive text tokens and flow-matching image denoising paths simultaneously. This unified approach improves performance by +5.73 average across seven benchmarks, surpassing GPT-4o on spatial reasoning and visual perception tasks despite using a smaller 7B parameter model.

Paper Primer

BRAID treats every reasoning turn—whether text or image—as a macro-action within a single decision process. It bridges the modality gap by mapping a shared trajectory-level advantage to modality-native gradients: a clipped surrogate objective for text and a reward-aware flow-matching loss for images.

To solve the long-horizon credit assignment problem, BRAID employs a vision-language model judge that provides dense, turn-level feedback on intermediate images. This "vision-thinking" reward scores images on correctness, fidelity, utility, and trustworthiness, steering the model toward visual intermediates that genuinely advance the reasoning chain.

BRAID consistently outperforms existing unified multi-modal models and proprietary baselines on spatial and visual reasoning.

Average improvement of +5.73 over the BAGEL baseline across seven benchmarks, with specific gains of +14.00 on SAT and +10.76 on V*Bench. Surpasses GPT-4o performance on these benchmarks using only a 7B parameter model.

Ablation studies confirm that both the unified RL loop and the vision-thinking process reward are essential. Removing the image-branch optimization (LNFT) is more damaging than removing the process reward, proving that bringing visual generation into the RL loop is the primary driver of the model's reasoning gains.

Why is it difficult to apply RL to both text and images in a single model?

Text generation uses discrete autoregressive likelihoods, while image generation via flow matching operates on continuous denoising paths. Naively combining them causes optimization collapse due to incompatible scales, which BRAID solves by normalizing advantages relative to modality-native baselines.

Does this approach require a new model architecture?

No. BRAID is a training framework built on top of existing hybrid AR-diffusion backbones (specifically BAGEL). It optimizes the existing model parameters end-to-end without changing the underlying architecture.

The core assumption is that visual generation acts as a "visual tool" for reasoning; BRAID succeeds because it forces the model to generate images that are not just visually coherent, but specifically useful for the reasoning task at hand.

Researchers can now optimize interleaved multi-modal reasoning end-to-end, moving away from supervised surrogates toward principled, reward-driven discovery of visual reasoning strategies.

Introduction and Motivation

Identifies why joint RL for text‑image reasoning is missing and introduces BRAID.

Existing reinforcement‑learning pipelines for unified multimodal models update only the textual branch, while image generation remains a frozen supervised component. This asymmetry blocks policy‑gradient flow across the full interleaved trajectory, preventing the model from learning how visual steps contribute to downstream reasoning.

Reasoning proceeds as a back‑and‑forth between language and vision: a text turn proposes a concept, an image turn visualizes it, and the next text turn refines the argument based on that picture.

To close this gap we propose BRAID, which casts the entire text‑image‑text chain as a unified two‑level Markov Decision Process. By defining a single trajectory‑level advantage, BRAID injects the same learning signal into both the autoregressive text policy and the flow‑matching image policy.

**Figure 1.** SFT, Half-optimized RL, and BRAID.

The core gap is the lack of a unified reinforcement‑learning objective that can propagate gradients through both text and image turns of an interleaved reasoning trajectory.

Related Work and Background

This section equips the reader with the background concepts needed to understand BRAID’s reinforcement‑learning formulation.

CoT decomposes a difficult problem into a sequence of explicit intermediate reasoning steps, letting the model “think out loud” before producing the final answer.

The BRAID Framework

Method details BRAID’s two‑level MDP, unified policy optimization, and vision‑thinking reward design.

Interleaved text‑image reasoning suffers from a sparse terminal reward that cannot pinpoint which intermediate turns actually drive the final answer, especially for non‑autoregressive image generation.

BRAID treats the alternating text‑image chain‑of‑thought as a single decision process, letting a unified policy generate both modalities while sharing a common notion of progress.

We abstract each reasoning turn as a macro‑action (text or image) and then unroll that macro‑action into a sequence of micro‑actions (tokens or diffusion steps), yielding a hierarchical Markov decision process.

Turn 1 state s₁ = (c, ∅). The policy samples a text chunk T₁ = “The object is a red cube”.

Turn 2 state s₂ = (c, T₁). The policy samples an image I₁ depicting a red cube.

Turn 3 state s₃ = (c, T₁, I₁). The policy samples a text chunk T₂ = “It sits on a table”.

Turn 4 state s₄ = (c, T₁, I₁, T₂). The policy samples an image I₂ showing the cube on a table.

Turn 5 state s₅ = (c, T₁, I₁, T₂, I₂). The policy emits the final answer `T_ans` = “Yes, the scene matches the description”.

This toy illustrates how each macro‑action (text or image) expands into a deterministic state update, enabling the same advantage signal to flow to every token or diffusion step within the turn.

How does this two‑level MDP differ from standard hierarchical RL formulations?

Standard hierarchical RL typically defines separate high‑level and low‑level policies with their own reward signals. Here the high‑level macro‑action is simply “emit a whole text chunk or a whole image”, but the low‑level micro‑actions inherit the *same* turn‑level advantage rather than a distinct sub‑reward, keeping the optimization unified across modalities.

Both text and image branches are updated together by feeding the same turn‑level advantage into modality‑specific surrogate losses, so learning stays synchronized despite different generation mechanisms.

Compute soft label: r₁ = $\sigma$(2.0 / 0.5) ≈ $\sigma$(4) ≈ 0.982.

Sample a diffusion step t=0.3, generate noisy image $x_{t}$=0.7 I₁ + 0.3 $\epsilon$.

Network predicts velocity v = $\theta$($x_{t}$,s₁,t).

Loss contribution: 0.5 · 0.982 · ‖v - ($\epsilon$ - I₁)‖² + 0.5 · (1‑0.982) · ‖v‖².

When the advantage is high, the loss heavily penalizes deviation from the direction that would denoise toward the target image, effectively steering generation toward rewarding visual content.

Why not use a separate reward for image generation instead of mapping the same advantage through a sigmoid?

Mapping preserves the *relative* ordering of advantages across turns, letting the same scalar guide both text and image updates while the sigmoid ensures the label stays in a stable [0,1] range suitable for the diffusion loss.

Beyond the final answer, we score each intermediate image on multiple visual criteria, producing a dense vector of rewards that directly informs the image‑generation loss.

**Figure 2.** Overview of BRAID. (a) Interleaved TIT-CoT reasoning formulated as a two-level MDP. (b) Unified policy optimization via GRPO for text and DiffusionNFT for image. (c) Decoupled advantage estimation combining terminal reward and vision-thinking process reward.

Empirical Evaluation

BRAID sets a new performance ceiling for 7B‑parameter multimodal models.

BRAID attains a 65.19 average score, out‑performing GPT‑4o (62.46) and Qwen2.5‑VL‑7B (62.11) while using only 7 B parameters.

Average across seven multimodal benchmarks reaches 65.19, the highest reported for a 7 B‑parameter model.

**Table 1.** Performance comparison on multimodal spatial reasoning and visual perception benchmarks. Best results in bold and second best underlined.

**Figure 3.** Maj@n and Avg@n curves on VStar and SAT benchmarks with varying number of samples n = {3, 5, 9, 17}. Complete results are available in Table 5.

**Figure 5.** Complete results are available in Table 7.

**Figure 6.** Complete results are available in Table 6.

We train on eight task families (Matterport3D, RealSee3D, Perspective, DirectionalQuery, 360+x, PositionedDirection, Jigsaw, VisualSearch) and split each into SFT and RL pools using three ratios (D2:1, D1:1, D∩), defaulting to the partial‑overlap D∩ regime.

Baselines include proprietary VLMs (GPT‑5.4, GPT‑4o, Qwen2.5‑VL) and competitive unified multimodal models (Janus‑Pro, Chameleon, BAGEL). RL training omits the KL loss term and uses rollout batch 64, update batch 32, eight rollouts per prompt.

Qualitative case studies (Appendix C) show that BRAID produces faithful intermediate images that directly support correct reasoning, whereas the LNFT‑free ablation generates distorted visuals and the rvis‑free variant inserts spurious artifacts.

When transferred to pure text‑to‑image prompts, only BRAID satisfies both spatial composition and numerical precision; the ablations fail to follow instructions or maintain numeric accuracy.

Ablation Study

Component‑wise ablations reveal which tricks drive BRAID’s gains.

BRAID treats interleaved text‑image reasoning as a two‑level MDP, letting a single RL policy optimise both modalities together.

Removing the visual‑reward term (rᵥᵢₛ) drops the overall average score by 2.5 points.

Full BRAID avg = 65.19; BRAID w/o rᵥᵢₛ avg = 62.68 (Table 4).

Removing the dense‑process reward (`L_NFT`) reduces the average score by 3.1 points.

Full BRAID avg = 65.19; BRAID w/o `L_NFT` avg = 62.10 (Table 4).

On VStar, BRAID’s majority‑vote at n = 17 outperforms SFT by 3.84 points.

BRAID Maj@17 = 65.97; SFT Maj@17 = 62.13 (Table 5).

On VStar, BRAID’s average‑vote at n = 17 outperforms SFT by 6.34 points.

BRAID Avg@17 = 67.53; SFT Avg@17 = 61.19 (Table 5).

The replay regime (𝔇∩) lifts SAT performance by 9.34 points over the D2:1 regime.

D2:1 RL SAT ≈ 55.81; 𝔇∩ RL SAT = 65.15 (Table 6 description).

**Figure 4.** Ablation results. Left: Final test accuracy across seven benchmarks. Middle: Training average accuracy. Right: Training average format reward. Complete results are available in Table 4.

Qualitative Analysis

Qualitative comparison of BRAID and its ablations on a VStar visual‑search case.

All BRAID variants answer the VStar license‑plate query correctly (100 % accuracy).

Rollout 1 and Rollout 2 of full BRAID, as well as the two ablated variants, each produce answer A.

The ablated variants still output the correct answer, but their visual intermediates are either hallucinated (w/o rvis) or blurred (w/o LNFT), exposing a fragile reliance on visual evidence that can fail under stricter evaluation.

**Figure.** A photograph of a white police van parked on a street, showing its rear license plate.

The image shows the rear view of a vehicle, specifically focusing on the license plate and the lower bumper area. The license plate is yellow with black text reading "OU68 EPD". The vehicle has high-visibility orange and yellow diagonal stripes on the rear doors and a yellow step bumper at the bottom.

A photograph showing the rear view of a white police van parked on a street. The van features high-visibility yellow and orange chevron markings on the rear doors and blue and yellow reflective strips along the side. The license plate, which reads "OU68 EPD", is clearly visible on the rear bumper. In the background, a brick building and a sidewalk with pedestrians are visible.

**Figure.** Generated Image.

The image shows a close-up of the rear section of a vehicle, likely an emergency service vehicle, featuring a yellow license plate with the registration "GI01 EPD". The vehicle body is white with high-visibility yellow and orange chevron markings. The rear bumper area appears damaged, with a white plastic component hanging down below the license plate.

Generalization Analysis

BRAID’s RL training improves text‑to‑image generation across spatial, counting, and creative prompts.

BRAID is the only method that generates exactly five balloons, achieving perfect numerical accuracy.

All baselines (BAGEL, SFT, w/o LNFT, w/o rvis) produce incorrect counts ranging from four to eight.

The vision‑thinking reward rvis guides the image branch to respect visual constraints, and the DiffusionNFT pathway (LNFT) is essential for propagating this signal.

**Figure 7.** Text-to-image generation comparison.

**Figure 8.** Text-to-image generation comparison across prompts testing spatial compositionality (Row 1), numerical accuracy (Row 2), and creative concept binding (Row 3).

Conclusion and Limitations

Limits of BRAID and avenues for future work.

BRAID treats interleaved text‑image reasoning as a unified two‑level MDP, letting a single policy optimise both modalities together.

The current implementation inherits BAGEL’s hybrid autoregressive‑diffusion backbone; moving to a fully autoregressive UMM would broaden applicability but requires redesigning the tokenisation and generation pipeline.

The vision‑thinking reward $r_{\text{vis}}$ is obtained from an external VLM judge, which adds inference cost and creates a reproducibility bottleneck; distilling this signal into a compact, jointly‑trained PRM could alleviate both issues.

Our formulation assumes a fixed interleaving pattern (text → image → text …). Allowing the policy to choose turn boundaries adaptively may yield richer reasoning strategies.

Beyond adaptive turns, a hierarchical advantage decomposition that attributes credit across multiple turns could further sharpen RL credit assignment for complex interleaved tasks.

**Table 2.** Per-task sample counts under the three data-allocation regimes. In $\mathcal{D}^{\cap}$, the RL pool extends the original RL split with 4,800 samples drawn from the SFT pool, yielding an overlap ratio $\omega \approx 0.32$.

**Table 1.** Hyperparameters used for training.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers