Concurrent Image Understanding and Generation: Self-Correcting Coupled Markov Jump Processes

Minh-Quan Le, Armand Comas, Alexandros Lattas, Stylianos Moschoglou, Pedro Vélez, Amit Raj, Aaron Germuth, Thabo Beeler, Dimitris Samaras, Di Qiu

A training-free sampler that couples text and image generation via cross-modal attention and self-correcting remasking.

How can we prevent modality drift in joint image-text generation by coupling their diffusion processes through a self-correcting feedback loop?

Existing parallel multimodal samplers treat text and image generation as independent processes, leading to cross-modal contradictions that cannot be corrected once tokens are committed. CO2Jump introduces a coupled birth-death jump process where each modality’s transition rates are dynamically adjusted based on the other’s confidence, using cross-modal attention to negotiate commitments within every denoising step. This approach achieves state-of-the-art performance on joint image editing and visual reasoning, with accuracy that scales monotonically as more denoising steps are added.

Paper Primer

CO2Jump implements a chain-rule decomposition of the joint reverse posterior, allowing the image branch to condition its updates on the text branch's current beliefs. It uses an entropy-gated mixture of self-confidence and cross-modal signals to trigger remasking jumps, effectively retracting tokens when cross-modal evidence turns against them.

CO2Jump improves joint multimodal generation performance across photographic and logical tasks.

Outperforms baseline samplers (MDM, ReMDM, MMaDA-Parallel) on ImgEditBench, JMaze, and JNono benchmarks. Achieves a 0.369 overall mAP on image-grounded understanding, compared to 0.335 for the best baseline.

Cross-modal coupling benefits compound across the sampling trajectory.

Performance scales monotonically with the number of denoising steps (NFE), whereas independent samplers plateau or degrade. At 512 NFE, CO2Jump outperforms the best baseline by up to 0.034 mAP, a gap that is negligible at 8 NFE.

Why does this approach require a remasking jump instead of standard diffusion?

Standard masked diffusion is unidirectional; once a token is unmasked, it cannot be revisited. The remasking jump enables a bidirectional birth-death process, allowing the model to retract earlier commitments when cross-modal contradictions are detected.

Does this method require retraining the underlying model?

No, CO2Jump is a training-free sampler. It operates on a frozen backbone by extracting latent cross-modal signals during the standard forward pass, requiring no architectural changes or auxiliary evaluators.

For researchers building multimodal systems, this paper demonstrates that concurrent generation is not just about parallel execution, but about active cross-modal negotiation; coupling modalities within the denoising loop is essential for logical consistency.

Introduction: The Need for Coupled Generation

We expose modality drift in multimodal diffusion and argue for a self‑correcting coupled generation loop.

Human cognition does not separate understanding and generation: a teacher’s speech and sketches continuously reshape each other. We bring this tightly coupled loop to artificial systems, aiming for simultaneous text and image creation rather than a sequential pipeline.

When text and image streams are updated independently, their internal states diverge, leading to incoherent or contradictory outputs.

Masked Diffusion Models (MDMs) predict all masked tokens in parallel, but existing samplers factor each denoising step so that text and image updates depend only on the previous joint state. The result is an interleaved rather than truly concurrent trajectory, and because MDMs cannot remask, any cross‑modal contradiction introduced early persists.

**Figure 1.** CO$_2$Jump in action: text and image co-author the answer. Three trajectories of CO$_2$Jump on image editing, maze, and nonogram solving, showing the joint state at an intermediate step $t$ and at the final step. The image-editing panel highlights the core mechanism: at step $t$ the text branch has already begun committing a target-image bounding box in text for the new object person; by the final step the image branch has placed the hiker exactly inside the finalized box (we overlay the bounding boxes from generated text on edited image). The text branch plans where the edit should land, and the image branch executes that plan within the same denoising trajectory — no second forward pass, no external grounder. Maze and Nonogram show the same coupled-refinement pattern: partial-path and partial cell-fill commitments at step $t$ converge with their text-side answers by the final step.

Cognitive coupling—letting generation and understanding inform each other continuously—should become a core design principle for multimodal AI.

Background: Masked Diffusion Models

Masked diffusion models define a continuous‑time corruption‑denoising loop that underlies the paper’s cross‑modal sampler.

Masked diffusion models provide a continuous‑time view of token corruption and denoising, forming the basis for later cross‑modal coupling.

They corrupt a clean sequence by gradually replacing tokens with a special “mask” state and train a network to reverse that process.

Remasking lets a token that has been unmasked revert to the mask state with probability $\sigma_{t}$, giving the reverse process a chance to correct earlier mistakes.

These background mechanisms—continuous‑time forward corruption, CTMC‑based reverse dynamics, and remasking—provide the formal substrate that the CO2Jump sampler later exploits for coupled image‑text generation.

Theoretical Foundations: Markov Jump Processes

Coupled jump processes let text and image tokens correct each other during sampling.

Independent reverse schedules for text and image cause modality drift, so the sampler must reconcile contradictory predictions on the fly.

An MJP is a continuous‑time stochastic walk that hops between discrete states, waiting a random exponential time before each jump.

How does an MJP differ from a generic continuous‑time Markov chain?

Both are defined by a rate matrix, but an MJP emphasizes the *jump* interpretation: each event changes the discrete state instantaneously after an exponential wait. In contrast, a generic CTMC may be presented in generator‑matrix form without highlighting the explicit waiting‑time sampling that Gillespie’s algorithm uses.

Exact simulation via Gillespie’s algorithm updates a single position per jump, which becomes prohibitive at modern sequence lengths; therefore samplers resort to $\tau$‑leaping, a parallel approximation that updates all positions simultaneously.

CMJP lets the text and image jump intensities talk to each other: each modality’s birth/death rates are modulated by the confidence of the opposite modality, creating a self‑correcting feedback loop.

At time $t$, the model predicts high confidence for the image ($c_{\text{img}}=0.9$) but low confidence for the text ($c_{\text{text}}=0.3$).

Cross‑modal gates are $\phi_{\text{text}}=0.3$ (down‑weighting text birth) and $\phi_{\text{img}}=0.9$ (up‑weighting image birth).

Using $\tau$‑leaping, both modalities attempt a jump: the image unmask event fires (0→1) with probability $0.9$, while the text remains masked because its birth rate is suppressed.

The new joint state becomes (0,1). In the next step, the text confidence rises (now $c_{\text{text}}=0.7$) because the image provides context, so $\phi_{\text{text}}=0.7$ and the text unmask event succeeds.

After two $\tau$‑leaps the joint reaches (1,1), having self‑corrected the initial text‑masking error.

The gating mechanism lets one modality rescue the other without waiting for an entire diffusion trajectory, turning modality drift into a rapid, localized correction.

How does CMJP differ from simply running two independent MJPs for text and image?

In independent MJPs each modality’s jump rates are computed from its own hidden state alone, so a mistake in one stream never influences the other. CMJP injects the opposite modality’s confidence as a multiplicative gate, so a low‑confidence text token can be forced to remask while a high‑confidence image token proceeds, enabling cross‑modal self‑correction within the same $\tau$‑leap.

Discrete diffusion methods such as D3PM, score‑entropy, and any‑order autoregressive formulations provide the backbone for discrete state generation.

Self‑correction via remasking has been tackled by predictor‑corrector samplers, training‑based remask estimators, and training‑free heuristics like ReMDM.

Concurrent multimodal samplers (e.g., UD‑VLA, MMaDA‑Parallel) update modalities independently within each step, lacking the instantaneous cross‑modal feedback that CMJP introduces.

The CO2Jump Sampler

Couple text and image updates through a self‑correcting birth‑death jump that shares confidence.

Independent denoising schedules let text and image drift apart, because each modality decides when to unmask without seeing the other.

The sampler treats text and image tokens as participants in a single birth‑death jump process, repeatedly asking “does this token look reliable enough?” and only committing when a cross‑modal confidence gate says so.

How does CO2Jump differ from a naïve two‑stage pipeline that runs text diffusion first and then conditions image diffusion on the generated caption?

In a two‑stage pipeline the image never influences the text confidence; CO2Jump interleaves the two modalities at every jump, letting low‑confidence image regions suppress premature text unmasking and high‑confidence text cues accelerate image birth events.

Death threshold set to $0.5$: tokens with $\lambda<0.5$ (second text token, first image token) are remasked.

Birth threshold set to $0.6$: only the second image token ($\lambda=0.70$) is eligible for unmasking.

After the first jump, the mask pattern becomes $[\,\text{unmasked},\ \text{masked},\ \text{masked},\ \text{unmasked}\,]$.

The model recomputes confidences on the new mask; the previously masked text token now receives a higher cross‑modal boost ($\lambda$ rises to $0.58$) and survives the next death check.

Process repeats until all $\lambda$ exceed the birth threshold, yielding the final joint sample.

The coupled gate lets a strong image cue rescue a weak text token, and a weak image cue can temporarily suppress a premature text unmask, achieving self‑correction without extra supervision.

The gate treats confidence as a shared currency: each modality contributes its own confidence, and the gate redistributes it according to how informative the other modality’s context is.

CO2Jump sampler – iterative birth‑death loop.

**Figure 2 | CO$_2$Jump sampler.** A single denoising step from $z_t$ to $z_s$. From one forward pass, the model produces per-token Self-Confidence for both modalities; cross-modal attention $A_t^{image \to text}$ propagates text confidence to image positions, and an entropy-based gate $\lambda$ mixes self and cross signals into Coupled Confidence. The Death jump remasks the lowest-confidence committed tokens, and the Birth jump reveals the highest-confidence masked tokens under the noise schedule.

Asymmetric Sequential Sampling

CO2Jump uses asymmetric chain-rule factorization to gate image updates with cross-modal confidence, correcting modality drift.

Standard samplers update text and image modalities independently, ignoring that the text decision provides immediate context for the image. We instead factorize the joint reverse posterior using the chain rule, allowing the image update to condition on the latest text state without requiring a second forward pass.

Text positions are scored by their pure self-confidence, while image positions use a gated mixture of self-belief and cross-modal signals to resolve uncertainty.

Perform a single forward pass to extract hidden representations and compute token-level self-confidence.

Compute the asymmetric scores: text uses self-confidence; images use Coupled Confidence gated by predictive entropy.

Determine the death quota based on the schedule and remask the lowest-scoring tokens in both modalities.

Determine the birth quota and unmask the highest-scoring tokens to maintain the signal-to-noise progression.

Datasets and Benchmarks

One‑million joint edits and two‑hundred‑thousand reasoning tasks form the CO2Jump benchmark suite.

The paper’s core premise couples image and text diffusion via a self‑correcting feedback loop; this section details the data that fuels that loop.

JEdit‑1M supplies a million joint image‑editing samples, the largest curated set for cross‑modal editing.

It merges 724 k pairs from ImgEdit with 368 k pairs from OmniEdit, then augments each with scene‑graph and thinking‑trace fields.

JEDIT‑1M is a million‑scale collection of image‑editing triples enriched with explicit scene‑graph semantics and a step‑by‑step reasoning trace.

**Figure 3.** **Dataset curation pipeline.** For JEDIT-1M, raw editing pairs are augmented by an oracle Qwen3-VL-235B that produces both the per-image scene-graph understanding and the thinking trace. For JMAZE-200K and JNONO-200K, source/target images and the structured understanding are produced algorithmically.

JMaze‑200K and JNono‑200K stress‑test joint generation on logically interlocked tasks: each input uniquely determines a solution, and both modalities can be verified against algorithmic ground truth.

Visual Reasoning Tasks

Dataset construction, supervision format, and CO2Jump benchmark results.

The visual‑reasoning corpora consist of square grids whose side length $N$ varies between 5 and 25. Each instance is generated by a mixture of pattern synthesizers that preferentially embed clues spanning multiple rows and columns, ensuring that solving the puzzle requires global reasoning.

For a given $N\times N$ grid the input image shows an empty board annotated only with marginal clue numbers; the target image fills the cells black so that every clue is satisfied. The accompanying thinking trace is produced by an oracle Gemini 3 Flash model conditioned on both the source and target images together with the algorithmic ground‑truth solution.

Supervision is written in a parallel‑decoding friendly form. Maze solutions are expressed as absolute $(r,c)$ coordinates rather than relative moves, and nonogram supervision propagates constraints bidirectionally across rows and columns instead of solving line‑by‑line.

Two held‑out test sets each contain 500 samples that deliberately exceed the training size range. JMaze‑Test500 spans sizes {3,…,22} (in‑distribution {6,…,20}, OOD {3,4,5,21,22}); JNono‑Test500 spans {3,…,27} (in‑distribution {5,…,25}, OOD {3,4,26,27}).

The primary evaluation metric is joint accuracy, which counts a sample as correct only when both the extracted text answer (via a tolerant Gemini extractor) and the generated image (compared side‑by‑side with the ground‑truth image by Gemini 3 Flash) are correct. Text‑only and image‑only accuracies are also reported for diagnostic purposes.

CO2Jump is benchmarked against three existing sampler families—MDM, ReMDM, and MMaDA‑Parallel—across the three visual‑reasoning tasks. All models are fine‑tuned from the same Lumina‑DiMOO checkpoint on 64 × H100 80 GB GPUs (batch size 512, learning rate $2\times10^{-5}$), differing only in the inference‑time sampler.

Every sampler follows a cosine $\alpha_t$ schedule; the remasking variants (ReMDM and CO2Jump) additionally apply a remasking schedule with $\sigma_t=0.01$ for $t\in[0.25,0.75]$ and zero elsewhere.

On the ImgEditBench, the headline metric is per‑section mAP@0.5:0.95 from a pseudo‑grounding pipeline. CO2Jump attains the highest joint mAP, surpassing all baselines while maintaining comparable or lower text perplexity and entropy.

Quantitative Performance

CO2Jump sets new records on multimodal grounding and image‑editing benchmarks.

We evaluate CO2Jump against five baselines on the extended ImgEditBench protocol, reporting text‑generation perplexity, image‑generation quality, and multimodal understanding mAP.

CO2Jump achieves the highest overall multimodal understanding mAP (0.369) across all samplers.

Table 1 shows CO2Jump’s overall mAP of 0.369, surpassing the next best overall mAP of 0.360 (Qwen3‑VL‑8B).

The empirical pattern—CO2Jump’s superior text grounding (target mAP 0.392) coincides with its best image‑editing score (1.93)—matches the chain‑rule decomposition: when text decisions directly inform image updates within a single step, the image‑side distribution tightens and generation quality improves.

Scaling and Robustness

CO2Jump scales sampling steps while visual reasoning accuracy climbs.

CO2Jump is the only sampler that improves monotonically on both ImgEditBench and overall mAP as the number of function evaluations ($NFE$) increases.

ImgEditBench rises from $1.72$ to $1.93$ and mAP from $0.074$ to $0.369$ when $NFE$ sweeps $8arrow512$; at $512$ NFE CO2Jump leads MDM, ReMDM, and MMaDA‑Parallel by $+0.015$, $+0.016$, and $+0.034$, respectively.

**Figure 5.** Scaling sampling steps. ImgEditBench (left) and overall mAP (right) vs. NFE. CO2Jump is the only sampler with monotonically rising curves; the gap to baselines widens with NFE.

Table 2 reports joint accuracy on the visual‑reasoning benchmarks JMaze‑Test500 and JNono‑Test500. CO2Jump attains the best score in every column—six metrics total—while MDM is second on Maze solving and MMaDA‑Parallel is second on Nonogram solving.

Ablation and Mechanism Analysis

Ablations show each CO₂Jump component contributes to consistent cross‑modal gains and OOD robustness.

We evaluate how each CO₂Jump mechanism affects performance on text, image, and joint tasks.

CO₂Jump achieves the highest scores on all six evaluation columns, with gains over the strongest baseline ranging up to +0.062.

Table 3 lists the full model leading in every metric; Figure 6 visualizes its consistent joint correctness.

On out‑of‑distribution grids CO₂Jump remains top‑performing, reaching 0.320 vs 0.312 on Maze and 0.175 vs 0.113 on Nonogram.

These OOD numbers are reported in the text and corroborated by the qualitative trajectories in Figure 7.

Uncoupled samplers degrade sharply OOD (MDM drops 55 %, MMaDA‑Parallel 21 %), whereas CO₂Jump’s performance stays roughly flat.

The percentage drops are calculated from the In‑Dist vs OOD scores reported for each baseline.

The ablation removes three CO₂Jump components: Shared Percentile Rank (a cross‑modal ranking used to align confidence scores), Entropy‑Based Gating $\lambda_{i}$mage (a gate that scales image updates by their Shannon entropy), and Self‑Correction (the feedback loop that revises earlier steps).

**Figure 6 | Qualitative comparison.** Top left: Nonogram (red X marks clue violations); Bottom left: Maze; Right: per-step MDM vs. CO2Jump trajectory. Only CO2Jump satisfies all clues and paths, and its joint trajectory stays consistent across both modalities.

**Figure 7.** (a) Per-step text/image entropy and the gate $\lambda_{image}$. The image starts confident (source prior), text starts uncertain; $\lambda_{image}$ rises from $\approx 0.05$ to $\approx 0.85$ as text commits. (b) Image-side remask events in (Self-Confidence, Cross Signal) space: most fire in the lower-left region; a smaller hotspot at high-Self / low-Cross corresponds to coupling-driven revocations.

Conclusion and Failure Modes

We recap CO2Jump’s coupling benefits and outline its current limits.

Standard multimodal diffusion models drift because text and image are generated on independent, irreversible schedules. CO2Jump closes that loop by feeding cross‑modal confidence back into each denoising step.

Figure 6 visualizes the uncoupling failure mode: every baseline breaks at least one Nonogram clue (red ✗) or wanders off the maze, while CO2Jump satisfies every clue and follows the ground‑truth path.

At step 160 of the MDM baseline the text answer is already correct, but the image has drifted because the image branch never sees the latest committed text. CO2Jump’s chain‑rule decomposition propagates each text commitment to the image scorer at the same step, forcing both modalities to converge.

Table 3 ablates each component. Removing Shared Percentile Rank reduces image‑edit fidelity from 1.93 to 1.87, showing that a common scale prevents Self‑Confidence or Cross Signal from arbitrarily dominating.

Dropping Entropy‑Based Gating leaves source‑mAP unchanged but cuts target‑mAP by 0.030 and overall mAP by 0.015, confirming the gate’s role in late‑stage sampling when the image is finally grounded by text.

Eliminating Self‑Correction is the most damaging: overall mAP drops by −3.2 and editing accuracy falls by 0.01, because without remasking the system cannot repair cross‑modal contradictions discovered late in the trajectory.

Figure 7 (a) tracks per‑step entropy and $\lambda_{\text{image}}$ for a sample edit. Early on $\lambda_{\text{image}}\approx0.05$, so the image relies on self‑confidence; as text entropy collapses, $\lambda_{\text{image}}$ rises to ≈0.85 and the image leans heavily on the text‑grounded cross signal—exactly the empirical signature of the chain‑rule decomposition.

Figure 7 (b) plots every image‑side remask event in the (Self‑Confidence, Cross Signal) plane. Most events cluster in the lower‑left where both signals are low, triggering the death‑jump (confidence $(1-\lambda)\cdot\text{Self}+\lambda\cdot\text{Cross}$ is tiny). A smaller hotspot in the high‑Self/low‑Cross quadrant corresponds to coupling‑driven unmasking, explaining the performance gap between full CO2Jump and the “‑ Self‑Correction” ablation.

In summary, SC‑CMJP provides a self‑correcting coupled Markov Jump Process that lets the two modalities negotiate every denoising step, and CO2Jump instantiates this recipe in a single‑pass, training‑free sampler. It achieves state‑of‑the‑art results on joint image understanding, editing, and visual reasoning, and its performance scales monotonically with the number of denoising steps, evidencing cumulative cross‑modal benefit.

The framework is modality‑agnostic: extending it to audio, video, or structured outputs only requires defining appropriate confidence signals and transition‑rate couplings, a direction we leave for future work.

Qualitative Trajectories

Qualitative CO2Jump examples illustrate joint editing, maze solving, and nonogram solving.

**Figure 8.** Joint image editing with concurrent understanding on JEDIT-1M. Prompt, model-generated thinking trace, scene-graph source/target analyses, and the source/generated images with predicted bounding boxes overlaid (boxes are the model's own text-side predictions, not ground truth). The text branch declares only the shell as a dynamic object — every other entity has identical bounding boxes in the source and target analyses — and the image branch realizes exactly this localized texture edit: scaly → smooth on the shell, with body, limbs, ground and rock preserved.

**Figure.** A maze navigation task showing the initial state (top) and the solved state (bottom).

**Figure.** A nonogram puzzle grid shown in two states: an empty 7x7 grid with row and column clues (top) and the solved grid with filled (black) and empty (white) cells (bottom). The row clues are on the left, and column clues are on the top, indicating the lengths of consecutive filled cell blocks.

Implementation Details

Exact prompts used for scene‑graph extraction and logic‑trace synthesis are listed.

Appendix D.1 records the exact system prompts used to extract pixel‑aligned scene graphs from image‑editing pairs and to generate logic‑based reasoning traces.

System prompt for pixel‑aligned scene‑graph extraction

The appendix also supplies the prompt that drives the CO2Jump sampler, a self‑correcting coupled Markov Jump Process for concurrent image understanding and generation.

Task: Synthesize a Logic-Based Reasoning Trace for Image Editing. ### Inputs: 1. Source and Target Images: Visual evidence. 2. Editing Prompt (The intended transformation): {`editing_prompt_var`} 3. Image Understanding (Bounding box coordinates and object labels for both states): {answer} ### Objective: Synthesize a logic-based reasoning trace that analyzes the visual transition relative to the Editing Prompt. Critically evaluate whether the target state successfully fulfills the prompt's instructions or deviates from the intended logic. ### Strict Reasoning Guidelines: Categorize Objects: Group objects into: Static (persistent coordinates/attributes), Dynamic (modified coordinates/attributes), and Deleted/Added (present in only one state). Quantify Change: Reference bounding box shifts or attribute updates as direct evidence of the transition. Critical Alignment Check: Explicitly compare the visual delta against the Editing Prompt. Identify hallucinations or instruction mismatches. Connect to Intent: Explain how the detections fulfill-or fail to fulfill-the specific requirements of the Editing Prompt. Constraint: Keep the response dense and information-rich, within a max of 350 words. Avoid generic phrases. Start with the direct logical flow. ### Output Format: &lt;think&gt; [Your concise logical trace here] &lt;/think&gt;

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers