RynnUnify: Towards Unified Embodied Brain
Minghao Zhu, Bohan Hou, Kun Xiang, Zixuan Wang, Yinzhou Tang, Yifei Yang, Zhikai Wang, Tianyi Zhang, Kehan Li, Haoyu Zhao, Siteng Huang, Leigang Qu, Sicong Leng, Dongchi Huang, Mingxiu Chen, Shijian Lu, Yue Wang, Xiaodan Liang, Zhenyu Zeng, Deli Zhao, Chengju Liu, Jianfei Yang, Xin Li
RynnUnify integrates embodied reasoning with video prediction to enable long-horizon visual planning and robot control.
How can a single unified model integrate multimodal reasoning with visual generation to improve long-horizon robot planning and embodied cognition?
Unified embodied models struggle to maintain consistency over long horizons because prediction errors accumulate and static visual predictions often overlook intermediate dynamics. RynnUnify addresses this by interleaving an autoregressive reasoner with a diffusion generator, using cross-chunk scene memory to maintain visual context across successive subtasks. The model achieves state-of-the-art embodied understanding among evaluated multimodal models and demonstrates 51.5% success on composite robot manipulation tasks.
Paper Primer
RynnUnify is a 16B parameter model that combines an 8B autoregressive reasoner with an 8B diffusion generator. The core mechanism is interleaved video-language reasoning: the reasoner predicts subtasks and visual prompts, which the generator then uses to predict temporally coherent video clips that serve as context for subsequent planning steps.
RynnUnify improves long-horizon visual planning performance.
Evaluation on the Rynn-VP benchmark. 9.75 points higher than the Cosmos 3 Nano baseline.
The agentic extension enhances anticipatory reasoning.
Evaluation on the Rynn-GEU benchmark.
Why is interleaved reasoning necessary for this model?
Static visual predictions often fail to capture intermediate dynamics, and errors accumulate over long horizons; interleaving allows the model to update its plan based on the visual consequences of preceding predictions.
What is the role of the "cross-chunk scene memory"?
It retains visual information—such as objects or scene regions that leave the field of view—by maintaining persistent reference frames and continuous video prefixes, ensuring visual consistency across subtask boundaries.
RynnUnify demonstrates that coupling language-based task planning with continuous video prediction creates a more robust foundation for embodied agents than either capability alone.
Introduction to RynnUnify
RynnUnify unifies reasoning and video generation to enable embodied understanding and long‑horizon planning.
Robots need a single model that can perceive the physical world, predict how actions will change it, and turn task intents into concrete motions. Current unified multimodal models can link perception and generation, but they falter when predictions must remain coherent over many steps, because errors accumulate and static visual forecasts miss intermediate dynamics.
RynnUnify stitches together an autoregressive reasoner and a diffusion generator so that each reasoning step can request a short video of the predicted future, and the next reasoning step consumes that video as evidence.
Beyond the base model we train an agent that decides when to answer directly and when to invoke the frozen RynnUnify to generate videos, using reinforcement learning with final‑answer feedback. We also adapt the full system into a World‑Action Model (WAM) that maps the generated visual evidence to robot joint commands.
To measure sustained visual planning we introduce Rynn‑VP, and to test anticipatory reasoning we add Rynn‑GEU. Across public embodied‑understanding benchmarks RynnUnify attains the highest average score, improves Rynn‑VP by 9.75 points over the strongest prior model, and its agentic variant lifts Rynn‑GEU by 6.0 points.
**Figure 1.** RynnUnify unifies embodied understanding, visual planning, agentic reasoning, and robot action.
Unifying autoregressive reasoning with diffusion‑based video generation lets a single model both imagine future scenes and use those imaginations to guide its own decisions.
Architecture Overview
Describes the MoT backbone, joint attention, and scene‑memory conditioning that enable interleaved reasoning and video generation.
When planning long‑horizon tasks the model must keep visual context alive across many sub‑tasks, yet the generator alone cannot infer what has already happened. The trick is to give the diffusion generator direct access to the reasoner’s textual embeddings and to a compact set of persistent visual tokens, so it can ground its predictions in both language and remembered scene snapshots.
MoT runs two independent transformer streams—one for autoregressive reasoning, one for diffusion‑based video generation—and stitches them together with a single joint‑attention layer that lets the generator read the reasoner’s hidden states.
Concatenate in the order required by the joint‑attention layer: [R₁,R₂,R₃, M₀₁,M₀₂, M₁₁,M₁₂, V₁,…,V₄].
The joint‑attention module builds a (9 + 4) = 13‑token matrix and computes attention scores for every query.
Only the queries belonging to the visual tokens (M\*, V\*) are updated; the reasoner tokens (R\*) remain static.
After the attention pass the generator proceeds with its diffusion update on V₁…V₄ while still seeing the unchanged R‑tokens and the memory tokens.
Because the reasoner tokens never change, the generator can repeatedly attend to a stable language context while progressively refining the visual latents.
How does MoT differ from a classic mixture‑of‑experts (MoE) that routes inputs to multiple expert networks?
MoE selects which expert processes each token, but the experts never see each other’s representations. MoT keeps both branches fully active and merges them via joint attention, so the generator can directly read the reasoner’s embeddings rather than being isolated behind a routing gate.
The reasoner branch is taken verbatim from Cosmos3: it consumes multimodal observations and language prompts, then emits a textual embedding sequence that encodes the current sub‑task, its anticipated visual evolution, and an estimated duration.
Visual inputs are first compressed by a causal video VAE (spatial factor 16, temporal factor 4) and then split into patch tokens. Both the scene‑memory frames and the main video tokens travel through the generator’s transformer stack.
Building the generator’s token sequence for a denoising step.
Cross‑chunk scene memory keeps four reference frames: an initial observation, two keyframes from earlier history, and the most recent frame. When history is short the initial frame is duplicated to fill the slots. During rollout the initial frame stays fixed while the other three are refreshed from the newly generated clips.
Three generation modes are supported by arranging fixed latents differently: (i) I2V fixes only the first latent frame, (ii) FL2V fixes both first and last frames, and (iii) continuation fixes a short prefix taken from the tail of the previous clip. The generator updates only the unfixed positions while still attending to the reasoner’s text.
Textual conditioning supplies the sub‑task instruction, a detailed visual description, and a duration. Temporal conditioning uses the 3‑D rotary position encoding of the backbone and allows variable‑length videos of the form 4n + 1 frames. For continuation the duration excludes the prefixed frames, so the generator knows exactly how many new frames to synthesize.
Data Sources and Composition
We assemble diverse fine‑tuning data that links language, visual prompts, and video clips for joint training.
The fine‑tuning corpus stitches together language instructions, structured visual prompts, and short video clips so that the Reasoner and Generator can be trained jointly.
**Table 1.** Overview of the data used for reasoner and generator fine-tuning in RynnUnify, covering general understanding, visual planning, and embodied perception and reasoning. We report sample counts for understanding and clip counts and approximate durations for video generation.
Training examples are built by weaving the task goal, a subtask instruction, a structured visual prompt, the corresponding video clip, and the operation’s duration into a single chronological sequence.
Goal: “Arrange the red cube on the blue platform and rotate it.”
Instruction 1: “Pick up the red cube.” Prompt 1 describes the cube’s initial pose; Clip 1 (4 s) shows the hand grasping the cube.
Instruction 2: “Place the cube on the blue platform.” Prompt 2 encodes the target location; Clip 2 (5 s) shows the cube settling on the platform.
Instruction 3: “Rotate the platform 90° clockwise.” Prompt 3 specifies the rotation angle; Clip 3 (6 s) shows the platform turning.
Durations recorded: 4 s, 5 s, 6 s respectively, attached to each subtask.
By presenting goal → instruction → prompt → clip → duration repeatedly, the model learns to anticipate visual outcomes and schedule time, which is essential for long‑horizon planning.
Generation-Enhanced Understanding (GEU)
GEU stitches generated videos into reasoning dialogs so the model learns to query its own visual predictions.
Reasoners excel at symbolic inference but stumble when the answer depends on visual dynamics that are not present in the input. Without a concrete visual cue, the model must hallucinate the future, leading to brittle predictions.
GEU turns a reasoning turn into a four‑step conversation: the assistant sees an observation + question, asks for a targeted “world‑modeling” instruction, receives a generated video that visualizes the requested future, and finally answers using that video as evidence.
Question: “Will the red cube fall after the blue cube is removed?”
World‑modeling instruction (generated by GPT‑5.6‑Luna): “Remove the blue cube and let the red cube react to gravity and any support loss.”
MiniMax‑H3 generates a 0.5‑second video where the blue cube disappears and the red cube tips over.
The reasoner watches the video, detects the red cube’s rotation, and outputs: “Yes, the red cube falls.”
The instruction guides the generator toward the *relevant* interaction while leaving the outcome to be discovered in the video, forcing the reasoner to ground its answer in visual evidence.
How does GEU differ from a standard multimodal dataset where the video is provided up‑front?
In ordinary multimodal training the video already contains the answer, so the model can shortcut by copying visible cues. GEU deliberately withholds the outcome in the instruction, forcing the generator to *produce* the future and the reasoner to *interpret* it, which trains a genuine visual‑to‑symbolic inference loop.
Phys builds question‑answer pairs from simulated object trajectories so the model learns to predict future motion from a short video prompt.
Mess captures cascading physical effects in cluttered tabletop scenes, teaching the model to anticipate indirect consequences of a local intervention.
Spatial creates QA pairs about position, direction, distance, and size, linking them to short videos that illustrate the queried geometry.
ActionReasoning pairs short observed‑continuation clips with planning‑style questions, teaching the model to infer intent, primitive selection, and affordances from visual transitions.
These datasets bind subtask‑level language instructions to spatial annotations (affordance points, placement areas) and to videos that visualize the corresponding manipulation.
Trajectory links a task‑conditioned instruction to a 2‑D image‑space trajectory, supervised by a short execution video.
All six pipelines share the same interleaved template, so the reasoner sees a uniform training signal regardless of domain.
Training Objectives and Stages
Two-stage pretraining teaches the reasoner then the generator to handle visual planning.
Long‑horizon rollout accumulates visual errors, so a generator trained only on perfect frames quickly diverges. The solution is a two‑stage pretraining pipeline that first endows the reasoner with spatio‑temporal skills, then teaches the generator to be robust to imperfect conditioning.
Supervise four capabilities: spatio‑temporal understanding, physical grounding, subtask planning, and video‑prompt formulation.
Present each example as a multimodal conversation: USER provides task context, ASSISTANT outputs <subplan>, <`world_modeling`>, and <duration> tags.
Apply an autoregressive loss over all assistant response tokens, excluding user turns and visual inputs.
Freeze the reasoner after convergence; its parameters become the fixed backbone for Stage 2.
Freeze the reasoner; its outputs (subplan and world‑modeling tags) become the textual conditioning for the generator.
Train three complementary generation tasks: image‑to‑video (I2V), first‑and‑last‑frame‑to‑video (FL2V), and cross‑chunk continuation.
Provide cross‑chunk scene memory: four reference frames (initial observation + three historical frames) are encoded and concatenated with the main video latent sequence.
Sample a prefix of $k \in \{2,4,8\}$ latent frames, corresponding to $5$, $13$, or $29$ actual video frames, and prepend it to the current clip, sharing the boundary frame once.
Limit the combined video to at most $241$ frames, ensuring the prefix occupies no more than half of the latent sequence.
Apply self‑forcing strategies (condition degradation, offline endpoint replacement, joint history & endpoint replacement) while still supervising the ground‑truth continuation.
Instead of always feeding perfect ground‑truth frames, the generator is occasionally given noisy or self‑generated frames so it learns to continue even when its own predictions are imperfect.
Apply condition degradation: add Gaussian noise ($\sigma$=0.1) to the start frame, slightly blurring the cube’s edges.
Feed the noisy start frame to the generator; it predicts the next 5 frames, producing a slightly jittery lift motion.
Replace the start frame of the next clip with the final frame of this prediction (the cube now partially lifted).
Run the generator again on the new clip; it refines the lift and completes the motion.
Even modest noise forces the generator to rely on learned dynamics rather than pixel‑perfect cues, reducing error buildup in long sequences.
How does self‑forcing differ from ordinary data augmentation?
Standard augmentation perturbs inputs but still compares the output to a ground‑truth target derived from the same clean input. Self‑forcing replaces parts of the conditioning with the model’s own predictions, so the generator must learn to continue from imperfect, self‑generated context while still being judged against the true future frames.
**Figure 3** Two-stage pretraining of RynnUnify. Stage 1 fine-tunes the reasoner for embodied understanding, spatial grounding, next-subtask prediction, and structured video-prompt generation. Stage 2 freezes the reasoner and fine-tunes the generator with self-forcing through condition degradation and offline replacement of starting frames and visual history, while retaining ground-truth future videos as supervision.
Agentic Reinforcement Learning
We introduce Branch‑Conditional Policy Optimization, an agentic RL framework that learns when to request visual generation to boost reasoning.
Reasoning alone often lacks the visual evidence needed to answer embodied questions, causing the model to guess or stall. The paper therefore needs a mechanism that decides *when* to call the frozen generator and *how* to use the produced video. Agentic Reinforcement Learning supplies that decision‑making loop.
BCPO treats the reasoner as an agent that can follow two alternative interaction paths—direct reasoning (D) or generation‑assisted reasoning (G)—and learns a policy that picks the path that yields higher task reward.
How does BCPO differ from standard PPO?
Standard PPO optimizes a single policy over a single action space. BCPO splits the action space into two *branches* and runs separate advantage estimations inside each branch, then uses the *difference* of the branch means as an additional supervision signal for the routing decision. The generator is never updated, so BCPO’s policy learns *when* to invoke an external tool rather than just *how* to act.
Compute mean reward per branch: $\mu_D = (2.0+1.5+1.8+2.2)/4 = 1.875$, $\mu_G = (3.0+2.7+3.2+2.9)/4 = 2.95$.
Normalize each trajectory’s advantage: $A_{D,i} = (r_{D,i}-\mu_D)/(\sigma_D+\epsilon)$, $A_{G,i} = (r_{G,i}-\mu_G)/(\sigma_G+\epsilon)$ (using the sample standard deviations).
Form the soft routing target $q_G = \sigma\big((\mu_G-\mu_D)/T_{\text{route}}\big)$; with $T_{\text{route}}=0.5$ this yields $q_G\approx0.88$.
Update the reasoner’s routing logits toward $q_G$ via masked binary cross‑entropy, while applying GSPO updates to the token policies inside each branch.
Even with only a few rollouts the agent can infer that the generation‑assisted path is consistently more rewarding and therefore learns to request visual evidence for similar future questions.
Beyond improving token policies, the agent must learn *when* to invoke the generator. This is done by turning the mean‑reward gap between branches into a soft target for the routing logits.
Both branches share a single task‑grounded reward that multiplies a protocol‑validity flag with a normalized answer‑quality score, ensuring that the routing signal directly reflects downstream performance.
**Figure 4** Agentic reinforcement learning for generation-enhanced embodied understanding. The agent dynamically chooses between answering directly and invoking the frozen RynnUnify model to generate visual evidence for further reasoning. Rewards combining format validity and answer quality guide within-branch GSPO and soft D/G routing. Only the agent is updated during reinforcement learning.
Training uses 22 k questions covering perception, physics, and spatial reasoning. Each update samples 64 questions with eight rollouts each (four per branch for routable questions), yielding 512 trajectories per step. The agent is initialized from the fine‑tuned 8 B reasoner and trained for one epoch with Adam at $10^{-6}$ while the generator stays fixed.
World-Action Model (WAM)
Extending RynnUnify, the World‑Action Model unifies reasoning, video prediction, and robot actions.
RynnUnify interleaves an autoregressive reasoner with a diffusion generator so that generated visual evidence can guide its own reasoning and action selection. The World‑Action Model builds on this by adding a dedicated action head that turns the generator’s video features into robot commands.
**Figure 5** Extending RynnUnify to a World-Action Model. The reasoner uses the task instruction, current observations, and visual memory to produce a subtask latent. This latent conditions the generator through layerwise attention. An action head reads the generator’s video features to predict action chunks. An auxiliary camera-pose head provides geometric supervision during training.
Think of the reasoner as a director writing a scene plan, the generator as a cinematographer filming the scene, and the action head as an editor cutting the footage into executable robot moves.
Generator receives $h_{\text{plan}}$ and attends to it while processing two future video tokens, producing video latent $V_1 = [[0.1,0.2],[0.3,0.4]]$.
A single denoising step adds Gaussian noise $\epsilon_v$ to $V_1$, yielding $\tilde V_1 = V_1 + 0.05\,\epsilon_v$.
The action head queries the hidden states of generator layers 5 and 6 with its action token, computing cross‑attention weights that focus on $\tilde V_1$.
Using flow‑matching, the head predicts a tiny velocity field $A_t = [[0.0,0.1],[-0.1,0.0]]$ for the robot’s end‑effector.
The predicted action chunk is assembled and sent to the robot controller, which moves the gripper toward the red cube.
This walk‑through shows how a single subtask latent steers both video synthesis and the downstream action, tightly coupling high‑level intent with low‑level motion.
How does the World‑Action Model differ from the original RynnUnify?
RynnUnify stops at generating a future video that can be inspected; WAM adds an action head that consumes the generator’s video features via cross‑attention to produce concrete robot commands, and it also trains an auxiliary camera‑pose head for geometric consistency. Moreover, after Stage I the reasoner is frozen, so later stages only adapt the generator and action components.
Stage I – Reasoner training: freeze the generator, train the reasoner’s language layers on subtask and skill annotations using token‑level cross‑entropy.
Stage II – Generator training: freeze the reasoner, train the generator conditioned on $h_{\text{plan}}$ with the same video and flow‑matching losses as the base model.
Stage III – Action‑head training: keep the reasoner frozen and also freeze the generator’s original action output layer; jointly train the generator (including the denoising step) and the new 18‑block action head, plus an auxiliary camera‑pose head.
Evaluation Benchmarks
We detail the new Rynn‑VP and Rynn‑GEU benchmarks that evaluate visual planning and generated‑evidence reasoning.
Rynn‑Unify’s embodied capabilities are measured with two new suites: Rynn‑VP, which tests long‑horizon visual planning, and Rynn‑GEU, which probes reasoning that leverages generated visual context.
During inference, the action head runs 16 Euler integration steps to produce a 48‑step action chunk; only the first 16 actions are emitted for execution, while the camera‑pose head is kept for training only.
Rynn‑VP evaluates visual planning on 200 manually collected egocentric scenarios. A model receives an initial image and a textual instruction, then generates either a full video rollout or an ordered set of step images. The protocol checks task progress, subtask dependencies, scene continuity, and physical plausibility without requiring a simulator.
Rynn‑GEU probes embodied reasoning that benefits from generated visual context. It comprises ten subsets covering spatial relations, physical prediction, chain reactions, action reasoning, affordance grounding, area grounding, trajectory prediction, and a “Mess” set of cluttered scenes.
Training data follow the same pipelines as the Rynn‑GEU training set, but evaluation scenes are disjoint (e.g., RoboTwin 2.0 for “Mess”, distinct assets for “Phys”). This prevents memorization of specific layouts.
To bias the benchmark toward questions that truly need future visual evidence, we filter candidates with GPT‑5.6‑Luna: we keep examples that improve markedly when the reference future video is available, while retaining a minority that remain answerable without it.
Table 2 aggregates results on public embodied‑understanding benchmarks that do not use generated video rollouts. Rynn‑Unify (our model) achieves the highest score among open‑source unified multimodal systems on the Embodied Cognition metric (67.1) while remaining competitive on other tasks.
**Figure 6.** Overview of Rynn-VP and Rynn-GEU. (a) Rynn-VP evaluates long-horizon visual planning across 200 task instances involving interdependent operations. (b) Rynn-GEU evaluates reasoning beyond observed states under both direct and generation-assisted settings, with six representative tasks illustrated here. The videos are ground-truth references, and the planning text is illustrative.
Summary of Results
Quantitative outcomes of the unified system across the full evaluation suite.
The evaluation covers 36 Reasoner tasks and 22 Generator tasks, providing a comprehensive view of performance.
Reasoner attains a peak benchmark score of 83.0, surpassing the Generator’s best score of 75.5 by 7.5 points.
Reasoner scores range from 0.27 to 83.0 across 36 tasks; Generator scores range from 3.3 to 75.5 across 22 tasks.
Embodied Understanding Performance
Embodied cognition benchmarks show RynnUnify leads UMMs with a 57.9 average score.
RynnUnify achieves the highest average score among evaluated unified multimodal models, reaching 57.9, surpassing the next best Cosmos 3 Nano (53.2).
Table 2 shows RynnUnify’s average of 57.9 versus 53.2 for Cosmos 3 Nano.
Embodied cognition is a model’s ability to reason about physical scenes, actions, and spatial relationships as if it were situated in the world.
**Table 2.** Comparison on public embodied understanding benchmarks without generated-video rollouts. Bold and underlining mark the best results among UMMs and all compared models, respectively. Avg. is the unweighted mean of ten metrics normalized to a 0–100 scale.
**Table 3.** Embodied reasoning on public benchmarks and Rynn-GEU. Interleaved Thinking uses generated visual context. Agentic Thinking augments RynnUnify-16B with an 8B RL-trained reasoner. Bold, underlining, and † indicate the best results among UMMs and their reasoning variants, direct-answering models, and all methods, respectively.
**Figure 11. Additional embodied cognition examples.** Six MindCube and ERQA cases cover viewpoint reasoning, camera motion, interaction grounding, and anticipated object motion. Each case presents the visual observations, question, and answer.
Visual Planning Capabilities
Interleaved visual planning boosts overall performance to 54.8, outpacing competing multimodal models.
RynnUnify attains an overall visual‑planning score of 54.8, a gain of +18 over the strongest prior unified multimodal model.