Orca: The World Is in Your Mind

Yihao Wang, Yuheng Ji, Mingyu Cao, Yanqing Shen, Runze Xiao, Huaihai Lyu, Senwei Xie, Euan Liu, Klara Tian, Tianfeng Long, Yichi Zhang, Zhengliang Cai, Ruike Chen, Jifan Zhao, Ruochuan Shi, Zihan Tang, Jing Lyu, Wenxing Tan, Ningbo Zhang, Yangtao Hu, Yuming Gao, Xiansheng Chen, Junkai Zhao, Congsheng Xu, Boan Zhu, Ziqi Wang, Yupu Feng, Qiongqiong Zhang, Yingli Zhao, Yulong Ao, Shaoxuan Xie, You Liu, Guocai Yao, Leiduo Zhang, Xiaodan Liu, Yunyan Zhang, Yance Jiao, Xinyan Yang, Jiaxing Wei, Xu Liu, Tengfei Pan, Shaokai Nie, Chunlei Men, Sen Cui, Xiaojie Jin, Hongyang Li, Jianlan Luo, Yao Mu, Yunchao Wei, Jun Yan, Hang Zhao, Xiaolong Zheng, Jiaming Li, Yonghua Lin, Tiejun Huang, Zhongyuan Wang, Pengwei Wang

Orca learns a unified world latent space via next-state prediction to support multimodal downstream tasks.

How can we build a unified "world foundation model" that learns a shared latent state representation across text, vision, and action, rather than training separate models for each task?

Current foundation models are often purpose-built for isolated tasks like next-token, next-frame, or next-action prediction, limiting their ability to generalize across diverse world-understanding scenarios. Orca shifts this paradigm by learning a unified world latent space through Next-State-Prediction modeling, using both unconscious learning from raw video and conscious learning from language-constrained events. Experiments show that this latent space scales effectively, consistently improving downstream performance in text generation, image prediction, and embodied action generation without task-specific backbone tuning.

Paper Primer

Orca models world states by predicting future or past states ($S_{t+\Delta}$) from a current state ($S_t$) under implicit dynamics and explicit conditions. It uses a frozen vision-language backbone and trains only lightweight modality-specific decoders to read out text, images, or actions from the learned latent space.

The core mechanism hinges on two complementary paradigms: unconscious learning captures dense, natural dynamics from continuous video without labels, while conscious learning uses language instructions to guide the model toward meaningful, sparse state transitions.

Stronger world latent representations directly improve downstream readout performance.

Probing checkpoints during pre-training shows that as pre-training data scale increases, performance on text, image, and action readouts consistently improves. Orca outperforms specialized baselines of comparable size across text generation, image prediction, and embodied action tasks.

The proposed learning paradigm is scalable.

Total loss curves for both 0.8B and 4B model sizes show a continuous downward trend as pre-training data scale increases, indicating the model does not quickly saturate. Training throughput reaches 2.91 samples/sec/GPU, a 4.4× acceleration over standard embodied community baselines like StarVLA.

Why prioritize Next-State-Prediction over the standard Next-Token-Prediction used in LLMs?

The authors argue that intelligence requires internalizing physical laws and dynamic evolution rather than just responding to instructions; state-transition modeling provides a unified paradigm for both observed and unknown domains.

What is the scope of the current Orca implementation?

Orca currently focuses on vision and language signals, using a pre-trained vision-language model backbone. It is an early exploratory milestone, with future work intended to incorporate broader physical signals like force, audio, and tactile feedback.

Researchers should view Orca as a shift toward general-purpose world foundation models where the latent space, rather than the task-specific head, is the primary carrier of intelligence. This suggests that scaling world-state modeling is a viable path to improving downstream capabilities in robotics and multimodal reasoning.

Introduction: The World Foundation Model

We frame the gap: existing models predict isolated tasks, while Orca aims for unified world‑state modeling.

Current foundation models are built for isolated tasks—next‑token, next‑frame, or next‑action prediction—so they cannot capture a coherent view of the world. Orca reframes the objective to predict the next latent world state, learning a unified World Latent Space that can be read out for language, vision, and action. It does so through two complementary paradigms, unconscious learning from raw video streams and conscious learning guided by language‑described events, while the backbone remains frozen and only lightweight decoders are trained.

A shared latent vector that encodes the underlying state of the environment, regardless of whether the input comes from vision, text, or action signals.

**Figure A1. Conceptual illustration of Orca.** Existing models are often organized around passive task-driven prediction, including next-token, next-frame, and next-action prediction. Orca shifts the modeling target toward next-state prediction, where multimodal world signals are used to learn a unified world latent. Unconscious learning captures dense natural dynamics from continuous observation, while conscious learning captures meaningful state transitions guided by language, events, and intentions. The learned world latent supports downstream readouts to language, vision, and action.

The key insight is shifting from task‑specific prediction to a unified world‑state model that supports diverse downstream modalities.

The Orca Architecture

Orca learns a unified world latent via unconscious and conscious paradigms to support multimodal decoders.

A single backbone must capture both dense natural dynamics and sparse semantic events across modalities, which prior models struggle to do.

Unconscious learning lets the encoder predict the next visual state solely from raw observations, capturing the continuous flow of the world.

How does unconscious learning differ from standard video‑prediction models?

Standard video prediction often conditions on past frames only; here the model treats the next frame as a teacher signal and learns a latent that directly predicts it, focusing on dense natural dynamics rather than a handcrafted loss.

Encode $v_0$ with the VLM → intermediate representation $h_0$.

Pass $h_0$ through two MLPs → $\hat{v}_\ell^{\,1}$.

Encode $v_1$ with the frozen vision encoder → $v_\ell^{\,1}$.

Compute loss $\|\hat{v}_\ell^{\,1} - v_\ell^{\,1}\|^2$ and back‑propagate.

Repeat for $v_1 arrow v_2$ to learn the transition dynamics.

Unconscious learning forces the model to capture the underlying physics of motion without any language cue.

Conscious learning conditions the state transition on an explicit language instruction, enabling the model to follow sparse, meaningful events.

Why add a language condition if the video already contains the target event?

The language cue tells the model *which* event to aim for; without it the model would only learn the most likely continuation, losing the ability to follow user‑specified commands.

Encode frame $v_0$ and the instruction “stop the ball” with the VLM → joint embedding $h$.

Two MLPs map $h$ to $\hat{v}_\ell^{\,2}$.

Encode the actual frame $v_2$ (where the ball is stopped) with the frozen encoder → $v_\ell^{\,2}$.

Compute loss $\|\hat{v}_\ell^{\,2} - v_\ell^{\,2}\|^2$ and update parameters.

The language condition forces the model to learn the causal effect of the command, not just the natural continuation of motion.

Collect multimodal inputs: video frames $v_t$ and optional language instructions $e_{t+\Delta}$.

Project each modality into the shared Vision‑Language Model (VLM) space.

Route the joint embedding through the unconscious‑learning branch (MLP stack) when no instruction is present.

Route the joint embedding through the conscious‑learning branch (MLP stack) when an instruction is present.

Fuse the two branch outputs into the unified world latent representation $S_t$.

Freeze the encoder after pre‑training; downstream decoders will read $S_t$.

Encoder forward pass – selects the appropriate learning paradigm.

**Figure 1. The Orca’s overall framework.** Orca follows an *Encoder-Decoder* architecture. Given multimodal world signals, the *Encoder* learns a world latent through two complementary paradigms: *unconscious learning* and *conscious learning*. Unconscious learning captures dense natural state transitions, while conscious learning captures sparse meaningful state transitions. To prove that the learned latent is effective, the *Encoder* is frozen after pre-training, and only the lightweight modality-specific decoders are trainable separately. The *Decoder* reads out the latent into text, images, actions, and other modalities.

**Figure 2. Overview of Encoder.** Orca learns a world latent representation through two learning paradigms. Unconscious learning uses video data to capture dense and natural state transitions. Conscious learning uses language instructions as explicit semantic conditions to capture sparse and meaningful state transitions.

**Figure 4.** Downstream readout architectures. To language reuses the LM head for text readout. To vision only trains an MLP adaptor and LoRA on top of a frozen SD3.5 to readout images. To action trains an MLP adaptor and a DiT-based Action Expert from scratch. Action Expert receives the latent, robot proprioception state, and noisy action to generate action chunks. The specific settings are shown in Appendix C.2.

Training Procedure

Orca trains via a two‑stage pipeline that first learns a world latent, then freezes it for modality‑specific readouts.

Training a single backbone to serve vision, language, and action demands both a rich world representation and efficient task‑specific adapters; naïvely fine‑tuning the whole model is prohibitively expensive.

First, a massive pre‑training phase learns a universal world latent from multimodal data; then the backbone is frozen and only lightweight readout heads for each downstream modality are trained.

Compute $L_{\text{obs}}$ on a single video frame pair → value 0.8.

Compute $L_{\text{evt}}$ on an event‑annotated pair → value 0.6.

Compute $L_{\text{vqa}}$ on a VQA sample → value 0.4.

Combine: $L = 0.5 \cdot 0.8 + 0.3 \cdot 0.6 + 0.2 \cdot 0.4 = 0.64$.

The example shows how the three losses interact; adjusting a single $\lambda$ directly scales that objective’s influence on the total gradient.

Insert a visual token $v_t$ and two learnable query tokens $q_1$, $q_2$ into the VLM backbone input stream.

For observation‑only transition, feed $v_t$ and $q_1$; the hidden state of $q_1$ passes through a two‑layer MLP to predict $\hat{v}_{t+1}$.

For event‑conditioned transition, also provide an event token $e_{t+\Delta}$ and $q_2$; the hidden state of $q_2$ predicts $\hat{v}_{t+\Delta}$.

For VQA, feed visual features $V$ and a question token $l_q$ to the LM head, which predicts the answer token sequence $l_a$ using standard next‑token cross‑entropy.

All three predictions contribute to the total loss $L$ (see the loss formula).

The pre‑training data comprise three complementary collections: 125 K hours of raw video (only one‑tenth used in this version), 160 M event annotations derived from multi‑level segmentation, and 11.5 M VQA pairs that couple language with visual context.

**Figure 3. Overview of pre-training data.** Orca's pre-training data includes video, event, and VQA data. A. Video Data supports 1) Observation-only state transition, A. Video Data and B. Event Data support 2) Event-conditioned state transition, and C. VQA Data supports 3) VQA response generation.

In the downstream stage the backbone remains frozen; only modality‑specific readout modules are trained, enabling rapid adaptation to language generation, image synthesis, or robot action without revisiting the expensive world‑latent pre‑training.

Language: feed a visual observation and an instruction into the frozen backbone, then use the LM head directly to generate the textual response.

Vision: pass the latent through an MLP adaptor, then condition a Stable Diffusion 3.5 decoder (with LoRA) to produce a pixel‑level image via multi‑step denoising.

Action: condition a DiT‑based Action Expert (trained with flow‑matching loss) on the adaptor‑processed latent and time‑embedded noisy action, then denoise to obtain the final action chunk.

Orca’s training stack is built on FlagScale with FSDP2 sharding, a Chunked Cross‑Entropy loss that avoids materialising full logits, activation recomputation for memory savings, and a forward/backward pre‑fetch schedule that overlaps communication with computation.

Training throughput rises from 0.66 to 2.91 samples / sec / GPU, a 4.4× speedup over the StarVLA baseline.

Measured on the same hardware (single GPU) with identical batch settings; the acceleration stems from the memory‑efficient loss, recompute, and communication‑aware scheduling.

Empirical Results

Orca’s unified world latent space scales and boosts downstream text, vision, and action tasks.

Orca extends the foundation‑model idea by learning a unified world latent space that captures state transitions across modalities. This section evaluates whether that latent space scales and supports downstream readouts.

We first test two core questions: (1) does scaling model size and pre‑training data improve the learning paradigm? (2) does a stronger latent space boost downstream readout performance?

Scaling model size and pre‑training data reduces total loss by 0.5, confirming Orca’s learning paradigm scales effectively.

Figure 5 shows loss decreasing from ≈0.7 to ≈0.2 as data increase for both 0.8 B and 4 B models.

Figure 6 demonstrates that as pre‑training data increase, readout performance on text, image, and action tasks improves for both model sizes, confirming a stronger world latent yields stronger downstream readouts.

**Figure 5.** Loss of model and data scaling.

**Figure 6.** Scaling behavior on downstream readouts performance.

Tables 1–4 provide the detailed numbers underlying the trends summarized above, while Table 5 reports the ablation study of the three pre‑training objectives.

Orca consistently outperforms comparable vision‑language models on text generation, image prediction, and action generation benchmarks.

Related Work

We categorize prior work by its primary learning objective, highlighting how each relates to Orca's world‑latent approach.

Self‑Supervised Learning methods are organized around latent world modeling, where predictive objectives focus on task‑relevant latent representations rather than raw pixels. JEPA‑style approaches such as I‑JEPA, VL‑JEPA, MC‑JEPA, and the V‑JEPA series progressively improve video and action understanding, while newer variants like LeJEPA, Causal‑JEPA, LeWorldModel, and GeoWorld enhance stability, scalability, and object‑centric causal reasoning.

Next Token Prediction research spans large language models (e.g., LLaMA 3.1, DeepSeek‑V4, Qwen 3, Kimi K2, GLM‑5, MiniMax‑M2.7, Phi‑4‑reasoning) that push scaling and reasoning, and multimodal large language models (e.g., LLaVA, Gemini 3.1, Qwen series, GPT‑5.4, Gemma 4, Kimi K2.5, LLaMA 4, Mistral Medium 3.5, RoboBrain series) that integrate vision and language. Works like Emu 3/3.5, BAGEL, Janus‑Pro, and Cosmos 3 further blur token and frame prediction by unifying multimodal generation under autoregressive frameworks.

Next Frame Prediction focuses on image and video generation models that map multimodal conditions to visual observations. Representative image generators include Nano Banana Pro, GPT Image 2, Qwen‑Image, and FLUX.2, while video generators such as Seedance 2.0, Sora, Cosmos‑Predict 2.5, and Wan 2.1 advance temporal coherence and controllability. Unlike these models, Orca does not aim to synthesize frames but to evaluate whether a target state obeys physical constraints and interaction dynamics.

Next Action Prediction research includes Vision‑Language Action models (OpenVLA, $\pi$0.5, $\pi$0.7, GR00T, VLA‑Adapter, SimpleVLA‑RL, VLA‑RFT) that learn manipulation from multimodal data, Video Action models (VPP, UVA, Mimic‑video, Cosmos‑Policy) that combine visual dynamics with policy learning, and World Action models (Motus, DreamZero, GigaWorld‑Policy, Being‑H0.7, VLA‑JEPA, JEPA‑VLA) that jointly model actions and future world states. Orca differs by first learning a world‑centric representation of temporal changes without action labels, enabling more efficient adaptation to embodied tasks.

Conclusion

Orca demonstrates a unified world latent space and outlines paths toward broader, more capable world models.

Orca introduces a world learner that first builds a shared representation of world states and then provides dedicated readout interfaces for language, vision, and action, shifting the modeling focus from next‑token prediction to next‑state prediction.

The current system is constrained by several limitations: it learns mainly from vision and language, relies on a frozen ViT‑space supervision, is restricted to 4 B/0.8 B model sizes, uses a limited vision benchmark, supervises only short‑horizon transitions, offers readouts for just three modalities, employs a modest three‑loss training objective, and evaluates on relatively easy embodied tasks.

Future work should broaden modality inputs, train a native world‑state model from scratch, devise a comprehensive state‑transition evaluation suite, enable a self‑evolutionary data‑training loop, and expand toward scientific domains such as quantum mechanics and life sciences.

Training Implementation Details

Implementation details for Orca’s pre‑training and downstream readout configurations.

This appendix records the exact hyper‑parameters and architectural choices used for Orca’s two training phases: the large‑scale pre‑training of the backbone and the modality‑specific readout fine‑tuning.

**Figure C1.** The implementation of Queries.

These tables fully specify the hyper‑parameters that reproduce the experiments reported in the main paper.

Training Infrastructure

We detail system-level optimizations that boost Orca’s training throughput.

Orca’s training combines visual embeddings, language modeling, future visual‑latent prediction, and action branches, which raises memory and communication demands beyond standard VLM training. To keep training stable at large scale we redesign the system infrastructure on FlagScale, focusing on distributed sharding, memory‑efficient execution, and overlapping communication with computation.

We replace DeepSpeed with FSDP2, which shards parameters, gradients, and optimizer states flexibly. Resharding discards redundant copies after use, cutting peak GPU memory, while lightweight visual blocks skip sharding to avoid extra communication.

Activation recompute checkpoints selected activation boundaries and reconstructs needed intermediates during backpropagation, swapping modest extra compute for a large memory reduction, enabling bigger batches.

Chunked cross‑entropy loss splits the token dimension, computing loss in blocks to avoid materializing the full logits tensor, which removes the peak‑memory spike caused by the log‑softmax in long‑sequence, large‑vocab settings.

Forward/backward pre‑fetching overlaps the all‑gather of parameters for the next layer with the current layer’s computation, reducing idle time from communication stalls and improving device utilization.

**Table D1.** As shown in Table D1, the optimized infrastructure achieves 2.91 samples/sec/GPU on H100 GPUs. This corresponds to a 3.0x improvement over the FSDP2 baseline of 0.97 samples/sec/GPU and a 4.4x improvement over the StarVLA (StarVLA Community, 2026) training pipeline.

Evaluation Benchmarks and Settings

Evaluation protocols for text, image, and robot‑action generation across diverse benchmarks.

We assess text generation on four video‑question benchmarks that probe action recognition, temporal dynamics, real‑world interaction, and 3‑D spatial reasoning.

Baselines span self‑supervised video models, unified tokenizers, and multimodal VLMs, providing a spectrum from latent world‑model to efficient edge models.

PRICE‑V0.1 evaluates instruction‑conditional image‑to‑image generation by requiring a model to transform an initial scene into a target state that respects physical constraints.

The benchmark draws from four real‑world interaction datasets—AgiBot‑World, HomeInteract, PE‑Video, and PSI‑Ego—covering robot and first‑person manipulations.

Generated images are judged by a closed‑source model suite (Gemini 3.1 Pro, GPT 5.4, Doubao Seed 2.0 Pro) and the open‑source Gemma 4‑31B, which assigns a 1‑5 score based on environment fidelity, accurate state change, and physical plausibility.

The evaluation prompt asks the judge to score the pair of images and instruction in JSON, emphasizing preservation of viewpoint, correct action execution, and avoidance of impossible physics.

Action readout is tested on a dual‑arm wheeled robot across five manipulation tasks, each with 200 recorded trajectories.

Two out‑of‑distribution regimes probe generalization: unseen table‑cloth/backgrounds (environment OOD) and novel objects or containers (object OOD).

Object OOD swaps the target object in each task (e.g., stacking bowls → stacking boxes, stamping paper → stamping a notebook).

Robot performance is measured by rule‑based stage scores (Table E2) and the PRM‑as‑a‑Judge diagnostic, which reports dense trajectory quality.

We compare Orca against V‑JEPA 2.1, Qwen3.5, and $\pi$0.5, freezing their backbones and training identical action experts on the same trajectories.

Action experts train for 20 k steps (batch 128) for Orca, V‑JEPA 2.1, and Qwen3.5, while $\pi$0.5 follows its official 30 k‑step (batch 32) schedule.

Detailed rule‑based scores (Table E3) and qualitative visualizations show Orca’s higher progress in failed attempts and its ability to recover from intermediate grasp failures.

**Figure E1.** PRICE-V0.1 Examples.

**Figure E2.** Real-robot benchmark. We evaluate the dual-arm wheeled robot on five manipulation tasks and construct OOD settings for environment and object generalization.

**Table E2.** Task-specific training and inference instructions.

**Table E2.** Scoring criteria for real-robot evaluation. Each task is evaluated within 60 seconds. If the robot becomes locked due to severe collision, or if the object falls and the task can no longer continue, evaluation is stopped. For each task, only the highest achieved score before termination is counted.

**Figure E3.** Failure with higher intermediate progress in Stamp. Orca grasps and transports the stamp toward the ink pad before dropping it near the end, while Qwen3.5 fails to maintain a meaningful stamp grasp and remains at low progress.

**Figure E4.** Failure with higher intermediate progress in Pull Out Tissue. Orca reaches the tissue-grasping stage and achieves substantially higher intermediate progress, while $\pi_{0.5}$ only approaches the tissue box and fails to grasp the tissue.

**Figure E5.** Failure with higher intermediate progress in Stacked Bowls. Orca advances through multiple bowl-stacking stages, while $\pi_{0.5}$ repeatedly fails to grasp the bowl and remains at lower progress.

**Figure E6.** Partial recovery after spoon-grasp failure in Scoop Sugar. Orca retries after failing to grasp the spoon and recovers some lost progress, while Qwen3.5 shakes in place without effective re-grasping.

**Figure E7.** Recovery through repeated spoon-grasp attempts in Scoop Sugar. Orca makes multiple recovery attempts and eventually grasps the spoon successfully, while JEPA remains largely stagnant with limited task progress.

**Figure.** A sequence of five frames showing a person unpacking items from a paper bag onto a table, followed by a multiple-choice question and model answers.

**Figure.** A sequence of five video frames showing a person interacting with a desk lamp and other objects on a table. Below the frames, a multiple-choice question asks if the lighting device is turned on at any point, with options A, B, and C provided. The correct answer is indicated as C (No), with both Orca and Qwen models selecting this option.

**Figure.** A film strip containing five frames showing the transformation of a 3D heart shape. Below the film strip, a multiple-choice question asks about the attribute change of the heart, with options A, B, C, and D provided, along with answers from two models (Orca and Qwen).

The image displays a film strip containing five sequential frames showing a forest scene transitioning through seasonal changes, followed by a multiple-choice question and model answers.

**Figure.** A film strip containing five sequential images showing a piece of toast with cream cheese, tomato, and greens being assembled or appearing on a wooden surface. Below the film strip, a multiple-choice question asks what is happening to the toast, with options A, B, and C provided, along with answers from "Orca" and "Owen".

**Figure.** A sequence of five frames showing a user interacting with a printer and a touchscreen interface.

**Figure F2.** More cross-benchmark examples of state transition.

**Figure F3.** Cross-benchmark examples of commonsense reasoning. The advantage of Orca is particularly pronounced in complex VQA scenarios that require reasoning beyond the visible scene and inferring hypothetical outcomes.

**Figure.** A sequence of five film strip frames showing a man in an office environment. The man is initially looking to the left, then turns his head and body to the right to face another person sitting at a desk in the background. Below the frames, a multiple-choice question asks to describe the video, with option B highlighted in green as the correct answer, supported by both Orca and Qwen models.

**Figure.** A sequence of five film frames showing a badminton court scene, followed by a multiple-choice question regarding the camera movement and the answers provided by two models, Orca and Qwen.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers