Multi-Agent Egocentric World Model with Fine-Grained Embodied Interaction
Dahyun Chung, Siyoon Jin, Hyunwook Choi, Honggyu An, Junyoung Seo, Hyunsung Kim, Seung Wook Kim, Seungryong Kim
ME-World generates synchronized, multi-agent egocentric video by coupling streams through joint denoising and shared environment memory.
How can we generate synchronized, consistent first-person video streams for multiple agents interacting in the same shared environment?
Existing world models generate egocentric video for a single agent, failing to maintain consistency when multiple agents interact in a shared environment. When one agent moves an object, independent models often produce conflicting outcomes or divergent scene layouts across different viewpoints. ME-World treats multiple ego streams as coupled observations of a single evolving world, jointly denoising them within a shared transformer sequence. It grounds this generation using shared action conditioning—which projects all agents' poses into each view—and a shared environment memory that warps historical observations into the current target viewpoint. On real and synthetic benchmarks, this approach improves shared-world consistency, identity preservation, and action control compared to independent generation or existing multi-agent baselines.
Paper Primer
The core challenge is that fine-grained embodied interactions—like two people manipulating a shared object—must appear consistent across different first-person viewpoints. ME-World solves this by moving from independent stream generation to joint multi-agent generation: it concatenates all agents' noisy latents into a single sequence, allowing the model's self-attention mechanism to exchange information between streams during the denoising process.
ME-World achieves superior shared-world consistency and interaction fidelity compared to adapted multi-agent baselines.
Quantitative evaluation on real and synthetic benchmarks using the proposed `S_env` (environment consistency) and `S_update` (state update consistency) metrics. ME-World consistently outperforms baselines like MultiWorld and MetaWorld in both environment and state-update consistency metrics while maintaining higher action control scores.
To ensure the model doesn't just generate plausible-looking but disconnected videos, it uses two grounding paths: stream-aligned geometric memory, which warps past observations into the current view to provide scene evidence, and cross-stream anchor memory, which provides unwarped reference images to maintain global scene appearance across all agents.
Why is "joint generation" necessary instead of just conditioning each agent on the same scene description?
Independent generation cannot guarantee that interaction-induced state changes—such as a moved object—propagate identically across views. Joint generation allows the model to resolve these updates once for the shared world state, ensuring all ego streams reflect the same outcome.
What is the scope of this model — does it only work for two agents?
The framework is general to $N$ agents. The authors demonstrate that adding a third agent simply requires appending an additional ego stream and its corresponding agent token to the joint sequence without modifying the underlying architecture.
Researchers building embodied AI can now move beyond single-agent simulation by using joint-stream denoising to maintain a persistent, consistent shared world across multiple viewpoints.
Introduction and Motivation
We expose why single‑agent egocentric models break in multi‑agent settings and outline our joint‑generation solution.
Egocentric world models predict first‑person observations from an agent’s actions, but almost all prior work assumes a single embodied agent. In real environments multiple agents act, interact, and share a common world state, so a model must generate synchronized ego streams that remain consistent across viewpoints, actions, and object updates.
Current multi‑agent approaches condition on coarse controls (e.g., camera trajectories or discrete commands) and therefore cannot capture fine‑grained embodied actions such as hand motions or head‑induced viewpoint changes, leading to three concrete failures: (i) actions appear inconsistent across agents, (ii) environments diverge in layout or appearance, and (iii) interaction‑driven state changes (e.g., a moved object) are not propagated to all views.
**Figure 1: Teaser.** Given multi-agent motions, head poses, character identities, and a shared world with first-frame initialization in (a), ME-World jointly generates synchronized ego streams (b) that remain consistent in character identity, scene appearance, and interaction outcomes across agents.
**Figure 2.** **Motivation.** Three challenges in multi-agent egocentric world modeling. Without shared action conditioning, (a) actions are inconsistent across ego streams. Without joint generation, independent ego streams show (b) environment inconsistency and (c) inconsistent state updates.
To overcome these gaps we introduce ME‑World, which jointly denoises all ego streams in a shared token sequence, conditions each stream on the target‑view poses of all agents, and grounds generation with a shared environment memory. This coupling enforces cross‑view action consistency, shared‑environment consistency, and consistent propagation of interaction‑induced state updates.
The key shift is moving from independent single‑agent egocentric modeling to a joint multi‑agent formulation that treats all ego streams as coupled observations of one evolving shared world.
Related Work and Approach
Contextualizes prior single‑agent models and introduces the multi‑agent ME‑World framework.
Prior work largely treats world modeling from a single perspective, while our approach unifies multiple egocentric views under a shared environment.
These models predict how a single embodied agent will see and act in its own coordinate frame, conditioning on that agent’s motion or hand‑camera control.
These extend world modeling to several agents that share a common environment, requiring the model to keep their observations and actions mutually coherent.
ME‑World generates synchronized ego streams for multiple agents by conditioning each stream on shared actions and a common environment memory, ensuring that all viewpoints agree on the underlying world state.
Shared Action Conditioning
Shared Action Conditioning aligns all agents' motions to enforce consistent ego streams.
When agents share an environment, generating their ego videos independently leads to contradictory views of the same interaction. Joint generation via a video diffusion transformer lets the streams exchange information, but without a shared action signal the model still cannot guarantee that each viewpoint reflects the same underlying motions.
Instead of feeding each ego stream only its own action, we feed every stream the full set of agents’ motions, so each view sees the same world‑changing events.
Agent A’s body vector becomes a $2\times2$ skeleton map with a bright pixel at the right‑hand location; Agent B’s body vector yields a similar map with a bright pixel at the left‑hand location.
Agent A’s head $30^\circ$ turn is turned into a Plücker ray map; Agent B’s $15^\circ$ turn yields its own ray map. Both maps are expressed in a shared world‑aligned frame.
Pose condition $X^{(A)}_{t,\text{pose}}$ is built by rasterizing Agent A’s own skeleton map and projecting Agent B’s skeleton map into Agent A’s camera view (result: two hand pixels appear).
Ray condition $X^{(A)}_{t,\text{ray}}$ concatenates Agent A’s ray map with Agent B’s ray map, giving the model a full picture of both agents’ viewing directions.
The concatenated condition is fed to the VAE encoder (pose) and MLP (ray), then merged with $Z^{(A)}$ before the DiT denoises the joint token sequence.
By injecting the other agent’s pose and ray information into each stream, the model can keep hand‑to‑hand contact and gaze alignment consistent across both viewpoints.
How does shared action conditioning differ from simply concatenating each agent’s private action vectors?
Private concatenation would still treat the actions as independent signals; shared conditioning explicitly projects other agents’ body poses into the target view and encodes all head‑induced camera rays in a common frame, giving the transformer cross‑agent geometry that a naïve concatenation lacks.
**Figure 3.** **Overall architecture** (a) For each agent, ME-World builds viewing ray conditions and shared-action conditions from agents' input and build shared environment memory from the both agents' history. (b) ME-World jointly generates synchronized ego streams.
Shared Environment Memory
Introduce a shared memory that grounds all agents in a common scene while preserving view‑specific detail.
Shared action conditioning makes each agent’s actions explicit, yet the model still lacks a common reference for scene layout and object state. Without such a reference, ego streams can diverge, producing inconsistent views of the same environment.
Think of a shared whiteboard that all agents can glance at, but each sees it from its own angle – the board supplies a common scene while each agent still gets view‑specific cues.
How does this memory differ from simply concatenating raw history images to the latent?
Raw concatenation would condition each stream only implicitly and discard geometry alignment; the stream‑aligned path explicitly reprojects the image into the target view, while the anchor path preserves unwarped appearance, together giving both aligned geometry and clean visual cues.
For Agent A, compute coverage of each $I_i$ after reprojection; $I_2$ yields the largest valid region, so $X_{hist}=I_2$.
Warp $I_2$ using A’s ray condition $X^{(A)}_{ray}$ → warped image $W_A$; mask dynamic regions with SAM3.
Encode $W_A$ with the frozen VAE to obtain condition vector $c_A$ and concatenate with latent $Z^{(A)}$.
For Agent B, $I_3$ gives the best coverage, producing warped image $W_B$ and condition $c_B$ similarly.
Select $K=2$ anchors: greedily pick $I_1$ (covers region not seen by $W_A$ or $W_B$) and $I_2$ (covers another uncovered region). Encode them to $z_{ref,1}$ and $z_{ref,2}$.
Form joint sequence $Z_{joint}^\tau = [Z_{gen}^\tau ; z_{ref,1} ; z_{ref,2}]$ and feed to the denoising transformer.
The two memory paths together ensure every part of the scene is represented either by a geometry‑aligned view or by a clean anchor, preventing holes that would otherwise appear in any single ego stream.
Evaluation Metrics
We report the three shared‑world consistency metrics and their experimental results.
Recall that ME‑World enforces cross‑view consistency by sharing action conditioning and a shared environment memory. This section evaluates how well the model maintains a coherent shared world across agents.
The three metrics quantify whether multiple agents truly share the same underlying environment, update it consistently, and preserve each other's identities, beyond what is forced by shared inputs.
**Figure 4.** Evaluation Protocol for Shared-World Consistency. We evaluate generated environments with $S_{env}$, state updates with $S_{update}$, and identity with $S_{id}$. Ground-truth geometry is used only to establish co-visible correspondences, while consistency is measured on generated streams.
Results and Ablations
ME-World outperforms baselines on shared‑world consistency and video quality.
Recall that ME‑World couples ego streams via shared action conditioning and a shared environment memory, addressing the inconsistency of independent ego generation. The following results quantify how this design improves consistency and video quality.
ME‑World achieves the highest Shared‑World Consistency across both real and synthetic benchmarks.
Tables 1 and 2 show it leads in every metric compared to multi‑view, single‑ego, and general world models.
**Figure 5.** Qualitative Comparison with Existing Methods. We compare ME-World on the real (top) and synthetic (bottom) benchmarks. Each column shows three timestamps from two synchronized ego streams. For multi-view baselines, the ground-truth conditioning stream is shown with reduced opacity. Best viewed zoomed in.
**Figure 6.** Qualitative Comparison with Multi-Agent World Models. We compare ME-World with existing multi-agent world models on the real benchmark. Each column shows three timestamps from two synchronized ego streams. ME-World better preserves a coherent shared environment and synchronized interactions across the two viewpoints. Best viewed zoomed in.
**Figure 7.** Qualitative Results of Ablation Studies. Removing individual components leads to inconsistencies in agent appearance, embodied actions, and the shared environment across ego streams. Red boxes highlight representative artifacts. Best viewed zoomed in.
Additional Qualitative Results
Additional qualitative results, extended generation settings, and metric details.
We supplement the main paper with extra qualitative examples and extended generation experiments that illustrate the robustness of the Egocentric World Model.
**Figure 8.** **Additional Qualitative Results on the Real Benchmark.** We show additional generations from diverse interaction sequences in real benchmark. Each example contains the two synchronized ego streams generated jointly by our model across multiple timesteps. Best viewed zoomed in.
**Figure 9.** Additional Qualitative Results on the Synthetic Benchmark. We show additional generations from diverse interaction sequences in synthetic benchmark. For each example, the two ego views are jointly generated by our model. The third-person view (TPV) is shown only as a visual reference to help interpret the shared scene and interaction, and is not generated or used as input by our model. Best viewed zoomed in.
**Figure 10.** **Multi-Agent and Autoregressive Generation.** We further demonstrate our model in more diverse generation settings, including multi-agent and long-horizon generation. Our model jointly generates three synchronized ego views for three interacting agents (top), and autoregressively generates two-agent sequences of 221 frames (bottom). For the three-agent results, the third-person view (TPV) is provided only as a visual reference to help interpret the shared scene and interaction, and is not generated or used as input by our model. Best viewed zoomed in.
**Figure 11.** **Metric validation.** We visualize the query regions evaluated by $S_{env}$ and $S_{update}$, their co-visible correspondences in the target stream, and the resulting DINOv3 cosine similarities. The consistent generation in (a) shows high correspondence similarity and receives higher metric scores, whereas the cross-view inconsistencies in (b) appear as localized similarity drops and lower scores. This correspondence provides qualitative evidence that the proposed metrics capture the intended cross-view consistency. Best viewed zoomed in.
Beyond the standard two‑agent, fixed‑length setting, we explore multi‑agent generation and autoregressive temporal extension.
Our model improves shared‑world consistency metrics over the single‑stream baseline.
Baseline `S_env` = 0.498, `S_update` = 0.402; our model `S_env` = 0.582, `S_update` = 0.495.
An environment conflict primarily harms the environment‑consistency metric.
Introducing a mismatched background reduces `S_env` by 0.090 and `S_update` by only 0.019.
A state conflict sharply degrades the state‑update consistency metric.
Replacing interaction‑induced regions drops `S_update` by 0.147 while `S_env` falls by 0.059.
We also evaluate camera trajectory error, hand and body action control, and video fidelity using established metrics.
Implementation Details
Implementation specifics: model init, training hyper‑parameters, evaluation metrics, and baselines.
Cosmos-Predict2.5 is a 2‑billion‑parameter multi‑view diffusion transformer that serves as the pretrained backbone for our system.
Training uses the rectified‑flow objective optimized with AdamW, a learning rate of $3 \times 10^{-5}$, and a global batch size of 4 distributed over four A100 GPUs.
At inference we run 35 denoising steps and apply a classifier‑free guidance scale of 7.0 to steer generation.
We evaluate camera control with Translation Error and Rotation Error, self‑action control with Hand‑F1 and Hand‑mIoU, other‑agent action control with Full‑Body‑F1 and PCK@10, and video quality with PSNR, SSIM, LPIPS, and FVD.
Baselines span three categories: multi‑view video generation models (GEN3C, AnyView‑DVS), single‑ego world models (EgoSim, JointControlVideo), and general world models (DreamX‑World, LingBot‑World), all used with their released checkpoints and no fine‑tuning.
For multi‑agent comparison we adapt four prior multi‑agent world models—MultiWorld, Solaris, $\gamma$‑World, and MetaWorld—using the same backbone, training data, and Shared Action Conditioning while preserving each method’s original multi‑agent mechanisms.
Tables 1 and 2 present the quantitative results on real and synthetic benchmarks, including selected metric values such as Translation Error = 12.376, Rotation Error = 4.736, Hand‑F1 = 0.537, and PSNR = 0.888 for the real benchmark.
Dataset and Training Details
Appendix Part 2 details the full experimental setup, data pipelines, and extra model variants.
Appendix Part 2 compiles the full experimental setup, dataset pipelines, and additional model variants referenced in the main text. It details single‑ego baselines, general‑world models, multi‑agent extensions, and the exact training and inference hyper‑parameters.
**Figure 12.** Dataset example. Examples are from (a) our generated synthetic dataset and (b) curated real-world CoMind dataset, showing synchronized ego streams and corresponding human poses.
**Figure 13.** Synthetic dataset generation pipeline. (a) We collect diverse 3D characters, environments, and paired motions, (b) apply the motions to characters, place them in the shared world space and attach egocentric cameras, and (c) render synchronized egocentric video pairs.
All hyper‑parameters, data processing steps, and model checkpoints are reproduced here to enable exact replication.