Vibeworlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End?

Yansong Ning, Jingwen Ye, Zhongkai Wu, Yang Sun, Yiqin Zhu, Xingyi Li, Weidong Zhang, Hao Liu

VibeWorlding provides a unified framework and RL training pipeline for end-to-end 3D world construction.

Can multimodal agents effectively construct and refine interactive 3D open worlds from natural language queries using a tool-use loop?

Multimodal agents struggle to construct 3D worlds because existing frameworks are fragmented, lack reliable verification, and rely on idealized queries that fail to capture real-world spatial complexity. The authors introduce VibeWorlding, a unified sandbox that integrates asset retrieval, editing, and rendering as tools, paired with a dual-constraint verifier that checks both physical feasibility and intent fulfillment. This framework enables agentic reinforcement learning (RL) post-training, which allows open-source models to match or exceed the performance of frontier closed-source models on complex 3D construction tasks.

Paper Primer

The core mechanism is a joint multimodal RL pipeline that optimizes agents using both text-only and image-conditioned queries. The agent interacts with a sandbox environment, receiving rendered visual feedback at each turn to refine its 3D scene, while a dual-constraint verifier provides the reward signal: physical checks for collisions and floating assets, and an MLLM-based judge for intent alignment.

RL post-training enables open-source models to outperform frontier closed-source models in 3D world construction.

VibeWorlder-30B-A3B achieves a 59.3% Pass@1 rate on VWE-BENCH, surpassing GPT-5.5 (57.3%) and Qwen3.8-Max (56.9%). The 30B-A3B model's performance represents a significant jump from its 13.6% base capability, with its lead most pronounced on verified, rule-checkable tasks.

Despite these gains, the authors identify a persistent bottleneck: precise 3D spatial editing. Even after RL, models frequently struggle with coordinate-frame errors—such as translating an object in the correct magnitude but the wrong direction—and over-editing, where the agent inadvertently deletes existing assets during refinement.

Why is 3D world construction harder to evaluate than standard text generation?

Unlike text, a 3D world must satisfy multifaceted, simultaneous constraints: it must be physically feasible (no collisions, grounded on surfaces), aesthetically coherent, and strictly aligned with the user's specific spatial intent.

What is the specific role of the "dual-constraint verifier" in the training process?

It acts as the reward signal for RL: it first filters out physically impossible worlds using geometric checks, then uses an MLLM judge to score whether the remaining worlds actually fulfill the user's intent, providing a more reliable signal than generic LLM evaluation.

Introduction to Vibe Worlding

We expose the gap between simple scene generation and full interactive 3D world construction.

Current multimodal agents are evaluated on overly simple, single‑turn queries, which prevents systematic analysis of how they reason over space, invoke 3D tools, and integrate visual feedback. Moreover, the lack of an open‑source sandbox hampers research on whether agentic reinforcement‑learning post‑training can close this gap.

The agent must turn a multimodal user request into a fully interactive 3D world by repeatedly planning, using asset‑manipulation tools, and checking visual feedback.

**Figure 1.** The overview of VWE-Bench and VibeWorlding-Gym.

The central challenge is bridging the gap between simple scene generation and fully interactive 3D world construction.

Definitions and Tooling

Defines the primitive tools and interaction loop that enable agents to build 3D worlds.

Current multimodal agents lack a concrete interface for manipulating a scene, so they resort to brittle end‑to‑end generation that fails on multi‑turn spatial reasoning. Introducing a minimal, deterministic toolbox gives the agent explicit, composable actions to edit the world step by step.

The five tools act like a kitchen’s basic utensils: `asset_retrieve` fetches an ingredient, `asset_add` places it on the table, `asset_rotate` turns the plate, `asset_translate` slides it across the surface, and `asset_delete` discards what is no longer needed.

How do these 3D tools differ from generic scene‑graph edit operations?

Scene‑graph edits typically modify abstract node attributes without guaranteeing a concrete geometry update. Each 3D tool here directly manipulates the underlying mesh and its pose, producing an observable rendered image after every call, which the agent can reason about.

`asset_retrieve`("00001") → returns the asset tuple (id, name, category, 388 faces, bbox).

`asset_add`("00001", pos=(1,0,0), rot=(0,0,0), scale=1.0) → the bookshelf appears at (1,0,0) with default orientation.

`asset_rotate`("00001", axis="y", angle=90) → the bookshelf now faces the opposite direction.

`asset_translate`("00001", delta=(0,2,0)) → the bookshelf moves upward by 2 units.

`asset_delete`("00001") → the scene becomes empty again.

The sequence shows that any complex layout can be built by chaining these five primitives; the agent only needs to decide the order and parameters.

The agent treats world building as a turn‑based dialogue: at each step it thinks, issues one or more tool calls, observes the updated scene, and repeats until it decides to stop.

**Definition 4. 3D tools.** To construct an interactive 3D world, the agent is provided with five tools: `asset_retrieve`, `asset_add`, `asset_rotate`, `asset_translate`, and `asset_delete`. These tools enable the agent to retrieve assets from the database, add assets to the scene, rotate assets, translate their positions, and remove assets from the 3D world, respectively. For example, `asset_rotate`("00001", "`Bookshelf_01`", (100, 140, 0), (0, 0, 0), (0, 90, 0), 1.0) rotates the bookshelf (located at (100, 140, 0)) 90° around the y-axis while keeping its original scale.

The VWE-BENCH Benchmark

VWE‑BENCH is assembled in three collaborative stages to supply assets, seed worlds, and queries.

Existing indoor‑scene datasets lack the breadth of assets and open‑ended queries needed for interactive world building. VWE‑BENCH fills this gap by coupling human artists with multimodal LLMs to produce a richly annotated corpus.

The benchmark works like an assembly line: artists define a catalog, generative models turn catalog entries into 3D meshes, and an MLLM‑driven critic turns completed worlds into diverse query‑answer pairs.

How does VWE‑BENCH differ from prior indoor‑scene datasets?

Earlier datasets provide static scenes and a fixed set of objects; VWE‑BENCH adds (1) a large, physically‑consistent asset library, (2) seed worlds that span a wide complexity range, and (3) multimodal queries that require agents to both construct and iteratively refine worlds.

Stage 1: Artists list the three assets; an image‑to‑3D model creates meshes for each.

Stage 2: Artists place the can and stool into World A, then add the tree to obtain World B.

Stage 3‑a (construction): An MLLM reads World A and generates “place a can and a stool in a room”.

Stage 3‑b (refinement, precise): An MLLM observes that World B adds a tree and produces the instruction “add a tree next to the stool”.

This miniature run shows how each stage adds a concrete, verifiable artifact that later becomes a query‑answer pair.

Stage 1 – Asset synthesis: annotators define inventory → generate images → convert to 3D meshes → filter & size‑annotate.

Stage 2 – Seed‑world annotation: artists assemble coarse worlds from the filtered assets, ensuring coverage of the full library.

Stage 3 – Reverse query synthesis: an MLLM reads each seed world, critiques it, and produces construction or refinement queries of varying granularity.

**Figure 2.** Illustrative VWE-BENCH data. (a) Asset examples ordered by physical size (small to large). (b) Seed 3D world examples ordered by complexity (low to high).

**Figure 3.** VWE-BENCH distributions. (a) assets across the 20 semantic categories; (b) assets are annotated physical-size class; (c) the seed world complexity varies.

VIBEWORLDING-GYM Environment

VIBEWORLDING‑GYM unifies a sandbox, verifier, and RL loop to train agents for 3‑D world construction.

Current agents stumble when asked to build coherent 3‑D scenes over many interaction turns; they lack a training loop that simultaneously checks physics and user intent.

VIBEWORLDING‑GYM stitches together a stable sandbox, a dual‑constraint verifier, and a joint multimodal RL loop so that agents can practice building worlds while being penalized only for truly wrong outcomes.

How does VIBEWORLDING‑GYM differ from a standard RL environment like OpenAI Gym?

Standard gyms expose a single scalar reward per step and assume a fixed observation space. VIBEWORLDING‑GYM instead returns multi‑view images, runs a two‑stage verifier (physics then LLM rubric), and only grants reward when *both* stages succeed, which forces the agent to master both geometry and intent.

Turn 1: Asset retrieve returns the chair (ID C) and table (ID T) candidates using the Qwen3‑Embedding‑4B retriever trained with InfoNCE loss.

Turn 2: Using the “add” tool, the agent places $T$ at coordinates $(1,1)$.

Turn 3: Using the “add” tool, the agent places $C$ at $(1,2)$, adjacent to $T$.

Verifier runs collision check – no overlap, passes.

Verifier runs height check – both assets rest on the floor, passes.

LLM rubric evaluates intent: the scene matches “chair next to table”, passes.

Reward = 1 (binary success) is fed to the RL update.

The example shows that a single successful episode requires the retriever, the geometric tools, and both verifier stages to line up; a failure in any stage yields zero reward.

The verifier treats world validity as a two‑step gate: first enforce hard physics, then assess soft semantic alignment.

Why not merge the two verifier stages into a single LLM prompt?

Mixing hard physics with open‑ended semantics would force the LLM to reason about precise geometry, a task it is not optimized for; separating them lets a fast Python check filter out obviously invalid worlds before the expensive LLM evaluation.

Collision check flags $S$ and $R$ as intersecting → fail.

Height check flags $L$ as unsupported (no terrain contact) → fail.

Because a physical check fails, the LLM rubric is never invoked and the episode receives reward 0.

The verifier’s short‑circuit logic saves compute by aborting semantic evaluation on obviously broken scenes.

Joint multimodal RL training loop using GRPO.

Experimental Setup

Key verification correlation and dataset split underpin the evaluation.

The automatic verifier’s reward score correlates with human intent‑fulfillment at $ρ = 0.88$ across all evaluated systems.

Spearman correlation computed over 8 frontier MLLMs, 3 agent‑scaffold frameworks, and 6 open‑backbone variants.

VWE‑BENCH is split into disjoint training and test worlds to prevent memorization; the training set contains 5,567 queries (1,129 SFT cold‑start, 4,438 world‑refinement, 1,007 RL) and the test set holds 254 queries (46 SFT, 208 RL).

Pass@1 measures the fraction of queries solved in a single rollout; an unverified query counts as passed only if all verifier dimensions succeed, while a verified query uses the rule‑based asset‑modification score as its Pass@1 value.

Human evaluation complements the automatic metric by blind annotation of (i) intent fulfillment, (ii) binary ecological plausibility, (iii) physical feasibility (using the collision result), (iv) factual hallucination in the final response, and (v) response intelligence on a 1–5 scale.

Baselines share the same sandbox tools, asset library, and verifier. Frontier MLLMs span 8 B–2.8 T parameters (GPT‑5.5, Gemini 3.5‑flash, Gemini 3.1‑pro, Claude‑Opus‑4.8, Kimi‑K3, Qwen3.8‑Max, Qwen3‑VL‑8B/30B‑A3B). Agent‑scaffold baselines re‑implement SceneWeaver, SAGE, and SceneAssistant on top of GPT‑5.5, isolating scaffold effects from post‑training.

Cold‑start fine‑tuning runs for 2 epochs on 122,880‑token sequences at $2\times10^{-5}$ learning rate; RL uses GRPO for 1 epoch with learning rate $5\times10^{-7}$, KL coefficient 0.05, no entropy bonus, and 8 rollouts per prompt. Rendering provides five $1280\times720$ images per turn; RL reward is supplied by Gemini 3.5‑flash. 8 B models train on a single node (8 × NVIDIA H20), while 30 B‑A3B models use three GPU nodes.

Main Results and Bottlenecks

VibeWorlder models achieve the highest overall Pass@1, with RL training closing the gap to frontiers.

The Vibe Worlding task demands multi‑turn spatial reasoning and tool use; current models still fall short, motivating the results presented here.

VibeWorlder‑30B‑A3B attains the highest overall Pass@1 on VWE‑BENCH.

Achieves 59.3% overall, surpassing GPT‑5.5 (57.3%) and Qwen3.8‑Max (56.9%).

**Figure 4.** Overall Pass@1 on VWE-BENCH across models on Verified and Unverified tasks.

**Table 4.** Pass@1 (%) by query sub-type on the Test split. For 3D world construction queries, the sub-types are Theme-only, Theme+elements (Th+El), Full-blueprint (Blupr), and Distractor (Distr). For 3D world refinement queries, the sub-types include asset edit (Precise/Fuzzy), Scene critique (Crit.), Scene guidance (Guid.), Scene restatement (Rest.), and Complex description (Cplx). The abbreviations in parentheses correspond to the column names in the table. “Overall” denotes the mean reward over all query types. “Human” denotes the holistic pass rate from a blind human study per model (a sample passes only if human annotators judge it physically feasible, intent-fulfilling, ecologically plausible, and free of response hallucination) shown in Appendix A.4. Agent-scaffold baselines (SceneWeaver, SAGE, SceneAssistant) use GPT-5.5 as the backbone.

**Figure 5.** Per-dimension pass rate across existing MLLMs, revealing which capabilities bottleneck. Each of the six capabilities (collision-freeness, height plausibility, ecological plausibility, 3D understanding, 3D reasoning, and retrieval plausibility) is quantified by our dual-constraint verifier.

RL Post-training Dynamics

We isolate each training stage’s impact by comparing the full pipeline to a cold‑start‑free ablation.

We isolate the contribution of each training stage by tracking VibeWorlder‑8B during multimodal RL post‑training and comparing to a cold‑start‑free ablation.

After a supervised fine‑tuning (SFT) “cold‑start”, the agent continues learning via reinforcement signals that incorporate multimodal feedback, letting it refine spatial reasoning and tool use.

How does “Multimodal RL Post‑training” differ from standard RL fine‑tuning?

Standard RL fine‑tunes on a single scalar reward, often ignoring multimodal cues. Here the reward is a composite of verified metrics (exact geometry checks) and unverified metrics (LLM‑based judgments), and the policy is initialized from an SFT checkpoint that already respects physical constraints.

SFT alone improves 3D understanding from 0.02 to 0.17.

Figure 6 shows the early solid curve (cold‑start) plateauing at ≈0.17 for 3D understanding.

Removing the cold‑start stage eliminates a 0.63 gain in 3D understanding, leaving the model at 0.17.

Dashed curves in Figure 6 (no‑cold‑start) never rise above 0.17 for 3D understanding.

Without cold‑start, 3D reasoning stays near 0.04, a 0.65 drop relative to the full pipeline’s 0.69.

Figure 6’s dashed line for 3D reasoning remains flat around 0.04.

Ablating cold‑start reduces verified‑query reward by 0.50, from 0.64 to 0.14.

Solid vs. dashed verified‑reward curves in Figure 6 illustrate this gap.

When cold‑start is omitted, unverified‑query reward drops by 0.33 (0.37 → 0.04).

Figure 6’s unverified‑reward dashed curve stays near 0.04 throughout training.

**Figure 6.** The Metric dynamics of the VibeWorlder-8B in multimodal RL training process. Solid curves are the flagship run with cold-start (i.e., RL from the cold-start policy), and dashed curves are the w/o cold-start ablation (i.e., RL from the base backbone).

**Figure 7.** Two dominant failure modes of frontier agents on VWE-BENCH. (a) Imprecise 3D distance editing: The query requires moving rock 18 “7 m forward.” The agent achieves the correct displacement magnitude but moves the object in the opposite direction, resulting in a 14 m positional error under a 0.5 m tolerance. The initial and edited renders (left and right) and coordinate visualization (center) highlight this directional inversion. (b) Overediting: The query requires adding a tree beside the chest. The agent correctly adds the new tree but silently deletes the existing tree, introducing an unrequested modification that violates the editing constraint.

Related Work and CLI Prototype

We situate our approach among prior MLLM‑driven 3D world construction systems and multimodal reinforcement‑learning methods.

MLLM‑based 3D world construction has progressed from Fixed‑Workflow pipelines, which stitch together specialized modules in a predetermined order, to Autonomous Agentic systems that let a single multimodal LLM plan, invoke tools, and iteratively refine scenes. Representative Fixed‑Workflow works include SceneCraft, which generates Blender scripts, and 3D‑GPT, which maps language to procedural parameters; autonomous approaches such as SceneWeaver, SAGE, and Vinedresser3D expose a unified tool suite for planning and self‑evaluation. Despite these advances, existing methods remain confined to idealized benchmarks and cannot robustly handle open‑ended real‑world user queries.

**Figure b.** Tool execution & response: the agent deletes the car and two buildings, re-renders to verify, and replies.

**Figure 8.** The VibeWorlding CLI prototype during a multi-turn refinement session on an urban scene, with the interactive terminal (left) shown alongside the browser-based GUI viewer (right). (a) First-turn reasoning: after the user asks to “remove the car in the middle of the road and delete the two green buildings,” the agent parses the request and enumerates the current 3D world state (trees, skyscrapers, military buildings, streetlights) from the scene’s 3D map. (b) Tool execution and response: the agent proactively invokes three delete tool calls, re-renders the 3D world from multiple views to verify that only the intended assets were removed, and returns a natural-language summary. After refinement, the 3D asset count decreases from 14 to 11 assets accordingly.

Reinforcement learning with verifiable rewards (RLVR) has become a standard way to improve long‑horizon reasoning and tool use, and recent multimodal RL systems extend this paradigm to vision‑language agents. Works such as Kimi K2.5 and OpenSearch‑VL demonstrate joint optimization of visual and textual modalities, while SceneRe‑Vis applies multi‑turn RL to 3D world building. However, no prior work has combined multimodal feedback (e.g., rendered images) with RL for open‑ended 3D world construction.

Case Study: World Construction

Agent builds a full urban street world in six turns, placing 34 assets across 11 types.

The agent constructs a complete modern urban street world in six turns.

The episode runs for five acting turns with four visual‑feedback rounds and terminates after the agent emits no tool call, yielding a final scene of 34 assets spanning 11 distinct types (see Figure 9).

Turn 1: retrieve flat asphalt street, two‑storey shop, red‑lantern string, modern high‑rise, and large advertising column assets.

Turn 2: place the road, arrange three shops with lanterns on the left, position two high‑rises on the right, and insert placeholder advertising columns.

Turn 3: delete placeholder columns, add electronic billboards, add larger lantern strings, and attach signboards to shop fronts.

Turns 4–5: add street lamps and a second lantern variant to enrich ambience.

Turn 6: emit no tool call, terminating the episode with a completed world.

**Figure 9.** Visual feedback across 3D world construction task. Each row is one round of environment feedback: the five fixed camera views the agent receives after its edits are applied. **Row 1** (seen when the agent reasons in turn 3): the planned layout is blocked out—central road, shop row, high-rises across the street—but the lanterns are barely visible and the distant pillars read as stone. **Row 2** (seen in turn 4): the agent has replaced its own placeholder pillars with billboards and enlarged the lanterns. **Row 3**: the final world, 34 assets over 11 types. Every panel in a column shares one crop box, so apparent motion is real motion.

Case Study: World Refinement

Agent deletes the nearest statue in one turn, achieving perfect Pass@1.

The agent attains 100 % Pass@1 on the 3D world‑refinement case, correctly deleting the statue nearest to the altar in a single turn.

Agent issued an `asset_delete` call targeting the statue at (18.78, 20.8, 2.0); the post‑turn map confirms its removal while the other statue remains.

The refinement episode supplies the agent with a full map and five orthogonal renders, then asks it to delete the statue closest to the altar. Because both candidates share the same asset ID, the decision hinges on computing Euclidean distances.

User issues a natural‑language query specifying the desired edit.

Agent parses the query, extracts the altar position, and computes distances to each Statue 01.

Agent selects the nearest statue and issues

Environment applies the deletion, re‑renders the scene, and returns the updated map.

Compute distance to A: $\sqrt{(14.13-17.5)^2 + (18.08-18.0)^2 + (2.0-1.0)^2} \approx 3.37$ m.

Compute distance to B: $\sqrt{(18.78-17.5)^2 + (20.8-18.0)^2 + (2.0-1.0)^2} \approx 3.08$ m.

Margin = $3.37 \text{m} - 3.08 \text{m} = 0.29$ m, so B is the nearest.

The agent must resolve sub‑meter differences; such precision exceeds the typical error band (≈0.5 m) reported for frontier agents.

1. delete the two "Theme-02 black iron pillar 02" [they read as stone pillars rather than electronic advertising columns, and slightly crowd the high-rises] 2. add the retrieved "Billboard 02 / 20 / 19" as distant electronic columns and shopfront signboards 3. add the retrieved "New Year lantern 01" and "Lantern 05" as more prominent red/warm lantern strings, hung continuously along the shops' outward faces Parameter check: all new ids came from earlier retrieve responses: 00654, 03069, 03068, 01243, 03100. [AGENT] tool calls `asset_delete`({"`modified_data`": [ {"name":"Theme-02 black iron pillar 02","pos":[6.4,8.9,0], "Extend":[0.75,0.7,2.7], "reason":"reads as a stone pillar rather than an electronic advertising column"}, {"name":"Theme-02 black iron pillar 02","pos":[7.5,9.25,0], "Extend":[0.75,0.7,2.9], "reason":"slightly crowds the high-rises and lacks advertising character"}]}) `asset_add`({"`modified_data`": [ /* 12 placements, abridged to 3 */ {"name":"Billboard 02","id":"00654","pos":[6.35,9.05,0], "Extend":[0.35,0.95,2.4],"rotate":[0,0,0], "reason":"first large vertical electronic advertising column on the distant open ground, pale teal panel echoing the modern city mood"}, {"name":"Billboard 19","id":"03068","pos":[2.65,1.65,1.65], "Extend":[0.75,0.25,0.55],"rotate":[0,0,90], "reason":"signboard over the first shopfront, raising storefront legibility"}, {"name":"New Year lantern 01","id":"01243","pos":[2.75,1.25,2.15], "Extend":[0.28,0.28,0.95],"rotate":[0,0,0], "reason":"more prominent warm-red lantern on the shops' outward face" ]})

**Figure 10:** Visual feedback across 3D world refinement task. **Row 1:** the five views given to the agent in turn 1. The world holds two instances of the same statue asset, 3.08 m and 3.37 m from the altar. **Row 2:** after the edit. The dashed marker locates the statue the agent deleted; every other asset is untouched. The scattered rocks and haystacks differ slightly between rows because the sandbox re-samples scatter-group instances at each render, not because the agent moved them.

Asset Retrieval Service

Details the graded retrieval pipeline, encoder training, and service evaluation for asset lookup.

Text‑to‑3D‑asset retrieval is graded, not binary. Queries receive assets in four grades: G0 (perfect match), G1 (right entity, wrong attributes), G2 (wrong entity, right attributes), and G3 (no match). The error cost is asymmetric: G2 distractors are most harmful and must be excluded from the top‑K, while G1 substitutes are tolerable.

The pipeline is interaction‑free and builds supervision from asset metadata in three stages: (1) asset‑grounded query synthesis, (2) hierarchical hard‑negative mining, and (3) two‑dimensional LLM grading. Each stage adds a layer of graded supervision without requiring human interaction.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers