Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modeling

Minghan Qin, Yuang Wang, Xiuyu Yang, Yushi Long, Yujian Zhang, Ruihuan Wang, Kai Ye, Yangang Zhang, Hang Li

Lucida reconstructs real indoor scenes as editable assets by deferring precision to a closed-loop VLM placement policy.

How can we reconstruct real indoor scenes as editable, simulation-ready 3D object assets using only posed RGB-D observations?

Composable scene modeling struggles because existing pipelines demand perfect geometry and clean segmentation from the start, which real, cluttered indoor captures rarely provide. Lucida keeps the modular parse-generate-place order but redistributes the requirements, allowing each step to consume only what a real capture reliably provides while deferring precision to a final, closed-loop refinement stage. This approach significantly improves reconstruction fidelity, raising strict alignment success (ADD-SB@0.05) by up to 25.6 percentage points over the strongest baseline.

Paper Primer

The system builds a scene graph by consolidating multi-view evidence into bundles for each object, which then guide an amodal generation module to synthesize complete, occlusion-free 3D assets. The core innovation is GizmoAct: a Vision-Language Model (VLM) policy that treats 3D object placement as a multi-turn GUI interaction, where the model iteratively manipulates a 3D gizmo to align the asset with the scene point cloud until it decides the alignment is sufficient.

Lucida achieves superior scene-level reconstruction accuracy compared to existing compositional methods.

On the R2S-Scene benchmark, Lucida achieves a scene F-Score of 0.924, significantly outperforming SAM 3D (0.794) and SceneGen (0.364). A 16.4% improvement in scene F-Score over the strongest baseline.

GizmoAct robustly aligns generated assets even when starting from coarse, inaccurate initial poses.

On the CA-1M dataset, GizmoAct raises the ADD-SB@0.05 success rate from 57.8% (RecGen) to 83.4%. A 25.6 percentage point increase in strict alignment success.

Why does this system use a multi-turn GUI interaction for object placement instead of a standard regression model?

Standard regression models struggle with the ambiguity of absolute pose estimation from noisy, cluttered inputs. By casting placement as a closed-loop interaction, the VLM can observe the consequences of its own incremental edits and decide autonomously when the asset is correctly aligned.

Does this method require perfect segmentation or clean, unoccluded views of objects to function?

No. Lucida is designed specifically to handle the limitations of real captures; it consolidates multi-view evidence to handle occlusions and uses a refinement policy that accepts mismatched assets and coarse initial poses, deferring the need for precision to the final placement step.

Introduction to Composable Scene Modeling

We frame composable scene modeling and expose why existing pipelines cannot meet real‑world capture constraints.

Composable scene modeling seeks to reconstruct real indoor environments as collections of editable 3D objects arranged as observed, enabling downstream robot simulation, AR/VR, and content creation. Existing pipelines split this task into parsing, asset generation, and placement, but each step presumes inputs—accurate geometry, unoccluded views, and perfectly matching assets—that cluttered real‑world captures rarely provide.

It is the process of turning a real indoor capture into a set of independent, fully editable 3D objects placed exactly where they appear in the scene.

Current pipelines assume that parsing yields precise instance masks and clean point clouds, that generation receives an unoccluded, centered view of each object, and that placement can rely on an asset that already matches the observation. In cluttered indoor footage, occlusions blur object boundaries, repeated furniture confuses instance association, and noisy depth makes clean point clouds scarce, causing errors to cascade through the pipeline.

**Figure 1 Teaser.** Given posed multi-view observations of a real indoor scene, our method constructs object-centric multi-view evidence, generates complete 3D assets, and places them into a composable reconstruction. The insets illustrate closed-loop rotation, translation, and scale refinement of object models against the observed scene geometry.

Lucida preserves the parse‑generate‑place order but redistributes requirements: parsing only needs to discover objects and collect multi‑view evidence, generation conditions on this evidence rather than on pristine crops, and placement tolerates mismatched assets by iteratively adjusting a gizmo in a closed loop.

The key shift is moving from monolithic meshes to editable object assets that can be individually refined.

The Lucida Pipeline

Lucida refines object poses through a closed‑loop gizmo policy that iteratively edits a rendered scene.

Estimating a precise 9‑DoF pose from noisy RGB‑D captures is fragile when done in a single step. Lucida therefore treats pose estimation as a closed‑loop interaction: it repeatedly renders the current object state, lets a vision‑language model suggest a tiny edit in the gizmo’s local frame, applies the edit, and iterates until alignment is achieved.

A scene graph is a network where each node stands for an object and edges encode spatial relations, turning a room into a linked collection of 3D items.

For every object we collect all RGB‑D views that see it, bundle them into a compact record, and use that record as the sole source of geometry and appearance for the rest of the pipeline.

**Figure 2. Overview of our method.** Given posed scene observations, the parsing stage constructs an object-centric scene graph with per-object multi-view evidence and coarse 3D estimates. For each object, the multi-view evidence is used to synthesize a complete object-centric image, which is then lifted into a 3D asset. GizmoAct places the asset through closed-loop 9-DoF gizmo manipulation, iteratively refining rotation, translation, and anisotropic scale until it decides that alignment is reached. The placed assets form an editable reconstruction of the captured scene.

Instead of predicting a final pose in one shot, GizmoAct repeatedly renders the current object, asks a vision‑language model to suggest a tiny edit expressed in the gizmo’s local frame, applies the edit, and loops until the render looks aligned.

Apply rotation: $R_1 = I \cdot R_{ZXY}(10°,0°,0°)$ yields a 10° yaw about the Z‑axis.

Update scale: $s_1 = (1,1,1) \odot (1.1,1,1) = (1.1,1,1)$.

Translate: $p_1 = (0,0,0) + R_1 \, (0.2,0,0) \odot (1,1,1) = (0.2\cos10°, 0.2\sin10°, 0)\approx(0.197,0.035,0)$.

Because translation is applied after rotation, the object moves along the newly oriented X‑axis, demonstrating why edits must be expressed in the gizmo’s current frame.

How does GizmoAct differ from a conventional pose‑regression network?

Standard regressors predict a single absolute pose from raw images, which is hard to learn because the correct pose is ambiguous and the loss landscape is highly non‑convex. GizmoAct instead predicts tiny incremental edits in the gizmo’s local frame, repeatedly re‑renders the result, and stops only when the visual alignment looks correct, turning a hard regression problem into a series of easy, locally observable adjustments.

**Figure 3. GizmoAct graphical observation and interaction trajectory.** (a) The observation at one turn. The instruction block specifies the adjustment target and the referring cue; the main views pair the raw input image with renderings that overlay the asset, its yellow 3D box, and the gizmo showing the object's local frame (red/green/blue axes), in which all edits are expressed. Semi-transparent green marks the model surface occluded by the scene point cloud. Auxiliary views add multi-view evidence, and orthographic views along the local axes support fine refinement. (b) A rollout in which the policy issues a flawed edit (red cross) and recovers from the resulting state (green checks). During SFT data construction, we simulate such cases through error injection and supervise only the recovery actions.

Render the current object state together with the gizmo, 3D box, and auxiliary views to form observation $o_t$.

Feed $o_t$ to the vision‑language model $\pi_\theta$ which emits an action $a_t$ (either `update_pose` with deltas or `stop`).

Apply the action using the incremental update equations, producing the next state $x_{t+1}$.

If the action is `stop`, terminate; otherwise return to the rendering step.

Example of a GizmoAct action emitted by the VLM.

**Figure 4.** Extended actions for coarse rotation. When the initial rotation error is large (step 1), the policy first issues `switch_obs`, which changes only the rendering mode: the next observation (step 2, enlarged in the bottom panel) shows the original-view pair together with six orthographic renderings of the asset along its signed local axes, exposing the sides the main views hide. From these views the policy predicts a single `permute_axis` action, which selects one of the 24 axis-aligned reorientations and removes the dominant rotation residual in one step. The remaining error is repaired by `update_pose` (step 3) before the policy predicts `stop` (step 4). The dashed arrow marks the alternative of predicting the whole correction with a single regular `update_pose`, skipping the permutation, which is less accurate.

Pose Estimation Setup

We detail the pose‑initialization step that underpins accurate 9‑DoF refinement.

Pose refinement must be evaluated without the confounding influence of upstream detection errors, so we isolate GizmoAct and examine how different initial pose guesses affect the final 9‑DoF estimate.

Boxer takes a coarse 2‑D bounding box and a depth map to produce a rough 3‑D pose, giving the refinement policy a sensible starting point.

How does Boxer Initialization differ from the Any6D or SAM 3D initializers?

Boxer builds the pose purely from back‑projected depth at the 2‑D box corners, whereas Any6D first estimates a 6‑DoF pose from RGB‑D cues and then augments it with a scale factor; SAM 3D instead regresses a full 3‑D shape and extracts a pose by aligning the shape to the depth map. Consequently, Boxer is geometry‑driven and deterministic, while the others rely on learned regressors that can introduce bias.

Back‑project each corner using its depth, yielding four 3‑D points near the cube’s corners.

Fit an oriented bounding box to these points; the fitted box has centroid (0.5, 0.5, 1.0) m and axes aligned with the world axes.

Compute translation as the centroid offset, rotation as the identity (axes already aligned), and scale as 0.2 m / model side length.

Combine translation, rotation, and scale into a 9‑DoF pose estimate ready for refinement.

Boxer provides a deterministic pose that captures the object’s coarse position and size, allowing the RL policy to focus on fine‑grained alignment rather than searching the entire pose space.

For completeness we also support Any6D‑style depth‑and‑mask initialization and SAM 3D‑derived initialization, but the experiments in this section use Boxer as the primary seed.

**Table 1.** Comparison of different initialization methods across three datasets (R2S-Object, CA-1M, and ADT) using metrics ADD-SB, ADD-SB@0.05, and 3D IoU.

Pose Estimation Performance

GizmoAct (max 4 views) sets new pose‑estimation accuracy records across all datasets.

We evaluate pose estimation using the ADD‑SB surface‑distance metric, its thresholded success rates, and 3D IoU for oriented‑box overlap.

ADD‑SB measures the average nearest‑neighbor distance between the predicted and ground‑truth object surfaces, expressed in meters.

GizmoAct (max 4 views) achieves the lowest ADD‑SB (0.017 m) on R2S‑Object, a 55 % reduction versus the strongest baseline.

Table 2 shows GizmoAct (max 4 views) at 0.017 m while the next best baseline (RecGen 2 views) is 0.038 m.

**Figure 6.** Qualitative comparison of object pose estimation on CA-1M, R2S-Object, and ADT. The leftmost column shows the scene image and a close-up with the target instance boxed. Blue renders the ground-truth model at the ground-truth pose; each method's model, posed by its prediction, is rendered jointly with this blue reference, so their mutual occlusion—together with the three axis-aligned close-ups—reveals the pose accuracy, and the GT column shows the reference alone. Lucida aligns the generated assets more closely with the observed instances across categories, shape complexity, and viewpoints.

We also test robustness to different pose initializations, keeping the same GizmoAct policy while varying the starting pose.

Boxer initialization yields the best overall ADD‑SB on R2S‑Object (0.022 m) and CA‑1M (0.026 m) among the three initializers.

Table 3 reports Boxer at 0.022 m (R2S‑Object) and 0.026 m (CA‑1M), lower than Any6D* and SAM 3D.

Full Scene Reconstruction

Our method cuts scene error and boosts reconstruction quality across all metrics.

Our approach attains the lowest scene Chamfer distance and the highest scene F‑Score among all baselines.

Table 4 shows a scene CD of 0.010 m² versus 0.022 for SAM 3D and 0.030 for SceneGen, and an F‑Score of 0.924 versus 0.794 and 0.682.

The table compares three methods (SceneGen, SAM 3D, and Ours) across several metrics: Scene CD (Pred-GT CD, GT-Pred CD, and total CD), Scene F-Score, BBox IoU, Object CD, and Object F-Score.

**Figure 7** GizmoAct pose-refinement trajectories under different initializations. For each dataset we show a reference image with the target instance boxed, and closed-loop rollouts of the same policy initialized by Boxer, Any6D*, and SAM 3D. Each cell shows the observation at that step (main views on top, orthographic views below); arrow labels denote the predicted action: an axis permutation (P) or the rotation (R), translation (T), and scale (S) components of `update_pose`. The final stop action is not shown. Any6D* uses only the depth-and-mask pose initialization from Any6D, without FoundationPose prediction or refinement. Across object categories and geometries, GizmoAct consistently refines each initialization to an accurate pose, outperforming the baseline estimates.

**Figure 8. Qualitative scene-level comparison.** For each scene, we show the multi-view capture, a scanned reference view, the selected evaluation instances, and reconstructions rendered from the main, side, and top viewpoints. SceneGen and SAM 3D receive the same reference image and instance masks, whereas Lucida uses per-instance multi-view evidence. The asterisk marks our reconstruction restricted to the object instances included in the comparison.

Ablation Studies

We quantify how each scene‑parsing component and training choice impacts detection and reconstruction quality.

We first isolate the three stages of the scene‑parsing pipeline (keyframe selection, evidence consolidation, and relation‑aware refinement) to see how each contributes to detection and reconstruction.

Uniform keyframe sampling hurts detection the most.

mAP drops from 0.597 to 0.516 when geometry‑aware selection is replaced by uniform sampling.

The table compares the performance of "Boxer initialization" versus "Random perturbation" across three datasets (R2S-Object, CA-1M, and ADT) using three metrics (ADD-SB, @0.05, and 3D IoU).

We next examine how the pose distribution used during RL training influences the GizmoAct policy.

**Table 6.** Effect of the RL training pose distribution. The two rows differ only in the initial-pose distribution used during RL training: poses produced by Boxer versus random perturbations. Both policies are evaluated from Boxer initialization, and matching the training distribution to the test-time initializer improves every reported metric. @0.05 denotes the ADD-SB success rate at that threshold; best per dataset in bold.

**Table 7.** Comparison of GizmoAct training strategies on the hard subset. SFT (no context) is trained with single-step supervision and no interaction history; all three policies are evaluated in the same closed-loop setting with at most 12 steps. RL improves every metric, with the largest gains on rotation accuracy. Evaluated on a hard subset of 110 asymmetric objects with large Any6D* initialization errors; best in bold.

RL training dramatically improves rotation accuracy on the hard subset.

Rotation error drops from 45.19° (SFT) to 20.98° (RL), and Rot.@5° rises from 52.7%/60.9% to 67.3%.

Related Work

We situate Lucida among prior composable scene, detection, and pose‑refinement approaches.

Composable scene modeling treats a captured environment as a set of editable 3D objects instead of a single monolithic mesh. Optimization‑based pipelines attach object structure to differentiable renderings via discovery, mask supervision, or Gaussian grouping, but they can only explain observed pixels, leaving occluded surfaces unrecovered. End‑to‑end approaches regress a full scene from images, yet synthetic supervision limits their transfer to real captures and joint shape‑pose prediction couples the two error sources.

Multi‑stage pipelines decompose the problem into segmentation, per‑object asset acquisition, and placement, often retrieving CAD models or generating meshes from image‑to‑3D networks anchored to depth. Because each stage assumes precise inputs—clean masks, unoccluded crops, depth consistent with the generated asset—early mistakes propagate forward with no correction mechanism. Lucida retains the modular decomposition but shifts precision to the output stage: parsing gathers multi‑view evidence, generation conditions on that evidence, and GizmoAct closes the loop on placement, tolerating coarse initializations.

Scene‑level 3D object detection aims to output a de‑duplicated inventory of oriented 3D boxes for a captured scene. Monocular detectors predict boxes from a single view but are fragile to occlusion and lack instance persistence. Multi‑view methods such as Boxer, BoxFusion, and Rooms from Motion either pool posed views into a volume or fuse per‑view detections; Lucida’s parse step instead detects on keyframes, fuses multi‑view observations, and verifies the scene graph against the full capture, achieving the highest reported mAP.

3D visual grounding localizes a referred object, traditionally via specialist detectors on reconstructed point clouds. Recent work leverages foundation models (PlaceIt3D, FirePlace, VULCAN, VLMPose) but either limits pose to 6‑DoF or relies on coarse language constraints. GizmoAct tackles the opposite regime: it aligns a generated asset to an observed instance with full 9‑DoF pose, using a closed‑loop policy that iteratively refines alignment against point‑cloud renderings.

**Figure 9.** Qualitative comparison of GizmoAct training strategies on two difficult examples. Each row shows the closed-loop rollout of the SFT, SFT (no context), or RL policy from the same initialization, evaluated as in Table 7; notation follows Figure 7. The SFT policies stop early with residual rotation errors or drift without converging, whereas RL keeps correcting the remaining rotation before stopping.

Conclusion

Summarizing contributions, limitations, and acknowledging support.

Lucida advances composable scene modeling by preserving the parse–generate–place pipeline while redistributing requirements to match what real captures reliably provide. The parse step discovers candidate instances on geometry‑aware keyframes, consolidates evidence over the full sequence, and refines a scene graph. The generate step amodally completes each instance, and GizmoAct—a VLM‑driven policy—aligns each generated asset through closed‑loop gizmo manipulation.

A remaining limitation is that objects missed during parsing cannot be recovered by later generation or grounding stages, and the current closed‑loop refinement operates only at the final placement step. Extending agentic refinement across the entire parse–generate–place pipeline toward a fully agentic framework is a key direction for future work.

The work is authored by Minghan Qin, Yuang Wang, Xiuyu Yang, Yushi Long, Yujian Zhang, Ruihuan Wang, Kai Ye, Yangang Zhang, and Hang Li, with affiliations to ByteDance Seed, Peking University, and Zhejiang University.

The authors thank Wei Li, Heng Dong, and Qifeng Zhang for providing the pretrained VLM and assisting with model training, and Lihao Liu, Baifeng Xie, and Yiming Qiao for help with simulation data, evaluation, and physical‑plausibility post‑processing.

**Figure 10.** Additional qualitative comparison of object pose estimation on CA-1M, R2S-Object, and ADT. Same layout and reading protocol as Figure 6: each method’s posed model is rendered jointly with the ground-truth posed model (blue), from a main view and three axis-aligned close-ups.

**Figure 11.** Additional GizmoAct trajectory examples on CA-1M, R2S-Object, and ADT. Same layout and notation as Figure 7: a reference image per dataset, followed by closed-loop rollouts from Boxer, Any6D*, and SAM 3D initializations.

**Figure 12.** Further GizmoAct trajectory examples on CA-1M, R2S-Object, and ADT. Same layout and notation as Figure 7: a reference image per dataset, followed by closed-loop rollouts from Boxer, Any6D*, and SAM 3D initializations.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers