Learning to Think and Explore in 3D World
Yijiang Li, Bingyang Wang, Nuno Vasconcelos
Act3D learns to navigate 3D environments and acquire task-relevant visual evidence through outcome-driven reinforcement learning.
How can vision-language models (VLMs) learn to actively explore 3D environments to answer spatial questions, rather than relying on pre-computed 3D representations?
Vision-language models (VLMs) typically rely on pre-provided 3D data like point clouds or multi-view images, which fails when the necessary evidence is occluded or missing from the initial viewpoint. Act3D treats the 3D scene as an external, interactive environment, using a two-stage training process: trajectory-supervised fine-tuning establishes basic navigation, followed by reinforcement learning (RL) that optimizes the policy based solely on final answer correctness. This approach achieves a 53.2% average accuracy across six benchmarks, outperforming both passive perception models and proprietary baselines while using 50–70% fewer input frames.
Paper Primer
The core mechanism is an interactive policy that alternates between reasoning and navigation: the model receives an egocentric 2D image, generates a reasoning step, and chooses a relative movement action (e.g., "move left") to update its viewpoint. The RL objective uses Group Relative Policy Optimization (GRPO) to assign credit for the final answer across the entire trajectory, effectively teaching the model to "search" for evidence without explicit supervision on where to look.
Act3D significantly improves performance on questions requiring extensive exploration.
Stratified analysis of ScanQA validation questions shows that performance gains over the base model increase from 0.6% for questions answerable from the initial view to 16.3% for those requiring significant navigation.
The model generalizes to unseen domains better than its supervised initialization.
When trained only on ScanQA, the model improved average accuracy on five held-out benchmarks by 2.1% over its supervised initialization, with specific gains of 6.0% on MP3D and 3.8% on SQA3D.
Why use relative camera movements instead of absolute coordinates?
Relative actions (e.g., "turn right") are scale-invariant and operate in the camera's local coordinate system, whereas absolute coordinates require the model to predict positions in a global reference frame that varies across scenes and is not directly observed.
Does more exploration always lead to higher accuracy?
Not necessarily. While explicit interaction rewards increase the frequency of exploration, they do not uniformly improve accuracy; the interaction reward primarily serves to encourage evidence acquisition, while outcome-based feedback guides the policy toward effective reasoning.
Outcome-based RL can successfully induce task-directed exploration in VLMs, allowing them to resolve spatial ambiguities and occlusions by actively acquiring evidence rather than relying on pre-packaged 3D representations.
The Challenge of 3D Reasoning
We expose the limitation of static 3D inputs and propose treating reasoning as active exploration.
Current 3D vision‑language models assume the full scene is available as a static input, e.g., a point cloud or multi‑view video. This passive perception limits the model’s ability to acquire missing evidence when the scene is only partially observable. We instead treat 3D reasoning as an active exploration task: a Vision‑Language Model (VLM) learns to navigate and interact with an environment to gather the visual information it needs.
**Figure 1.** Learning to reason and explore in 3D. Unlike prior methods with passive perception which assumes receive complete 3D representations as input, Act3D treats 3D representations as interactive environments, learning to alternate between reasoning and actions to actively explore the 3D world before committing an answer.
These baselines give the model a fixed set of visual inputs—such as a complete 3D scan or a pre‑recorded video—without any opportunity to request additional views.
In many real‑world scenarios the agent only sees a subset of the scene from its current viewpoint, so crucial objects may be hidden.
Shifting from static inputs to active interaction lets the model acquire missing evidence on demand.
The Act3D Framework
We train a VLM to explore 3D scenes via a two‑stage policy that learns navigation and answering.
We adopt a two‑stage training pipeline. First, supervised fine‑tuning (SFT) on curated interaction trajectories gives the model a coherent reason‑act‑observe behavior. Then reinforcement learning with a verifiable reward refines the policy to explore task‑directedly, using only the final answer quality as supervision.
The framework casts 3D reasoning as an active‑exploration POMDP where a VLM moves a camera, gathers visual evidence, and answers a question once enough information is collected.
Why model 3D reasoning as a POMDP instead of assuming a full 3D representation is given?
Because the scene is hidden and only observable through limited camera views; a POMDP explicitly captures the need to gather information before answering, whereas a full‑representation assumption would ignore the exploration challenge.
The policy decides where to move the camera (relative motions) and when to stop and answer, conditioning on all past observations and actions.
Unit step for forward is $\delta\xi = 1$ m, so $n = \text{clip}(\text{round}(2.4/1),1,8) = 2$; the camera moves $2 \times 1 = 2$ m forward.
New position $e_{1} = \Pi_B(e_0 + 2\,\text{m}\,u_{\text{forward}})$; since the room is 2 m wide, the clamp $\Pi_B$ keeps $e_{1}$ inside the room.
Yaw rotation unit step is $30^\circ$, so $n = \text{clip}(\text{round}(45/30),1,8) = 2$; the view direction rotates $2 \times 30^\circ = 60^\circ$ to the right.
Resulting orientation $f_{2} = \text{Rot}_{\hat{z}}(60^\circ) f_{1}$.
The policy then emits $\text{STOP}(y)$, terminating the episode.
This example shows how a single scalar step size $d$ is quantized into an integer multiplier $n$, guaranteeing that each action respects the discrete action space while still approximating the desired continuous motion.
How does the relative parameterization differ from absolute actions, and why is it preferred?
Relative actions describe motion in the camera’s own frame (e.g., “move left 2 steps”), which is invariant to the scene’s global scale and avoids predicting unseen absolute coordinates. Absolute actions would require the policy to guess world‑frame positions that are not directly observable.
Supervised fine‑tuning uses curated interaction trajectories to give the VLM a sensible starting policy before reinforcement learning.
Why is a supervised fine‑tuning stage needed before reinforcement learning?
Because RL alone would start from a random policy, leading to poor exploration and slow convergence; SFT provides a reasonable prior that accelerates learning of effective exploration strategies.
The reward is computed only from the final answer quality, encouraging the policy to explore only as much as needed to improve that answer.
Why does the reward ignore intermediate viewpoints instead of providing step‑wise feedback?
Because the ultimate goal is answer correctness; intermediate signals could bias the policy toward actions that look good locally but do not gather the most useful information for the final answer.
**Figure 3.** Training data composition. Distribution of SFT trajectories and RL questions across datasets, with representative examples of curated spatial reasoning questions.
Performance and Benchmarks
Act3D’s training and evaluation show it outperforms baselines across 3D reasoning benchmarks.
We evaluate the full Act3D pipeline—from trajectory‑supervised initialization to outcome‑driven reinforcement learning—under a uniform interactive protocol.
Act3D achieves the highest average accuracy (53.2 %) across six 3D reasoning benchmarks, surpassing the Qwen 3.5‑9B base model by +22.0 %.
Table 1 shows Act3D (mixed‑data) with 53.2 % average, while the base model attains 31.2 %.
GRPO treats a batch of rollouts as a group, computes a normalized advantage for each rollout, and updates the policy with PPO‑style clipping while penalizing deviation from a reference policy.
How does GRPO differ from standard PPO?
Standard PPO computes an advantage per timestep, whereas GRPO computes a single standardized advantage per rollout group and applies the same clipped ratio to every token in that rollout. GRPO also adds an explicit KL penalty toward a reference policy, which PPO typically omits.
**Table 1.** Performance comparison on six 3D reasoning benchmarks. We compare Act3D with existing active and passive methods. Green highlights indicate benchmark domains included in model training.
**Table 2.** ScanQA Acc. given different initial observation.
Act3D outperforms passive baselines across all benchmarks.
Analysis of Exploration Dynamics
We dissect Act3D components to see how each affects interactive 3D reasoning performance.
Act3D frames 3D reasoning as an active exploration task: a Vision‑Language Model (VLM) must move and look around to collect missing visual cues instead of receiving a full 3D map up front.
**Figure 4.** Learning dynamics of Act3D. (a) ScanQA-only validation. (b) Mixed-data validation gains relative to SFT initialization. (c) Mean number of interactions for the final reward. Panel (c) is in-training rollout statistics; panels (a) and (b) are validation accuracies, and no value should be read across the two.
**Figure 5.** Performance gains across different exploration demands. ScanQA validation questions are grouped by a geometry-based estimate of the interactions needed to reach the answer object from the initial viewpoint. Stacked bars show the accuracy improvements from SFT over the base model and subsequent RL over SFT.
**Figure 6.** Qualitative analysis. We highlight emergent exploration behaviors. Arrows denote the executed actions, and text excerpts beneath illustrate the reasoning of the policy.
RL over SFT initialization raises average accuracy from 48.6 % to 53.2 % (+4.6 %).
Mixed‑data training improves five of six benchmarks, with the largest gains on ARKit (+14.1 %) and SPBench‑MV (+8.1 %).
Training RL solely on ScanQA still generalizes: average accuracy climbs from 47.8 % to 50.6 % (+2.8 %).
Cross‑domain gains appear on four other benchmarks, notably MP3D (+6.0 %) and SQA3D (+3.8 %).
Validation accuracy improves from 40.2 % to 45.5 % (+5.3 %) as RL steps increase (Figure 4a).
Corresponding interaction count rises from ~3.3 to ~8.3 per episode (Figure 4c), linking more exploration to better performance.
Higher exploration demand yields larger gains; the hardest group (7+ actions) sees a +16.3 % improvement, with RL contributing an extra +10.9 % over SFT.
Stratifying ScanQA questions by estimated action count reveals a monotonic increase in combined SFT+RL benefit.
Using a single rendered view (41.0 %) matches the performance of a bird’s‑eye view plus four frames (36.7 %), a drop of –4.3 %.
Adding a BEV to a single view actually reduces accuracy to 36.7 %.
Rendered observations outperform nearest‑view retrieval by +10.6 % average accuracy under matched training conditions.
Retrieval introduces pose mismatches that hurt the action–observation consistency crucial for interactive reasoning.
Relative camera actions achieve 41.4 % accuracy versus 36.7 % for absolute pose prediction, a –4.7 % drop when using absolute actions.
Relative actions align with the local, sequential nature of observation‑conditioned exploration.
Supervised fine‑tuning (SFT) initialization adds +3.2 % accuracy over RL without SFT (39.5 % → 42.7 %).
SFT‑initialized RL also raises the average interaction count from 2.7 to 8.7 per episode.
Adding a small interaction reward (+0.05 per step) boosts SPBench‑MV accuracy by +6.4 %.
While the reward raises interaction frequency, it slightly harms training‑domain performance (0.3–2.3 % loss).
Implementation Details and Robustness
Key performance numbers and qualitative observations for Act3D.
The final experiments confirm that Act3D consistently outperforms both passive baselines and prior active‑exploration methods, while also revealing how different design choices affect performance.
**Figure 7.** Rendered versus retrieved observations. Comparison of pose-aligned renders and nearest available real images from ScanNet. Distance and angular offsets indicate discrepancies between the requested and retrieved camera poses.
**Figure 8.** ScanQA Acc. of different action parameterizations.
**Figure 9.** Effect of supervised initialization on RL. We report ScanQA Acc. vs. average number of interactions per episode with and without SFT initialization. RL-only results are shown after 80 and 470 training steps.
**Figure 10.** Effect of interaction reward. ScanQA Acc. with and without the per-interaction reward across held-out and training benchmarks. Numbers below the bars indicate the average number of interactions per episode.
Qualitative examples demonstrate that the policy selects exploration actions tailored to the specific question, gathering evidence that directly resolves ambiguities in the scene.