Qwen-Drive-1.0: An Initial Step Towards a Vision-Language Foundation Model for Autonomous Driving

Xin Zhou, Zongchuang Zhao, Zhibo Yang, Mingsheng Li, Humen Zhong, Shuai Bai, Du Chu, Ruizhe Chen, Zhaohai Li, Jun Tang, Qiuyue Wang, Mingkun Yang, Jiazhao Zhang, Dayiheng Liu, Dingkang Liang, Xiang Bai

Qwen-Drive-1.0 unifies 3D perception, VQA, and motion planning in a single vision-language foundation model.

How can we unify 3D perception, general vision-language reasoning, and motion planning into a single foundation model for autonomous driving?

Autonomous driving systems often rely on separate modules for perception and planning, which complicates integration and forces a trade-off between driving performance and general-purpose reasoning. Qwen-Drive-1.0 integrates an external bird's-eye-view perception head and a diffusion-based planning expert into a frozen pretrained vision-language model, using a staged training recipe to align these tasks without architectural changes. The model achieves competitive 3D detection and motion planning performance while retaining general vision-language capabilities, enabling unified cockpit and driving integration.

Paper Primer

The core challenge is that textual supervision in standard vision-language models (VLMs) fails to constrain 3D spatial layout, while extensive domain adaptation often triggers catastrophic forgetting of general world knowledge. Qwen-Drive-1.0 solves this by attaching an explicit 3D perception probe and a planning expert to a shared VLM, using a staged training recipe to preserve broad visual understanding.

The method hinges on a dual-module architecture: a bird's-eye-view (BEV) perception head that lifts image features into a 3D volume for explicit scene probing, and a diffusion-based Planning Expert that conditions on cached VLM representations to generate future ego trajectories. The BEV head acts as a mail sorter: it reads image features and projects them into a shared 3D grid, allowing the model to perform detection and occupancy prediction without altering the underlying VLM.

Qwen-Drive-1.0 achieves highly competitive 3D perception and motion planning performance while maintaining general-purpose VLM capabilities.

The model reaches 43.95 mAP on nuScenes 3D detection and a 7.91 Rater Feedback Score on the Waymo Open Dataset end-to-end benchmark. It significantly outperforms the base Qwen3.5-4B model in driving scene understanding while preserving general instruction-following performance.

Why use an external perception head instead of relying on the VLM's internal spatial reasoning?

The authors observe that pretrained VLM features support visual-text alignment but do not directly expose the 3D structure required for driving; an explicit probe is necessary to constrain 3D layout and depth.

How does the model avoid catastrophic forgetting during domain adaptation?

The staged training recipe combines driving-specific supervision with general-purpose vision-language data, and the model keeps the VLM architecture frozen during the perception and planning training stages.

This paper demonstrates that a single foundation model can effectively handle both autonomous driving tasks and general cockpit interactions, potentially reducing the compute and hardware costs of future integrated vehicle platforms.

Introduction and Motivation

We introduce Qwen-Drive-1.0, a unified vision‑language model for driving that adds perception and planning heads, preserving the VLM.

Recent autonomous‑driving work has abandoned hand‑crafted, task‑specific pipelines in favor of end‑to‑end learning systems. Vision‑language‑action models exploit large‑scale pretrained VLMs to fuse scene understanding, reasoning, and control, promising better generalization to rare or out‑of‑distribution situations.

However, two fundamental gaps remain. First, textual VQA supervision does not force the model to predict explicit 3D structure—depth, occupancy, or object geometry—so the resulting representations can be vague in space. Second, heavy domain‑specific fine‑tuning tends to overwrite the broad visual and world knowledge acquired during pretraining, leading to catastrophic forgetting of capabilities that are crucial for out‑of‑distribution reasoning.

The core problem is to build a single model that can both reason about driving scenes in 3D and retain the wide‑range visual‑language abilities of a pretrained VLM.

The shift from modular pipelines to a unified foundation model unlocks both comprehensive 3D perception and retained general vision‑language capability for autonomous driving.

Unified Model Architecture

Qwen-Drive-1.0 integrates a shared vision-language backbone with specialized perception and planning modules for unified driving intelligence.

Qwen-Drive-1.0 unifies 3D perception, reasoning, and motion planning by augmenting a shared Vision-Language Model (VLM) with two external modules: a Bird's-Eye-View (BEV) perception head and a Planning Expert. The shared backbone processes diverse inputs—from single images to temporal sequences—while the external heads translate these features into geometric predictions and future ego trajectories.

**Figure 2.** Unified architecture of Qwen-Drive-1.0 for 3D perception, visual question answering, and motion planning. A shared vision encoder and VLM support text generation, while the external BEV perception head and Planning Expert produce geometric predictions and future ego trajectories.

The BEV perception head constructs a top-down Bird's-Eye-View (BEV) representation—a 3D map of the driving scene—by lifting 2D image features into 3D space and aggregating them onto a shared ego-centric grid.

The Planning Expert treats motion planning as a conditional generation task, using a diffusion transformer to predict a clean future trajectory from noisy waypoint tokens.

Four-Stage Training Recipe

Four-stage recipe trains perception, language, and planning components in a staged fashion.

The model must acquire 3‑D perception, language reasoning, and planning abilities, but learning them all at once overwhelms the shared backbone. Splitting training into focused stages lets each new head adapt before being exposed to the full multitask load.

We decompose learning into four sequential phases, each freezing the parts that are already useful and only updating the newly added module.

Stage 1 – Initialize the BEV perception head; train with loss $L_{\text{perc}}$ while vision encoder and VLM stay fixed.

Stage 2 – Jointly train perception head, vision encoder, and VLM on mixed perception ($L_{\text{perc}}$) and VQA ($L_{\text{ntp}}$) samples; use a 20× higher learning rate for the BEV head.

Stage 3 – Freeze vision encoder and VLM; train the Planning Expert on trajectory loss $L_{\text{plan}}$ with optional textual reason $r$.

Stage 4 – Reinforcement‑learning fine‑tune the Planning Expert using task‑level rewards; only stochastic trajectory steps are optimized.

During reinforcement learning we inject controlled noise only into the last three integration steps, letting the planner explore diverse futures while preserving the pretrained flow for earlier timesteps.

Initialize waypoints $\tau(0)=0$ for all five positions.

Compute deterministic flow $v_\theta\Delta t$ for each step (here $v_\theta=0.5$ gives a shift of $0.05$ per step).

At $k=7$, evaluate score correction $s_\theta$ using Eq. 9, yielding a small pull toward the clipped endpoint $\hat{\tau}=0.8$.

Form the mean $\mu(7)=\tau(7)+0.05+0.03^2 s_\theta$ and add a low‑frequency perturbation $\sigma_7\Phi Z_7$ (with $Z_7$ sampled from $\mathcal{N}(0,1)$).

Repeat the same for $k=8$ and $k=9$, each time using the updated waypoint as input.

Final waypoints $\tau(10)$ reflect the deterministic flow plus three smooth perturbations, producing a slightly curved trajectory.

Restricting noise to the last three steps yields diverse end‑of‑trajectory shapes while keeping early waypoints anchored to the pretrained flow, which stabilizes training.

Why inject stochasticity only into the final three Euler steps instead of the whole trajectory?

Perturbations near $t=1$ have a direct impact on the emitted trajectory because later integration steps cannot overwrite them. Adding noise earlier would be quickly damped by subsequent deterministic updates, offering little diversity while increasing variance in the loss.

**Figure 4.** Four-stage training recipe of Qwen-Drive-1.0. Stages 1 and 2 adapt the shared vision-language pathway, first initializing the BEV perception head and then using perception and VQA supervision to update the vision encoder and VLM. Stages 3 and 4 train the Planning Expert on top of these fixed representations, first by flow matching and then by reward-based optimization. Flames indicate trainable modules, and snowflakes indicate fixed modules.

Data Organization and Alignment

Align heterogeneous perception, language, and planning data before joint training.

Joint training of perception, vision‑language, and planning heads fails unless the heterogeneous task definitions and spatial grids are reconciled. Directly mixing raw datasets would corrupt supervision because class taxonomies and voxel coordinates differ across sources.

The trick is to align labels and voxel coordinates across datasets before mixing them, so a single model can consume all sources without confusing supervision.

Map D₁: car → vehicle, truck → vehicle, bicycle → bicycle, pedestrian → pedestrian.

Map D₂: vehicle → vehicle, bike → bicycle, person → pedestrian, sign → (dropped as source‑specific).

After mapping, both datasets contain only the three shared classes.

During training, loss for “sign” is computed only on D₂ samples because D₁ lacks that label.

Unifying at the coarsest compatible granularity lets us mix datasets without forcing impossible class correspondences, while still preserving source‑specific information for classes that appear in only one source.

**Table (a).** Task-specific unification

**Figure b.** Occupancy label processing

**Figure 6.** Vision-language data and the Stage 2 training mixture. (a) Input-format distribution of the 3.09M filtered public driving samples before Stage 2 subsampling. (b) Composition of the 1.54M Stage 2 training set before repetition. (c) Representative scenes spanning diverse road environments, illumination, and weather conditions.

How does the Data Recipe differ from simply concatenating all raw datasets?

Naïve concatenation would mix incompatible class vocabularies and voxel coordinate systems, causing the model to receive contradictory supervision. The Data Recipe first maps each source to a shared label space and aligns predictions with source‑specific grids, so the loss is computed only on compatible classes and the spatial meaning of each voxel remains consistent.

Heterogeneous data alignment—unifying taxonomies and spatial grids before mixing—is the key to successful multi‑task training.

3D Perception Performance

Qwen-Drive-1.0 sets new 3D perception benchmarks on nuScenes and OpenScene.

We evaluate on the official nuScenes validation set and the 16‑log OpenScene split. For 3D detection we report mean average precision (mAP) and a modified nuScenes detection score (NDS) over seven classes, matching predictions to ground‑truth boxes by BEV center distance. Semantic occupancy is measured by mean IoU (Occ mIoU) and RayIoU, while BEV map segmentation uses mean IoU over foreground map classes.

We compare against reproduced single‑frame variants of BEVFormerV2, its unified multi‑task version BEVFormerV2*, and PETRv2, all using ResNet‑50 or the SigLIP‑Qwen vision encoder at 896 × 512 resolution. Our BEV perception head omits rig‑specific camera embeddings, enabling a single model to run across both the six‑camera nuScenes rig and the eight‑camera OpenScene rig.

Qwen-Drive-1.0‑SFT attains the highest NDS on the nuScenes validation set.

Achieves 34.13 NDS, surpassing all reproduced baselines.

**Figure 7.** Qualitative results of Qwen-Drive-1.0-SFT on the OpenScene (a, b) and nuScenes (c, d) validation splits. Each row shows 3D detection, semantic occupancy, and BEV map segmentation results.

Driving Visual Question Answering

Driving VQA shows large gains of Qwen‑Drive‑1.0‑SFT over prior models.

Qwen‑Drive‑1.0‑SFT attains the highest driving‑VQA average of 69.43, beating the strongest baseline by 5.91 points.

Table 2 shows Qwen‑Drive‑1.0‑SFT at 69.43 versus Qwen3.5‑4B at 63.52 and all other models lower.

The improvement is distributed: LingoQA rises +7.40, SURDS +13.18, and Ego3D distance‑estimation RMSE drops 40.9 % to 7.78, indicating richer physical understanding beyond raw visual features.

**Table 1.** Performance comparison of various models on driving QA, spatial understanding, and causal reasoning benchmarks.

On causal‑reasoning benchmarks Qwen‑Drive‑1.0‑SFT reaches an average of 58.30, a 36.25‑point lead over the second‑best Gemma4‑12B.

Table 2 reports 58.30 for Qwen‑Drive‑1.0‑SFT versus 22.05 for Gemma4‑12B.

General‑purpose VLMs cap at 4.01 on this metric, while InternVL3.5‑8B fails to produce parsable JSON, underscoring the difficulty of causal grounding without targeted supervision.

**Figure 8.** Qualitative comparison of driving VQA capabilities. Green and red highlight correct and incorrect content, respectively. Questions and responses are abridged for space, with complete examples provided in Appendix C.

General Vision-Language Understanding

General VQA results show Qwen‑Drive‑1.0‑SFT retains vision‑language ability while improving spatial reasoning.

Qwen‑Drive‑1.0‑SFT loses less than one point on the knowledge‑reasoning benchmarks compared to Qwen‑3.5‑4B.

Table 3(a) shows averages 66.41 versus 67.40 for the baseline.

The results demonstrate that adapting the model for driving does not sacrifice its broad multimodal reasoning; instead, it strengthens spatial understanding while maintaining competitive planning performance.

**Table 1.** Performance comparison of various models across different benchmarks. (a) General multimodal capabilities and (b) Spatial understanding and grounding.

**Table 1.** Comparison of driving performance on the validation and test splits. The table presents metrics for various methods, including ADE (Average Displacement Error) at 3s and 5s, and RFS (Reasoning-based Driving Score). The table is divided into two parts: (a) Validation split and (b) Test split.

Motion Planning and Reinforcement Learning

We benchmark motion planning across open‑loop, pseudo‑closed‑loop, and closed‑loop settings, achieving top scores.

Recall that Qwen‑Drive‑1.0 unifies 3D perception, vision‑language reasoning, and motion planning by augmenting a pretrained VLM with dedicated BEV Perception Head and Planning Expert.

Qwen‑Drive‑1.0 attains the highest Predictive Driver Model Score (PDMS) on NAVSIM and the highest Rater Feedback Score (RFS) on the WOD‑E2E test split.

PDMS reaches 100.0 (best‑of‑N) and RFS reaches 7.91, surpassing all baselines.

The table presents a comparison of various autonomous driving methods across several performance metrics: RL (Reinforcement Learning), NC (Navigation Compliance), DAC (Driving Action Consistency), EP (Efficiency Performance), TTC (Time-to-Collision), Comf. (Comfort), and PDMS (Planning and Decision-Making Score). The methods are categorized into those without RL and those with RL, with specific variants of Qwen-Drive-1.0 highlighted.

**Table 6.** Pseudo-closed-loop motion planning on NAVSIM v1.1 navtest. We report the Predictive Driver Model Score (PDMS) and its no-collision (NC), drivable area compliance (DAC), ego progress (EP), time-to-collision (TTC), and comfort (Comf.) components. RL indicates reinforcement learning. ‡ denotes best-of-N selection with N = 6, where the candidate with the highest PDMS is chosen for each scene.

**Table 7.** Closed-loop planning on 916 AlpaSim (NVIDIA et al., 2025) scenarios using PAI-AV-NuRec (Wu et al., 2025) version 26.02. Params. denotes all parameters excluding the LLM token embeddings. All comparison methods are reproduced under the same evaluation setting.

**Figure (a).** Open-loop planning on WOD-E2E test split and PhysicalAI-AV

**Figure 10.** Qualitative effect of reinforcement learning on the same NAVSIM left-turn scene. (a) Qwen-Drive-1.0-SFT with reasoning before reinforcement learning. (b) Qwen-Drive-1.0-RL. The prediction is shown in red and the recorded future in green in the camera and BEV views.

**Figure 11.** Ablation of the reinforcement learning data mixture and reward design. (a) Reinforcement learning on NAVSIM alone. (b) Joint reinforcement learning on NAVSIM, WOD-E2E, and PAI-AV. The source-specific reward uses PDMS for NAVSIM, RFS for WOD-E2E, and ADE for PAI-AV. The shared-ADE variant adds a displacement reward to every data source.

**Figure 12.** Effect of PAI-AV planning data scale. The Planning Expert is trained in Stage 3 using only PAI-AV, with the number of training samples increasing from 0.17M to 1.38M. Both 5 s Avg. ADE and Avg. FDE decrease consistently on the standard 644-example split and the leakage-free 700-frame subset.

**Figure 13.** Qualitative perception outputs of Qwen-Drive-1.0-SFT on unseen camera rigs. WOD-E2E is shown in (a, b) and PAI-AV in (c, d). These datasets provide no unified perception ground truth, so all panels show predictions only.

Conclusion and Limitations

We recap the main contributions and outline key limitations and future directions.

Qwen-Drive-1.0 builds on a pretrained Vision‑Language Model (VLM) and augments it with a BEV Perception Head and a Planning Expert, thereby unifying 3D perception, language reasoning, and motion planning in a single architecture.

Across open‑loop, pseudo‑closed‑loop, and closed‑loop evaluations, the system attains highly competitive performance in perception, scene understanding, and planning, while largely preserving the general vision‑language capabilities of the underlying VLM.

Nevertheless, the planning module still struggles with multi‑timescale causal reasoning: it can misinterpret simultaneous causes (e.g., a distant red light versus an imminent pedestrian) and exhibit a 1–2 s decision lag. Moreover, the generated trajectories do not always stay faithful to the textual rationale, suggesting a need for explicit consistency supervision. Future work should therefore incorporate multi‑timescale causal models and tighter alignment between rationale and trajectory, as well as better cross‑task representation sharing through unified input formats and joint optimization.

Reinforcement Learning Reward Definitions

Defines the reward formulas used for reinforcement‑learning stage 4.

This appendix spells out the per‑source reward functions that drive the final reinforcement‑learning stage.

All three sources share the same displacement‑error backbone; NAVSIM and WOD‑E2E add their respective task‑level scores, while PAI‑AV relies exclusively on multi‑horizon displacement terms.

Additional Qualitative Visualizations

Additional visualizations of Qwen-Drive-1.0-SFT capabilities.

Camera‑Based 3D Grounding asks the model to recover metric 3‑D boxes from a single image. In the urban turning scene, Qwen‑Drive‑1.0‑SFT localizes three vehicles and returns their box parameters in the requested JSON format.

Traffic‑Signal Detection requires distinguishing signal type, orientation, and relevance to the current intersection. The model outputs twelve detections, each with a tight 2‑D bounding box and the required attributes.

Roadwork Detection expands the object vocabulary to temporary traffic‑control devices and construction items. Qwen‑Drive‑1.0‑SFT identifies thirteen instances, including work vehicles, tubular markers, and cones.

Road Element Recognition goes beyond localization to interpret traffic‑rule semantics. The model correctly selects the meaning of a single dashed yellow line on a Chinese expressway, while eight baselines answer incorrectly.

Reasoning‑Based Motion Planning predicts future trajectories with a concise rationale. Given multiview temporal observations and a navigation command, the model produces a one‑sentence reasoning and an eight‑point trajectory that closely follows the ground truth.

Detailed Driving VQA Examples

Appendix C provides full prompts, ground‑truth answers, and model outputs for four detailed VQA cases.

C.1 (a) Temporal Understanding and Agent‑State Estimation (LingoQA): The prompt asks for the number of parked vehicles in a video sequence, expecting the answer “Zero” (or “None”). Qwen‑Drive‑1.0‑SFT correctly reports no parked vehicles, while Qwen3.5‑4B and MiMo‑Embodied‑7B incorrectly claim two and three vehicles respectively.

C.2 (b) Causal Reasoning for Planning (PAI‑AV‑CoC): The task provides four temporally spaced frames and ego trajectories, asking for a brief reasoning of the future driving decision. The ground‑truth decision is to stop behind the lead vehicle at the stop sign. Qwen‑Drive‑1.0‑SFT correctly outputs this stop command, whereas Cosmos‑Reason2‑32B and Alpamayo‑1.5‑10B incorrectly predict straight‑ahead motion.

C.3 (c) Cross‑View Spatial Distance Perception (Ego3D‑Bench): The query asks for the distance in meters between a dark‑colored sedan in the back‑right view and a beige sedan in the front view, requiring the answer in a \boxed{} tag. The ground‑truth distance is 22.93 m. Qwen‑Drive‑1.0‑SFT estimates the distance as $\boxed{22}$, while Cosmos‑Reason2‑32B and UniDriveVLA‑8B both output $\boxed{15}$, deviating from the true value.

C.4 (d) Traffic‑Road Recognition (VLADBench): Given five sequential frames from a Chinese road scene, the question asks which ego lane the vehicle occupies in the final frame, selecting from several lane types. The correct answer is “left turn lane”. Qwen‑Drive‑1.0‑SFT correctly identifies the left turn lane, whereas UniDriveVLA‑8B and Alpamayo‑1.5‑10B incorrectly answer “straight lane” and an unrelated code string, respectively.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers