Evaluating Multimodal LLMs as Generalist Vision-Language-Action Agents for Drone Control: Commanding, Approaching, Tracking and Searching

Jaewoo Park, Minyoung Lee, Sukmin Seo, Moonbin Yim, Hyunwook Yoon, Dohoon Ryu, Daehee Kim, Myungseo Song, Jihyuk Byun, Seunggyu Chang, Taeho Kil, Jiseob Kim, Bado Lee, Geewook Kim

DroneCATS-Agent evaluates MLLMs as autonomous drone pilots, revealing that small models fail at protocol adherence, not navigation.

How do off-the-shelf Multimodal LLMs perform when tasked with direct, zero-shot drone control using only visual input and text-based action prompts?

Existing drone control systems offload decision-making to external detectors or hard-coded thresholds, masking whether the underlying model actually understands its mission. DroneCATS-Agent treats the Multimodal Large Language Model (MLLM) as a swappable pilot that must independently decide when to search, deliberate, and declare arrival. Evaluations show that while small open models navigate to targets as reliably as frontier models, they fail by declaring arrival prematurely or not at all, exposing a critical gap in protocol discipline.

Paper Primer

The agent uses a pointing interface where the model emits one of four primitives: **go** (pixel and depth), **rotate** (yaw), **think** (deliberation), or **finished** (arrival declaration). This architecture is model-agnostic, requiring no fine-tuning or function-calling schemas, and relies on a geometric controller to translate the model's high-level choices into flight setpoints.

Small models possess sufficient spatial perception for navigation but lack the protocol discipline to sustain a mission.

Qwen3.5-9B enters the success radius in 90% of episodes—outperforming frontier models—yet succeeds in only 35% of them due to erratic arrival declarations. The gap between navigation success and mission success is largest at the 2B–9B parameter scale, where models frequently declare arrival at 1.28x the start distance or never declare at all.

Multi-drone commanding amplifies model-specific failures, with smaller models failing to distinguish between distinct camera views.

In a 4-drone fleet setting, Qwen3.5-9B emits identical coordinates for all four views in 70% of cases, whereas frontier models like Gemini 3.7 Flash maintain distinct control streams. Gemini 3.7 Flash achieves an 80% success rate in commanding tasks, while GPT-5 drops to 20% and smaller open models fail entirely.

Why does this paper move the "arrival" decision inside the model?

Prior systems use external distance thresholds to terminate episodes, which hides whether the model actually knows it has succeeded. By requiring the model to declare arrival, the authors can measure if an agent is capable of self-reporting completion, a prerequisite for delegating tasks to autonomous fleets.

Does test-time deliberation (the "think" action) improve performance?

No. Across 1,797 invocations, the "think" action provided no consistent performance gain over standard steps, suggesting that current test-time compute strategies are poorly suited for real-time physical control loops where latency is a collision risk.

Introduction: MLLMs as Drone Agents

We expose why current MLLMs fall short as drone controllers and define the gap to close.

Multimodal Large Language Models (MLLMs) have become strong perceivers of images and video. Yet perception alone does not close the loop: a drone must choose its next physical action from observations, and that action determines the next observation. Existing systems (e.g., TypeFly, PIVOT, SPF, Fly0, OnFly) increasingly constrain the model’s decision‑making through fine‑tuning or function‑calling schemas, limiting what the model can actually control.

Embedding the whole action space directly in the prompt lets an off‑the‑shelf MLLM act as a drone controller without any task‑specific fine‑tuning or external function calls.

The experimental gap emerges when small open‑source models reach the target region more often than frontier models but then either declare arrival too early or never declare it at all. This shows that navigation accuracy is not the bottleneck; the protocol discipline encoded in the prompt is.

The shift is from treating MLLMs as pure perceivers to viewing them as active controllers that must follow a disciplined action protocol.

Related Work

We survey prior MLLM‑driven drone systems, benchmarks, and multi‑drone efforts.

Modern drone control research increasingly leverages off‑the‑shelf multimodal LLMs, avoiding task‑specific fine‑tuning. This section situates our work among those systems, the benchmarks they use, and the sparse literature on multi‑drone agents.

Type‑Fly treats the MLLM as a high‑level planner that emits symbolic detections, while a separate low‑level controller executes the actual flight commands.

A general visual‑prompting scheme that reduces control to selecting among image‑derived candidate actions; later adopted as a baseline in drone systems.

Recasts control as visual grounding; the MLLM emits 2‑D waypoints that are lifted to 3‑D commands.

Emits a grounding tuple (point, region, relation token, confidence) at ~0.5 Hz, delegating trajectory generation to an Ego‑Planner at 50 Hz.

Separates goal generation from progress monitoring and verifies model proposals before execution, raising success from 26.4 % to 67.8 %.

Benchmarks combinations of a text LLM with a vision‑language model on search‑and‑approach tasks; the best pairing reaches 40 % success.

Scores six sub‑skills across thirteen MLLMs; items are multiple‑choice and dynamic scenarios are graded step‑by‑step by human raters.

Focuses on tracking performance for UAVs; provides dedicated metrics for sustained target following.

Vision‑language‑action model that learns the action space within its weights from demonstrations.

Benchmarks search capabilities for UAVs; evaluates the ability to locate targets from aerial imagery.

Evaluates aerial search and rescue scenarios, emphasizing robustness to occlusions and dynamic environments.

Instruction‑following benchmark for aerial navigation; includes a closed‑loop evaluation protocol.

Vision‑language‑action model with the action space baked into its weights, trained from demonstrations.

Chains question answering, observation planning, and fine‑grained control on a real quadrotor; uses a hierarchical task decomposition.

Embodied reasoning benchmark from egocentric flight video; actions are multiple‑choice items.

Introduces a dual‑altitude cooperative VLN task for two drones, focusing on coordinated navigation.

Evaluates 40 MLLMs on 14,610 multiple‑choice items built from synchronized multi‑view images; no model issues actions.

Runs a closed loop over multiple UAVs, using one LLM per drone that reasons over structured local state.

**Table 1.** Comparison of existing benchmarks and DroneCATS.

The DroneCATS Agent Framework

Describes how a prompt‑only MLLM drives a drone via four high‑level actions.

The agent wraps any off‑the‑shelf MLLM in a prompt that defines a tiny action space, letting the model output a JSON command while all low‑level flight logic stays unchanged.

How does this differ from typical function‑calling interfaces?

Function‑calling requires the model to invoke a predefined API with typed arguments; here the model simply emits a free‑form JSON describing a high‑level action, and the controller interprets it without any schema enforcement, making the interface model‑agnostic.

The entire drone command set is encoded as natural‑language descriptions inside the prompt, so the model “points” to a pixel and optionally supplies a depth or yaw value to select an action.

Why is the predicted depth treated as a step‑size rather than a calibrated metric?

The model predicts depth from a single RGB frame without camera intrinsics, so the value is only reliable as a relative advance; the controller uses it as a proposal for closed‑loop motion rather than as an absolute distance measurement.

Receive the egocentric RGB frame, the natural‑language instruction, and the last five JSON actions.

Parse the model’s JSON output to identify which of the four primitives was selected.

If the action is

The flight controller’s standard cascade tracks the setpoint continuously, converting it into attitude and rotor commands.

DroneCATS‑Agent loop – pseudo‑code for one control step.

**Figure 2.** The DroneCATS-Agent loop. The action space exists only in the prompt, so a model plugs in with no fine-tuning and no function-calling schema, and swapping it changes nothing else. Beyond pointing, the model decides when to look around, when to spend more computation, and when it is done. Below the chosen action, a rule-based controller translates it into a setpoint — geometry solved once per step — and the flight stack’s cascade tracks that setpoint continuously, so nothing between the JSON object and the rotors is model-specific (Section 3.2).

We evaluate five off‑the‑shelf MLLMs as candidates for the DroneCATS‑Agent: GPT‑5, Claude, Gemini, Qwen 3.5, and Cosmos3‑Edge.

The DroneCATS Benchmark

The DroneCATS Benchmark defines tasks, success criteria, and evaluation settings for drone control experiments.

The benchmark fixes the DroneCATS‑Agent across all models and varies only explicit scene variables, scoring every episode with a single unified arrival‑declaration rule.

How does this benchmark differ from typical drone benchmarks that change evaluation metrics per task?

Most benchmarks vary the success threshold (e.g., distance or dwell time) for each task, making cross‑task comparison ambiguous; DroneCATS keeps the metric constant and varies only the scene, so all four cells are judged by the same arrival‑declaration rule.

The axis enumerates four concrete drone tasks: Approaching a static visible target, Tracking a moving visible target, Searching for a hidden static target, and Search‑and‑Track for a hidden moving target.

Why is the “search‑and‑track” task treated as a distinct cell rather than just a combination of search and tracking?

Because the drone must first locate a hidden moving target (search) and then maintain visual contact (track); the benchmark records this compounded difficulty as its own cell to avoid conflating the two separate challenges.

**Figure 3** The DroneCATS task axis. Target motion and initial visibility combine into four task types. In each pictogram the wedge is the drone’s field of view, the filled dot is a target inside it, the dashed circle outside the wedge is a target whose location is unknown, and the arrow marks a moving target.

**Algorithm 1** The DroneCATS success criterion, used unchanged for all four task types. **Require:** arrival declarations $D$, each a triple $(t, d, v)$ of time, distance to the target and visibility, the last two measured against ground truth at $t$; radius $\delta$; episode cap $T_{max}$ 1: $D \leftarrow$ entries of $D$ with $t \leq T_{max}$, ordered by $t$ $\triangleright$ a declaration logged during shutdown does not count 2: **for** $(t, d, v) \in D$ **do** 3: $\quad$ **if** $d \leq \delta$ and $v$ **then** 4: $\quad \quad$ **return** Success, $NE \leftarrow d$ $\quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \triangleright$ the first qualifying declaration decides 5: $\quad$ **end if** 6: **end for** 7: **return** Fail, $NE \leftarrow d$ at the last action $\quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \triangleright$ reported where the episode ended, declared or not

Organisational settings split the benchmark into a one‑drone condition, where a single MLLM controls one platform, and an N‑drone commanding condition (N = 4) that requires the model to emit a separate command for each drone in a single response.

Episodes are built in AirSim on two Unreal Engine maps (residential and campus). The one‑drone suite runs ten episodes per map per task type (80 episodes total), and the commanding suite adds 20 episodes, yielding 100 episodes overall. Metrics reported include Success Rate (SR), Oracle Success Rate, the nearest‑error (NE), minimum distance, approach ratio, and the ratio of declared to start distance.

Experimental Results and Failure Analysis

Key performance shifts and failure analyses for Action Space Prompting.

Embedding the full action space in the prompt lets off‑the‑shelf MLLMs control drones zero‑shot, as introduced earlier.

Action Space Prompting raises the approaching success rate from 40 % to 65 %.

Table 3 shows Gemini 3.7 Flash achieving 65 % SR, while the same model drops to 40 % when the target is hidden from the first frame.

Episodes can fail for six distinct reasons: never grounding the referent, grounding without closing distance, oscillating without progress, arriving without declaring, declaring prematurely, or exhausting the step budget.

**Figure 1.** Approaching episode example. The model is given the instruction and the egocentric frame and returns one action per step; the circle marks the point it emitted, which stays on the target as the target grows in view, and the fourth panel is an external view of the drone holding position after it declared arrival. The band below is the same flight from above, with the $\delta$ ring on the target and the declaration marked, so the strip and the criterion can be read against each other.

The dominant failure mode is premature or missing declaration rather than navigation error.

Table 3 and Table 4 reveal large OSR–SR gaps; Figure 14 shows small open models with up to a 90 % declaration gap.

**Figure 13** Per-model failure-mode distribution over the 80 one-drone episodes: success and the five diagnosed failure modes of Section 5.5. The timed-out band includes the did-not-move episodes (a simulator fault, see text), so the bars partition all 80. Frontier failures concentrate in mistimed declarations; the small open models spread across every mode.

**Figure 6.** The same episode flown twice — nh approaching, "go to the cyclist" — Gemini 3.7 Flash left, Qwen3.5-2B right, on one plate at one scale. Both start 16.1 m out, fly the same line in, and cross into $\delta$ within a step of each other ($\times$). Only the ending differs: the frontier model declares and holds 2.8 m short, while the 2B model closes to 2.53 m, declares nothing and drives through to the timeout. The ring is a horizontal projection of a three-dimensional criterion, so stretches of the higher-flying 2B trace sit inside it but outside $\delta$.

**Figure 7** Approaching, decomposed. Reaching within $\delta$, declaring arrival, and succeeding, over the 20 episodes of that cell, so every bar is a multiple of 5 points. Frontier models lose episodes because they do not arrive; the small open models decouple the three quantities in both directions.

**Figure 9.** One `think` step in two episodes. Left, the frame that triggered it; centre, what the next, thinking-on step returned, quoted from the log; right, a later frame with the model's emitted point and where the episode ended. Above, a `think` that helped — Qwen3.5-9B, the roster's best case, not a typical one: lost and drifting out to 15 m, the model reads the scene correctly, turns round, re-acquires the car and reaches 2.0 m. Below, five spent in a row and wasted — Qwen3.5-4B, 5.0 m from the quadrotor it was told to follow: four of the five thinking-on replies are the one line `{"action":"think"}`, the fifth a 90° turn, after which it aims past the target at the tree line (6.6 m ahead; the chosen pixel is background at 217 m) and never declares.

In the commanding suite (Section 5.3), four drones share a single context and must disambiguate a target among several candidates; Table 5 shows that Gemini 3.7 Flash retains high success (80 % SR) while small open models either fail to reach or declare.

Variance analysis (Section 5.4) confirms that the observed spread in Table 3 matches binomial sampling noise (≈ 10 % points), and throughput sensitivity experiments reveal that slower control loops disproportionately hurt moving‑target tasks.

Test‑time deliberation (Section 5.6) introduces a “think” action; across 1,797 invocations it yields negligible performance change, with only a handful of episodes showing a clear benefit.

Appendix: Implementation Details

Appendix provides episode specifications, controller constants, prompt details, metrics, audits, and extra results.

Episodes are authored in a browser‑based JSON editor, recording target assets, paints, instructions, poses, start yaw, distance, bearing, and visibility flags. Two maps are used: a residential map with colour‑named targets placed 16 m from the start, and a campus map with generic fixtures requiring disambiguation; each map contributes ten episodes per task cell for a total of eighty episodes.

The commanding episodes add multiple drones and a list of candidate targets (four per episode) that differ only by inscriptions. All candidates are visible from the start poses, and the nearest pair is 5.5–11.2 m apart while the grid centre lies 12–32 m from the named target. The runner respawns candidates, measures ground height, and verifies each spawn before the episode begins.

Back‑projection uses a pin‑hole camera with horizontal field of view $\phi = 90^\circ$ and resolution $W \times H = 1280 \times 720$. A model point $(u,v)$ normalized to $0\!-\!1000$ maps to pixel $(p_x,p_y) = (uW/1000, vH/1000)$, then to body‑frame displacement $dx = p_x - W/2$, $dy = d$, $dz = H/2 - p_y$ (right, forward, up). The controller issues at most one turn (if horizontal bearing exceeds $8^\circ$) and one move, clamping the remaining displacement to $3\,$m and holding the velocity for $2.5\,$s. Depth‑dependent bands scale the step to $0.6$ under $5\,$m and to zero under $2\,$m, causing the drone to brake.

Rotation commands are rate‑based at $60^\circ\!/$s for a duration of $|\theta|/60$ seconds; the prompt allows up to $90^\circ$ per step but $215$ of $6401$ rotations exceed this, mostly from a single model. The simulator is AirSim 1.8.1 with the SimpleFlight controller; the forward camera is offset $0.5\,$m forward and $0.1\,$m down from the body origin.

All models run zero‑shot with an $8192$‑token output cap, temperature $0.4$, and for Gemini models top‑p $0.95$ and top‑k $40$. Extended reasoning is disabled by default and only enabled for the step following a “think” action.

The system prompt (Listing 1) defines the action space and is prepended to each user message; three fields are substituted each step: {instruction}, {`point_fmt`} (coordinate order), and {history} (last five actions). History lines are rendered as compact JSON‑like strings, e.g., “GO x=612 y=430 depth=9m”.

For OpenAI‑compatible endpoints the request JSON contains a text part $P_t$ and a base64‑encoded JPEG of the egocentric image $I_t$; for the native Gemini endpoint the same pair is passed as a text item followed by an inline JPEG part. Images are resized only if width exceeds $640$ px and JPEG‑encoded at quality $85$.

Metrics include distance to target centre and to the nearest point on its 3D bounding box, visibility (view‑frustum and occlusion test), navigation error (distance at decisive measurement), and approach ratio (fraction of start distance closed). Per‑drone records are pooled for episode verdicts, and the same settings apply across the suite.

Offline verifier audit re‑derived every verdict from raw logs; alternative scoring rules (first‑declaration only, dwell‑window, trajectory proximity, episode cap) would have altered many successes and changed the paper’s conclusions.

Figure 10 visualizes the four distinct failure modes—never grounded, grounded without closing distance, oscillation, and timeout—each with first‑person and top‑down views, highlighting how agent behavior diverges from successful navigation.

Figure 11 contrasts a successful division‑of‑labour episode (Gemini 3.7 Flash) where each drone receives a distinct target point with a failed coordinate‑copying episode (Qwen 3.5‑4B) where the same point is sent to all drones, illustrating why copying leads to failure.

Figure 12 extends the comparison by showing full‑episode trajectories for both fleets, emphasizing that the copy‑error prevents any drone from reaching the target while the coordinated approach succeeds.

Figure 13 presents detailed flight paths for the two fleets, marking the successful coordinated approach of Gemini 3.7 Flash versus the repeated mis‑targeting of Qwen 3.5‑4B.

Figure 4 displays a successful episode for each task type on a common map, aligning egocentric frames with top‑down traces to illustrate the model’s navigation behavior.

Figure 5 compares two failure modes—late declaration without arrival and early declaration without meeting the success radius—showing how the arrival test can be the bottleneck.

Figure 8 aggregates drone‑outcome percentages across models, with diamonds indicating episode success rates; Gemini 3.7 Flash achieves the highest episode success.

Table 7 lists the benchmark parameters—pose trace, visibility, detection radius, declaration pose, and success radius—along with their concrete definitions used throughout the evaluation.

Additional results break down one‑drone successes by map, showing that approaching is harder on the campus map while tracking benefits from the slower campus target; failure mode counts are also reported.

The figure illustrates four failure modes in a navigation task: "Never grounded the referent," "Grounded, never closed distance," "Oscillated without progress," and "Ran out of steps." Each mode is represented by a pair of images: a first-person view showing the current step and target, and a top-down map view showing the agent's path and the target's 5m radius threshold.

**Figure b.** Qwen3.5-4B copies one answer to all four drones — same point, same depth

**Figure 11.** One step of the commanding setting. Each row shows the four views exactly as the model received them, with the emitted go points ringed. (a) Division of labour (Gemini 3.7 Flash, successful episode): the four points land on four different look-alikes, so one response splits the candidates. (b) Coordinate copying (Qwen3.5-4B, failed episode): one response gives every drone the same pixel and the same depth. The views share nothing — the point is a distant car in one view and a house wall in another — so at most one of the four commands can be grounded. Copying at this exactness is how the small open models command: Qwen3.5-9B emits an identical point for all four drones in 70% of its all-go steps and 27B in 58%, while the frontier models sit at 0–1%. Both rows are mid-episode steps on the residential map.

**Figure 12** The same commanding episode flown by two fleets — residential map, "Find the blue car with license plate 4017", four candidates, identical start grid. *Top*: the four onboard views at an early step, with the commanded go point ringed and printed. Gemini 3.7 Flash assigns one car per drone (step 3); Qwen3.5-27B emits the same point and depth four times — (448, 357), 12 m — once into each of four different views (step 1). *Bottom*: the flights, north running left to right, one colour per drone, traces lightly smoothed for print; dots mark starts, squares where each drone ended, the green star the accepted declaration, the red cross the failed episode's final one. Panel (a) is drawn up to its accepted declaration; panel (b), which never succeeds, up to the 300-second timeout. (a) The fleet works down the row of look-alikes together while one drone breaks off to the target; the verifier accepts its declaration at 77 s. (b) The fleet inspects the three look-alikes — passing within a metre of two of them — but no drone ever comes within 8 m of the target; three of the four end piled beside the look-alike with plate 3218, and the fleet declares 114 times, every every one rejected by the distance test.

**Figure 4** One successful episode per task type, same map and same model throughout, so only the scene differs. Three egocentric frames, cropped toward the point the model emitted, sit over a top-down trace on an aerial photograph of the ground flown. The wedge is the camera's field of view in the first frame, so the target sits inside it on the two panels whose task starts with it visible and outside it on the two that do not; its radius is a symbol, not a distance. Where the model turned before setting off, a thin arc carries how far. The two rows run at 28.7 and 86 m across, so each panel carries a scale bar, and the $\delta = 5$ m ring is at true scale throughout.

**Figure 5** The two ways a declaration goes wrong, one episode each, both on nh and both starting 16.1 m out. (a) Qwen3.5-2B is 4.4 m out and inside $\delta$ at step 7, emits go, and never declares in 34 steps. (b) Gemini Robotics-ER 2 declares at step 3 of 87, 11.4 m out with the car dead centre, and never enters $\delta$ at all: what fails is the arrival test, not perception. Each strip ends on the marked frame, and the two windows differ, so each trace carries its own scale bar.

**Figure 8.** Commanding, decomposed over drone-outcomes rather than episodes: the three categories of Figure 7, over the eighty drone-outcomes of each model (twenty episodes × four drones). Diamonds mark the episode success rate, which the stacked bar cannot express, since one drone’s qualifying declaration settles the episode: Gemini 3.7 Flash wins 80% of episodes out of 36% of drone-outcomes.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers