ReferTrack: Referring Then Tracking for Embodied Visual Tracking

Hanjing Ye, Tianle Zeng, Jiazhao Zhang, Shaoan Wang, Zibo Zhang, Weisi Situ, Yuchen Zhou, Yonggen Ling, Hong Zhang

ReferTrack improves embodied visual tracking by grounding target identification in indexed image-space bounding boxes rather than abstract latents.

How can an embodied agent reliably track a specific target in a crowded scene using only natural language instructions and a single forward-facing camera?

Embodied visual tracking agents often struggle to maintain focus in crowded scenes because their reasoning occurs in abstract spatial latents that are difficult to supervise and weakly aligned with visual evidence. ReferTrack solves this by framing identification as a selection task: the model evaluates an indexed catalog of detected bounding boxes to pick the target, then injects the selected target's geometric history into the visual stream via temporal-viewpoint-bbox indicator (TVBI) tokens. On the EVT-Bench suite, this approach achieves state-of-the-art single-view performance, outperforming multi-camera baselines on identification-heavy tasks without requiring reinforcement learning.

Paper Primer

The core mechanism is a two-stage reasoning process: first, the model selects a target from a discrete catalog of detected bounding boxes using a single Refer-CoT token; second, it uses this selection to condition future motion planning by injecting the target's historical coordinates into the visual history. This "referring-then-tracking" paradigm forces the model to ground its reasoning in explicit image-space detections, effectively closing the loop between identification and navigation.

ReferTrack achieves state-of-the-art performance on the EVT-Bench single-view tracking splits.

The model scores 89.4% SR on single-target, 73.3% on distracted, and 74.1% on ambiguity tracking, surpassing previous single-view methods and matching multi-camera baselines. It provides absolute gains of +22.9% SR on the ambiguity tracking split compared to the strongest single-view baseline, TrackVLA++.

Why is the "indexed catalog" approach more effective than previous abstract reasoning methods?

By treating identification as a multiple-choice selection over detected bounding boxes, the model aligns its reasoning with the grounding mechanisms of vision-language models, making the process easier to supervise and less prone to the errors inherent in abstract latent-space reasoning.

Does this method require reinforcement learning to achieve these results?

No. ReferTrack relies entirely on supervised fine-tuning (SFT) using a 1:1 mix of navigation trajectories and a custom Refer-QA dataset, demonstrating that explicit referring capabilities can compensate for the lack of costly RL-based policy optimization.

The Challenge of Embodied Tracking

We expose why conflating target identification with tracking hampers embodied visual tracking.

A mobile robot must continuously follow a specific person described in natural language, using only its forward‑facing camera.

Current vision‑language‑action (VLA) policies merge identification and planning into a single chain‑of‑thought that operates in abstract spatial latents. This entanglement makes supervision hard and often misaligns the reasoning with concrete image‑space detections, leading to failures in crowded environments. ReferTrack addresses the gap by first selecting a target from an indexed set of bounding boxes (the Refer‑CoT step) and then feeding the chosen box through a sliding‑window queue of TVBI tokens before predicting waypoints, thereby decoupling identification from tracking.

**Figure 1.** **ReferTrack** formulates embodied visual tracking as *referring then tracking*: forward-view detections are organized as indexed bboxes, and a single Refer-CoT token selects the instructed pedestrian before trajectory prediction. This facilitates image-grounded reasoning within a unified, end-to-end policy.

Decoupling target identification from trajectory prediction yields more reliable embodied visual tracking.

Prior Approaches and Field Context

We situate ReferTrack among vision-language navigation, tracking, and grounding literature, and outline its high-level architecture.

Related work spans three strands: vision‑language navigation, embodied visual tracking, and referring‑grounding for catalog‑based identification. Each line of prior art informs a component of ReferTrack, from TVI‑style temporal tokens to indexed‑bbox selection via a Refer‑CoT token. Below we summarize the most relevant systems.

Agents navigate to language‑specified goals by processing visual observations and textual instructions. Recent work moves from prompting off‑the‑shelf LLMs with scene descriptions to end‑to‑end fine‑tuning of vision‑language models on navigation trajectories.

Mobile agents must follow a dynamic target described in natural language using only onboard vision. Early pipelines separate perception from control, while recent Vision‑Language‑Action (VLA) models fuse language, vision, and action into a single policy.

Referring expression comprehension (REC) localizes objects from free‑form language. Open‑vocabulary detectors and grounding models bridge language to image regions, and multimodal LLMs can output or select bounding boxes.

EVT‑Bench is a benchmark suite that evaluates embodied visual tracking across diverse environments, target descriptions, and motion patterns.

Method overview formalizes the task: at each timestep $T$, given an instruction $L$ and a forward‑view observation stream $O_{1:T}$, the agent predicts a trajectory $W_T$ of waypoints $w_i = (x, y, \theta) \in \mathbb{R}^3$. ReferTrack encodes visual history $E^{V}_{1:T}$, augments it with TVBI tokens, builds an indexed candidate catalog $C_T$ of detected bboxes, and concatenates these with language embeddings $E^{L}$ before feeding them to a large language model that first emits a Refer‑CoT token $E^{\text{refer}}_T$ and then produces action embeddings $E^{A}_T$, which are decoded by an ActionHead into $W_T$.

The ReferTrack Architecture

How ReferTrack isolates target selection from motion planning using a dedicated Refer‑CoT token.

Embodied visual tracking fails when the agent mixes target identification with trajectory prediction, leading to brittle behavior. ReferTrack solves this by inserting a dedicated Refer‑CoT token that first selects the target before any motion planning occurs.

The visual stream is processed by two complementary pretrained encoders—SigLIP for global semantics and DINOv2 for fine‑grained texture—whose features are concatenated to give a richer representation.

How does this differ from simply stacking two identical encoders?

Stacking identical encoders would duplicate the same feature space, offering no new information. Here the two encoders are trained on different pretext tasks (image‑text vs. self‑supervised vision), so their outputs complement each other—one provides semantic context, the other supplies fine‑grained structure.

Instead of reasoning over a long textual chain, the model emits a single token that explicitly indexes the chosen target from a catalog of detections, making the selection step compact and directly supervised.

The LLM evaluates each candidate token $\langle\text{ped}_k\rangle$ against the instruction and scores them.

It selects $\langle\text{ped}_1\rangle$ because $b_1$ has the smallest $x$‑coordinate, matching “left”.

The chosen box $b_1$ is stored as $E_{\text{refer}}$ and fed to the trajectory head.

The Refer‑CoT token collapses a potentially ambiguous multi‑candidate reasoning problem into a single, deterministic index, which the planner can rely on without further disambiguation.

Why not let the LLM generate a free‑form description of the target instead of an indexed token?

A free‑form description would be noisy and hard to map back to a concrete bounding box, especially under tight latency constraints. The indexed token guarantees a one‑to‑one correspondence with a detected bbox, enabling precise geometric conditioning for downstream planning.

Run the dual‑encoder (SigLIP + DINOv2) on the current frame to obtain concatenated visual features.

Apply grid‑pooling to produce $V_{\text{fine}} \in \mathbb{R}^{64 \times C}$ (fine‑grained) and $V_{\text{coarse}} \in \mathbb{R}^{4 \times C}$ (coarse‑grained) tokens.

Project the token sequence with a two‑layer MLP $P_{\text{vision}}$ into the LLM latent space, yielding $E_{1:T}=P_{\text{vision}}(V_T)$.

Insert TVBI tokens between frame groups: for each timestamp $t$, compute $E_{\text{TVBI}}(t)=E_{\text{TVI}}(t)+P_{\text{bbox}}(b_t)$.

Maintain a sliding window of the latest $H$ frames; the current frame’s fine tokens receive only TVI cues, while historical frames receive TVBI cues containing the queued target boxes.

The model is given a fixed‑size list of indexed pedestrian detections plus a sentinel entry, so it can reason over a bounded set of concrete targets.

After the Refer‑CoT token selects a target, its bounding box is stored in a FIFO queue that supplies geometric context to future frames via TVBI tokens.

Table 1 (see the paper) reports the quantitative gains of ReferTrack on the EVT‑Bench suite, confirming that the decoupled Refer‑CoT design yields state‑of‑the‑art success rates while keeping collision rates low.

**Figure 2.** Overview of ReferTrack. ReferTrack first grounds the language instruction and visual stream by selecting one indexed bbox from the current detections, then predicts tracking waypoints conditioned on this Refer-CoT decision. The selected bbox is stored as target-specific memory and injected into future visual history through TVBI tokens.

The core trick is a single Refer‑CoT token that cleanly separates target identification from trajectory planning, enabling robust, memory‑driven tracking.

Training and Supervision

Training ReferTrack combines navigation and referring data with a two‑stage supervised fine‑tuning pipeline.

Training ReferTrack involves two complementary datasets and a two‑stage supervised fine‑tuning pipeline.

ReferTrack is trained on a total of 2.6 M samples, equally split between navigation and Refer‑QA data.

Navigation data provides 1.3 M expert trajectories; Refer‑QA data supplies 1.3 M synthetic referring examples.

Prior EVT methods typically train on only navigation data, whereas ReferTrack’s equal‑weight Refer‑QA component supplies explicit visual grounding supervision.

Cross‑entropy supervision of the Refer‑CoT target index is the key driver of accurate target selection.

Ablation and Component Analysis

Refer‑CoT isolates target selection, so we test how each component affects performance.

Recall that ReferTrack decouples target identification from motion planning by inserting a Refer‑CoT token that explicitly selects the target before predicting its trajectory.

DT describes scenarios where the target is occluded or other agents distract the tracker, making pure motion‑prediction unreliable.

**Table 2.** Ablation on EVT-Bench DT. Single forward-view setting. †Oracle TVBI uses ground-truth target bboxes without Refer-CoT. Parentheses denote absolute changes w.r.t. ReferTrack (YOLO-X).

Using perfect target bounding boxes (TVBI w/ GT bbox) raises success rate by +8.2 points to 81.5 %.

Table 2 shows SR = 81.5 (+8.2) versus the baseline 73.3 %.

Removing both Refer‑CoT and TVBI drops success rate by ‑17.6 points to 55.7 %.

Table 2 reports SR = 55.7 (‑17.6) for the “w/o Refer‑CoT & TVBI” variant.

Omitting TVBI alone reduces success rate by ‑2.9 points to 70.4 %.

Table 2 lists SR = 70.4 (‑2.9) for the “w/o TVBI” variant.

The large gain of the oracle TVBI variant over the full model, together with the modest gap to the expert policy, indicates that target identification—not motion planning—is the dominant failure mode in distracted settings.

When TVBI is removed but Refer‑CoT remains, success and tracking rates fall only slightly, confirming that the Refer‑CoT token already provides a strong identification signal; the additional temporal bbox memory further stabilizes tracking.

Implementation and Training Details

Details on data curation, refer‑QA generation, and training hyper‑parameters.

We generate expert trajectories using a custom oracle controller that accesses the Habitat simulator state. At each step it queries the geodesic shortest‑path planner, densifies the path, and selects a local waypoint that keeps the robot at a preferred 1.2 m following distance, switching to a backward goal when the robot gets too close. The resulting base velocities (vx, 0, $\omega$) are recorded together with robot and target poses, yielding 330 K samples for Single‑Target Tracking, 330 K for Ambiguity Tracking, and all 640 K Distracted Tracking samples, for a total of 1.3 M navigation samples.

Refer‑QA data are built on the SYNTH‑PEDES dataset: each image is cropped to 384 × 384, 2–3 pedestrians are pasted at random scales (0.75–1.5), and each receives a catalog ID from 0–19 with its bbox and caption; a negative caption with an all‑zero bbox is also added. During training the candidates are shuffled into the indexed catalog used for navigation, with a fixed ⟨NO EXIST⟩ slot appended. The instruction prompt asks the model to “find ⟨caption⟩ in the video” and to output the corresponding bbox‑index token or ⟨NO EXIST⟩.

Training follows a two‑stage SFT schedule. Stage 1 freezes the language and vision encoders while training the vision projector for one epoch on general QA data with a learning rate of $1 \times 10^{-4}$. Stage 2 jointly trains navigation and Refer‑QA for 20 K steps (global batch size 256) using AdamW, a cosine decay schedule, and a linear warm‑up; the LLM learning rate is $2 \times 10^{-5}$ and all modules except the frozen vision encoders are updated.

Inference runs on a remote GPU server that receives JPEG‑compressed RGB frames from the robot’s forward‑facing RealSense D455 camera together with the language instruction. The server decodes each frame, updates the detector/tracker, builds the indexed bbox catalog, and returns the predicted trajectory and selected target slot. To keep latency low, only the latest pending frame is kept while older frames are dropped.

We evaluate ReferTrack on two robot platforms: the Unitree Go2 quadruped and the Unitree G1 humanoid, both equipped with a single forward‑facing RealSense D455 camera and a portable Wi‑Fi module. The ReferTrack model itself runs on a high‑performance remote server, while the robots stream observations and receive trajectory commands.

**Figure 6.** Robot platforms. The ReferTrack is deployed on a remote high-performance server. Both robot platforms are equipped with a single forward-facing camera (Intel RealSense D455 camera) and a portable Wi-Fi for communication.

During deployment, feature extraction is parallelized: DINO and SigLIP features for the current frame are computed in separate Python threads on distinct CUDA streams and synchronized before concatenation. The concatenated features are grid‑pooled into 64 fine tokens for the current frame and 4 coarse tokens for history; coarse tokens are enqueued in a fixed‑length visual‑history queue, while the Refer‑CoT‑selected bbox is stored in a target‑bbox history queue and injected only into future TVBI tokens. A pure‑pursuit controller on the robot converts the predicted trajectory into linear and angular velocity commands.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers