ENEAS: Embedding-guided Neural Ensemble for Adaptive Segmentation

Javier del Pino, Salvador Rodríguez, Alejandro Garabito, Javier Álvarez, Chema Garabito

ENEAS uses a cascaded verification pipeline to filter semantic distractors in video segmentation.

How can we prevent temporal hallucinations and semantic drift in text-promptable segmentation models when tracking objects in video?

Foundation models for segmentation often fail in uncurated video, where they confuse visually similar objects—like statues or paintings—with the actual target, such as a person. ENEAS addresses this by decoupling geometric proposal from semantic verification: it uses a fast embedding filter to discard clear non-matches and invokes a vision-language model (VLM) only to adjudicate ambiguous candidates. On challenging datasets, this approach improves precision from 11.1% to 94.7% compared to state-of-the-art foundation models, effectively eliminating ontological hallucinations.

Paper Primer

The method operates in two modes: instance tracking and semantic discovery. For tracking, it extends the SeC architecture with text-driven initialization, using temporal memory to maintain identity even when the target disappears or undergoes extreme scale changes. For discovery, it cascades three stages: a permissive region proposal, a sigmoid-based embedding filter, and a conditional VLM judge that acts as a semantic referee.

ENEAS achieves near-perfect semantic precision in ontologically ambiguous scenes.

In the Church Statues benchmark, ENEAS-2B reached 94.7% precision compared to 11.1% for SAM 3. A four-fold improvement in F1-score over the state-of-the-art baseline.

The cascaded verification architecture minimizes computational overhead.

By restricting VLM activation to an uncertainty interval, the system maintains high accuracy while reducing latency by 35% compared to a VLM-only approach. VLM activation rates drop significantly in standard scenes, allowing for real-time-adjacent performance.

Why is a VLM judge necessary if visual embedding models are already highly capable?

Visual embeddings rely on structural priors and often fail to distinguish between objects that look alike but are ontologically different, such as a hyper-realistic statue and a living person. The VLM provides a cognitive layer that analyzes global context and material properties to resolve these ambiguities.

Does this method require re-prompting when new instances enter the scene?

No. Unlike propagation-only trackers, ENEAS re-evaluates every frame during semantic discovery, allowing it to detect and segment new instances as they appear without additional user input.

Introduction and Motivation

We expose why current promptable segmentation fails in video and introduce ENEAS to fix it.

Promptable segmentation models excel at producing pixel‑accurate masks from a text query, yet in video they routinely hallucinate masks on background, fragment objects under extreme zoom, and misclassify semantically similar artifacts such as statues or reflections.

When an object leaves the field of view, the model either keeps a stale mask or snaps to a visually similar distractor, creating a phantom “ghost” that never existed.

Relying solely on visual features causes the model to label visually similar but semantically different entities—e.g., a statue or a painting—as the queried object.

The figure illustrates the Eneas framework, which takes various inputs including text prompts, still images, and video clips. The central panel displays the Eneas logo and describes it as a "unified, text-promptable method for instance tracking and semantic discovery." The right panel shows an output example of a segmented scene where a specific penguin is highlighted in blue against a snowy background.

The core failure mode of current promptable segmentation models in video is the combination of temporal hallucination and semantic drift.

Related Work

Survey of promptable segmentation, open‑vocabulary detection, and vision‑language integration.

Promptable segmentation began with the Segment Anything Model (SAM), which turned fixed‑vocabulary masks into point‑, box‑, and mask‑based prompts. Subsequent SAM 2 and SAM 3 added video memory and concept‑level prompting, while a host of set‑prediction transformers (DETR, DINO, Mask2Former, OneFormer) and open‑vocabulary detectors (Grounding DINO, YOLO‑World) expanded the vocabulary to arbitrary text.

Grounded SAM first runs an open‑set detector to locate candidate regions, then feeds those regions to SAM so that the final masks are conditioned on the original text prompt.

Early works such as Faster R‑CNN and Mask R‑CNN used fixed class sets, while set‑prediction transformers (DETR, DINO, Mask2Former, OneFormer) introduced end‑to‑end mask prediction. SAM broke the closed‑set barrier by learning from a billion‑mask corpus and supporting arbitrary prompts; SAM 2 added a streaming memory bank for video, and SAM 3 introduced noun‑phrase‑driven concept segmentation.

Vision‑language pre‑training (CLIP, ALIGN) and grounded distillation introduced open‑vocabulary detection. Methods such as Grounding DINO and YOLO‑World fuse language and vision for real‑time detection, while LSeg, OpenSeg, ODISE, SAN, X‑Decoder, and OpenSeeD align per‑pixel embeddings or decoder representations with textual cues.

Contrastive models (CLIP, ALIGN) and their successors (SigLIP, SigLIP 2, NaFlex, NaViT) improve semantic alignment and localisation. Instruction‑tuned models (LLaVA, BLIP‑2, InternVL, PaliGemma, Qwen‑VL series) can answer visual questions, while smaller judges (Qwen3‑VL) are used in ENEAS to resolve ambiguous embeddings via constrained decoding.

NeRF and 3D Gaussian Splatting assume static scenes; transient objects cause floaters. Extensions (NeRF‑in‑the‑Wild, RobustNeRF, NeRF On‑the‑go, SpotlessSplats, WildGaussians) model transients with per‑image latents or robust losses. Parallel work lifts 2D segmentation and language cues into 3D, but ENEAS removes distractors before reconstruction using explicit masks and semantic definitions.

The ENEAS Framework

Method details the verification loop and tracking pipeline that eliminate hallucinations.

Text‑promptable segmentation often produces spurious masks or loses object identity across frames. ENEAS combats this by inserting an embedding‑guided verification loop that filters proposals against the prompt’s semantic intent.

Florence‑2 turns the natural‑language prompt into a set of image‑region proposals by matching the prompt’s language embedding against visual features extracted from each frame.

How does Florence‑2 differ from earlier grounding models like CLIP‑based region proposal?

Unlike CLIP, which scores whole images, Florence‑2 evaluates dense spatial embeddings, allowing it to localize arbitrary nouns at pixel‑level resolution without a separate detection head.

Think of the loop as a security guard that checks each candidate region’s ID (its embedding) before letting it pass to the mask generator – only confident matches are accepted, ambiguous ones are sent for a deeper interview.

Why not simply raise a single confidence threshold instead of using a verification stage?

A single threshold would either let many false positives through (if low) or discard many true instances (if high). The verification stage lets the system keep high‑confidence matches while still rescuing ambiguous but correct candidates with a more expensive, focused model.

Region $r_1$ exceeds $\tau_{\text{acc}}$ and is immediately accepted.

Region $r_2$ falls below $\tau_{\text{rej}}$ and is discarded.

Region $r_3$ lies in the uncertainty interval $[0.60,0.80)$, so it is sent to semantic verification.

The second‑stage model returns $v(r_3)=1$, so $r_3$ is promoted to the accepted set.

The two‑threshold scheme preserves low latency for obvious cases while allocating compute only to genuinely ambiguous proposals.

Ground the prompt on a reference frame $I_{t_0}$ to obtain the initial region $r_0 = G(I_{t_0}, p)$.

Initialize the SeC tracker with $r_0$ and its high‑level representation.

For each subsequent frame $I_t$, the tracker $T$ uses its memory $H_t$ to propagate the mask: $M_t = T(I_t \mid r_0, H_t)$.

If the target disappears, $T$ outputs an empty mask, preventing drift to look‑alikes.

The final masks are produced by the segmentation operator $S$ applied to the propagated region.

For every frame $I_t$, invoke Florence‑2 to generate candidate regions $R_t = G(I_t, p)$.

Score each candidate with the embedding model, yielding $s(r)$ for $r\in R_t$.

Apply thresholds $\tau_{\text{rej}}$ and $\tau_{\text{acc}}$ to split candidates into accept, discard, or verify groups.

Send the verify group to the semantic verifier $Qwen3$; obtain binary verdicts $v(r)$.

Form the accepted set $A_t = \{r\mid s(r)\ge\tau_{\text{acc}}\}\cup\{r\mid v(r)=1\}$.

Run the segmentation operator $S$ on each $r\in A_t$ to produce masks $M^{(i)}_t = S(I_t, r_i)$.

**FIG. 1** ENEAS decision flow for semantic discovery. For every frame $I_t$ and prompt $p$, region proposal yields candidate regions $R_t$; embedding verification scores each candidate with $s(r)$. Candidates above $\tau_{acc}$ are accepted directly and those below $\tau_{rej}$ are discarded; only the uncertainty interval in between is routed to semantic verification, where a vision-language model returns a verdict $v(r)$. The accepted set $A_t$ is turned into one binary mask per instance by mask generation. Each stage is costlier than the previous one and sees fewer candidates.

Experimental Results Overview

ENEAS outperforms prior methods on the SA‑Co/VEval benchmark.

ENEAS beats SAM 3 on four of the five evaluated metrics, achieving the highest overall score.

Table 2 shows ENEAS leading in HOTA, DetA, AssA, and TETA, while SAM 3 only leads in LocA.

**Fig. 2** Temporal Robustness and Spatial Integrity Comparison. The target is the “blue painting”. Row 1 (Grounded SAM): Suffers from severe drift when the target is occluded or absent, incorrectly segmenting curtains and hair (red crosses). It also fails to capture the full object during close-ups (last column). Row 2 (SAM 3): Improves temporal stability but suffers from spatial fragmentation in the close-up, segmenting internal details instead of the full object instance. Row 3 (ENEAS): correctly handles target absence by outputting no mask and maintains spatial coherence, capturing the entire object even under extreme scale changes.

Design Evolution and Justification

We ablate each design choice to quantify its impact on recall, precision, and latency.

We iteratively removed or altered each component of the pipeline to expose its contribution to recall, precision, and runtime.

Generate a caption for every detected object and feed a single multi‑step prompt that iterates over numbered bounding boxes.

Why does the exhaustive captioning pipeline incur such high latency?

It forces the model to process a long multi‑step prompt for every frame, which the vision‑language model can only execute in roughly 15 s, making it unsuitable for real‑time video.

Apply a global softmax over the target text and a set of negative classes to filter region proposals.

How does softmax normalization suppress the target score?

Because all class probabilities must sum to 1, the presence of highly salient negatives steals probability mass, driving the target’s score below a usable threshold.

Treat each image‑text pair as an independent binary classification using a pairwise sigmoid loss, decoupling scores from other concepts.

Why does sigmoid‑based matching avoid the score suppression seen with softmax?

Each class is evaluated separately, so no probability mass is shared; the target’s confidence remains high regardless of other objects.

Isolate the target region with black masking and replace deep reasoning with a structured analytic prompt that enforces critical‑thinking constraints.

Why does disabling deep reasoning and using a structured prompt cut latency?

The model no longer generates a sequence of reasoning tokens; it performs a single fixed‑format check, dropping runtime from seconds to ~1 s.

Optimized design reduces processing time by ≈93 % compared to the exhaustive baseline.

Latency drops from 15 s to 1 s per frame.

Semantic Discovery in Clutter

ENEAS continuously discovers and segments new instances without semantic drift.

Recall that ENEAS adds an embedding‑guided verification loop to curb temporal hallucination and enforce semantic intent.

ENEAS outperforms SAM 3 on three of five semantic discovery metrics, demonstrating superior robustness to clutter.

Table 3 reports higher HOTA, AssA, and TETA scores for ENEAS compared with SAM 3.

**Figure 3.** Continuous Discovery and Robustness to Clutter. Comparison of Grounded SAM versus ENEAS in multi-instance discovery. Top Rows (Prompt: “real person”): Grounded SAM hallucinates masks on the background curtains and fails to maintain consistent IDs. ENEAS correctly segments only the valid subjects throughout the sequence. Bottom Rows (Prompt: “chair”): In a highly cluttered moving scene, Grounded SAM suffers from catastrophic over-segmentation by labeling boxes and tables as chairs. ENEAS exhibits superior semantic filtering, exclusively segmenting the target objects while ignoring significant geometric clutter.

Semantic Rigor and SAM 3 Comparison

ENEAS achieves near‑perfect semantic purity while boosting overall F1.

Promptable segmentation often confuses visual similarity with true identity; ENEAS adds a verification loop that checks whether a region truly matches the semantic intent.

SAM 3 is a large‑scale foundation model that predicts masks for any textual prompt, relying primarily on visual cues.

Ontological filtering discards proposals that do not belong to the target concept’s real‑world category, regardless of visual similarity.

ENEAS‑4B reaches 97.5 % precision and 87.6 % F1 on the Church Statues benchmark, a four‑fold F1 improvement over SAM 3.

Table 1 shows the full precision/recall/F1 breakdown for SAM 3, ENEAS‑2B, and ENEAS‑4B.

**Fig. 4** Semantic Ambiguity Challenge I – Artistic Representations. The prompt is “real person”. Row 1 (SAM 3): Lacks ontological filtering, repeatedly segmenting the large pop-art painting as a person due to visual similarity. Row 2 (ENEAS): leverages VLM verification to correctly identify the painting as a non-living representation, focusing only on real human subjects. Note that both models struggle in the extreme close-up (last column) due to the lack of surrounding context, but ENEAS minimizes false positives across the full sequence.

Ablation Study: Component Necessity

We quantify how each ENEAS component affects precision, recall, and F1 on the Church Statues testbed.

Church Statues provide dense ontological ambiguity—hyper‑realistic sculptures appear alongside real humans—making them a stricter probe of semantic reasoning than COCO, LVIS, DAVIS, or OVIS.

Component‑necessity ablations compare five configurations ranging from a pure RPN baseline to the full ENEAS‑4B model.

ENEAS‑4B delivers the highest overall F1 while restoring baseline recall.

Table 4 shows ENEAS‑4B achieving 87.6 % F1 versus 18.6 % for the RPN baseline.

Threshold‑sensitivity ablations on the 2B model reveal a “Robust Mode” (0.10–0.90) that maximizes semantic purity with a 35 % latency reduction compared to a VLM‑only configuration.

In less ambiguous scenes the same adaptive thresholds can be relaxed, yielding higher throughput without sacrificing the strict artifact‑rejection needed for Church Statues.

**Tab. 4** Impact of Semantic Verification Modules on Church Statues

Ablation Study: Efficiency and Latency

We quantify how each component’s removal reshapes speed, activation, and accuracy.

We now probe two orthogonal axes: the uncertainty interval that governs VLM activation, and the capacity of the VLM itself. Each ablation reports the resulting latency, activation rate, and F1‑score.

Fast‑mode on Moving Boxes runs three times faster than the robust configuration on Church Statues.

Latency drops from $3.29\,$s (robust) to $1.14\,$s (fast‑mode), a $3\times$ speedup.

Upgrading from the 2B to the 4B VLM raises F1 by $4.8$ points while incurring a $52\%$ latency increase.

Table 6 shows F1 $82.8\% arrow 87.6\%$ and latency $3.29\,$s $arrow 5.02\,$s.

Hardware and Evaluation Setup

Describes the test hardware and evaluates ENEAS robustness to occlusion and target loss.

The experiments run on a single NVIDIA L4 GPU (Ada Lovelace, 24 GB VRAM) paired with an Intel Xeon CPU offering 12 virtual cores at 2.20 GHz.

This cloud inference setup mimics a cost‑effective production environment, and all reported latency numbers cover the entire pipeline, including data loading and CPU‑bound preprocessing such as normalization and resizing.

We assess ENEAS’s ability to handle instance disappearance, occlusion, and scale changes, using the Blue Painting video where the target object undergoes severe viewpoint shifts and partial occlusions.

Baseline trackers that rely on per‑frame detection, such as Grounded SAM, tend to drift in identity and hallucinate masks on background elements when the target vanishes or is heavily occluded.

By propagating temporal memory through the SeC architecture, ENEAS enforces identity constraints and correctly emits a True Negative (zero mask) whenever the specified instance is not visible, preventing false‑positive accumulation.

When the object fills the frame in extreme close‑ups, prior models fragment the segmentation into high‑frequency details, whereas ENEAS maintains the semantic integrity of the whole “painting” and produces a complete mask.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers