A Glance Is All You Need: Single-Pass Fine-Grained Image Captioning with SimLoss

Suryaansh Jain, Rahasya Barkur, Vishal G, Ryan Rossi, Franck Dernoncourt, Jack Wang, Koustava Goswami, Nedim Lipka, Puneet Mathur, Samyadeep Basu, Seunghyun Yoon

SimLoss aligns VLM hidden states with frozen image embeddings to enable single-pass, fine-grained captioning.

How can we train vision-language models to generate fine-grained, attribute-rich captions in a single pass without relying on expensive multi-stage sampling or reward-based reinforcement learning?

Vision-language models often produce fluent but generic captions, omitting the specific attributes, textures, and spatial relations that define an image. Multi-stage pipelines can recover these details, but they impose significant latency costs by requiring repeated verification and rewriting steps. SimLoss addresses this by aligning the model's internal hidden-state representation with a frozen image embedding during training. This contrastive objective forces the captioner to retain discriminative visual information before any text is generated, without requiring human-written fine-grained targets. SimLoss FFT achieves the highest precision of any evaluated method while nearly matching the F1 score of multi-stage pipelines, all while running roughly 20× faster.

Paper Primer

SimLoss treats fine-grained captioning as a representation-alignment task: it uses a frozen image encoder to provide a dense visual signal, training the VLM to map its internal hidden states into the same embedding space. The core move is a contrastive InfoNCE loss that forces the model to distinguish its source image from in-batch alternatives, effectively "compressing" visual evidence into the model's internal state before decoding begins.

SimLoss FFT achieves the highest precision among all evaluated methods.

Evaluation on the IIW-400 benchmark shows SimLoss FFT reaching 0.8485 precision, outperforming both multi-stage pipelines and reward-optimized baselines. It achieves this precision while maintaining single-pass inference, running roughly 20× faster than the CapMAS multi-stage pipeline.

Embedding-space supervision recovers the F1 performance of multi-stage verification.

SimLoss FFT reaches an F1 score of 0.7023, nearly identical to the 0.7025 score of the CapMAS multi-stage pipeline. The performance gap is negligible (0.0002), effectively closing the quality divide between single-pass and multi-pass systems.

Why is this approach preferred over simply training on more detailed human-written captions?

Fine-grained human captions are scarce and often contain stylistic artifacts or unverifiable claims. SimLoss avoids the need for these targets entirely by using the image itself as the supervision signal, preventing the model from inheriting the omissions or hallucinations present in human-written or pipeline-generated text.

What is the difference between the FFT and GRPO variants?

SimLoss FFT is fully differentiable, allowing gradients to flow directly from the embedding alignment into the model's adapters. SimLoss GRPO is a reward-based variant designed for black-box settings where the embedding model is inaccessible, which improves recall but is less precise than the differentiable FFT approach.

Fine-grained captioning does not require inference-time pipelines; embedding-space supervision can bake visual grounding directly into a single-pass model, enabling high-precision description at a fraction of the latency.

Introduction

Vision‑language models are fluent but miss fine‑grained visual details.

Modern vision‑language models (VLMs) excel at producing fluent, high‑level captions, yet they routinely omit the fine‑grained visual attributes—counts, textures, materials, and spatial relations—that make an image truly specific.

Captions that enumerate concrete visual details—how many objects, what textures, which materials, and where things are relative to each other—rather than only naming the most salient objects.

Recent multi‑stage systems recover some of these missing details by generating, decomposing, verifying, and rewriting captions, but each additional stage multiplies inference latency, making them impractical for interactive or large‑scale deployment.

SimLoss addresses this gap with a reference‑free embedding‑space objective: during training a frozen image encoder produces a dense visual embedding, while the VLM’s hidden‑state representation is projected into the same space and aligned via an InfoNCE contrastive loss, providing dense supervision before any text is decoded.

We instantiate SimLoss in two forms: SimLoss FFT, which back‑propagates through a locally available embedding model for fully differentiable training, and SimLoss GRPO, which treats the embedding model as a black‑box reward and optimizes via policy gradients.

The core trade‑off is between caption fluency and visual detail: SimLoss shifts the balance toward richer detail without sacrificing single‑pass speed.

Related Work

We survey prior work on fine‑grained captioning and similarity‑based supervision.

Image captioning progressed from encoder–decoder models with visual attention to instruction‑tuned vision‑language models, yet fine‑grained captioning remains challenging.

Methods that explicitly prompt models to describe attributes, relations, and scene structure beyond salient objects.

Decomposes a generated caption into atomic claims, verifies each claim against visual evidence, and rewrites the caption using only supported content.

Uses external detection and visual question‑answering tools to correct captions after generation.

Aggregates patch‑level visual features to recover local detail lost in global attention.

Reference‑free caption evaluation that measures image‑text similarity using a pretrained CLIP model.

Uses CLIP similarity as a reward signal during training to encourage distinctive, fine‑grained captions.

Self‑retrieval objectives that retrieve similar images to guide caption generation, but naive optimization may reduce faithfulness.

Techniques that detect or prevent object‑level hallucinations in generated captions.

Reward‑optimized and perception‑aware captioners that incorporate learned feedback to improve detailed caption quality.

SimLoss shares the similarity‑supervision intuition of these works but aligns the captioner’s continuous representation with a frozen image embedding before decoding, using an InfoNCE contrastive loss rather than post‑hoc scoring.

The Fine-Grained Captioning Gap

Fine-grained captioning demands detailed visual grounding, exposing a gap between training captions and evaluation needs.

Typical captions such as “a cat sitting on a table near a lamp” name the dominant objects but omit discriminative details like the cat’s markings or the lamp’s ceramic base. Fluent vision‑language models readily drop these fine‑grained attributes, leaving the caption under‑specified.

A caption that records visible attributes, counts, textures, materials, object parts, and spatial relations, while avoiding unfounded details.

Standard supervision from MS COCO provides five human captions per image, averaging only 10.0 words and rarely mentioning attributes or textures. In contrast, the ImageInWords (IIW) dataset offers hyper‑detailed descriptions averaging 171.2 words—about seventeen times longer. CapMAS pipeline captions, used only for evaluation, average 186.0 words, contain 29.3 atomic propositions of which 22.3 are true, yielding a factuality ratio of 0.766.

We evaluate captions with a dual protocol: precision measures the fraction of atomic propositions verified by a multimodal judge (factuality), while recall measures the proportion of image‑derived multiple‑choice questions answered correctly from the caption alone (coverage). Their harmonic mean is reported as F1. Additionally, we report CLAIR, a reference‑based LLM metric normalized to $[0,1]$, as well as caption length and latency.

The ideal objective is to maximize mutual information $I(v;\hat{y})$, equivalently minimizing the residual uncertainty $H(v \mid \hat{y})$. Conventional cross‑entropy loss $L_{\text{CE}}$ aligns the model with a single reference caption $y^{*}$, which is itself a lossy projection of the image; thus low $L_{\text{CE}}$ does not guarantee preservation of fine‑grained details.

SimLoss bridges this supervision gap by discarding the COCO captions and instead aligning the VLM’s hidden representation $z_{\text{vlm}} = W\,h_{\theta}(v, x)$ with the frozen image embedding $z_{\text{img}} = E_I(v)$ via a contrastive objective. This representation‑level proxy encourages the model to retain enough information to retrieve the correct image among alternatives, thereby enforcing dense visual grounding before decoding.

**Figure 1.** Example of a human-written fine-grained caption. Fine-grained captioning requires visually specific details. A generic caption may identify the main objects while omitting attributes, materials, counts, textures, and spatial relationships that distinguish the image.

SimLoss: Embedding-Space Distillation

Method introduces SimLoss, a differentiable embedding alignment that trains VLMs to retain fine‑grained visual detail.

Standard caption‑level supervision either relies on human‑written captions or on high‑variance policy‑gradient rewards, both of which struggle to preserve fine‑grained visual attributes. SimLoss replaces those brittle signals with a continuous alignment between the model’s internal representation and a frozen image embedding.

SimLoss forces the captioning model to keep dense visual information by matching its pooled hidden state to a frozen image embedding, rather than learning from noisy text targets.

Project $h_1$ and $h_2$ (identity) to obtain $z_1^{V}=h_1$, $z_2^{V}=h_2$.

Compute cosine similarities: $s_{11}=1.0$, $s_{22}=1.0$, $s_{12}=0.0$, $s_{21}=0.0$ (orthogonal embeddings).

With temperature $\tau=0.1$, the softmax numerator for each positive is $\exp(1.0/0.1)=\exp(10)\approx 2.2\times10^{4}$.

The denominator for each example is $\exp(10)+\exp(0)=2.2\times10^{4}+1\approx2.2\times10^{4}$, yielding a loss $\approx -\log(1)=0$.

If $z_1^{V}$ were slightly misaligned, e.g., $z_1^{V}=[0.7,0.3,0.0,0.0]$, then $s_{12}=0.2$, the denominator grows, and the loss becomes positive, driving the model to correct the misalignment.

Even tiny deviations from the frozen target quickly inflate the loss because the softmax is sharply peaked at low temperature, forcing the VLM to preserve fine‑grained visual cues.

How does SimLoss differ from standard contrastive learning that also uses InfoNCE?

Standard contrastive learning treats both sides as learnable embeddings; SimLoss keeps the image encoder frozen and only adapts the VLM’s hidden state and a lightweight projector, so the training pressure directly targets visual fidelity rather than learning a symmetric embedding space.

FFT refines SimLoss by restricting gradient flow to the LoRA adapters and the projector, leaving the massive base model untouched while still receiving a fully differentiable training signal.

Why not fine‑tune the entire VLM instead of using LoRA adapters?

Full‑model fine‑tuning would require updating billions of parameters, dramatically increasing compute and risking catastrophic forgetting of the pretrained knowledge; LoRA confines updates to a tiny low‑rank matrix, preserving the original capabilities while still allowing the SimLoss signal to reshape the visual representation.

When the embedding model is inaccessible for back‑propagation, SimLoss can be turned into a reward‑based variant (GRPO) that treats the cosine similarity between a frozen image embedding and a generated caption embedding as a scalar reward.

FeedQuill PPO serves as a baseline that optimizes a composite reward combining unit‑level F1, global CLIP alignment, and CIDEr similarity, demonstrating that external caption‑quality metrics can improve fine‑grained generation without direct representation alignment.

**Figure 2.** SimLoss fully differentiable fine-tuning (FFT). The frozen encoder produces image targets $z_i^I$, while Qwen2.5-VL's token states are mean-pooled and projected to $z_j^V$. Cosine scores for all batch pairs are separated explicitly into matched identities ($s_{ii}$, green) and other identities ($s_{ij}$ for $j \neq i$, red) before entering InfoNCE. The loss updates only LoRA and the projector, without caption targets. At inference, the encoder, projector, and contrastive loss are removed.

Experimental Results

Key experimental results comparing SimLoss variants to baselines on fine‑grained captioning.

The central premise—that standard vision‑language models sacrifice visual detail for fluency—has been restated, and now we assess how SimLoss restores fine‑grained grounding.

CLAIR (Captioning Language‑Image Alignment and Recall) measures how tightly a generated caption aligns with visual attributes, rewarding precise visual grounding.

SimLoss FFT matches CapMAS’s F1 score within 0.0002 while using a single‑pass inference.

Table 3 shows CapMAS F1 = 0.7025 and SimLoss FFT F1 = 0.7023.

**Table 3.** Results on IIW-400. SimLoss FFT has the highest precision and nearly matches multi-stage CapMAS in F1 with single-pass inference; SimLoss GRPO has the highest CLAIR and recall. Length is mean $\pm$ standard deviation in words.

**Table 1.** Cross-source embedding similarity on IIW-400. Entries are mean matched-pair cosine similarities in a frozen Qwen3-VL-Embedding space.

**Table 2.** Cross-source embedding similarity on MS COCO. Entries are mean matched-pair cosine similarities in a frozen Qwen3-VL-Embedding space.

Mechanism Analysis

Key ablations reveal why SimLoss variants excel and where alternatives fall short.

This ablation suite isolates each training signal and inference‑time component to measure its impact on fine‑grained captioning quality.

SimLoss GRPO attains the highest recall ( $0.6015$ ) among all methods.

Recall is measured with the CLAIR metric on the test split; all other variants fall below $0.55$.

Because the GRPO reward is applied after discrete token generation, the model can inflate visual coverage by adding plausible details, which dilutes precision.

CapMAS yields a marginal F1 advantage of $+0.0002$ over SimLoss FFT ( $0.7025$ vs $0.7023$ ).

F1 scores are computed on the CLAIR benchmark; the difference is statistically insignificant.

The multi‑stage verification pipeline of CapMAS incurs a latency gap that makes SimLoss FFT a more attractive quality‑latency trade‑off.

SimLoss FFT produces the shortest average captions ( $114$ words), far below the plain baseline ( $347$ words) and CapMAS ( $189$ words).

Mean caption length is reported across the evaluation set; standard deviation is also lowest for SimLoss FFT.

Shorter captions indicate that SimLoss FFT compresses the necessary visual information into fewer tokens, akin to lossless compression of an image description.

Overall, the ablations show that direct dense alignment (SimLoss FFT) avoids the fragile LLM‑judge pipeline, achieves superior precision and near‑optimal F1, and does so with markedly shorter, more efficient captions.

Qualitative Analysis

Qualitative caption analysis shows how SimLoss captures scene hierarchy and fine details.

SimLoss variants answer nine more questions than Plain VLM on Figure 6 (32 vs. 23).

Qualitative analysis shows richer scene and ground‑level detail captured by SimLoss.

Across all evaluated figures, SimLoss consistently produces more spatially organized and detail‑rich captions, leading to higher coverage and precision than the baselines.

**SimLoss FFT, 122 words.** "This image captures a serene and majestic mountain landscape. In the foreground, there is a wooden bench placed on a grassy area, inviting viewers to sit and enjoy the view. The bench is simple in design, with a backrest and seat made of natural wood, blending harmoniously with the natural surroundings.

**SimLoss FFT, 108 words.** The image shows a train platform with a series of car carriers loaded with luxury cars. The train is labeled “MILSPED AML,” which likely stands for a company that specializes in the transportation of vehicles by rail. The cars are neatly arranged on the carriers, with some covered in protective plastic to prevent damage during transit. The platform has a modern design with a checkered pattern on the edge, and the sky above is clear with a few scattered clouds. The train is stationary, and the platform appears to be empty except for the cars. The overall scene suggests a professional and organized process for transporting high-value vehicles.

**SimLoss FFT, 132 words.** "This image captures a serene forest scene during what appears to be the spring season, as indicated by the abundance of white flowers blooming on the trees. The foreground features a large tree with a thick trunk and numerous branches, heavily laden with white blossoms. The branches extend outward, creating a dense canopy of flowers that dominate the view. The ground is covered with a mix of green grass and fallen leaves, suggesting a transition between seasons.

**SimLoss FFT, 133 words.** The image shows a section of a tiled wall with two mosaic designs of hats. The tiles above the hats feature a number “23” in a mosaic pattern, with blue and white tiles forming the digits. Below the number, there is a row of smaller, multicolored tiles that create a border around the number.

`aar_test_04629`

Prompt Configurations

Provides the full set of captioning, evaluation, and reward prompts used in the experiments.

This appendix enumerates every prompt employed in the study, organized into four groups: captioning prompts for the vision‑language model, evaluation prompts for precision/recall/CLAIR, inference‑pipeline prompts for the CapMAS baseline, and reward/judge prompts for reward‑optimized baselines.

The single‑pass captioning prompt, used across all methods, is simply “Describe this image in detail.”

CapMAS generates five diverse captions per image by issuing five detailed‑description prompts, each phrased slightly differently to encourage varied outputs.

Evaluation of factuality uses a GPT‑4o judge that extracts atomic propositions $G(\hat y)$ from a candidate caption $\hat y$ and checks each against the image $v$ and reference description $r$.

Recall measures how many of the ground‑truth questions $Q(v)$ can be answered correctly from the generated caption $\hat y$.

The harmonic mean $F1 = \frac{2PR}{P+R}$ combines precision $P$ and recall $R$, while the CLAIR metric scores overall caption‑image alignment on a 0–100 scale.

Figure 7 illustrates a decaying log example used to highlight how SimLoss FFT can over‑specify details that are plausible but unverifiable.

**SimLoss FFT, 113 words.** "The image shows a close-up view of a decaying log. The log is covered with a layer of pine needles and small pine cones, indicating it is part of a forest floor. The wood appears weathered and broken, with a dark, hollow center that suggests it has been hollowed out by decay or possibly by an animal. The texture of the wood is rough and uneven, with cracks and splits visible throughout. The pine needles and cones are scattered around the log, adding to the natural debris on the forest floor. The overall appearance of the log gives a sense of age and decomposition, typical of a fallen tree in a forest environment."

**SimLoss FFT, 112 words.** The image depicts a mannequin dressed in medieval armor and attire. The mannequin is wearing a chainmail hauberk, which is a type of armor made of interlinked metal rings. Over the hauberk, there is a red surcoat, which is a loose-fitting garment often worn over armor for protection and visibility. The surcoat is secured with a white sash tied around the waist. The mannequin also wears chainmail gauntlets on the arms and chainmail boots that cover the legs up to the knees. A helmet with a visor is worn on the head, providing head protection. The mannequin is holding a sword in its right hand, which is sheathed at the hilt. In

**SimLoss FFT, 130 words.** "The image shows three cars parked on the side of a street in a residential area. The car on the left is a red Porsche, which appears to be an older model with a classic design. The middle car is a white SUV, which looks more modern and is parked behind the Porsche. The car on the right is a silver vintage car, possibly a classic Porsche as well, given its shape and design. The cars are parked in front of a house with a light-colored exterior and a tree in the background. The street is lined with trees and bushes, and there is a fence partially visible on the left side of the image. The overall scene suggests a quiet neighborhood with a mix of classic and modern vehicles."

**SimLoss FFT, 121 words.** “The image depicts a large, intricately carved wooden sculpture of a deer. The sculpture is highly detailed, with the antlers and body of the deer crafted from wood that has been carefully shaped to mimic the natural form of the animal. The antlers are particularly elaborate, with multiple branches and points, giving the sculpture a dynamic and lifelike appearance. The deer's head is turned slightly to the side, and its mouth appears to be open, as if it is mid-roar or mid-breathe.

The image shows a wooden sign mounted on a rustic wooden post. The sign reads "All weapons must be peace tied...." and features a painted illustration of a dagger. Below the sign is a wooden box or holder.

Table 3 reports that SimLoss FFT yields the shortest captions while attaining the highest precision, indicating that brevity does not sacrifice factual accuracy.

Example (a) Plain VLM produces a 340‑word caption that repeats information and adds speculative details not grounded in the image.

Example (b) CapMAS generates a 133‑word caption that infers the purpose of a wooden box without visual evidence.

Example (c) SimLoss FFT delivers a 123‑word caption that covers the same questions with less redundancy and achieves 100 % measured precision.

The final prompt asks the model to decompose a given caption into atomic propositions, one per line, to enable precision evaluation.

A long‑form Plain VLM caption example follows, illustrating the type of output that the proposition‑verification prompt will process.

Prompt Examples

This appendix lists the prompt templates that drive caption evaluation and refinement.

CapMAS presents a detailed visual description of a circular stone carving: a textured, light‑colored surface bearing two intertwined, stylized human‑like figures with simplified facial features, decorative crown‑like patterns, and a muted gray background that emphasizes the carving’s lines.

SimLoss FFT describes a similar circular motif etched into a textured surface, highlighting two intertwined faces with exaggerated eyes, swirling patterns around the eyes, and a sense of movement reminiscent of ancient tribal art.

The first prompt asks the model to output a list of “True/False” answers matching the number of propositions, using the exact format “1. True/False” on separate lines.

The recall prompt instructs the model to answer IIW‑400 multiple‑choice questions using only the candidate caption, replying “I don’t know” when the caption lacks sufficient information.

The CLAIR prompt requests a 0–100 score (with a JSON “score” field) indicating how likely a candidate caption set describes the same image as a reference set, accompanied by a brief reason.

D.3 outlines the CapMAS inference‑pipeline prompts: Stage 2 merges multiple captions, Stage 3 decomposes the merged caption into atomic propositions, Stage 4 fact‑checks each proposition via a VLM, and Stage 5 rewrites the caption using only verified facts.

D.4 describes the reward and judge prompts: FeedQuill breaks captions into atomic units for precision/recall scoring, while PAPO relies on a perception‑consistency loss; SimLoss FFT uses a frozen‑encoder embedding‑alignment objective, and SimLoss GRPO employs a black‑box embedding similarity reward.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers