VGCap: Grounding What Matters

TL;DR News articles mention many entities, but only some are visible in the image. VG-Cap selectively identifies and conditions on visually grounded entities, keeping irrelevant context out of the caption. It achieves state-of-the-art performance on the OpenEvents-V1 dataset.

Overall Score

0.6219

+0.0122 vs ENRIC

CIDEr

0.265

+0.008 vs ENRIC

CLIPScore

0.858

+0.025 vs ENRIC

Entities per caption

7.59 → 6.76

same 104-word length

OpenEvents-V1 public test · same retrieval backbone as ENRIC

Abstract

News image captioning requires models to connect visual evidence with external article context to identify depicted people, locations, events, and their broader significance. Existing retrieval-augmented methods typically inject all named entities extracted from the retrieved article into the captioning pipeline. However, news articles often mention secondary actors and background events that are not depicted, leading to unsupported descriptions and inefficient use of the caption budget. We introduce VG-Cap, a retrieval-augmented news captioning framework that selectively incorporates entities according to their estimated visual groundedness. Our Entity Heuristic Scorer ranks candidate entities using their presence across visual and textual context, position within the article, and semantic type. A unified gating mechanism applies the same entity selection to both prompt-time entity conditioning and post-generation enrichment, preventing rejected entities from being introduced through these explicit entity pathways. We further introduce Two-Stage Length Control mechanism that combines bounded generation with entity-preserving semantic normalization. Experiments on the OpenEvents-V1 dataset demonstrate that VG-Cap consistently outperforms the strongest baselines. The results further show that selective entity grounding improves the relevance and semantic alignment of generated news captions by reducing unnecessary contextual entities.

Contributions

VG-Cap framework

A retrieval-augmented framework for event-enriched captioning that makes the use of external knowledge selective and visually grounded.

Entity Heuristic Scorer

Scores article entities by contextual presence, document position and semantic type. One selection gates both prompt injection and post-generation enrichment.

Two-Stage Length Control

Bounded generation followed by entity-preserving semantic normalization reduces the length mismatch without dropping key event information.

Method

Two stages: article retrieval (fixed, from ENRIC), then context-grounded caption generation. Click a step to see it in the figure.

Overview of VG-Cap: article retrieval and re-ranking, then context construction, entity heuristic scorer, caption generation and two-stage length control.

1 · Article Retrieval & Re-ranking

VG-Cap reuses ENRIC's retrieval backbone, unchanged. InternVL-14B encodes the query image and the corpus (>400K images), and the top-K (K = 15) candidates are kept by cosine similarity.

An uncertainty detector checks three signals: the entropy of the top-K distribution, the gap between the top-1 and top-2 scores, and the top-1 similarity. For uncertain queries only, caption-based text similarity is blended in:

\[ s_{\mathrm{rerank}}(q,x_i) = \lambda\, s_{\mathrm{img}}(q,x_i) + (1-\lambda)\, s_{\mathrm{text}}(q,x_i) \]

The highest-scoring candidate gives the retrieved article. Retrieval is identical to ENRIC in every experiment.

2 · Context Construction

The article covers much more than the depicted scene, so VG-Cap organizes the available information into three groups:

\[ \mathcal{C} = \{\, \mathcal{C}_{\mathrm{article}},\ \mathcal{C}_{\mathrm{image}},\ \mathcal{C}_{\mathrm{entity}} \,\} \]

Article context: a summary plus question–answer pairs (who, where, when, why, how).
Image context: a VLM-generated description of the query image plus the editorial caption of the retrieved image.
Entity context: named entities from the article. This is the only group filtered by the scorer.

3 · Entity Heuristic Scorer

Candidates come from NER over the article plus LLM keyword lists, merged by substring-aware deduplication. Each candidate gets a groundedness score:

\[ S(e) = \max\!\Big(0,\ \sum_{f\in\mathcal{F}} w_f\,\phi_f(e)\Big) \]

\[ \mathcal{E}^{*} = \operatorname{TopK}_{K=10}\big(\{\, e\in\mathcal{E} : S(e)\geq 3.0 \,\}\big) \]

4 · Two-Stage Length Control

Length is regulated twice: the prompt bounds generation to a range, then a normalizer brings the caption to the target length L* = 104 words.

\[ \hat{y} = G_\theta\big(q, \mathcal{C}, \mathcal{E}^{*};\ L_{\min}, L_{\max}\big), \qquad y = \mathcal{N}\big(\hat{y};\ L^{*}\big) \]

5 · Unified Entity Conditioning

The same selected set \(\mathcal{E}^{*}\) is passed to both entity pathways:

\[ \hat{y} = G_\theta\big(q,\ \mathcal{C}_{\mathrm{article}},\ \mathcal{C}_{\mathrm{image}},\ \mathcal{E}^{*}\big), \qquad y = \mathcal{N}\big(\hat{y},\ \mathcal{E}^{*}\big) \]

An entity rejected by the scorer cannot come back through post-generation enrichment.
An entity selected as grounded stays available during normalization.

Qualitative Examples

Captions with all article entities (Unfiltered · 0.6166) compared with VG-Cap (VG-Cap · 0.6219). Both use the same retrieval, backbone and 104-word target. Hover an entity chip to find it in the captions.

Results

OpenEvents-V1 public test. Overall Score is the weighted harmonic mean of the retrieval and captioning metrics.

Comparison with state-of-the-art

VG-Cap: 0.6219 (rank 1 of 7, +0.0122 vs ENRIC). Best Overall Score, with the gain coming entirely from the captioning stage.

* Overall Score computed from the official evaluation formula (not reported by the original authors).

Ablation: building up VG-Cap

Every component adds to the Overall Score. The selector's gain comes with no change in length, so it reflects what is said, not how much.

BibTeX

@inproceedings{phan2027vgcap,
  title     = {Grounding What Matters: Selective Entity Conditioning
               for Event-Enriched News Image Captioning},
  author    = {Phan, Van-Phuc and Pham, Minh-Tan and Tran, Mai-Khiem and Do, Trong-Le and
               Nguyen, Hai-Dang and Nguyen, Tam V. and Tran, Minh-Triet and Le, Trung-Nghia},
  booktitle = {SOON},
  year      = {2027}
}