VNCultureVQA Benchmarking VQA for Vietnamese Cultural Events

Van-Phuc Phan1,2† Minh-Tan Pham1,2† Trung-Nghia Le1,2‡
1University of Science, Ho Chi Minh City, Vietnam 2Vietnam National University, Ho Chi Minh City, Vietnam

†Equal contribution  ·  ‡Corresponding author: ltnghia@fit.hcmus.edu.vn

arXivSoon Hugging Face

Representative images from VNCultureVQA — festivals, traditional arts, heritage sites and community life.

28,655
cultural articles
107,947
image–article pairs
691,830
QA pairs
6
cognitive levels
8
Vietnamese news outlets

Overview

Abstract

Visual Question Answering (VQA) in low-resource languages remains largely disconnected from the cultural and contextual knowledge required to interpret real-world imagery. Existing Vietnamese VQA benchmarks primarily treat images as isolated inputs, limiting evaluation to visual recognition rather than knowledge-grounded multimodal reasoning. We introduce VNCultureVQA, a large-scale benchmark for Vietnamese cultural understanding that jointly grounds images and questions in real-world cultural articles, linking each image to its source article, human-authored caption, and contextual knowledge. VNCultureVQA contains 107,947 image–article pairs and 691,830 question–answer pairs spanning six cognitive levels, from visual recognition to global contextual synthesis.

The benchmark supports knowledge-grounded generative VQA under progressively enriched contextual settings, enabling evaluation of visual understanding and culturally grounded multimodal reasoning. We further establish three progressively enriched evaluation settings, ranging from image-only VQA to article-grounded VQA and supervised domain adaptation. Experiments with representative Vietnamese and multilingual vision-language models show consistent gains from contextual grounding, while consensus and lexical scores degrade steadily as reasoning demands increase and supervised domain adaptation yields uneven improvements, revealing persistent limitations in current multimodal models.

Dataset

Construction Pipeline

Each article is modelled as a unified multimodal knowledge source: full body text, embedded images and human-authored captions are kept together, so every question is grounded in the narrative its image came from.

Five-stage pipeline: web crawl, caption generation with Qwen3-VL-8B, question generation and answer generation with Gemini 3 Pro, and verification with Llama 3 8B.
Multi-stage construction pipeline. Articles crawled from eight Vietnamese news platforms are captioned with Qwen3-VL-8B, turned into questions and article-grounded answers with Gemini 3 Pro, and verified against the source article with Llama 3 8B.

Six-level cognitive taxonomy

Questions are organised into three meta-groups. Purely visual questions are deliberately capped at 8.2% to mitigate visual shortcuts and language priors; 92.0% of questions require multimodal integration or higher-level reasoning.

Visual-only questions are first detected with keyword and part-of-speech analysis and validated zero-shot with Qwen3-VL-8B. The remaining questions are scored with Chain-of-Thought prompting, and the final difficulty score is a weighted ensemble in which the higher Gemini weight reflects its stronger reasoning on Levels 5–6:

$$S_{\text{final}} = 0.7 \cdot S_{\text{Gem}} + 0.3 \cdot S_{\text{Qwen}}$$

Statistics

Dataset at a Glance

To our knowledge, the largest Vietnamese VQA dataset grounded in real-world news articles. Splits are stratified at the image level (5 : 2 : 2 : 1), and Test 1 → Test 3 increase in reasoning difficulty.

Overall statistics of VNCultureVQA per official split.

Images per news outlet

107,947 images from eight professionally edited Vietnamese platforms.

Image semantic categories

Share of images (%) assigned during caption generation.

Events/Ceremonies dominate (43.2%), reflecting the news-centric source material; Traditional Arts and Culture add a further 33.7%, spanning tangible monuments, intangible traditions and daily community life.

Difficulty level distribution and composition per split.
Difficulty across splits. Harder test splits concentrate Level 5–6 questions. TODO: figure labels read Val1/Val2/Test — regenerate as Test 1/2/3
Histograms of question and answer length in characters and words.
Question & answer length. Questions average 16.9 words and answers 32.6 words — open-ended, generative responses rather than single-word answers.

Examples

Dataset Samples

Each sample links an image to its source article, caption and a Vietnamese question–answer pair.

Results

Benchmark Results

Four open-source VLMs across three context levels and three difficulty-stratified test splits.

L1 image + question  ·  L2 + caption + article (zero-shot)  ·  L3 fine-tuned
Highlighted cells mark the best value per column within each split (lowest for latency); underlined cells mark the runner-up.

Limitations

Discussion & limitations

Gemini-generated QA pairs carry risks of stylistic homogeneity and hallucination, and web-sourced content may partially inflate zero-shot performance through pretraining-corpus overlap — a contamination risk that InfoSeek addressed with Unseen-Entity splits not yet provided here. Monolingual coverage limits cross-lingual evaluation compared with SEA-VQA and CVQA.

On the societal side, the dataset supports culturally aware vision-language models and the digital preservation of Vietnamese heritage for education, though LLM-generated content may encode factual errors. Reliance on major publications introduces representational bias toward dominant cultural narratives, partially mitigated by detailed source documentation and human verification of the full test split.

Citation

BibTeX

@inproceedings{TODO_vnculturevqa2027,
  title     = {VNCultureVQA: Benchmarking VQA for Vietnamese Cultural Events},
  author    = {Phan, Van-Phuc and Pham, Minh-Tan and Le, Trung-Nghia},
  booktitle = {TODO},
  year      = {2027}
}