cultural-awareness-eval

name	cultural-awareness-eval
description	Assesses the ability of large multimodal models to identify the geographical origin (country, subregion, or continent) of an image based on visual cultural cues. It probes implicit stereotypical associations and geographic bias in vision-language models. Use when the user wants to benchmark on Dalle Street, Dollar Street, MaRVL, or asks about evaluating this task. Reports classification accuracy.
metadata	{"skill_kind":"dataset_eval","source_arxiv":2407.02067,"bibtex_key":"mukherjee2024crossroads","confidence":"high"}

cultural-awareness-eval

Crossroads of Continents: Automated Artifact Extraction for Cultural Adaptation with Large Multimodal Models — Mukherjee et al. (2024) (arXiv:2407.02067, 2024)

What this evaluates

Assesses the ability of large multimodal models to identify the geographical origin (country, subregion, or continent) of an image based on visual cultural cues. It probes implicit stereotypical associations and geographic bias in vision-language models.

Datasets

• Dalle Street — total 9935; splits: test (-1)
• Dollar Street — total ?; splits: test (-1)
• MaRVL — total ?; splits: test (-1)

Metrics

• classification accuracy (primary) — range: percent
  • Percentage of correct predictions where the model's open-ended text output is mapped to a UN geoscheme region and compared against the ground truth label. Calculated at country, subregion, and continent levels, as well as union (correct at any level) and intersection (correct at all levels) accuracies.

Input / output format

Input: A single image depicting a cultural scene or artifact.

Output: Open-ended text response predicting the geographical region (country, subregion, or continent) represented in the image.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    pred_regions = [map_to_un_region(p) for p in predictions]
    gold_regions = gold_labels
    correct = sum(1 for p, g in zip(pred_regions, gold_regions) if p == g)
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Models frequently trigger content policy violations instead of answering, which must be explicitly tracked as a separate category rather than treated as random errors.
• Accuracy varies significantly by geographic level; reporting only country-level accuracy underestimates model capability, as subregion and continent accuracies are substantially higher.
• Economic disparities affect performance differently across continents; lower-income quartiles show higher accuracy in Africa/Asia but lower in the Americas, requiring stratified reporting.

Evidence (verbatim from paper)

We process generated text to map it to one of the geographical sub-regions or a policy violation case and then compare it with true labels by mapping country information to geographical regions, which gives us classification accuracy as a quantitative metric for measuring success.

Citation

@misc{mukherjee2024crossroads,
  title={Crossroads of Continents: Automated Artifact Extraction for Cultural Adaptation with Large Multimodal Models},
  author={Mukherjee et al. (2024)},
  year={2024},
  note={arXiv:2407.02067}
}

• arXiv: 2407.02067