gaea-geolocation-aware-conversational-model

name	gaea-geolocation-aware-conversational-model
title	GAEA: A Geolocation Aware Conversational Assistant
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2503.16423
keywords	["geolocation","conversational-ai","multimodal-models","image-geolocalization","visual-question-answering"]
description	Build a conversational AI that combines image geolocalization with contextual geographical knowledge. GAEA enables users to query precise GPS locations from images while receiving conversational responses about places, their attributes, and regional context—outperforming GPT-4o by 7.2% on geography-aware visual QA tasks.

Core Concept

GAEA addresses a critical gap in large multimodal models: while LMMs can answer questions about images, they fail at specialized geolocation reasoning. GAEA fuses three capabilities: image-to-location prediction, geographical understanding, and conversational interaction. Unlike prior geo-localization models that only return coordinates, GAEA provides rich contextual knowledge about predicted locations—ideal for forensics, navigation, social media analysis, and creative applications requiring location-aware reasoning.

Architecture Overview

The framework integrates three core components:

• Vision-Language Backbone: Qwen2.5-VL as the base LMM, which combines joint image-text embeddings with separate image and text encoders for multimodal understanding
• Dataset Foundation: GAEA-1.4M—a 800k-image dataset with 1.4M question-answer pairs curated from OpenStreetMap attributes and geographical context clues
• Evaluation Benchmark: GAEA-Bench with 3.5k diverse image-text pairs testing multiple question types (MCQs, true/false, long-form VQA, short-form VQA)
• Training Strategy: Single-stage fine-tuning using LoRA-style adaptation to inject geolocation-specific knowledge without full retraining

Implementation Steps

1. Dataset Preparation with OpenStreetMap Integration

The dataset leverages OSM attributes to extract geographical context clues. This example shows how to structure queries from location metadata:

import json
from typing import Dict, List

def create_geo_qa_pairs(osm_attributes: Dict, image_path: str) -> List[Dict]:
    """
    Generate diverse question-answer pairs from OpenStreetMap attributes.
    Combines location metadata with image features for geolocation reasoning.
    """
    qa_pairs = []

    # Multiple-choice questions about location types
    mcq = {
        "type": "mcq",
        "question": f"What type of location is shown in {image_path}?",
        : [osm_attributes[], , , ],
        : osm_attributes[]
    }
    qa_pairs.append(mcq)

    
    tf = {
        : ,
        : ,
        : 
    }
    qa_pairs.append(tf)

    
    svqa = {
        : ,
        : ,
        : osm_attributes.get(, [])
    }
    qa_pairs.append(svqa)

     qa_pairs