modality-adaptive-reasoning-visualizations

name	modality-adaptive-reasoning-visualizations
title	MARVIS: Modality Adaptive Reasoning over VISualizations
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.01544
keywords	["Vision-Language Models","Multimodal Reasoning","Latent Embedding Visualization","Audio Processing","Tabular Data"]
description	Enable small vision-language models to reason over diverse data types by converting latent embeddings into visual representations, achieving specialized performance without domain-specific training.

MARVIS: Training-Free Multimodal Reasoning via Embedding Visualization

The challenge of building versatile AI systems that work across multiple modalities—vision, audio, tabular data, and biology—typically requires either specialized models for each domain or large foundation models with limited efficiency. MARVIS demonstrates that a 3B parameter vision-language model can match or exceed much larger systems by transforming latent embeddings from any modality into visual representations. This training-free approach leverages the spatial reasoning capabilities that VLMs naturally excel at.

The key insight is that embeddings from any modality (audio spectrograms, biological sequences, tabular encodings) can be visualized in ways that preserve semantic structure. A VLM can then interpret these visualizations using the same reasoning mechanisms it applies to images, effectively adapting to new domains without retraining.

Core Concept

MARVIS operates on a simple but powerful principle: latent embeddings contain rich semantic information regardless of their original modality. Rather than training modality-specific decoders, the approach renders embeddings as images using techniques like heatmaps, spectrograms, or graph layouts. The VLM's spatial reasoning and fine-grained visual understanding then become universal interpreters for any domain.

This removes the traditional requirement for domain-specific pretraining or fine-tuning. The model reasons about structure, relationships, and patterns in the visual representation without knowing it originated from audio or tabular data. The method naturally preserves privacy since the original data is never exposed—only its learned embedding representation is visualized.

Architecture Overview

The system comprises three main components:

• Embedding Extractor: Modality-agnostic encoder that produces dense vector representations from any input type (audio, images, tables, sequences)
• Visualization Mapper: Converts latent embeddings into image-space representations suitable for VLM interpretation, handling dimensionality reduction and spatial encoding
• Vision-Language Reasoner: Standard VLM (3B parameters in the paper) that interprets visualized embeddings using its native visual reasoning capabilities

The separation of concerns allows swapping embedding extractors for different modalities while reusing the same VLM, creating a truly modality-adaptive system.

Implementation

The embedding visualization process maps high-dimensional latent vectors to 2D or 3D spatial representations that preserve semantic structure.

Create a mapping function that converts embeddings to visual space:

import numpy as np
 sklearn.decomposition  PCA
 PIL  Image

 ():
    
    
    embedding_flat = embedding.flatten()

    
    target_pixels = shape[] * shape[]
     (embedding_flat) < target_pixels:
        embedding_flat = np.pad(embedding_flat, (, target_pixels - (embedding_flat)))
    :
        embedding_flat = embedding_flat[:target_pixels]

    
    heatmap = embedding_flat.reshape(shape)
    heatmap_norm = (heatmap - heatmap.()) / (heatmap.() - heatmap.() + )
    heatmap_uint8 = (heatmap_norm * ).astype(np.uint8)

    
    image = Image.fromarray(heatmap_uint8, mode=)
     image

 ():
    
    
    pca = PCA(n_components=)
    coords_2d = pca.fit_transform(embedding.reshape(, -))

    
    img = Image.new(, shape, color=)
    pixels = img.load()

    
    x = ((coords_2d[, ] / (np.(coords_2d[, ]) + )) * shape[] //  + shape[] // )
    y = ((coords_2d[, ] / (np.(coords_2d[, ]) + )) * shape[] //  + shape[] // )

    
    radius = 
     dx  (-radius, radius):
         dy  (-radius, radius):
            px, py = x + dx, y + dy
              <= px < shape[]   <= py < shape[]:
                dist = np.sqrt(dx** + dy**)
                intensity = ((, (radius - dist) / radius * ))
                pixels[px, py] = (, pixels[px, py] - intensity)

     img