lumine-generalist-agents-3d-worlds

name	lumine-generalist-agents-3d-worlds
title	Lumine: Open Recipe for Building Generalist Agents in 3D Open Worlds
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2511.08892
keywords	["Vision-Language Models","Game AI","Embodied AI","Human-Like Interaction","Zero-Shot Transfer"]
description	Build generalist agents for 3D open worlds using vision-language models with human-like interaction patterns and adaptive reasoning—achieving zero-shot transfer across games without fine-tuning through pixel-level perception and high-frequency action generation.

Build Generalist Agents for Open-World 3D Environments

Game-playing agents typically require game-specific training or scripted logic. Lumine demonstrates that pretrained vision-language models can play complex 3D open-world games by mimicking human interaction patterns: processing visual frames at 5 Hz, generating keyboard-mouse actions at 30 Hz, and adaptively invoking reasoning only when necessary. A single agent trained on one game transfers zero-shot to others without fine-tuning.

The approach completes five-hour story sequences on par with human efficiency, establishing a concrete step toward generalist embodied agents for open-ended environments.

Core Concept

Lumine treats game-playing as a straightforward vision-to-action problem: given pixels, generate human-like actions (keyboard/mouse). The key innovations are:

1. Human-like interaction frequency - Process at 5 Hz (human perception) for perception but generate actions at 30 Hz (motor frequency)
2. Adaptive reasoning - Invoke chain-of-thought only when needed (complex decisions), not constantly
3. Zero-shot transfer - Train once on one game; use directly on others with no game-specific tuning

This approach relies on strong vision-language models' inherent understanding of physics, game mechanics, and strategy—learned during general pretraining.

Architecture Overview

• Vision Encoder: Processes game frames to understand state and environment
• Perception Buffer: Maintains frame history for context (last 4-5 frames at 5 Hz)
• Reasoning Module: VLM chain-of-thought for complex decisions (adaptive invocation)
• Action Generator: Converts reasoning to keyboard/mouse commands
• Action Executor: Sends actions to game at 30 Hz motor frequency
• Adaptive Control: Routes between reactive (fast) and deliberative (slow) modes

Implementation Steps

Step 1: Frame Capture and Perception Pipeline

Capture game frames and feed them to vision-language model.

import cv2
import numpy as np
from collections import deque
import time

class PerceptionPipeline:
    """
    Handles frame capture and perception preprocessing.
    """

     ():
        
        .game = game_process
        .perception_freq = perception_freq
        .frame_size = frame_size
        .perception_interval =  / perception_freq

        
        .frame_buffer = deque(maxlen=)
        .last_perception_time = 

     () -> np.ndarray:
        
        
        screenshot = .game.get_screenshot()
        frame = cv2.resize(screenshot, .frame_size)
         frame

     () -> [, ]:
        
        current_time = time.time()

        
         current_time - .last_perception_time < .perception_interval:
               

        
        frame = .capture_frame()
        .frame_buffer.append(frame)
        .last_perception_time = current_time

         {
            : frame,
            : (.frame_buffer),
            : current_time
        }