wildworld-dataset

name	wildworld-dataset
description	WildWorld large-scale action-conditioned world modeling dataset with 108M+ frames from a photorealistic ARPG game, featuring per-frame annotations, 450+ actions, and explicit state information for generative world modeling research.
triggers	["use WildWorld dataset","load WildWorld ARPG data","work with WildWorld annotations","WildWorld world modeling","action conditioned video dataset","WildBench benchmark evaluation","WildWorld frame annotations","generative ARPG dataset"]

WildWorld Dataset Skill

Skill by ara.so — Daily 2026 Skills collection.

What WildWorld Is

WildWorld is a large-scale action-conditioned world modeling dataset automatically collected from a photorealistic AAA action role-playing game (ARPG). It is designed for training and evaluating dynamic world models — generative models that predict future game states given past observations and player actions.

Key Statistics

Property	Value
Total frames	108M+
Actions	450+ semantically meaningful
Monster species	29
Player characters	4
Weapon types	4
Distinct stages	5
Max clip length	30+ minutes continuous

Per-Frame Annotations

Every frame includes:

• Character skeletons — joint positions for player and monsters
• Actions & states — HP, animation state, stamina, etc.
• Camera poses — position, rotation, field of view
• Depth maps — monocular depth for each frame
• Hierarchical captions — action-level and sample-level natural language descriptions

---

Project Status

⚠️ As of March 2026, the dataset and WildBench benchmark have not yet been released. Monitor the repository for updates.

# Watch the repository for dataset release
# https://github.com/ShandaAI/WildWorld

---

Repository Setup

# Clone the repository
git clone https://github.com/ShandaAI/WildWorld.git
cd WildWorld

# Install dependencies (when benchmark code is released)
pip install -r requirements.txt

---

Expected Dataset Structure

Based on the paper and framework description, the dataset is expected to follow this structure:

WildWorld/
├── data/
│   ├── sequences/
│   │   ├── stage_01/
│   │   │   ├── clip_000001/
│   │   │   │   ├── frames/          # RGB frames (e.g., PNG)
│   │   │   │   ├── depth/           # Depth maps
│   │   │   │   ├── skeleton/        # Per-frame skeleton JSON
│   │   │   │   ├── states/          # HP, animation, stamina JSON
│   │   │   │   ├── camera/          # Camera pose JSON
│   │   │   │   └── actions/         # Action label files
│   │   │   └── clip_000002/
│   │   └── stage_02/
│   └── captions/
│       ├── action_level/            # Per-action descriptions
│       └── sample_level/            # Clip-level descriptions
├── benchmark/
│   └── wildbench/                   # WildBench evaluation code
├── assets/
│   └── framework-arxiv.png
├── LICENSE
└── README.md

---

Working with the Dataset (Anticipated API)

Loading Frame Annotations

import json
import os
from pathlib import Path
from PIL import Image
import numpy as np

class WildWorldClip:
    """Helper class to load a WildWorld clip and its annotations."""

    def __init__(self, clip_dir: str):
        self.clip_dir = Path(clip_dir)
        self.frames_dir = self.clip_dir / "frames"
        self.depth_dir = self.clip_dir / "depth"
        self.skeleton_dir = self.clip_dir / "skeleton"
        self.states_dir = self.clip_dir / "states"
        self.camera_dir = self.clip_dir / "camera"
        self.actions_dir = self.clip_dir / "actions"

    def get_frame(self, frame_id: int) -> Image.Image:
        frame_path = self.frames_dir / f"{frame_id:06d}.png"
        return Image.open(frame_path)

    def get_depth(self, frame_id: int) -> np.ndarray:
        depth_path = .depth_dir / 
         np.load(depth_path)

     () -> :
        skeleton_path = .skeleton_dir / 
         (skeleton_path)  f:
             json.load(f)

     () -> :
        
        state_path = .states_dir / 
         (state_path)  f:
             json.load(f)

     () -> :
        
        camera_path = .camera_dir / 
         (camera_path)  f:
             json.load(f)

     () -> :
        action_path = .actions_dir / 
         (action_path)  f:
             json.load(f)

     ():
        
        frame_files = (.frames_dir.glob())
         frame_path  frame_files[start:end]:
            frame_id = (frame_path.stem)
             {
                : frame_id,
                : .get_frame(frame_id),
                : .get_depth(frame_id),
                : .get_skeleton(frame_id),
                : .get_state(frame_id),
                : .get_camera(frame_id),
                : .get_action(frame_id),
            }



clip = WildWorldClip()
 sample  clip.iter_frames(start=, end=):
    frame_id = sample[]
    state = sample[]
    action = sample[]
    ()

PyTorch Dataset

import torch
from torch.utils.data import Dataset, DataLoader
from pathlib import Path
import json
import numpy as np
from PIL import Image
import torchvision.transforms as T

class WildWorldDataset(Dataset):
    """
    PyTorch Dataset for WildWorld action-conditioned world modeling.
    
    Returns sequences of (frames, actions, states) for next-frame prediction.
    """

    def __init__(
        self,
        root_dir: str,
        sequence_length: int = 16,
        image_size: tuple = (256, 256),
        stage: str = None,
        split: str = "train",
    ):
        self.root_dir = Path(root_dir)
        self.sequence_length = sequence_length
        self.image_size = image_size

        self.transform = T.Compose([
            T.Resize(image_size),
            T.ToTensor(),
            T.Normalize(mean=[0.485, 0.456, 0.406],
                        std=[0.229, 0.224, 0.225]),
        ])

        # Discover all clips
        self.clips = self._discover_clips(stage, split)
        self.samples = ._build_sample_index()

     ():
        clips = []
        stage_dirs = (
            [.root_dir /  /  / stage]
             stage
             ((.root_dir /  / ).iterdir())
        )
         stage_dir  stage_dirs:
             stage_dir.is_dir():
                 clip_dir  (stage_dir.iterdir()):
                     clip_dir.is_dir():
                        clips.append(clip_dir)
        
        split_idx = ((clips) * )
         clips[:split_idx]  split ==   clips[split_idx:]

     ():
        
        samples = []
         clip_dir  .clips:
            frames = ((clip_dir / ).glob())
            n_frames = (frames)
             start  (, n_frames - .sequence_length, .sequence_length // ):
                samples.append((clip_dir, start))
         samples

     ():
         (.samples)

     ():
        clip_dir, start = .samples[idx]
        frames_dir = clip_dir / 
        frame_files = (frames_dir.glob())[start:start + .sequence_length]

        frames, actions, states = [], [], []
         frame_path  frame_files:
            frame_id = (frame_path.stem)

            
            img = Image.(frame_path).convert()
            frames.append(.transform(img))

            
            action_path = clip_dir /  / 
             (action_path)  f:
                action_data = json.load(f)
            actions.append(action_data.get(, ))

            
            state_path = clip_dir /  / 
             (state_path)  f:
                state_data = json.load(f)
            states.append([
                state_data.get(, ),
                state_data.get(, ),
                state_data.get(, ),
            ])

         {
            : torch.stack(frames),            
            : torch.tensor(actions, dtype=torch.long),   
            : torch.tensor(states, dtype=torch.float32),  
        }



dataset = WildWorldDataset(
    root_dir=,
    sequence_length=,
    image_size=(, ),
    split=,
)

loader = DataLoader(dataset, batch_size=, shuffle=, num_workers=)

 batch  loader:
    frames = batch[]   
    actions = batch[] 
    states = batch[]   
    ()
    

Filtering by Action Type

# Action categories in WildWorld
ACTION_CATEGORIES = {
    "movement": ["walk", "run", "sprint", "dodge", "jump"],
    "attack": ["light_attack", "heavy_attack", "combo_finisher"],
    "skill": ["skill_cast_1", "skill_cast_2", "skill_cast_3", "skill_cast_4"],
    "defense": ["block", "parry", "guard"],
    "idle": ["idle", "idle_combat"],
}

def filter_clips_by_action(dataset_root: str, action_category: str) -> list:
    """Find all frame indices that contain a specific action category."""
    root = Path(dataset_root)
    results = []
    target_actions = ACTION_CATEGORIES.get(action_category, [])

    for clip_dir in root.glob("data/sequences/**"):
        if not clip_dir.is_dir():
            continue
        for action_file in sorted((clip_dir / "actions").glob("*.json")):
            with open(action_file) as f:
                data = json.load(f)
            if data.get("action_name") in target_actions:
                results.append({
                    : (clip_dir),
                    : (action_file.stem),
                    : data.get(),
                })
     results


skill_frames = filter_clips_by_action(, )
()

---

WildBench Evaluation

# WildBench evaluates world models on next-frame prediction quality.
# Expected metrics: FVD, PSNR, SSIM, action accuracy

class WildBenchEvaluator:
    """Evaluator for world model predictions on WildBench."""

    def __init__(self, benchmark_dir: str):
        self.benchmark_dir = Path(benchmark_dir)
        self.metrics = {}

    def evaluate(self, model, dataloader):
        from torchmetrics.image import StructuralSimilarityIndexMeasure, PeakSignalNoiseRatio

        ssim = StructuralSimilarityIndexMeasure()
        psnr = PeakSignalNoiseRatio()

        all_psnr, all_ssim = [], []

        for batch in dataloader:
            frames = batch["frames"]       # (B, T, C, H, W)
            actions = batch["actions"]     # (B, T)
            states = batch["states"]       # (B, T, S)

            # Use first T-1 frames to predict the T-th frame
            context_frames = frames[:, :-1]
            context_actions = actions[:, :-1]
            target_frame = frames[:, -1]

            with torch.no_grad():
                predicted_frame = model(context_frames, context_actions, states[:, :-1])

            all_psnr.append(psnr(predicted_frame, target_frame).item())
            all_ssim.append(ssim(predicted_frame, target_frame).item())

        return {
            "PSNR": np.mean(all_psnr),
            "SSIM": np.mean(all_ssim),
        }

---

Citation

@misc{li2026wildworldlargescaledatasetdynamic,
      title={WildWorld: A Large-Scale Dataset for Dynamic World Modeling with Actions and Explicit State toward Generative ARPG}, 
      author={Zhen Li and Zian Meng and Shuwei Shi and Wenshuo Peng and Yuwei Wu and Bo Zheng and Chuanhao Li and Kaipeng Zhang},
      year={2026},
      eprint={2603.23497},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2603.23497}, 
}

---

Resources

• Project Page: https://shandaai.github.io/wildworld-project/
• arXiv Paper: https://arxiv.org/abs/2603.23497
• YouTube Demo: https://www.youtube.com/watch?v=9vcSg553r2g
• GitHub: https://github.com/ShandaAI/WildWorld

---

Troubleshooting

Issue	Solution
Dataset not yet available	Monitor the repo; dataset release is pending as of March 2026
Frame loading OOM	Reduce sequence_length or image_size in the Dataset
Missing annotation files	Check that all subdirs (frames, depth, skeleton, states, camera, actions) are fully downloaded
Slow DataLoader	Increase num_workers, use SSD storage, or preprocess to HDF5
Benchmark code not found	The benchmark/wildbench directory will be released separately — watch the repo