hdf5-pde-data-loading

name	hdf5-pde-data-loading
description	Patterns for loading PDE simulation datasets (PDEBench, PhiFlow, JAX-CFD) from HDF5 files. Handles layout detection (single tensor vs separate variables), spatial/temporal downsampling, multi-variable systems, HuggingFace and DaRUS data sources, and efficient PyTorch DataLoader creation. Use when preparing PDE data for neural operator training.
category	data-engineering
version	1.0.0
author	Synthetic Sciences
license	MIT
tags	["HDF5","Data Loading","PDE","PyTorch","DataLoader","Downsampling","PDEBench"]
dependencies	["h5py","numpy","torch","huggingface-hub"]

HDF5 PDE Data Loading

When to Use

• Loading PDE simulation datasets stored in HDF5 format
• PDEBench, PhiFlow, JAX-CFD, or custom simulation outputs
• Multi-variable systems (density, velocity, pressure, etc.)
• Need to downsample spatial/temporal dimensions for memory

HDF5 Layout Detection

PDE datasets come in two common layouts:

Layout 1: Single "tensor" dataset

with h5py.File(path, "r") as f:
    if "tensor" in f:
        ds = f["tensor"]  # shape: [N, T, X, C] or [N, T, X]

Layout 2: Separate variable datasets

with h5py.File(path, "r") as f:
    # Keys like: "density", "Vx", "pressure", or "t", "x", "u"
    rho = f["density"][:]  # [N, T, X]
    vel = f["Vx"][:]
    prs = f["pressure"][:]
    data = np.stack([rho, vel, prs], axis=-1)  # [N, T, X, 3]

Robust loading (handles both):

def load_pde_hdf5(path, res_x=1, res_t=1):
    """Load PDE data from HDF5, handling multiple layouts."""
    with h5py.File(path, "r") as f:
        print(f"Keys: {list(f.keys())}")

        if "tensor" in f:
            ds = f["tensor"]
            raw_shape = ds.shape
            if len(raw_shape) == :
                N, T, X, C = raw_shape
            :
                N, T, X = raw_shape; C = 

            X_ds = X // res_x
            T_ds = T // res_t  res_t >   T
            data = np.empty((N, X_ds, T_ds, C  (raw_shape)==  ), dtype=np.float32)

             s  (, N, ):  
                e = (s + , N)
                chunk = ds[s:e, ::res_t, ::res_x]
                 (chunk.shape) == :
                    data[s:e, :, :, ] = np.transpose(chunk, (, , ))
                :
                    data[s:e] = np.transpose(chunk, (, , , ))
        :
            
            var_keys = []
             k  (f.keys()):
                 (f[k], h5py.Dataset)  (f[k].shape) >= :
                    var_keys.append(k)

            arrays = [f[k][:, ::res_t, ::res_x]  k  var_keys]
            raw = np.stack(arrays, axis=-)  
            data = np.transpose(raw, (, , , )).astype(np.float32)

        
         grid_key  [, , ]:
             grid_key  f:
                grid = np.array(f[grid_key], dtype=np.float32)[::res_x]
                
        :
            grid = np.linspace(, , data.shape[], dtype=np.float32)

     data, grid