library-integration

name	library-integration
description	Guides adding support for new image/video augmentation libraries to the benchmark suite. Use when integrating a new library, adding library support, or when the user mentions adding a new augmentation library to test.

Library Integration

Add support for new augmentation libraries to the benchmark suite.

Integration Checklist

- [ ] Create transform implementation file
- [ ] Create requirements file
- [ ] Register matrix spec maps, requirements, and environment groups
- [ ] Add or update config examples if the library affects paper/common run configs
- [ ] Add or update matrix/config/job tests
- [ ] Test with sample data
- [ ] Generate baseline results
- [ ] Update documentation

Step 1: Create Transform Implementation

Create benchmark/transforms/{library}_impl.py:

"""Transform implementations for {library}."""

# Import library
import {library}

# Define library name
LIBRARY = "{library}"

def __call__(transform, image):
    """Apply transform to image.

    Adapt this to library's calling convention:
    - Some use transform(image)
    - Some use transform(image=image)
    - Some require specific formats
    """
    return transform(image)

# Define transforms to benchmark
TRANSFORMS = [
    {
        "name": "HorizontalFlip",
        "transform": {library}.HorizontalFlip(),
    },
    {
        "name": "VerticalFlip",
        "transform": {library}.VerticalFlip(),
    },
    # Add more transforms...
]

Image Loading

Add loader to benchmark/utils.py if needed:

def get_image_loader(library: str):
    """Get appropriate image loader for library."""
    if library == "new_library":
        def load_new_library(path):
            # Load in library's expected format
            return img
        return load_new_library

Step 2: Create Requirements File

Create requirements/{library}.txt:

{library}>=1.0.0
numpy>=1.19.0
opencv-python>=4.5.0

Add to requirements/requirements.txt if base dependencies needed.

Step 3: Register Matrix Support

For image benchmarks

Add the library to the image spec and requirement maps in benchmark/matrix.py:

IMAGE_SPECS["newlib"] = "benchmark/transforms/newlib_impl.py"
IMAGE_REQUIREMENTS["newlib"] = "requirements/newlib.txt"

For video benchmarks

Add the library to VIDEO_SPECS and VIDEO_REQUIREMENTS in benchmark/matrix.py. If it can share dependencies with existing libraries, add it to the relevant ENV_GROUPS entry instead of creating a separate venv. If it needs a non-standard backend, add a BenchmarkJob.backend value in benchmark/jobs.py and route it through benchmark/orchestrator.py; do not add a library-specific branch to benchmark/cli.py. If it needs different media defaults or slow-skip thresholds, add them to benchmark/policy.py, not to individual runners. If it needs new user-facing run options, add them to BenchmarkRunConfig in benchmark/config/models.py first, then update YAML loading/overrides in benchmark/config/resolve.py, dry-run expansion in benchmark/config/plan.py, and examples under configs/. Keep result filename changes in benchmark/output_naming.py, and detached GCP path changes in benchmark/cloud/paths.py.

Update tests when the matrix changes:

python -m pytest tests/test_matrix.py tests/test_config_models.py tests/test_config_plan.py tests/test_jobs_orchestrator.py

Step 4: Test Integration

# Prefer a config-first smoke run, with CLI overrides for temporary values.
python -m benchmark.cli plan --config configs/examples/local_rgb_micro_cpu.yaml --num-items 10 --num-runs 1
python -m benchmark.cli run --config configs/examples/local_rgb_micro_cpu.yaml \
  --output test_output/newlib \
  --num-items 10 \
  --num-runs 1

# Verify JSON output
python -c "import json; print(json.load(open('test_output/newlib/image-rgb/micro/newlib_micro_results.json'))['metadata']['library_versions'])"

Step 5: Generate Baseline Results

# Full benchmark run
python -m benchmark.cli run --config configs/examples/local_rgb_micro_cpu.yaml \
  --output output/newlib_rgb_micro \
  --num-items 2000 \
  --num-runs 5

Step 6: Update Documentation

Create docs/images/newlib_metadata.yaml or docs/videos/newlib_metadata.yaml:

library_name: NewLib
version: "1.0.0"
description: Brief description of the library
documentation: https://newlib.readthedocs.io
repository: https://github.com/org/newlib

Run comparison:

./tools/update_docs.sh

Common Issues

Import Errors

• Ensure library is in requirements file
• Check virtual environment activation
• Verify compatible Python version

Transform Failures

• Check transform API matches library version
• Verify image format (RGB vs BGR, HWC vs CHW)
• Test transforms individually first

Performance Issues

• Verify thread settings (OMP_NUM_THREADS=1)
• Check warmup convergence
• Monitor early stopping conditions

Video Library Integration

For video libraries, create {library}_video_impl.py and adapt:

import numpy as np


def __call__(transform, video):
    """Apply transform to one clip. Shape conventions:
    - (T, H, W, C) for Albumentations (NumPy) — use batch API when available
    - (T, C, H, W) for torch / Kornia tensors
    """
    # Albumentations: native multi-frame API (one param draw per clip)
    return np.ascontiguousarray(transform(images=video)["images"])

For Kornia/torchvision, see kornia_video_impl.py / torchvision_video_impl.py. Video loaders differ — check benchmark/utils.py (get_video_loader) and benchmark/video_runner.py.