transform-spec-validator

name	transform-spec-validator
description	Validates custom transform specification files for benchmark runner, checking LIBRARY, __call__, and TRANSFORMS definitions. Use when creating custom transforms, validating transform specs, or when the user mentions transform validation or spec files.

Transform Spec Validator

Validate custom transform specification files for the benchmark runner.

Required Structure

Transform spec files must define three components:

# 1. Library name
LIBRARY = "library_name"

# 2. Call function
def __call__(transform, image_or_video):
    """Apply transform to data."""
    return transform(image_or_video)

# 3. Transforms list
TRANSFORMS = [
    {
        "name": "TransformName",
        "transform": LibraryTransform()
    }
]

Quick Validation

# Validate spec file
python -c "
import importlib.util
spec = importlib.util.spec_from_file_location('test', 'my_transforms.py')
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)

# Check required attributes
assert hasattr(module, 'LIBRARY'), 'Missing LIBRARY'
assert hasattr(module, '__call__'), 'Missing __call__'
assert hasattr(module, 'TRANSFORMS'), 'Missing TRANSFORMS'
assert callable(module.__call__), 'Module.__call__ not callable'

print('✓ Valid spec file')
"

Component Validation

1. LIBRARY

Must be a string matching library name:

# ✓ Good
LIBRARY = "albumentationsx"

# ✗ Bad
LIBRARY = None
library = "albumentationsx"  # Wrong name

2. call

Must be callable and accept transform + data:

# ✓ Good - Standard pattern
def __call__(transform, image):
    return transform(image)

# ✓ Good - Keyword arguments
def __call__(transform, image):
    return transform(image=image)

# ✓ Good - Format conversion
def __call__(transform, image):
    # Convert if needed
    result = transform(image)
    return result

# ✓ Good - Albumentations video (T, H, W, C), batch API
def __call__(transform, video):
    import numpy as np

    return np.ascontiguousarray(transform(images=video)["images"])

# ✗ Bad - Wrong signature
def __call__(transform):  # Missing image parameter
    return transform()

3. TRANSFORMS

Must be list of dicts with 'name' and 'transform' keys:

# ✓ Good
TRANSFORMS = [
    {
        "name": "HorizontalFlip",
        "transform": A.HorizontalFlip(p=1.0)
    },
    {
        "name": "Rotate",
        "transform": A.Rotate(limit=45, p=1.0)
    }
]

# ✗ Bad - Missing keys
TRANSFORMS = [
    {
        "transform": A.HorizontalFlip()  # Missing 'name'
    }
]

# ✗ Bad - Wrong structure
TRANSFORMS = [
    A.HorizontalFlip()  # Not a dict
]

Naming Conventions

Transform Names

Use descriptive names that include key parameters:

# ✓ Good - Clear and specific
{"name": "HorizontalFlip", ...}
{"name": "Rotate(limit=45)", ...}
{"name": "GaussNoise(var_limit=(10,50))", ...}
{"name": "ToGray(method=weighted_average)", ...}

# ✗ Bad - Too generic
{"name": "Transform1", ...}
{"name": "Rotate", ...}  # When testing multiple rotation limits

The name appears in result files and comparison tables.

Testing Spec Files

Test with Sample Data

import importlib.util
import numpy as np

# Load spec
spec = importlib.util.spec_from_file_location('test', 'my_transforms.py')
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)

# Create test image
test_image = np.random.randint(0, 255, (256, 256, 3), dtype=np.uint8)

# Test each transform
for t in module.TRANSFORMS:
    name = t['name']
    transform = t['transform']

    try:
        result = module.__call__(transform, test_image)
        assert result is not None
        assert result.shape == test_image.shape
        print(f"✓ {name}")
    except Exception as e:
        print(f"✗ {name}: {e}")

Test with Runner

# Quick test with small dataset
python -m benchmark.cli run \
  --config configs/examples/local_rgb_micro_cpu.yaml \
  --data-dir /path/to/test/images \
  --output test_output/custom_spec \
  --spec my_transforms.py \
  --num-items 10 \
  --num-runs 1

Check output:

import json
with open('test_output.json') as f:
    data = json.load(f)

# Verify all transforms ran
for name in data['results']:
    metrics = data['results'][name]
    if not metrics['supported']:
        print(f"✗ {name}: not supported")
    elif metrics.get('early_stopped'):
        print(f"⚠ {name}: {metrics['early_stop_reason']}")
    else:
        print(f"✓ {name}: {metrics['median_throughput']:.2f} img/sec")

Common Issues

Import Errors

Problem: Module not found when runner loads spec file

Fix: Install library in venv before running:

# Check library in requirements
cat requirements/library.txt

# Or install manually in test venv
python -m venv .venv_test
source .venv_test/bin/activate
pip install library
python my_transforms.py  # Test imports

Transform Failures

Problem: Transform raises exception during benchmark

Causes:

• Incorrect image format (RGB vs BGR, HWC vs CHW)
• Invalid parameters
• Version incompatibility

Debug:

# Add to spec file temporarily
def __call__(transform, image):
    print(f"Image shape: {image.shape}, dtype: {image.dtype}")
    result = transform(image)
    print(f"Result shape: {result.shape}, dtype: {result.dtype}")
    return result

Probability Issues

Problem: Transform sometimes doesn't apply (p < 1.0)

Fix: Set probability to 1.0 for deterministic benchmarking:

# ✓ Good - Always applies
transform = A.HorizontalFlip(p=1.0)

# ✗ Bad - Random application
transform = A.HorizontalFlip(p=0.5)

Example Spec Files

Image Transforms

See:

• benchmark/transforms/albumentationsx_impl.py
• examples/custom_image_transforms.py

Video Transforms

See:

• benchmark/transforms/albumentationsx_video_impl.py
• benchmark/transforms/kornia_video_impl.py
• examples/custom_video_specs_template.py

Parametric Testing

Test same transform with different parameters:

TRANSFORMS = [
    # Test blur kernel sizes
    {"name": "GaussianBlur(kernel=3)", "transform": A.GaussianBlur(blur_limit=(3,3), p=1)},
    {"name": "GaussianBlur(kernel=7)", "transform": A.GaussianBlur(blur_limit=(7,7), p=1)},
    {"name": "GaussianBlur(kernel=15)", "transform": A.GaussianBlur(blur_limit=(15,15), p=1)},

    # Test noise levels
    {"name": "GaussNoise(var=10)", "transform": A.GaussNoise(var_limit=(10,10), p=1)},
    {"name": "GaussNoise(var=50)", "transform": A.GaussNoise(var_limit=(50,50), p=1)},
]

Analyze with:

python tools/analyze_parametric_results.py results.json

Validation Checklist

Before running benchmarks:

- [ ] LIBRARY defined as string
- [ ] __call__ function defined and callable
- [ ] TRANSFORMS is list of dicts
- [ ] Each dict has 'name' and 'transform' keys
- [ ] Transform names are descriptive
- [ ] All transforms have p=1.0
- [ ] Imports work in isolation
- [ ] Test transforms on sample data
- [ ] Quick benchmark run succeeds