// Validates whether benchmark artifacts cover the paper's required RGB micro and RGB DataLoader sections. Use when checking missing RGB runs, deciding what to run next, validating gcp_runs/output folders, or preparing paper tables.
Automates running image/video augmentation benchmarks for single or multiple libraries, validates outputs, generates comparison reports, and updates documentation. Use when running benchmarks, comparing library performance, or when the user mentions benchmark, benchmark.cli, pyperf, GCP benchmark runs, or performance testing.
Updates benchmark documentation with latest results including README tables, speedup plots, and library metadata. Use when updating documentation, generating comparison tables, or when the user mentions update_docs.sh or documentation generation.
Triage detached GCP benchmark runs, DONE/FAILED sentinels, VM cleanup, vm.log, gcp_last_run.json, and partial result downloads. Use when GCP benchmark logs mention DONE, FAILED, exit_code.txt, VM disappeared, STOPPING, gcloud machine type errors, or missing artifacts.
Guides adding support for new image/video augmentation libraries to the benchmark suite. Use when integrating a new library, adding library support, or when the user mentions adding a new augmentation library to test.
Executes the paper benchmark plan for RGB, multichannel, DataLoader, and video benchmarks. Use when the user mentions the paper benchmark, deadline plan, machine matrix, RGB micro, multichannel, DataLoader, video GPU, c4/c4d/g2 machines, or what to run next.
Analyzes benchmark results to identify slow transforms, warmup issues, and performance regressions. Compares speedups across libraries and generates optimization recommendations. Use when analyzing performance, investigating slow benchmarks, or comparing library results.
| name | paper-coverage-validator |
| description | Validates whether benchmark artifacts cover the paper's required RGB micro and RGB DataLoader sections. Use when checking missing RGB runs, deciding what to run next, validating gcp_runs/output folders, or preparing paper tables. |
Use tools.check_paper_coverage to avoid manual folder inspection.
python -m tools.check_paper_coverage gcp_runs output
Validate the RAM-only reduced production-path pass:
python -m tools.check_paper_coverage --profile ram-reduced gcp_runs output
Require optional libraries such as DALI:
python -m tools.check_paper_coverage gcp_runs output --require-optional-libraries
image-rgb/micro: AlbumentationsX, torchvision, Kornia, Pillow; summary JSON.image-rgb/pipeline: AlbumentationsX, torchvision, Kornia, Pillow.9-channel and video benchmarks are still supported by the codebase, docs, and transform sets, but they are not required for the current RGB paper artifact.
--profile ram-reduced intentionally checks only:
memory_dataloader_augment.It does not require decode DataLoader, 9-channel, or video runs.
results/ from the GCS run prefix.