benchmark-packing

name	benchmark-packing
description	Benchmark chunkah packing algorithm changes against a series of OCI images.

Benchmark Packing

Overview

This skill guides benchmarking of packing algorithm changes by chunking a series of consecutive OCI images and measuring layer reuse between them. The key tools are:

• tools/chunk-image-series.py -- chunks a series of images
• tools/analyze-layer-reuse.py -- compares layer sharing across the chunked series

Pipeline

1. Build chunkah

After making code changes, build the container image:

just buildimg --no-chunk

This produces localhost/chunkah:latest.

2. Prepare source images

Source images must be in containers-storage. Either:

From OCI archives:

for archive in /path/to/*.ociarchive; do
    version=$(echo "${archive}" | grep -oP 'PATTERN')
    skopeo copy "oci-archive:${archive}" "containers-storage:localhost/myrepo:${version}"
done

From a registry:

chunk-image-series.py can pull directly from a registry:

tools/chunk-image-series.py docker://quay.io/fedora/fedora-coreos \
    --tag-filter '43.*' --limit 10 ...

3. Chunk the series

tools/chunk-image-series.py containers-storage:localhost/myrepo \
    --tag-filter '43.*' \
    --prefix myrepo-chunked \
    --chunkah-image localhost/chunkah \
    --force \
    -- --prune /sysroot --max-layers 128

Key flags:

• --force to overwrite results from prior runs
• --limit N to control how many images to process
• Extra chunkah args go after --

4. Analyze layer reuse

Pass all chunked images as separate positional arguments:

image_args=()
for i in $(seq 0 9); do
    image_args+=("containers-storage:localhost/myrepo-chunked:${i}")
done
tools/analyze-layer-reuse.py --json "${image_args[@]}" > results.json

IMPORTANT: do NOT use brace expansion inside quotes (e.g. "...:{0,1,2}") -- it won't expand. Always build the argument list explicitly.

The JSON summary fields are:

• avg_reuse_ratio -- fraction of data shared (0.0 to 1.0)
• avg_download_bytes -- average new data per update

5. Clean up chunked images

Always clean up chunked images after capturing results:

for i in $(seq 0 9); do
    podman rmi "localhost/myrepo-chunked:${i}" 2>/dev/null || true
done

6. Compare against original packing

Always measure the original (un-chunked) images as a baseline:

image_args=()
for tag in $(podman images --filter reference='localhost/myrepo' \
    --format '{{.Tag}}' | sort); do
    image_args+=("containers-storage:localhost/myrepo:${tag}")
done
tools/analyze-layer-reuse.py --json "${image_args[@]}" > original.json

Comparing multiple configurations

To compare a code change against the current defaults:

1. Chunk with the baseline chunkah (quay.io/coreos/chunkah:dev)
2. Capture and save JSON results
3. Clean up chunked images
4. Build local chunkah with your changes
5. Chunk with localhost/chunkah
6. Capture and save JSON results
7. Clean up chunked images
8. Compare the two JSON files

The source images and original baseline can be reused across runs.

Pitfalls

Duplicate tags

chunk-image-series.py creates temporary localhost/tmp-chunk-src tags during chunking. These are cleaned up on exit, but if a prior run was interrupted, stale tags may remain and cause duplicate entries. Before running analysis, verify tags are clean:

podman images --filter reference='localhost/myrepo' --format '{{.Tag}}' | sort

If you see duplicate tags, remove them before proceeding. A red flag in results is min_download_bytes: 0 -- this means two consecutive images were identical (likely duplicates).

Sample size sensitivity

Results can vary significantly depending on the specific time window of builds tested. Always test against multiple independent corpora (e.g. different image types, different time periods) before concluding a change is beneficial. An improvement on one sample may not generalize.

Layer count matters

The default --max-layers is 64. Results at 64 layers vs 128 layers can differ substantially. Always compare configurations at the same layer count.