benchmarkoor

# Benchmarkoor Run benchmarkoor on Linux with native OverlayFS without mutating the pristine State Actor datadir or pre-populated snapshot. Put a kernel-enforced read-only bind mount in front of the pristine directory and keep all writable overlay data in a disposable external directory. ## Read current upstream material Consult these sources before adapting a current suite: - [Local benchmarking guideline](https://hackmd.io/@QmVpC8TxQ8a1nTCW46EsEQ/SygwzXUrMl) - [benchmarkoor configuration reference](https://github.com/ethpandaops/benchmarkoor/blob/master/docs/configuration.md) - [benchmarkoor-tests configs](https://github.com/ethpandaops/benchmarkoor-tests) Record the benchmarkoor, benchmarkoor-tests, Erigon, and State Actor image revisions. Do not assume rolling branches, fixture URLs, or image tags still identify the same code. ## Preserve these invariants - Obey the repository's `AGENTS.md`. For Erigon work, build from a dedicated Git worktree rather than the primary checkout. - Designate the completed State Actor directory or pre-populated snapshot as pristine and never use it as a `direct` runner datadir. - Use `method: overlayfs` for these Linux benchmark hosts. - Configure `source_dir` to a verified read-only bind mount of the pristine datadir, never to the original writable path. - Perform recursive inspection, size measurement, and hashing through that read-only bind. Reading files through the writable path can update access times on a `relatime` filesystem. - Keep OverlayFS `upperdir`, `workdir`, and `merged` paths outside the pristine and lower trees. - Reject descendant mounts below the pristine datadir before creating its bind. A plain bind does not include child mounts and can otherwise expose incomplete or hidden underlying state. - Verify the lower mount is `ro` before startup and remains `ro` throughout the run. - Treat every other datadir method as an alternative that requires an explicit user request; never inherit an upstream datadir default silently. - Do not stop unrelated workloads merely to improve benchmark isolation. Report contention and ask before changing out-of-scope processes. ## Discover paths and pin revisions Resolve these common inputs from the current host and requested suite. They are required inputs, not defaults copied from a previous machine: If checkout paths were not supplied, inspect only the task-approved workspace roots, identify Git repositories from their remotes and contents rather than their directory names, and clone a missing benchmarkoor or benchmarkoor-tests repository into a new task-owned path. Use `git rev-parse --show-toplevel`, `git remote -v`, and the expected `go.mod` or `configs/` layout to validate each candidate. Do not search the host for datadirs. ```bash BENCHMARKOOR_SRC=${BENCHMARKOOR_SRC:?set the benchmarkoor checkout} TESTS_DIR=${TESTS_DIR:?set the benchmarkoor-tests checkout} ERIGON_WT=${ERIGON_WT:?set the dedicated Erigon worktree} CONTEXT_NAME=${CONTEXT_NAME:?set the exact context directory name} SUITE_KIND=${SUITE_KIND:?set to compute or stateful} RUN_ROOT=${RUN_ROOT:?set a new task-owned runtime directory} RESULTS_DIR=${RESULTS_DIR:?set a durable task-owned results directory outside RUN_ROOT} BENCHMARKOOR_SRC=$(realpath -e -- "$BENCHMARKOOR_SRC") TESTS_DIR=$(realpath -e -- "$TESTS_DIR") ERIGON_WT=$(realpath -e -- "$ERIGON_WT") RUN_ROOT=$(realpath -m -- "$RUN_ROOT") RESULTS_DIR=$(realpath -m -- "$RESULTS_DIR") case "$RESULTS_DIR/" in "$RUN_ROOT/"*) exit 1 ;; esac case "$RUN_ROOT/" in "$RESULTS_DIR/"*) exit 1 ;; esac LOWER_DIR="$RUN_ROOT/lower-ro" OVERLAY_TMP="$RUN_ROOT/overlay-runtime" CACHE_DIR="$RUN_ROOT/cache" BENCHMARKOOR_SOURCE_BIN="$BENCHMARKOOR_SRC/bin/benchmarkoor" ERIGON_SHORT_COMMIT=$(git -C "$ERIGON_WT" rev-parse --short=12 HEAD) DOCKER=(env -i PATH=/usr/sbin:/usr/bin:/sbin:/bin \ /usr/bin/docker --host unix:///var/run/docker.sock) ``` Use this sanitized `DOCKER` command array for every Docker CLI operation in the workflow. It pins inspection, image preparation, and preflight checks to the same system daemon used by the guarded launcher, regardless of the invoking account's current Docker context or environment. Use the exact context name requested by the user, such as a future devnet directory; do not select a similarly named `-full` or older devnet. For a State Actor build, choose a new `STATE_ROOT` and set `PRISTINE_DIR="$STATE_ROOT/erigon"`. For a pre-populated run, require the operator to designate the existing `PRISTINE_DIR`; never infer which valuable datadir is pristine by scanning the host. Canonicalize it with `realpath -e` before creating mounts. Require `RUN_ROOT`, cache, results, OverlayFS writable layers, and build contexts to be outside the pristine tree and every protected lower. Keep the durable `RESULTS_DIR` disjoint from the disposable `RUN_ROOT` so runtime cleanup cannot remove benchmark artifacts. Stop on path overlap instead of silently choosing another directory. Before selecting a base image or writing an override, follow [Discover the current files](references/compute-stateful-suites.md#discover-the-current-files) to resolve the exact runner, context globals, clients file, and instance from the checked-out revision. Verify repository state and record commits: ```bash git -C "$BENCHMARKOOR_SRC" status --short --branch git -C "$BENCHMARKOOR_SRC" rev-parse HEAD git -C "$TESTS_DIR" status --short --branch git -C "$TESTS_DIR" rev-parse HEAD git -C "$ERIGON_WT" status --short --branch git -C "$ERIGON_WT" rev-parse HEAD ``` Fetch the authoritative Erigon and benchmarkoor-tests branches immediately before creating the worktree and resolving the suite, then record both commits. Re-fetch after a long run. If a remote tip advanced, report both hashes and describe the intervening changes; call the result "latest at launch," not the current tip. For benchmarkoor-tests, also compare the selected runner-config blob and fixture source because the repository can advance without changing the suite that ran. Do not silently move the worktree or relabel an already-produced result. ## Build benchmarkoor ```bash cd "$BENCHMARKOOR_SRC" || exit 1 make build-core "$BENCHMARKOOR_SOURCE_BIN" version ``` Use the source build because datadir handling, result formats, pre-run handling, and State Actor orchestration change across benchmarkoor revisions. Before a long suite whose rollback strategy is `container-recreate`, verify the selected benchmarkoor revision eagerly releases the previous datadir after removing its container. At benchmarkoor `709ad43`, cleanup stayed deferred until process teardown, so a stateful run retained one OverlayFS mount and upper layer per completed fixture. Local patch `ea92a86` was validated to release each previous mount before preparing the next; use an upstream equivalent when available. Do not infer safety from a successful short command alone: smoke-test the actual binary and assert the live disposable-overlay count never exceeds one. A pre-populated run may also retain the one fixed staging overlay described below; do not count that fixed upper as a per-test upper. ## Generate a separate pristine State Actor datadir ```bash STATE_ROOT=${STATE_ROOT:?set a new State Actor output root} STATE_ROOT=$(realpath -m -- "$STATE_ROOT") PRISTINE_DIR="$STATE_ROOT/erigon" ``` Stop if `$PRISTINE_DIR` already exists and its ownership is unclear. Choose a new path rather than using `--force` against a valuable snapshot. Append a local override after all upstream configs: ```yaml global: env: STATE_DIR: /absolute/path/to/new-state-actor-root builder: cleanup_on_start: false state_actor: container_runtime: docker runner: container_runtime: docker cleanup_on_start: false live_reporting: enabled: false ``` Do not put `runner.client.datadirs` in this build-only override. Datadir isolation is a run-time concern, and the later run-only override supplies the protected OverlayFS source. From the benchmarkoor-tests repository, build only Erigon: ```bash rg --files "$TESTS_DIR/configs/datadirs/state-actor" | rg '/(global|builder)\.ya?ml$' | sort STATE_ACTOR_GLOBAL_CONFIG=${STATE_ACTOR_GLOBAL_CONFIG:?select the matching global config} STATE_ACTOR_BUILDER_CONFIG=${STATE_ACTOR_BUILDER_CONFIG:?select the matching builder config} STATE_ACTOR_BUILD_OVERRIDE=${STATE_ACTOR_BUILD_OVERRIDE:?set the build-only local override} GLOBAL_CONFIG=$(realpath -e -- "$TESTS_DIR/configs/global.yaml") STATE_ACTOR_GLOBAL_CONFIG=$(realpath -e -- "$STATE_ACTOR_GLOBAL_CONFIG") STATE_ACTOR_BUILDER_CONFIG=$(realpath -e -- "$STATE_ACTOR_BUILDER_CONFIG") STATE_ACTOR_BUILD_OVERRIDE=$(realpath -e -- "$STATE_ACTOR_BUILD_OVERRIDE") BUILD_USER=$(id -un) BUILD_HOME=$(getent passwd "$(id -u)" | cut -d: -f6) test -n "$BUILD_USER" test -d "$BUILD_HOME" cd "$TESTS_DIR" || exit 1 env -i \ PATH=/usr/sbin:/usr/bin:/sbin:/bin \ HOME="$BUILD_HOME" \ USER="$BUILD_USER" \ LOGNAME="$BUILD_USER" \ DOCKER_HOST=unix:///var/run/docker.sock \ BENCHMARKOOR_BUILDER_CLEANUP_ON_START=false \ BENCHMARKOOR_RUNNER_CLEANUP_ON_START=false \ "$BENCHMARKOOR_SOURCE_BIN" build \ --config "$GLOBAL_CONFIG" \ --config "$STATE_ACTOR_GLOBAL_CONFIG" \ --config "$STATE_ACTOR_BUILDER_CONFIG" \ --config "$STATE_ACTOR_BUILD_OVERRIDE" \ --limit-state-actor-target=erigon \ --rebuild-on-diff ``` State Actor can exceed its final-size projection while creating `streamsort-*` intermediates and explicit templates. Reserve peak working headroom and trust only the completed filesystem measurement after temporary data is cleaned up. One historical build peaked near 1,056.5 GiB and settled at 516.83 GiB; treat that 2.04x ratio as capacity evidence, not a bound. Wait for a successful exit and inspect CPU and disk activity instead of treating a quiet finalization phase or ETA as a failure. After success, canonicalize and recheck the path boundary before installing the read-only bind: ```bash PRISTINE_DIR=$(realpath -e -- "$PRISTINE_DIR") case "$RUN_ROOT/" in "$PRISTINE_DIR/"*) exit 1 ;; esac case "$PRISTINE_DIR/" in "$RUN_ROOT/"*) exit 1 ;; esac ``` Install the bind in the protection section before inspecting the completed tree. Once that section has set `INTEGRITY_ROOT`, inspect only through that path: ```bash jq . "$INTEGRITY_ROOT/state-actor-manifest.json" jq . "$INTEGRITY_ROOT/.benchmarkoor-build.json" find "$INTEGRITY_ROOT" -maxdepth 1 -type f -printf '%f %s bytes\n' | sort ``` The manifest records State Actor metadata and the resolved builder-image digest; the sidecar records benchmarkoor's rebuild fingerprint. Do not rerun State Actor against this path after declaring it pristine. ## Measure pristine and overlay capacity First identify the filesystems containing the pristine tree and planned writable overlay. The fresh `RUN_ROOT` need not exist yet, so resolve its nearest existing ancestor for this planning check. Defer recursive allocated/apparent-size measurement until the read-only bind in the protection section is mounted; large database files may be sparse, and reading them through the writable path can update atime: ```bash OVERLAY_FS_PROBE="$RUN_ROOT" while test ! -e "$OVERLAY_FS_PROBE"; do next_probe=$(dirname -- "$OVERLAY_FS_PROBE") test "$next_probe" != "$OVERLAY_FS_PROBE" || exit 1 OVERLAY_FS_PROBE="$next_probe" done test -d "$OVERLAY_FS_PROBE" test -w "$OVERLAY_FS_PROBE" df -B1 --output=size,used,avail,target "$PRISTINE_DIR" "$OVERLAY_FS_PROBE" findmnt -T "$PRISTINE_DIR" findmnt -T "$OVERLAY_FS_PROBE" ``` Do not assume OverlayFS needs a second full copy for this workload, but do not assume the upper will stay small either. Before the first run, use the larger of the pristine datadir's allocated and apparent sizes as the conservative copy-up budget, then require operational headroom beyond it. Apparent size covers writes that allocate holes in sparse lower files. During the run, measure the actual upper layer and stop before the filesystem becomes critically full. A historical full compute run observed about 1.8 GiB of upper-layer data over a 516.83 GiB lower; this is capacity evidence, not a bound for a different source or client revision. ## Build an image from the exact Erigon worktree Build the binary from the exact commit under test: ```bash cd "$ERIGON_WT" || exit 1 make erigon ./build/bin/erigon --version ERIGON_BINARY_SHA256=$(sha256sum ./build/bin/erigon | awk '{print $1}') [[ "$ERIGON_BINARY_SHA256" =~ ^[0-9a-f]{64}$ ]] || exit 1 ``` When a compatible image for the same branch already exists, a small overlay image avoids rebuilding the unchanged container environment. Create a dedicated build-context directory containing the built `erigon` binary and this Dockerfile: ```dockerfile ARG BASE_IMAGE FROM ${BASE_IMAGE} COPY --chown=0:0 erigon /usr/local/bin/erigon ``` Set `BASE_IMAGE` to the selected context's compatible client image pinned as `repository@sha256:digest`; obtain it from the effective selected instance, not from a prior devnet. Build and verify that the image contains the byte-identical local binary: ```bash BASE_IMAGE=${BASE_IMAGE:?set the selected client base image pinned by digest} case "$BASE_IMAGE" in *@sha256:*) ;; *) exit 2 ;; esac SUITE_RUNNER_CONFIG=$(realpath -e -- "${SUITE_RUNNER_CONFIG:?select the suite runner config}") CLIENTS_CONFIG=$(realpath -e -- "${CLIENTS_CONFIG:?select its clients config}") INSTANCE_ID=${INSTANCE_ID:?select the exact Erigon instance} IMAGE_SCOPE=$( { printf 'base=%s\0instance=%s\0binary=%s\0' \ "$BASE_IMAGE" "$INSTANCE_ID" "$ERIGON_BINARY_SHA256" sha256sum "$SUITE_RUNNER_CONFIG" "$CLIENTS_CONFIG" } | sha256sum | cut -c1-16 ) IMAGE_TAG="erigon-local:$ERIGON_SHORT_COMMIT-$IMAGE_SCOPE" IMAGE_CONTEXT=${IMAGE_CONTEXT:?set the task-owned image build context} IMAGE_CONTEXT=$(realpath -e -- "$IMAGE_CONTEXT") IMAGE_IID_FILE=$(mktemp /tmp/benchmarkoor-image-id.XXXXXX) test "$(stat -Lc '%u:%g:%a:%h' -- "$IMAGE_IID_FILE")" = \ "$(id -u):$(id -g):600:1" || exit 1 if ! "${DOCKER[@]}" build --iidfile "$IMAGE_IID_FILE" \ --build-arg BASE_IMAGE="$BASE_IMAGE" -t "$IMAGE_TAG" "$IMAGE_CONTEXT"; then unlink -- "$IMAGE_IID_FILE" exit 1 fi LOCAL_IMAGE_ID=$(tr -d '[:space:]' <"$IMAGE_IID_FILE") unlink -- "$IMAGE_IID_FILE" [[ "$LOCAL_IMAGE_ID" =~ ^sha256:[0-9a-f]{64}$ ]] || exit 1 "${DOCKER[@]}" run --rm "$LOCAL_IMAGE_ID" --version test "$("${DOCKER[@]}" run --rm --entrypoint sha256sum "$LOCAL_IMAGE_ID" \ /usr/local/bin/erigon | awk '{print $1}')" = "$ERIGON_BINARY_SHA256" ``` Resolve and pin the base-image digest. If the branch changes the container environment or is incompatible with that base, build the repository's complete Dockerfile instead. Do not rely on a rolling remote image's Erigon binary when benchmarking a local branch. The tag scope includes the base digest, selected instance and config blobs, and binary digest, but tags remain mutable daemon aliases. Capture this build's immutable ID through its unique `--iidfile`; never resolve the tag afterward because another task can retag it between those operations. Put `LOCAL_IMAGE_ID` in every run override and retain `pull_policy: never`; verify the persisted run records that same image ID before accepting results. ## Create a run-only override Copy the complete chosen instance from the current `clients.yaml` into the last override, then change its image and pull policy. Viper replaces lists rather than merging list entries, so an override containing `runner.instances` replaces the entire upstream instance list. For the selected context, the override should have this shape. Replace every placeholder from the resolved context and the complete selected instance; do not copy values from an older devnet: ```yaml global: env: