workload-h5bench

name	workload-h5bench
description	H5Bench-specific knowledge: CMake build quirks, annotation edge cases (assert/else-if brace insertion), INI config format, dftracer integration, and Cray HDF5 compatibility. Load this skill whenever working with h5bench.

Cross-references: [[dftracer-annotation-lessons]] [[dftracer-pipeline]] [[software-hdf5]] [[software-mpi]]

Source repo: https://github.com/hariharan-devarajan/h5bench

---

Binary Names (look here first — wrong name = silent failure on compute nodes)

All h5bench binaries live under build_ann/ (annotated) or build/ (original). The binary names do NOT always match the workload label. Use this table every time:

Workload label	Binary name
write	h5bench_write
read	h5bench_read
append	h5bench_append
overwrite	h5bench_overwrite
write_unlimited	h5bench_write_unlimited
hdf5_iotest	h5bench_hdf5_iotest
exerciser	h5bench_exerciser

CORRECTION (confirmed 2026-07-10 against hariharan-devarajan/h5bench @ master): There is NO write_normal_dist / h5bench_write_var_normal_dist binary or config variant in this fork's current CMakeLists.txt or source tree — grepping the whole source tree for normal_dist/var_normal/NORMAL returns nothing. This entry was stale (likely inherited from a different h5bench fork/version) — do not plan around it; it does not exist. The real, buildable workload set from this fork's add_executable(...) calls is: write, write_unlimited, overwrite, append, read, hdf5_iotest, exerciser (7 total, not 8).

Always verify with:

ls <WS>/build_ann/h5bench_*

hdf5_iotest and exerciser are OFF by default — need explicit CMake flags

CMakeLists.txt gates both behind options that default OFF:

option(H5BENCH_EXERCISER "Enable Exerciser benchmark" OFF)
option(H5BENCH_METADATA  "Enable Metadata benchmark"  OFF)   # gates hdf5_iotest
option(H5BENCH_ALL       "Enable all benchmarks"      OFF)   # turns both ON

A plain cmake -S source -B build (no extra flags) silently builds only 5 binaries (write, write_unlimited, overwrite, append, read) with no error or warning about the missing exerciser/hdf5_iotest targets. Always pass -DH5BENCH_ALL=ON (or the two individual -DH5BENCH_EXERCISER=ON -DH5BENCH_METADATA=ON flags) to get the full binary set, and verify all 7 are present after build — don't assume a clean make output means every target was configured.

---

Build

Cray HDF5 chid_t typo breaks dftracer/brahma (C++ frontend)

/opt/cray/pe/hdf5-parallel/1.14.3.7/cray/20.0/include/H5Apublic.h:932:29:
error: unknown type name 'chid_t'; did you mean 'hid_t'?
gmake[5]: *** [CMakeFiles/brahma.dir/...] Error 1

Root cause: Cray-patched HDF5 (and upstream 1.14.3) have a typo on line 932 of H5Apublic.h — H5Aread_async declares chid_t instead of hid_t. Only the C++ compiler rejects it; IOR's C frontend tolerates it.

Fix:

# Download vanilla HDF5 1.14.3 (GitHub 404s on this system; use HDF Group FTP):
curl -fkL https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.14/hdf5-1.14.3/src/hdf5-1.14.3.tar.gz \
  -o hdf5-1.14.3.tar.gz
tar xf hdf5-1.14.3.tar.gz && cd hdf5-1.14.3
CC=mpicc ./configure --prefix=<ws>/hdf5_1.14 --enable-parallel \
  --enable-shared --enable-build-mode=production --with-zlib=/usr
make -j8 && make install

# Patch the typo:
sed -i 's/H5Aread_async(chid_t attr_id/H5Aread_async(hid_t attr_id/' \
  <ws>/hdf5_1.14/include/H5Apublic.h

# Update session.json to point at <ws>/hdf5_1.14 and re-run session_install_dftracer.

Still present in 1.14.5: confirmed 2026-07-10 that vanilla HDF5 1.14.5 (not just 1.14.3) still ships the same chid_t typo at H5Apublic.h:926 — this is not yet fixed upstream. Apply the same sed patch regardless of which 1.14.x version is fetched; don't assume newer patch releases have resolved it.

dftracer build requires Cray PE runtime libs on LD_LIBRARY_PATH, not just at link time

When dftracer/h5bench binaries are built with cce/20.0.0, running them later needs the same Cray runtime lib dirs on LD_LIBRARY_PATH, not just during the build: /opt/cray/pe/cce/20.0.0/cce/x86_64/lib and /opt/cray/pe/cce/20.0.0/cce-clang/x86_64/lib (for libmodules.so.1, libfi.so.1, libcraymath.so.1, libf.so.1). Write these into the same setup_dftracer_env.sh wrapper used for build, and source it before every run/smoke test too — not just before session_install_dftracer/session_build_annotated.

Note: MPI compatibility warning — MPICH 9.0.1 is outside brahma's tested range; MPI-IO interception is disabled but POSIX and app-level annotation tracing work.

---

CMake library name mismatch: -ldftracer vs -ldftracer_core

/usr/bin/ld: cannot find -ldftracer: No such file or directory

Root cause: dftracer installs as libdftracer_core.so, not libdftracer.so. session_install_dftracer patches CMake to link -ldftracer which fails.

Fix: After session_install_dftracer, patch build files:

sed -i 's/-ldftracer\b/-ldftracer_core/g' build_ann/CMakeCache.txt
find build_ann/CMakeFiles -name "link.txt" \
  -exec sed -i 's/-ldftracer\b/-ldftracer_core/g' {} \;
grep -r "ldftracer[^_]" build_ann/   # must return nothing

Link order: -ldftracer_core must appear AFTER all .o files. When CMake places it before objects (via CMAKE_EXE_LINKER_FLAGS), move it:

sed -i 's/ -ldftracer_core//g' link.txt
# Then append it at the END of the same line as the cc command (not a new line).

Important: Do NOT put -ldftracer_core on its own line in link.txt. cmake -E cmake_link_script treats each line as a separate command; a line with only -ldftracer_core is silently ignored.

---

Patching CMakeCache.txt triggers MPI re-detection failure

Root cause: Editing CMakeCache.txt causes cmake to re-run configure, which fails to find MPI in a container environment.

Fix: After editing CMakeCache.txt, add entries to skip MPI re-detection:

MPI_C_WORKS:BOOL=TRUE
MPI_CXX_WORKS:BOOL=TRUE

---

Annotation Pitfalls

clang_add_braces corrupts assert() macro call-sites

assert(pconfig->version ==
{
0)
}
Compiler: "error: expected ')' before '{' token"

Root cause: glibc's assert(expr) expands to an IfStmt with a NullStmt then-body. _collect_braceless treated the NullStmt as an unbraced body and split multi-line macro arguments.

Fix in source_parser.py _collect_braceless():

if kind == "IfStmt":
    _then_is_null = (len(inner) >= 2 and inner[1].get("kind") == "NullStmt")
    for i, child in enumerate(inner):
        if i == 0: continue  # condition
        if _then_is_null: continue  # assert()-style — skip ALL bodies

---

clang_add_braces inserts standalone { before else if

} else
{
if (condition) {
Compiler: "error: expected expression before '{' token"

Root cause: In _collect_braceless, else-if bodies (index ≥ 2 that are themselves IfStmt) were being wrapped, inserting { before the else keyword.

Fix in source_parser.py _collect_braceless():

if i >= 2 and child.get("kind") == "IfStmt":
    continue  # else-if: skip wrapping; recursion handles inner IfStmt

---

DFTRACER_C_INIT(NULL, NULL, -1) causes segfault

Segmentation fault in initialize_main() at fgets call immediately after startup.

Root cause: The third argument is cast to (int*)0xffffffffffffffff → segfault.

Fix: Always use NULL for the process_id argument:

DFTRACER_C_INIT(NULL, NULL, NULL)

---

DFTRACER_C_METADATA / DFTRACER_CPP_METADATA are 3-arg macros, not 2-arg

annotate_add_app_metadata (C dialect) previously emitted the 2-arg form DFTRACER_C_METADATA("app", "h5bench_write"), which fails to compile — the real macro in dftracer/include/dftracer/dftracer.h is DFTRACER_C_METADATA(name, key, val) (3-arg; DFTRACER_CPP_METADATA has the same 3-arg shape). name is a bare C identifier, not a string literal — it's used internally for ##name token-pasting to declare a local variable, so it must be unique within scope.

Confirmed 2026-07-10 on h5bench_write.c; manually patched to:

DFTRACER_C_METADATA(dft_meta_app, "app", "h5bench_write");

Fixed permanently in the annotate_add_app_metadata MCP tool (src/dftracer_agents/mcp_tools/tools/session/annotation_validate.py), which now generates a unique dft_meta_<key>-style identifier per metadata call — no more hand-patching needed in future sessions.

---

hdf5_iotest.c and h5bench_exerciser.c annotate cleanly

Confirmed 2026-07-10: unlike h5bench_write.c, neither file triggers the clang_add_braces assert()/else-if brace-insertion issues above — no special handling needed for these two files.

---

clang_syntax_check needs explicit include dirs for this session's toolchain

In this session's environment, clang_syntax_check only validates cleanly when extra_include_dirs is passed pointing at BOTH:

• the venv's dftracer headers, e.g. <WS>/venv/lib/python3.13/site-packages/dftracer/include
• the session's source-built HDF5 include dir: <WS>/hdf5_1.14/include

Without both, syntax check fails to resolve dftracer/dftracer.h and/or HDF5 headers even though the actual build succeeds (the build's own Makefile/CMake already wires these paths; clang_syntax_check does not inherit them automatically).

---

Config Format

h5bench_write expects INI key=value, NOT the JSON sample files

Passing a JSON file from samples/ causes a segfault inside fgets(). The JSON files are for the Python runner (h5bench.py); the binary needs INI:

cat > /tmp/h5bench.cfg << 'EOF'
MEM_PATTERN=CONTIG
FILE_PATTERN=CONTIG
TIMESTEPS=3
DELAYED_CLOSE_TIMESTEPS=0
COLLECTIVE_DATA=NO
COLLECTIVE_METADATA=NO
NUM_DIMS=1
DIM_1=1048576
DIM_2=1
DIM_3=1
EOF
mpirun -np 2 ./h5bench_write /tmp/h5bench.cfg /tmp/test.h5

h5bench_read needs a PRE-EXISTING file matching its config dims — it does not create one

h5bench_patterns/h5bench_read.c (main(), argv[2] = "data file to read") opens argv[2] and expects a dataset already shaped per the config's NUM_DIMS/DIM_1/etc. Pointing it at a fresh/empty path does NOT fail loudly — it produces an HDF5 error-storm (H5Sget_simple_extent_dims: "not a dataspace", H5Dclose: "not a dataset ID", H5Gclose: "not a group ID", ...) yet the process still exits 0 and dftracer still writes a (bogus) trace. A baseline collection run can look "successful" (5/5 reps captured, non-empty trace) while every rep is silently measuring an error path, not real read I/O — always grep the run log for these signatures before trusting a read baseline.

Fix: always run a two-phase write-then-read pattern, never point h5bench_read at a fresh/empty path:

# Phase 1 (untraced setup): create the file read.cfg expects, matching dims/pattern
DFTRACER_ENABLE=0 ./h5bench_write write_for_read.cfg read.h5
# Phase 2 (traced benchmark): now the file exists with the right shape
DFTRACER_ENABLE=1 DFTRACER_LOG_FILE=... ./h5bench_read read.cfg read.h5

write_for_read.cfg should use MEM_PATTERN=CONTIG/FILE_PATTERN=CONTIG and the same NUM_DIMS/DIM_1 as read.cfg. Only enable dftracer for phase 2 so the trace reflects the read benchmark, not the setup write.

---

Dataset Sizing (memory threshold rule)

Smoke tests and trace collection MUST move >50% of each node's physical memory to/from the filesystem to avoid OS page-cache effects (see [[dftracer-annotation-lessons]] R9).

Tuolumne (AMD MI300A APU), 192 ranks, 2 nodes:

• MemTotal per node: ~502 GiB
• Required total data: >502 GiB
• DIM_1=33554432 (32M float32) × 192 ranks × 4B × 4 timesteps = 768 GiB ✓
• DIM_1=16777216 (16M float32) × 192 ranks × 4B × 4 timesteps = 384 GiB ✗