software-pecan

name	software-pecan
description	PECAN (Pose Classification, part of PECAN/MILAN pose-classification/binding-affinity codebase) — PyTorch+PyTorch-Geometric EGNN training on HDF5 shard data. Build, annotation status, dataset access, and measured dftracer optimization findings. Load this skill for any PECAN/PDBspheres/PDBBind session.

Cross-references: [[dftracer-io-optimization]] [[dftracer-compute-optimization]] [[dftracer-communication-optimization]] [[dftracer-memory-optimization]] [[system-tuolumne]]

PECAN is annotated/instrumented by dftracer (this skill uses software- naming, not workload-, per the software-vs-workload naming convention). Source repo: https://czgitlab.llnl.gov (kim63/pecan_milan), sibling app MILAN in the same repo shares this skill's build/env notes but has a separate (unannotated) codepath — see below.

Codebase layout — which files are actually the baseline path

pecan/trainer.py + pecan/dataset.py (Dataset_PDB) are the ANNOTATED, ACTIVE baseline path for the PDBspheres/PDBBind pose-classification workload. milan/*.py, *_old.py, and pecan/dataset_dyad*.py/pecan/datacopy_dyad*.py (DYAD in-transit staging path) are NOT annotated and NOT part of the default local-I/O run — don't be misled by grep/graph-BFS neighbors into thinking they're in scope. model/egnn.py (the nn_type=3 EGNN model) only needs torch_geometric core, NOT torch_scatter/torch_sparse — but model/model_trainer.py unconditionally imports ALL model variants at module load (including sgcnn.py/ggcnn.py, which DO need torch_scatter/torch_sparse), so both extensions must still be built even though only EGNN is exercised at runtime.

Build (Tuolumne / MI300A / ROCm 6.3.1)

• torch ROCm wheels (repo.radeon.com/rocm/manylinux/rocm-rel-6.3.1/) only exist for cp39/cp310/cp311 — use python/3.11.5 module, NOT the Tuolumne-default python/3.13.2.
• torch_scatter/torch_sparse (vendored source at pyg-rocm-build/{pytorch_scatter-2.1.2,pytorch_sparse-0.6.18} in the parent install repo, copy before building — never build against a read-only source tree) must be built with PYTORCH_ROCM_ARCH=gfx942 (MI300A only) — without it, HIP compiles for 12 architectures and takes 30-60+ min per package instead of ~5-8 min.
• torch_scatter builds fine with Cray mpic++ (CC=mpicc CXX=mpic++); torch_sparse's CPU extensions use -fopenmp and crash at IMPORT time with undefined symbol: _cray$mt_kmpc_fork_call_with_flags when linked with Cray's OpenMP runtime — rebuild with module load gcc-native/13; CC=gcc CXX=g++ (plain system GCC) instead.
• Both packages' setup.py install post-build step throws a spurious error: [Errno 2] No such file or directory on this filesystem (NFS-ish rename race) — harmless; the wheel is already in dist/*.whl by that point, pip install it directly.
• pip install pyyaml separately — not pulled in by any other PECAN dependency, but main_app.py's load_config() needs it.

Dataset access — a real per-file permission gotcha, not directory-wide

PDBspheres/PDBBind data on VAST ($VAST_ROOT/data_pecan/, or a differently-permissioned copy at $VAST_ROOT/... owned by a different group). The pre-generated CSV index (pdbspheres_all_0000000_all.csv) can be OWNER-ONLY rw------- even though the actual .hdf5 shard files in the SAME directory are group-readable rw-rw----. If the index CSV is inaccessible, don't give up — pass train_fns (a list of .hdf5 file paths) instead of train_csvs; Dataset_PDB.__init__'s h5-driven fallback path derives the same metadata directly from the HDF5 files and auto-generates its own index CSV.

Real app bugs found and fixed (2026-07-20, keep this code path fixed going forward)

1. Dataset_PDB.__init__'s h5-driven path writes its auto-derived index CSV back into the SAME directory as the source .hdf5 files (fn_prefix + "_all.csv") — always fails with PermissionError when the dataset is on a read-only-to-this-session PFS mount. Fixed via a PECAN_INDEX_CSV_DIR env var (falls back to original behavior, then tempfile.gettempdir() on PermissionError) so the derived index lands in a writable location instead.
2. The crystal-structure branch of Dataset_PDB.__init__ (h5-driven path, use_crystal=True) appended a 6-element list while __getitem__ unpacks 7 — crashed with ValueError: not enough values to unpack (expected 7, got 6) the first time a crystal-structure entry was iterated. Fixed by adding the missing trailing 0.0 (score) field.

DDP launch on Tuolumne (flux)

• flux run -N<nodes> -n<total_tasks> -g<gpus_per_task> is the ONLY valid flag combo — mixing --tasks-per-node (a "per-resource" option) with --gpus-per-task (a "per-task" option) errors Per-resource options can't be used with per-task options.
• MASTER_ADDR MUST be resolved identically on every rank — flux getattr local-uri returns a local:// socket path, NOT a hostname; using it (or each rank's own $(hostname)) makes every rank compute a DIFFERENT MASTER_ADDR, silently hanging NCCL/RCCL rendezvous forever with ZERO output (Python's stdout is block-buffered when non-tty, so a live-but-hung job looks identical to a not-yet-started one). Correct recipe (every rank agrees on the same first-node hostname):

  NODELIST=$(flux job info $FLUX_JOB_ID R | python3 -c \
    "import json,sys; print(json.load(sys.stdin)['execution']['nodelist'][0])")
  export MASTER_ADDR=$(flux hostlist -n 0 "$NODELIST")
  

• Always run multi-rank flux-launched Python with python -u (unbuffered) — a live 16-rank job can look completely silent for 1-2 minutes while torch/torch_geometric/h5py/dftracer import across the network filesystem on all ranks concurrently; without -u you can't tell "slow import" from "hung."
• dftracer_service node-counter daemon — use -n<n> -c1, NOT --tasks-per-node 1. flux run -N<n> --tasks-per-node 1 dftracer_service start <dir> reserves ALL requested nodes EXCLUSIVELY by default on this flux config ("exclusive": true in the resolved jobspec, even with no --exclusive flag passed) — it silently blocks the training job (or a later stop job) from co-scheduling onto those same nodes. Fixed recipe (validated 2026-07-20):

  flux run -N<n> -n<n> -c1 dftracer_service start <dir>   # 1 task/node, 1 core each — NOT exclusive
  flux run -N<n> -n<n> -c1 dftracer_service stop <dir>    # same pattern to stop
  

  Verify with flux job info <id> jobspec (no "exclusive": true) and flux resource list before assuming co-location works. Also confirmed: the per-hostname .pid file fix in dftracer's 3e6fc42 develop commit (dftracer_server_<hostname>.pid, replacing a single shared dftracer_server.pid that used to cause "No running server found") is real — stop now cleanly sends SIGINT to every node's server once combined with the non-exclusive invocation above.
• Before crediting any optimization-variant wall-time delta, verify the variant run matches the baseline's SCALE (rank/process count) and actually COMPLETED (no job.exception cancel in its event log). A half-scale or cancelled run's wall time is worthless for comparison — fall back to work-normalized per-operation ratios (e.g. opens-per-__getitem__), which are scale- and truncation-robust, when a clean equal-scale re-run isn't available before the allocation expires.

Measured dftracer findings (2026-07-20, 16-rank / 4N x 4GPU DDP, PDBspheres_v2_split8,

150 shards, batch_size=8, num_workers=2, 2 epochs)

Diagnosed bottleneck ranking (real diagnose() output, analyzer_preset="generic"): hdf5 layer CRITICAL and worsening (severity_score=0.988, prevalence=73%) — dominated by H5Oopen (577s cumulative, 4.6M calls) and H5Fopen (490s cumulative, 48,384 calls) — i.e. METADATA/OPEN overhead from Dataset_PDB.__getitem__ re-opening the same small HDF5 shard files fresh on every call, not read bandwidth (H5Dread itself only 58s). Fix: a per-worker h5py.File handle cache (_get_h5, keyed by filepath, with __getstate__ dropping the cache dict so no fd crosses the DataLoader fork) + persistent_workers=True. Validated mechanistically: -76.5% file-open rate (work-normalized H5Fopen/H5Aread ratio, robust to the scale/truncation issues that affected the wall-time measurement — see the flux-alloc note above). communication_io (severity 0.996) is the SAME root cause via a different dft_event_logging wrapper around the same code — expect the handle-cache fix to resolve both together.

GPU compute (model-forward+model-backward, cat=compute) was the single LARGEST per-rank cost once measured directly (776s aggregate, bigger than the entire 412s preprocess bucket) — it did NOT show in any analyzer preset's Layer Breakdown (posix/dlio/generic all lack a compute layer bucket entirely; see dftracer-compute-optimization), which looked like "zero compute cost" but was a preset-bucketing artifact, not a tracing gap. Compute optimization candidates ranked by potential (none measured this session — see dftracer-compute-optimization for the full table): AMP bf16 autocast (gfx942 native bf16) > torch.compile (risk: PyG dynamic shapes) > precompute/cache graph adjacency to disk (invariant across epochs, no augmentation)

MIOpen autotune > torch.cdist swap for pairwise_distances.

communication-except-io/cpu-gpu-transfer and NUMA/memory levers are all confirmed NOT the bottleneck for this app (25.58s aggregate comm cost = <1% of wall; MI300A unified HBM makes pin_memory/staging-copy tuning structurally inert; NUMA/launcher affinity confirmed inert on this system for a 4th workload class by inference from prior KB). Do not re-investigate these dimensions for PECAN without new evidence — I/O (HDF5 metadata storm) dominates by 2+ orders of magnitude and should stay the primary optimization target.

Full 4-dimension optimizer pass (2026-07-21, baseline5 trace, post-handle-cache, 15 real

diagnose() findings, 14 high/critical)

I/O — second-order HDF5 metadata storm confirmed UNDER the handle-cache fix. The per-worker file-handle cache eliminated H5Fopen (1.0 → 0.03/sample, verified) but HDF5 remains the #1 persistent finding (94.7M events / 2020.87s aggregate, persistence=15 — present across nearly the whole run). Root cause is now CLIENT-SIDE h5py PER-CALL OVERHEAD, not file opens: H5Oopen object navigation (~16/sample, h5[pdbid][h5_dcom][poseid] group traversal) + 5 scalar-attribute reads every __getitem__ (num_hbonds/hpbond/habond/sbond/pbond, each dragging H5Aopen_by_name+H5Aget_type+H5Aget_space+H5Sget_simple_extent_dims+H5Aread+ H5Tclose+H5Sclose) + ubiquitous h5py bookkeeping (H5Eset_auto2 6.2M calls = ~51% of all HDF5 events; H5Iis_valid/H5Idec_ref ref-counting). Next levers (cited Mohan et al., VLDB 2021, https://arxiv.org/abs/2007.06775), NOT YET APPLIED — need a validation run before crediting: (a) hoist the 5 scalar attrs into data_list at __init__ time so __getitem__ reads from the list instead of h5_data.attrs (removes ~40-50% of per-sample metadata events every epoch); (b) per-worker decoded-sample cache keyed by (pdbid, poseid) for epoch-2+ reuse (samples are invariant in graph modality — no augmentation there; must guard OFF for modality==1, which DOES apply random affine augmentation per sample). stdio_ops_slope scored critical (0.999) but is a BURST-RATE ARTIFACT, not real overhead — 0.85s total = 0.008% of wall time, 49,919 cheap fopen64/fclose probe events. Do not act on it.

MEASURED 2026-07-22 (baseline5 scale, 4N×4GPU DDP, live validation run, annotated/source/pecan/dataset.py): both (a) attr-hoist and (b) decoded-sample cache were applied TOGETHER (not isolated — allocation time ran out before a per-proposal breakdown). Correctness verified standalone first: 0 mismatches across 50 rows comparing hoisted attr values against direct h5py.attrs reads. The csv_filepaths init branch was confirmed OUT OF SCOPE and left untouched — train_csvs: null in this run's config, so that code path is dead for this workload; the decode-cache guard for modality==1 augmentation is inert for the same reason (active config is nn_type: 3, graph modality, no per-sample augmentation) but was kept in place for safety. Combined measured delta: epoch-2 (steady-state) wall time 31.54s → 28.77s (-8.8%); data-load-h5 event count -49.1% (26,112 → 13,280), mean event duration -14.5%. (Epoch-1 delta was NOT used as the primary metric — confounded by __init__ overhead and a concurrent job sharing the allocation.) Single replicate — treat as directional; a clean isolated #1-vs-#2 breakdown and more replicates are still needed before fully crediting either sub-change.

Incident note (2026-07-22): running the I/O and compute optimizer agents in PARALLEL on separate allocations against the SAME shared annotated/source/ tree caused one NCCL remote process exited crash — the compute agent's concurrent edit to model/model_trainer.py (adding torch.compile) meant different ranks in the I/O agent's job loaded different versions of that file mid-run, breaking DDP sync. Recovered by setting PECAN_TORCH_COMPILE=0 (an env-gate the compute agent had already added) rather than touching the other agent's file. See feedback-shared-source-tree-race for the general lesson.

MEASURED 2026-07-22 (isolation run, hoist-only, baseline5 scale, 4N×4GPU DDP): added a PECAN_DECODE_CACHE env-var gate (default "1" = prior combined-on behavior, no-op unless set) around the per-worker decoded-sample cache in _get_h5/__getitem__ so proposal #2 (decode cache) could be disabled independently of proposal #1 (attr-hoist, left always-on). With PECAN_DECODE_CACHE=0 (attr-hoist ONLY): epoch-2 wall time 31.544s → 33.146s (no measured win, +5.1%, single replicate/noisy); data-load-h5 event count 26,112 → 26,336 (essentially unchanged, +0.9%) — expected, since attr-hoist only shortens per-event work inside __getitem__, it does not skip/merge events like the decode cache does. By subtraction against the combined measurement (both proposals: -8.8% wall time, -49.1% event count), the decode cache (proposal #2) accounts for essentially ALL of the measured win; attr-hoist alone showed no resolvable wall-time benefit at 1 replicate. Treat as directional — needs ≥5 replicates on an isolated (non-shared) allocation before fully crediting either sub-change; do not yet conclude attr-hoist is worthless, only that its effect (if any) is smaller than this run's noise floor.

Pitfall confirmed again (2026-07-22): the first isolation-run attempt hit the exact shared-source-tree race from the incident note above — PECAN_TORCH_COMPILE defaults to "1" in model/model_trainer.py, and a concurrently-running compute-optimizer job on a different allocation was mid-edit on that file, causing an InternalTorchDynamoError: ModuleNotFoundError: No module named 'torch.distributed.tensor._ops' on some ranks and a cascading NCCL DistBackendError. Fix: always export PECAN_TORCH_COMPILE=0 explicitly in any I/O-optimizer run script while a compute-optimizer job may be running in parallel against the same annotated/source/ tree, rather than relying on the default.

RESOLVED 2026-07-22 (replicate confirmation, same allocation window as the isolation run above): ran one more replicate of the combined config (io_opt1_r3: 28.953s, 13,056 data-load-h5 events) and two more replicates of the hoist-only config (hoistonly_r2: 31.9998s/26,112 events; hoistonly_r3: 32.5846s/26,112 events).

• Combined (both proposals ON) — CONFIRMED. 2 replicates: 28.766s and 28.953s (mean 28.859s, CV 0.65%). Delta vs. the single baseline5 measurement (31.54s) is -8.5% (range -8.8% to -8.2%) — the originally-reported -8.8% win is real, not single-replicate noise.
• Hoist-only (attr-hoist alone, PECAN_DECODE_CACHE=0) — CONFIRMED NO WIN, revised from "noisy/directional". 3 replicates: 33.146s, 32.000s, 32.585s (mean 32.577s, stdev ~0.57s, CV 1.8% — low noise). All three land at or above the 31.54s baseline (mean +3.3%); none show a wall-time improvement. data-load-h5 event count is deterministic and IDENTICAL across all 3 hoist-only replicates (26,112) and matches baseline5's own count exactly — consistent with attr-hoist only shortening per-event work inside