dftracer-agents

Communication-component (MPI/collective/network) bottleneck-to-optimization mappings, papers, and L1/L2/L3 strategies for the dftracer optimization pipeline

Compute-component bottleneck-to-optimization mappings, papers, and L1/L2/L3 strategies for the dftracer optimization pipeline

Key literature, bottleneck-to-optimization mappings, and strategies for the dftracer I/O optimization pipeline

Memory-component bottleneck-to-optimization mappings, papers, and L1/L2/L3 strategies for the dftracer optimization pipeline

VPIC-Kokkos-specific knowledge: build/annotation quirks, the two-stage deck-compile pattern, deck sizing for smoke vs validation runs, and the measured MPI-communication-bound diagnosis on Tuolumne. Load this skill whenever working with vpic-kokkos (github.com/lanl/vpic-kokkos).

Install and build dftracer into the session using the MCP install tool and record tool pitfalls early.

Diagnose traces and convert symptoms into actionable bottlenecks with MCP tools.

Cross-session, citation-backed knowledge base of every MEASURED dftracer optimization, partitioned into system-centric (L3), software-centric (L2), and workload-centric (L1) findings. Load this FIRST in any optimization loop to avoid re-deriving what is already known.

Complete C annotation rules for dftracer — START/END placement, comp types, linker setup, coverage verification, and the full checklist

Language-agnostic dftracer annotation rules — Rule 0 (what to skip), coverage, error paths, backend priority order, and wrapper handling

mpifileutils (dcp/dsync/drm/dwalk/dcmp/dtar/dchmod/dstripe/etc.) — parallel, MPI-based replacements for serial Linux filesystem utilities (cp/rsync/rm/ find/diff/tar/chmod) on HPC parallel filesystems. Load this skill whenever a filesystem operation (copy, sync, delete, compare, archive, permission change, restripe, inventory) touches a large (multi-GB to TB-scale) dataset on a cluster/shared parallel filesystem — prefer these over standard serial Linux utilities at that scale.

H5Bench-specific knowledge: CMake build quirks, annotation edge cases (assert/else-if brace insertion), INI config format, dftracer integration, and Cray HDF5 compatibility. Load this skill whenever working with h5bench.

IOR-specific knowledge: build quirks, annotation pitfalls, ROMIO tuning on VAST NVMe storage, and optimal dftracer run configuration for IOR 4.0.0. Load this skill whenever working with IOR.

Critical annotation rules, corner cases, and top-10 known mistakes for dftracer — read this before every annotation session

Install and privilege rules for dftracer sessions — never use sudo, always install to userspace paths; autotools pkg-config integration

Interactive dftracer annotation pipeline. Clones the app, builds it, annotates all source files using the clang-based MCP annotation tools (clang_annotate_project / clang_annotate_file + clang_syntax_check + clang_lint_annotations), builds the annotated version, runs a smoke test, collects traces, and produces optimization proposals.

Reference for running applications with dftracer in PRELOAD or HYBRID mode. Covers correct environment variable setup, DATA_DIR rules, LD_PRELOAD wiring, MPI forwarding, and trace verification. Use before any session_run_with_dftracer call or when debugging missing trace categories (POSIX/HDF5/MPI).

TOP PRIORITY — always use dftracer MCP utils tools for any trace work; never use raw bash/python/gzip scripts to read or process .pfw/.pfw.gz files

MPI and MPI-IO (ROMIO) optimization strategies, Cray MPICH environment variable wiring, flux proxy env propagation rules, and dftracer MPI tracing requirements. Load this skill for any MPI or collective I/O work.

ScaFFold (LBANN PyTorch benchmark, 3D U-Net on procedurally-generated fractals). Python/PyTorch DDP app running on Tuolumne MI300A. Environment setup, shared venv pattern for AI/ML workloads, MPI4PY Cray linkage, ROCm paths.

Lessons learned from dftracer annotation sessions — real errors, root causes, and exact fixes. Loaded by each per-file annotation subagent at startup. Updated by the pipeline recipe after every session (Step 8).

How to use and update the lessons-learned cache at .goose/lessons-learned.md — proactive lookup before any build/install/annotate step

MANDATORY end-of-session privacy validation. Scans everything the session added to the git-tracked trees (memory, lessons, skills, agent definitions) for identifying content — usernames, absolute user paths, emails, flux job ids, session UUIDs, node hostnames — and redacts it deterministically with the privacy_scan / privacy_redact MCP tools. Load this before writing to ANY skill, lesson, memory, or agent file, and run it as the last step of every session.

MANDATORY pipeline self-profiling via MLflow. Every dftracer agent brackets its work with profile_step_begin / profile_step_end so the session records what each step cost in wall time, tokens, dollars, and retries. The main thread binds the session once with profile_bind and writes the report with profile_report. Load this skill before executing any pipeline step.

Flash-X-specific knowledge for the dftracer pipeline — build system (Python setup + GNU Make), Sedov 3D test problem, dangling symlinks in shallow clones, HDF5/flash.par pitfalls. Load this skill whenever working with Flash-X.

The complete dftracer manual annotation process — 5 passes, coverage requirements, annotation_logs format, build prep, and mandatory Pass 5 review

Context/token economy for every dftracer agent. Use the graph tools (graph_query / graph_ensure) to LOCATE code and compute change blast-radius instead of reading whole files into context. Load this before any tool call, annotation, optimization, or pipeline step that would otherwise open source files.

Annotate ML/DL Python workloads end-to-end with dftracer AI/ML region decorators. Detects frameworks and ROCm/HIP requirements, installs dftracer with correct flags, annotates all Python files, builds/runs smoke test, collects traces, and auto-updates the lessons file with any new pitfalls discovered during the session.

DFTracer documentation links, artifact log locations, step numbering, and how to read session logs

HDF5 optimization strategies (L2 software tuning), version compatibility, ALWAYS building HDF5 from source (never Cray/system module), Cray HDF5 pitfalls, and dftracer HDF5 tracing setup. Load this skill for any HDF5 or parallel I/O work.

Project-level annotation router that scopes files and dispatches by language.

Build the original application session with the smallest useful context and record build lessons early.

Project-level router for dftracer stage subagents, model selection, and tool delegation.

Smoke test rules for dftracer — single-process only, environment variables, DFTRACER_INIT conflict warning, and trace file paths

Detect the runtime system, modules, allocation shape, and environment assumptions for dftracer sessions.

Install and run Pegasus workflows (e.g. montage-workflow-v3) on HPC systems without a site-wide HTCondor pool, using Pegasus MPI Cluster (PMC) under a Flux/Slurm allocation instead of Condor+DAGMan. Covers userspace install of Condor (for pegasus-plan tooling only), Pegasus 5.x, rebuilding pegasus-mpi-cluster against the system MPI, site/ properties configuration for PMC-only code generation, dftracer tracing wiring, and every pitfall hit getting this working on Tuolumne (Cray PE / Flux). Load this skill for any Pegasus workflow work, not just Montage.

POSIX I/O optimization strategies: application-level buffering (L1), readahead and library hints (L2), OS/filesystem tuning (L3), and dftracer DATA_DIR configuration for capturing POSIX events. Load this skill for any POSIX I/O bottleneck work.

Rabbit near-node-flash storage accelerators on Tuolumne — DW directives via Flux -S "#DW ...", picking SHM vs XFS vs GFS2 vs Lustre by data-sharing scope, and how to use them as an L3 filesystem optimization to accelerate data access.

C++ annotation rules for dftracer — RAII guard, REGION macros for main, UPDATE usage, and quick checklist

Python annotation rules for dftracer — decorator usage, initialize/finalize, comp types, class methods, and quick checklist