dftracer-pipeline

name	dftracer-pipeline
description	Interactive dftracer annotation pipeline. Clones the app, builds it, annotates all source files using the clang-based MCP annotation tools (clang_annotate_project / clang_annotate_file + clang_syntax_check + clang_lint_annotations), builds the annotated version, runs a smoke test, collects traces, and produces optimization proposals.

Lessons file: /workspaces/dftracer-agents/.agents/skills/dftracer-annotation-lessons/LESSONS_LOG.md (load the compact rules with skill_load(name="dftracer-annotation-lessons"); load the accumulated entries with skill_load(name="dftracer-annotation-lessons", file="LESSONS_LOG.md"))

Read the lessons file before doing anything else. Apply every lesson that matches the current app or language.

══════════════════════════════════════════════════════════════════════ !! ANNOTATION MODE — MANDATORY RULE (READ BEFORE ANY OTHER STEP) !! ══════════════════════════════════════════════════════════════════════

ALWAYS annotate using MCP clang tools. NEVER do manual annotation.

CORRECT flow: 1. clang_annotate_project() ← primary: annotates all files at once 2. clang_syntax_check() per file ← validation 3. clang_lint_annotations() per file← validation 4. If check/lint FAILS for a specific function → call clang_annotate_file() with that function excluded or with comp_overrides to correct it. This is "manual correction" — targeted, single-function only.

FORBIDDEN at all times (do NOT do these, ever): ✗ Read a source file → manually compose macros → Write/Edit the file ✗ Use Bash gcc/g++ -fsyntax-only to check annotations ✗ Call session_annotate_c_file / session_annotate_cpp_file (deprecated) ✗ Rewrite or re-annotate an entire file with Edit/Write tools ✗ Fall back to "manual mode" when an MCP call fails — instead, re-call clang_annotate_file() for only the failing function with overrides.

If clang_annotate_project itself fails to run (tool error, not annotation error): diagnose the tool error, do NOT switch to manual annotation. Report the tool failure to the user and stop.

PYTHON AI/ML ANNOTATION — FUNCTIONS TO NEVER ANNOTATE:

The following Python patterns must not receive dftracer decorators. Remove any auto-placed decorators from them before running:

✗ @staticmethod functions must NEVER carry @_dlp.log_static — dftracer's decorator passes self as the first positional arg via *args, causing "multiple values for argument" errors when the static method has keyword parameters. Instead instrument the body with a CONTEXTUAL region (dft_fn and the dft_ai objects are context managers):

    @staticmethod
    def f(...):
        with DFTracerFn("<cat>", name="f"):      # generic
            ...
    @staticmethod
    def backward(...):
        with dft_ai.compute.backward():          # semantic: keep the AI API
            ...

`python_annotate_file` / `python_annotate_ai_file` now emit this
automatically; `validate_annotations` flags any surviving @log_static.

✗ @numba.njit / @numba.jit / @cuda.jit compiled kernels — numba CPUDispatcher objects do not support inspect.getfullargspec; dftracer's log decorator fails at decoration time with "TypeError: unsupported callable".

✗ @torch.jit.script decorated functions — same reason as numba.

✗ len on Dataset subclasses — DataLoader calls len O(10000+) times per epoch; annotating it overflows dftracer's C-level event buffer and causes SIGABRT in DataLoader workers. Leave len unannotated.

DataLoader worker segfault (on cleanup, or during dftracer finalize)? Two known root causes and the full gdb/core-dump debugging procedure are in a separate reference file — load it only if this crash actually occurs:

skill_load(name="dftracer-pipeline", file="dataloader_crash_debugging.md")

For either root cause, the fix is the same: simply skip the decorator; the function will still be called normally and its callers (which ARE annotated) will capture the timing.

DFANALYZER PRESET SELECTION — DLIO vs POSIX:

dfanalyzer supports two analysis presets: • posix — generic POSIX I/O workload (default for C/C++/Fortran HPC apps) • dlio — deep learning workload (understands epoch/fetch_data/data_loader/ checkpoint/compute layers; use for PyTorch/TF/JAX/DALI/horovod apps)

The pipeline auto-detects the preset via _detect_analyzer_preset(): • If source code imports any of: torch, tensorflow, jax, keras, flax, mxnet, horovod, deepspeed, megatron, FSDP, dali, dlio_benchmark, lightning → preset is automatically set to dlio • Otherwise → posix

When calling mcp__dftracer__analyze manually, always pass the correct preset: • DL workload: analyzer/preset=dlio • Generic HPC: analyzer/preset=posix

The dlio preset produces semantically richer bottleneck names (e.g. reader_posix_read_ops_slope instead of posix_read_ops_slope for DataLoader workers) and understands training-phase patterns that posix does not.

══════════════════════════════════════════════════════════════════════

SESSION HYGIENE — SKILL UPDATES AND CONTEXT MANAGEMENT:

While waiting for long-running jobs (smoke test, dftracer run, optimization iteration), always use the idle time to update skill files with lessons learned so far in the session. Do not wait until the end — capture pitfalls as soon as they are resolved so they are not lost to context compaction.

Protocol: 1. After any bug fix or new pitfall discovered, immediately update the relevant skill file (.claude/commands/*.md) with the lesson. 2. While a flux job is running and output has not yet appeared, update skills. 3. If conversation context exceeds ~60%, update all relevant skill files with current lessons, then run /compact to free context before continuing. 4. After /compact, re-read the task list and resume from where you left off.

Skill files to update when relevant: • dftracer-ml-annotate.md — new Python annotation pitfalls • dftracer-pipeline.md — new pipeline protocol rules • flux-alloc.md — new Flux job management lessons

══════════════════════════════════════════════════════════════════════ STEP 1 — GATHER INPUTS (if not supplied via arguments) ══════════════════════════════════════════════════════════════════════

If the user invoked this with named arguments (run_id=…, url=…, etc.), use those directly. Otherwise ask one question at a time and wait:

Q1: "What is the Git URL of the application you want to annotate?" → Store as APP_URL.

Q2: "Which branch or tag? (default: main)" → Store as REF (use "main" if blank).

Q3: "Smoke test command? (leave blank to auto-detect)" → Store as SMOKE_CMD.

Q4: "Extra CMake/configure build flags? (leave blank to skip)" → Store as EXTRA_FLAGS.

If a run_id was supplied skip Q1–Q4 and jump to Step 3.

Print: "Starting pipeline for <APP_URL> @ "

Note: every MCP tool call that runs a pipeline step returns timing fields started_at, ended_at, and duration_s in its result. Collect these into a running STEP_TIMINGS list as steps complete:

STEP_TIMINGS = []   # append {step, started_at, ended_at, duration_s} after each step

══════════════════════════════════════════════════════════════════════ STEP 2 — SESSION SETUP (MCP tools) ══════════════════════════════════════════════════════════════════════

2a. Create session and build the original source:

session_create(url=APP_URL, ref=REF)
→ store RUN_ID, WS (workspace path)

STRICT RULE: every step from here on writes only into the paths this
session owns (baseline/, annotated/, opt<n>/, artifacts/, tmp/,
dataset/) — never a path a step invents on its own. Get exact paths from
session_get_run_paths or session.json["paths"], not by hand-building
strings like "ws/build_ann". See dftracer-cheatsheet S0.

2b. Check HDF5 version before configuring:

h5cc --version 2>/dev/null || h5pcc --version 2>/dev/null || \
  find /usr -name "H5public.h" | xargs grep H5_VERS_INFO 2>/dev/null | head -1

REQUIRED: HDF5 ≥ 1.14.x. If the system HDF5 is 1.10.x or 1.12.x:
  - Build HDF5 1.14 from source into <WS>/hdf5_1.14/ (see dftracer-install skill)
  - Add "-DHDF5_DIR=<WS>/hdf5_1.14" to EXTRA_FLAGS for all cmake steps
  - Set HDF5_DIR and LD_LIBRARY_PATH in every subsequent shell command

HDF5 1.14 unlocks: H5Pset_page_buffer_size with MPIO VFD,
async VOL (H5Fcreate_async), improved collective metadata flush,
and the full posix_close_ops_slope fix path.

2c. Configure + build the original source:

session_configure(run_id=RUN_ID, extra_cmake_flags=EXTRA_FLAGS)
session_build_install(run_id=RUN_ID)

2c. Install dftracer into the session (cmake mode, with MPI + HDF5 auto-detected from the project source):

session_install_dftracer(run_id=RUN_ID)

On failure → print the cmake/pip error and stop.

IMPORTANT for Python/AI/ML apps: dftracer and the app MUST share the
same venv (``ws/install/``).  ``session_install_dftracer`` enforces this
automatically for ``build_tool=python`` projects — it installs dftracer
into ``ws/install/`` and never creates a separate ``ws/venv/``.
If the app venv does not exist yet, it is created by this step.

2d. Copy source to annotated/ workspace:

session_copy_annotated(run_id=RUN_ID)

2e. Baseline annotated build (no macros yet — verifies the build system patch works before any annotation):

session_build_annotated(run_id=RUN_ID,
  extra_cmake_flags=<same flags as 2b>)

On failure → show cmake/make errors and stop.

Print: "Setup complete. RUN_ID=<RUN_ID> Baseline build PASSED."

2f. Validate session structure before annotating anything:

session_validate_structure(run_id=RUN_ID)

If clean=false → session_reorganize_structure(run_id=RUN_ID, dry_run=False)
then re-run session_validate_structure to confirm clean=true before
proceeding to Step 3. Never annotate into a drifted workspace.

══════════════════════════════════════════════════════════════════════ STEP 3 — WHOLE-PROJECT ANNOTATION (clang MCP tools) ══════════════════════════════════════════════════════════════════════

The preferred path is a single project-level call that discovers all C/C++ files, determines entry points, filters trivial functions by AST cost, and inserts macros in bottom-to-top line order:

clang_annotate_project(
  run_id      = RUN_ID,
  language    = "c",          # or "cpp" for C++ projects
  init_args   = "NULL, NULL, NULL",
  exclude_patterns = ["test/", "tests/", "vendor/", "third_party/"]
)

This call: • Discovers every .c / .cpp / .cxx / .cc under annotated/ • Skips files matching exclude_patterns (plus the always-excluded /CMakeFiles/, /.git/ paths) • Annotates library/inner files first (is_entry=False) • Annotates entry-point files last (is_entry=True) so INIT/FINI land around main() • For each file, internally calls clang_extract_functions to get an authoritative function map, then clang_estimate_function_cost per function to decide annotate vs. skip (score ≥ 20 or lifecycle rule) • Inserts macros in a single in-memory pass (no intermediate writes) • Writes each file exactly once via clang_write_annotated_file

Print the result: number of files annotated, functions annotated vs skipped.

When to use per-file annotation instead

If clang_annotate_project reports errors for specific files, OR if you need comp= overrides for particular functions, switch to per-file mode for those files only:

# 1. Extract the function map
clang_extract_functions(run_id=RUN_ID, filepath=<file>)

# 2. For any function where you want to override the auto comp:
clang_estimate_function_cost(run_id=RUN_ID, filepath=<file>,
  function_name=<name>)
# → review the cost_info and decide comp= category manually

# 3. Annotate the file with optional overrides
clang_annotate_file(
  run_id         = RUN_ID,
  filepath       = <file>,
  is_entry       = <True if file contains main()>,
  language       = "c",        # or "cpp"
  init_args      = "NULL, NULL, NULL",
  comp_overrides = '{"fn_name": "comm", "other_fn": "io"}'
)
# clang_annotate_file writes the file in one in-memory pass;
# call clang_write_annotated_file only if the tool says it is
# needed to flush (check the response).

Do NOT manually read files, insert macros with Edit/Write, or run shell gcc -fsyntax-only commands. All of that is handled by the MCP tools.

══════════════════════════════════════════════════════════════════════ STEP 4 — PER-FILE VALIDATION (clang MCP tools) ══════════════════════════════════════════════════════════════════════

After annotation (project-level or per-file), validate every annotated C/C++ file using the two MCP validation tools. Run both for each file:

clang_syntax_check(run_id=RUN_ID, filepath=<file>)
clang_lint_annotations(run_id=RUN_ID, filepath=<file>)

clang_syntax_check rules: • Uses the real gcc/g++ front-end with a dftracer stub header and the session's MPI + dftracer include paths — no manual -I flags needed. • PASS → move to next file. • FAIL → fix ONLY the exact lines named in the compiler error output: - Extract function name and line number from "error:" lines. - Call clang_annotate_file again for ONLY that function using comp_overrides (e.g. to skip it or override the comp= value). - NEVER rewrite the whole file manually. NEVER touch functions that already pass syntax check. - Retry clang_syntax_check. Max 2 targeted fixes per file. - On 2nd failure: strip the single failing function's macros by re-calling clang_annotate_file with that function excluded, then mark it as PENDING in a comment. Move on.

clang_lint_annotations rules check: L1 — DFTRACER_C_INIT before DFTRACER_C_FUNCTION_START in main() L2 — comp= UPDATE_STR within 3 lines after every START L3 — DFTRACER_C_FINI before MPI_Finalize in main() L4 — no END immediately before MPI_CHECK / NCMPI_CHECK L5 — no END at global scope

LINT violations → use clang_insert_line to fix ONLY the reported line; never re-annotate the whole file. Then re-lint to verify.

Print per-file status: ✓ ( functions annotated, lint PASSED)

══════════════════════════════════════════════════════════════════════ STEP 5 — BUILD ANNOTATED VERSION (MCP tools) ══════════════════════════════════════════════════════════════════════

5a. Set DFTRACER_INIT mode:

Primary (always try first):
  DFTRACER_INIT_ENV = {"DFTRACER_INIT": "FUNCTION"}

FUNCTION mode works for both C/C++ and Python:
- C/C++: DFTRACER_C_INIT / DFTRACER_C_FINI macros in source
- Python: dftracer.initialize_log() / _dft_log.finalize() + decorators

Fallback (only if FUNCTION produces an empty trace or crashes):
  dftracer_lib=$(python -c "import dftracer; import os; print(os.path.join(os.path.dirname(dftracer.__file__), 'lib', 'libdftracer_preload.so'))")
  DFTRACER_INIT_ENV = {"DFTRACER_INIT": "HYBRID",
                       "LD_PRELOAD": "<dftracer_lib>"}

PRELOAD-only mode is never used — annotations must always be present.

Important: NEVER set DFTRACER_INIT=0 — it disables POSIX-level tracing.
All values are CASE-SENSITIVE uppercase strings.

5b. Build and install annotated version:

session_build_annotated(run_id=RUN_ID,
  extra_cmake_flags=<same flags as 2b>)

On failure:
  1. Extract failing function(s) from the compiler error.
  2. Re-annotate only those files using clang_annotate_file with
     the failing function excluded (add it to comp_overrides with
     a sentinel, or use exclude).
  3. Re-run syntax check + lint for the fixed file.
  4. Retry session_build_annotated. Max 2 retries.
  5. If still failing → escalate to user with exact error lines.

5c. Run smoke test:

session_run_smoke_test(run_id=RUN_ID, command=SMOKE_CMD,
  subfolder="build_ann")

MPI/OpenMPI as root? Add env:
  OMPI_ALLOW_RUN_AS_ROOT=1, OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1

Flux proxy systems (Tuolumne, etc.): if SMOKE_CMD contains ``flux proxy``,
the MCP tool automatically wraps the payload in a script under
``<ws>/tmp/run_smoke_test.sh`` that sources lmod init and then runs the
command.  Never pass inline ``bash -c "module load ..."`` to flux proxy —
it fails to propagate module state into subprocesses.

On failure: if DFTRACER symbols in error → re-annotate + retry.
Otherwise ask: "Smoke test failed (non-annotation issue). Continue? [yes/stop]"

══════════════════════════════════════════════════════════════════════ STEP 6 — ANNOTATION REPORT + USER CONFIRMATION ══════════════════════════════════════════════════════════════════════

session_annotation_report(run_id=RUN_ID)

Print: ┌─────────────────────────────────────────────────────────┐ │ ANNOTATION REPORT — <RUN_ID> │ │ Files: annotated │ │ Functions: / skipped: │ │ comp: io= comm= mem= cpu= │ │ Build: PASSED Smoke test: PASSED │ │ Annotated source: workspaces/<RUN_ID>/annotated/ │ ├─────────────────────────────────────────────────────────┤ │ STEP TIMINGS (phase 1) │ │ Step Duration │ │ step_1_clone N.NNNs │ │ step_2_detect N.NNNs │ │ step_3_configure N.NNNs │ │ step_4_build_install N.NNNs │ │ step_5_smoke_test N.NNNs │ │ step_6_copy_annotated N.NNNs │ │ step_7_patch_build N.NNNs │ │ step_8_annotate N.NNNs │ │ Timing file: workspaces/<RUN_ID>/step_timings.json │ └─────────────────────────────────────────────────────────┘

Read timing data from the step_timings field in the tool result (or from workspaces/<RUN_ID>/step_timings.json after the pipeline completes) and append each entry to STEP_TIMINGS.

Ask: "Proceed with dftracer trace run? [yes / no / fix ]"

"no" → stop, print artifact location. "fix " → re-annotate that file using clang_annotate_file with comp_overrides derived from the feedback, re-run lint + syntax check, rebuild, re-run smoke test, show updated report, ask again. "yes" → continue to Step 7.

══════════════════════════════════════════════════════════════════════ STEP 7 — TRACE COLLECTION + ANALYSIS (MCP tools) ══════════════════════════════════════════════════════════════════════

7a. Create the trace output directory before running:

On LLNL systems (Tuolumne, Lassen, etc.) all trace output MUST go to
Lustre, not NFS.  ``session_run_with_dftracer`` auto-routes to Lustre
when ``/p/lustre5/$USER/workspaces/`` exists.  Verify before running:

    mkdir -p /p/lustre5/$USER/workspaces/<app>/{traces,fractals,datasets,runs}

``session_run_with_dftracer`` writes ``DFTRACER_LOG_FILE`` to the Lustre
path and creates a symlink at ``<WS>/traces/`` for downstream tools.
If Lustre is unavailable (containers, non-LLNL), traces land in ``<WS>/traces/``.

7b. Run with dftracer:

session_run_with_dftracer(run_id=RUN_ID, command=SMOKE_CMD,