dftracer-io-optimization

name	dftracer-io-optimization
description	Key literature, bottleneck-to-optimization mappings, and strategies for the dftracer I/O optimization pipeline

MANDATORY: Exhaustive Dimension Checklist (walk ALL 12, every session — user instruction 2026-07-10)

Every row in the resulting opt_proposal_table MUST carry app_impact_pct (% of current application wall time, measured or bounded-estimate) and system_impact_pct (throughput/bandwidth/utilization effect, 0 if not measured/applicable) — columns: #, Strategy, Description of Optimization, App Impact, System Impact, Weighted Score (50/50 weighted, auto-sorted descending by the tool). Never omit these fields.

The optimizer must not stop at the top-ranked bottleneck's obvious fix. Every optimization loop walks this full checklist, every time, and records a verdict for EACH category — not just the ones that turned out to matter. This is the enumeration source for the "Show the WHOLE option space" proposal table (see dftracer-optimizer agent). Verdict per category is one of: Applied & measured (result + citation), Applicable, not measured (concrete reason), or Not applicable (concrete technical reason — wrong FS_TYPE, no literature found for this access pattern, structurally inert given the app's I/O shape, admin-only/already-maxed per the system skill, etc.). Never silently omit a category, and never mark "not applicable" without first running the literature search for it.

1. L1 buffering (write coalescing/batching/double-buffering)
2. L1 caching (chunk cache, app-level LRU, page-cache hints)
3. L1 prefetching (async I/O, HDF5 async VOL, posix_fadvise, background threads)
4. L1 stage-in/stage-out (faster intermediate storage, pattern unchanged)
5. L2 MPI-IO/ROMIO (collective/two-phase buffering, data sieving, cb_nodes/multiplier)
6. L2 HDF5 (chunk cache sizing, metadata block size, collective metadata ops, page buffering)
7. L2 other I/O middleware (NetCDF/PnetCDF, compression filters, framework dataloader overlap)
8. L3 filesystem striping/layout (Lustre lfs setstripe/PFL/Data-on-MDT, GPFS/BeeGFS block size)
9. L3 near-node/burst-buffer acceleration (Rabbit/DW, node-local NVMe, /dev/shm, VAST cache)
10. L3 OS/kernel tuning (readahead, VM/dirty-ratio, I/O scheduler, NUMA binding — check tunability first)
11. Metadata-path tuning (Lustre metadata striping, Data-on-MDT, directory sharding)
12. Compute/communication overlap (async checkpointing, overlapping compute with I/O)

See the L1/L2/L3 Strategy sections below for the detailed technique catalog behind each checklist number, and the FS_TYPE-gated tables for which system/filesystem combinations each technique even applies to.

MANDATORY: Tool-First Analysis and Explicit Separation

Tool-First Rule

Every analysis and diagnosis step MUST attempt MCP tools before falling back to manual methods. The canonical tool order is:

1. mcp__dftracer__analyze — dfanalyzer trace analysis
2. mcp__dftracer__diagnose — dfdiagnoser bottleneck scoring
3. mcp__dftracer__comparator — compare two runs
4. mcp__dftracer__event_count — event count summary
5. mcp__dftracer__reader — trace metadata reader
6. mcp__dftracer__session_analyze_traces — session-scoped analysis

If tools are unavailable, ask the user to start the dftracer MCP server. If tools error, fix the tool or wiring before using custom Bash/Python.

Pipeline eagerly — don't wait for the whole batch (MANDATORY, user instruction 2026-07-10)

When a multi-workload or multi-replicate baseline is only partially complete, start analysis/diagnosis/optimization on whatever workloads ARE ready now — do not block the entire pipeline on the slowest or broken unit. Track which units are included in a given pass vs. still pending, and feed the pending ones in as they complete rather than re-running from scratch. State the partial coverage explicitly in every report so downstream synthesis doesn't mistake a partial result for the complete picture.

Explicit Separation Requirement

In every report, create a table separating:

• TOOL FINDINGS: Results from MCP tools (dfanalyzer, dfdiagnoser, comparator)
• MANUAL ANALYSIS: Results from custom Bash/Python (only when tools fail)

Never conflate the two. Label each finding with its source.

Citation-Backed Optimizations (MANDATORY)

Every optimization proposal MUST carry a verifiable paper citation. Use the Built-in Citations section below (WisIO, Drishti, GLANCED-IO, etc.), or search arXiv / Semantic Scholar. The citation must include: authors, title, venue/year, and a URL (arXiv PDF, ACM DOI, or IEEE Xplore). If no paper is found after 10 search attempts, mark the proposal as UNSUPPORTED and do not apply it.

NEVER "Do Less" as Optimization

The following are FORBIDDEN:

• "Reduce checkpoint frequency" or "write fewer checkpoints"
• "Reduce plot variables" or "write less data"
• "Do less I/O", "do less compute", "do less communication", "use less memory"
• Any proposal whose core mechanism is reducing the amount of work done

Why: Doing less is not a solution. The goal is to make the SAME work run faster (better bandwidth, lower latency, higher throughput), not to avoid the work. If the bottleneck is write-time, propose buffering, async I/O, collective I/O, compression with faster algorithms, or stripe tuning — never "write less."

---

MANDATORY: Two optimization axes — layer × metric_scope

Every optimization sits on two independent axes, not one:

Axis	Values	Answers
layer (scope in opt_kb)	L1 workload / L2 software / L3 system	who inherits this finding
metric_scope	app (default) / system	which metric moved

metric_scope="app" means the measured metric came from the app's own trace — epoch time, I/O time, app-observed bandwidth. metric_scope="system" means a filesystem/system-level outcome — aggregate achieved bandwidth, reduced filesystem load. On this pipeline "system metric" is currently a trace-derived proxy (aggregate bytes/sec computed from the app's own trace), not real OST/MDT-side telemetry — there is no Lustre-admin monitoring access here. Don't present it as more precise than it is.

Non-degradation guard (MANDATORY). A system-level optimization is only worth keeping if it did not cost the app anything. opt_kb_record(metric_scope="system", ...) REQUIRES the paired app_metric/app_before/app_after for the SAME change — the tool rejects a system-scoped entry without it. If the paired app metric regressed more than 2%, the tool force-sets the verdict to regression (guard_triggered: true) regardless of how good the system-side number looks. Treat guard_triggered: true exactly like a normal regression: revert the change. A system optimization that degrades the app is never a win.

---

MANDATORY: Datasets Must Live on Lustre, Never NFS

Application datasets (training data, fractals/checkpoints/runs, any file the app reads/writes repeatedly during a run) must be placed on Lustre (/p/lustre5/$USER/...), never on NFS-backed paths (e.g. /usr/WS*, /g/g*, /collab/...). NFS is a single-server filesystem with no striping — it hard-caps aggregate bandwidth/IOPS regardless of how well the application layer (num_workers, prefetch_factor, persistent_workers) is tuned. This was confirmed directly on Tuolumne: a ScaFFold fractal dataset sitting on NFS (cz-ws2-nfs-new.llnl.gov) showed critical posix_*_ops_slope bottlenecks and only ~41% compute/I-O overlap even with L1-optimal DataLoader settings; moving the same dataset to Lustre with matched striping removed the ceiling entirely without any app-level code change.

Before running any benchmark/training job, check where the dataset directory actually lives:

stat -f <dataset_dir>          # look for "Type: nfs" vs "Type: lustre"
df -T <dataset_dir>             # filesystem type column

If it's NFS, copy it to Lustre and repoint the app's data-dir argument — do not proceed with performance analysis on an NFS-resident dataset, since any I/O bottleneck found there may just be "wrong filesystem," not a real app/library issue.

Set Lustre striping to match the access pattern

Striping must be sized to the actual per-file size and per-I/O transfer size observed in the trace, not applied blindly:

Access pattern (from trace)	Stripe count	Stripe size
Many small files, each < stripe size (e.g. ML per-sample files, KB–few MB)	1 — striping a tiny file across multiple OSTs adds overhead with no parallelism benefit; parallelism instead comes from many files being spread across the directory/filesystem	leave at filesystem default
Few large shared files (checkpoints, HDF5, single big dataset file) read/written by many ranks concurrently	OST count, capped 8 for < 1 GB files, 16+ for very large files	1–4 MB, aligned to the app's read/write transfer size (see dftracer_info/diagnose *_avg_transfer_size)
Sequential large writes (checkpoints)	4–8	4 MB (matches ROMIO cb_buffer_size default of 64 MB / 16 aggregators)

Apply with lfs setstripe -c <count> -S <size> <dir> before any file is created in that directory (striping is set at file-creation time and cannot be changed retroactively without rewriting the file). For a directory of many small per-sample files (the common ML dataset case), the correct call is simply lfs setstripe -c 1 <dataset_dir> — do NOT default to a high stripe count "for safety"; it actively hurts small-file access.

Verify after copying data in: lfs getstripe -c <dataset_dir> should report the value you set, and lfs getstripe <sample_file> should show a single OST for small-file datasets.

Distinguish op-count-bound I/O from bandwidth-bound I/O before picking a lever

Confirmed on vpic-kokkos benchmark.cxx at 128 ranks (2026-07-14): 340.85M POSIX ops moved only 325.0 MB total (sub-byte average transfer size) yet consumed just 1.65s of a 210.2s job — the I/O was real but structurally negligible, and its shape was op-count-bound (huge number of tiny writes), not bandwidth-bound. ROMIO/Lustre-striping levers target bandwidth-bound patterns (few large transfers) and would do nothing here; the only lever that would help an op-count-bound pattern is write coalescing at the app level (batch many small writes into fewer, larger ones) — a compute-optimization-adjacent app-code change, not a filesystem/MPI-IO tuning knob. Always check whether total I/O time is a meaningful fraction of job time AND whether the bottleneck is few-huge-ops or many-tiny-ops before reaching for a striping/ROMIO fix — see workload-vpic-kokkos for the full worked example.

Related Skills

Software-specific strategies are also available in dedicated skills:

• [[software-mpi]] — MPI-IO/ROMIO details, Flux env propagation, Cray MPICH
• [[software-hdf5]] — HDF5 version compatibility, chunk/cache tuning, build from source
• [[software-posix]] — POSIX readahead, Lustre striping, OS/VM tuning, ops_slope bottlenecks
• [[workload-vpic-kokkos]] — worked example of an op-count-bound, structurally negligible I/O profile

Workload-specific results:

• [[workload-ior]] — quantified ROMIO results on VAST NVMe (Tuolumne)
• [[workload-h5bench]] — HDF5 CMake build and annotation pitfalls

System-specific accelerators (L3 near-node storage tiers):

• [[system-tuolumne-rabbit]] — Rabbit near-node flash on Tuolumne (XFS/GFS2/Lustre via Flux -S "#DW ..."); stage hot data onto a local flash tier to relieve network-Lustre bottlenecks

---

Key Reference Papers

WisIO (primary bottleneck → optimization guide)

Citation: Yildirim, Izzet, Hariharan Devarajan, Anthony Kougkas, Xian-He Sun, and Kathryn Mohror. "WisIO: Automated I/O Bottleneck Detection with Multi-Perspective Views for HPC Workflows." In Proceedings of the 39th ACM International Conference on Supercomputing, pp. 749–763. 2025.

What it provides:

• Multi-perspective bottleneck taxonomy covering: sequentiality, small-I/O, read/write time fraction, metadata overhead, fetch pressure, stragglers
• Per-category root cause + optimization strategy mapping
• Quantified thresholds for high/critical classification

Bottleneck → optimization mapping from WisIO:

WisIO category	Metric in dfdiagnoser	Primary fix (L1→L3)
small-io	small_io_pct	L1: buffer reads; L2: collective I/O; L3: stripe
sequentiality	rand_pct / seq_pct	L1: sort access; L2: prefetch hint; L3: readahead
read-time	read_time_pct	L1: async I/O; L2: cb_buffer; L3: blockdev setra
write-time	write_time_pct	L1: async write; L2: dirty tuning; L3: vm.dirty
metadata	metadata_time_pct	L1: cache stats; L2: HDF5 metadata opt; L3: MDT
fetch-pressure	fetch_pressure	L1: DataLoader workers; L2: prefetch_factor
stragglers	epoch_straggler	L1: sort by size; L2: persistent_workers

Drishti (user-facing optimization guidance)

Citation: Bez, Jean Luca, Hammad Ather, and Suren Byna. "Drishti: Guiding end-users in the I/O optimization journey." In 2022 IEEE/ACM International Parallel Data Systems Workshop (PDSW), pp. 1–6. IEEE, 2022.

What it provides:

• End-user-facing optimization suggestions organized by I/O category
• Three-level suggestion structure (application / library / system) matching our L1/L2/L3
• Estimated improvement ranges per suggestion type

Drishti suggestion model:

Drishti category	L1 (app)	L2 (software)	L3 (system)
small-io	buffer reads, batch writes	ROMIO cb_buffer, HDF5 chunks	lfs setstripe -S 4m
metadata	cache stat(), open once	H5Pset_coll_metadata_write	lfs mkdir -c N (DNE)
sequentiality	sort indices, posix_fadvise	romio_ds_read=enable	blockdev --setra 4096
shared-file	independent file per rank	romio_cb_read=enable	increase OST count

Optimization Metric Selection Guide

Use this when the agent must suggest which metric to optimize:

Dominant bottleneck(s)	Recommended metric	Rationale
read_time_pct or write_time_pct ≥4	time	Latency is the binding constraint
read_bw_mean or write_bw_mean low	bandwidth	Throughput is the binding limit
small_io_pct or rand_pct ≥4	iops	Operation rate limits throughput
metadata_time_pct ≥4	metadata_ops	Metadata is the bottleneck