Menu
Key literature, bottleneck-to-optimization mappings, and strategies for the dftracer I/O optimization pipeline
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Communication-component (MPI/collective/network) bottleneck-to-optimization mappings, papers, and L1/L2/L3 strategies for the dftracer optimization pipeline
Compute-component bottleneck-to-optimization mappings, papers, and L1/L2/L3 strategies for the dftracer optimization pipeline
Memory-component bottleneck-to-optimization mappings, papers, and L1/L2/L3 strategies for the dftracer optimization pipeline
VPIC-Kokkos-specific knowledge: build/annotation quirks, the two-stage deck-compile pattern, deck sizing for smoke vs validation runs, and the measured MPI-communication-bound diagnosis on Tuolumne. Load this skill whenever working with vpic-kokkos (github.com/lanl/vpic-kokkos).
Install and build dftracer into the session using the MCP install tool and record tool pitfalls early.
Diagnose traces and convert symptoms into actionable bottlenecks with MCP tools.
| name | dftracer-io-optimization |
| description | Key literature, bottleneck-to-optimization mappings, and strategies for the dftracer I/O optimization pipeline |
Every row in the resulting opt_proposal_table MUST carry app_impact_pct (% of current application wall time, measured or bounded-estimate) and system_impact_pct (throughput/bandwidth/utilization effect, 0 if not measured/applicable) — columns: #, Strategy, Description of Optimization, App Impact, System Impact, Weighted Score (50/50 weighted, auto-sorted descending by the tool). Never omit these fields.
The optimizer must not stop at the top-ranked bottleneck's obvious fix. Every optimization
loop walks this full checklist, every time, and records a verdict for EACH category — not
just the ones that turned out to matter. This is the enumeration source for the "Show the
WHOLE option space" proposal table (see dftracer-optimizer agent). Verdict per category
is one of: Applied & measured (result + citation), Applicable, not measured
(concrete reason), or Not applicable (concrete technical reason — wrong FS_TYPE, no
literature found for this access pattern, structurally inert given the app's I/O shape,
admin-only/already-maxed per the system skill, etc.). Never silently omit a category, and
never mark "not applicable" without first running the literature search for it.
posix_fadvise, background threads)cb_nodes/multiplier)lfs setstripe/PFL/Data-on-MDT, GPFS/BeeGFS block size)/dev/shm, VAST cache)See the L1/L2/L3 Strategy sections below for the detailed technique catalog behind each checklist number, and the FS_TYPE-gated tables for which system/filesystem combinations each technique even applies to.
Every analysis and diagnosis step MUST attempt MCP tools before falling back to manual methods. The canonical tool order is:
mcp__dftracer__analyze — dfanalyzer trace analysismcp__dftracer__diagnose — dfdiagnoser bottleneck scoringmcp__dftracer__comparator — compare two runsmcp__dftracer__event_count — event count summarymcp__dftracer__reader — trace metadata readermcp__dftracer__session_analyze_traces — session-scoped analysisIf tools are unavailable, ask the user to start the dftracer MCP server. If tools error, fix the tool or wiring before using custom Bash/Python.
When a multi-workload or multi-replicate baseline is only partially complete, start analysis/diagnosis/optimization on whatever workloads ARE ready now — do not block the entire pipeline on the slowest or broken unit. Track which units are included in a given pass vs. still pending, and feed the pending ones in as they complete rather than re-running from scratch. State the partial coverage explicitly in every report so downstream synthesis doesn't mistake a partial result for the complete picture.
In every report, create a table separating:
Never conflate the two. Label each finding with its source.
Every optimization proposal MUST carry a verifiable paper citation. Use the Built-in Citations section below (WisIO, Drishti, GLANCED-IO, etc.), or search arXiv / Semantic Scholar. The citation must include: authors, title, venue/year, and a URL (arXiv PDF, ACM DOI, or IEEE Xplore). If no paper is found after 10 search attempts, mark the proposal as UNSUPPORTED and do not apply it.
The following are FORBIDDEN:
Why: Doing less is not a solution. The goal is to make the SAME work run faster (better bandwidth, lower latency, higher throughput), not to avoid the work. If the bottleneck is write-time, propose buffering, async I/O, collective I/O, compression with faster algorithms, or stripe tuning — never "write less."
Every optimization sits on two independent axes, not one:
| Axis | Values | Answers |
|---|---|---|
layer (scope in opt_kb) | L1 workload / L2 software / L3 system | who inherits this finding |
| metric_scope | app (default) / system | which metric moved |
metric_scope="app" means the measured metric came from the app's own trace —
epoch time, I/O time, app-observed bandwidth. metric_scope="system" means a
filesystem/system-level outcome — aggregate achieved bandwidth, reduced
filesystem load. On this pipeline "system metric" is currently a trace-derived
proxy (aggregate bytes/sec computed from the app's own trace), not real
OST/MDT-side telemetry — there is no Lustre-admin monitoring access here. Don't
present it as more precise than it is.
Non-degradation guard (MANDATORY). A system-level optimization is only worth
keeping if it did not cost the app anything. opt_kb_record(metric_scope="system", ...) REQUIRES the paired app_metric/app_before/app_after for the SAME
change — the tool rejects a system-scoped entry without it. If the paired app
metric regressed more than 2%, the tool force-sets the verdict to regression
(guard_triggered: true) regardless of how good the system-side number looks.
Treat guard_triggered: true exactly like a normal regression: revert the
change. A system optimization that degrades the app is never a win.
Application datasets (training data, fractals/checkpoints/runs, any file the
app reads/writes repeatedly during a run) must be placed on Lustre
(/p/lustre5/$USER/...), never on NFS-backed paths (e.g. /usr/WS*,
/g/g*, /collab/...). NFS is a single-server filesystem with no striping
— it hard-caps aggregate bandwidth/IOPS regardless of how well the
application layer (num_workers, prefetch_factor, persistent_workers) is
tuned. This was confirmed directly on Tuolumne: a ScaFFold fractal dataset
sitting on NFS (cz-ws2-nfs-new.llnl.gov) showed critical posix_*_ops_slope
bottlenecks and only ~41% compute/I-O overlap even with L1-optimal
DataLoader settings; moving the same dataset to Lustre with matched striping
removed the ceiling entirely without any app-level code change.
Before running any benchmark/training job, check where the dataset directory actually lives:
stat -f <dataset_dir> # look for "Type: nfs" vs "Type: lustre"
df -T <dataset_dir> # filesystem type column
If it's NFS, copy it to Lustre and repoint the app's data-dir argument — do not proceed with performance analysis on an NFS-resident dataset, since any I/O bottleneck found there may just be "wrong filesystem," not a real app/library issue.
Striping must be sized to the actual per-file size and per-I/O transfer size observed in the trace, not applied blindly:
| Access pattern (from trace) | Stripe count | Stripe size |
|---|---|---|
| Many small files, each < stripe size (e.g. ML per-sample files, KB–few MB) | 1 — striping a tiny file across multiple OSTs adds overhead with no parallelism benefit; parallelism instead comes from many files being spread across the directory/filesystem | leave at filesystem default |
| Few large shared files (checkpoints, HDF5, single big dataset file) read/written by many ranks concurrently | OST count, capped 8 for < 1 GB files, 16+ for very large files | 1–4 MB, aligned to the app's read/write transfer size (see dftracer_info/diagnose *_avg_transfer_size) |
| Sequential large writes (checkpoints) | 4–8 | 4 MB (matches ROMIO cb_buffer_size default of 64 MB / 16 aggregators) |
Apply with lfs setstripe -c <count> -S <size> <dir> before any file is
created in that directory (striping is set at file-creation time and cannot
be changed retroactively without rewriting the file). For a directory of
many small per-sample files (the common ML dataset case), the correct call
is simply lfs setstripe -c 1 <dataset_dir> — do NOT default to a high
stripe count "for safety"; it actively hurts small-file access.
Verify after copying data in: lfs getstripe -c <dataset_dir> should report
the value you set, and lfs getstripe <sample_file> should show a single
OST for small-file datasets.
Confirmed on vpic-kokkos benchmark.cxx at 128 ranks (2026-07-14):
340.85M POSIX ops moved only 325.0 MB total (sub-byte average transfer
size) yet consumed just 1.65s of a 210.2s job — the I/O was real but
structurally negligible, and its shape was op-count-bound (huge number
of tiny writes), not bandwidth-bound. ROMIO/Lustre-striping levers target
bandwidth-bound patterns (few large transfers) and would do nothing here;
the only lever that would help an op-count-bound pattern is write
coalescing at the app level (batch many small writes into fewer, larger
ones) — a compute-optimization-adjacent app-code change, not a
filesystem/MPI-IO tuning knob. Always check whether total I/O time is a
meaningful fraction of job time AND whether the bottleneck is few-huge-ops
or many-tiny-ops before reaching for a striping/ROMIO fix — see
workload-vpic-kokkos for the full worked example.
Software-specific strategies are also available in dedicated skills:
Workload-specific results:
System-specific accelerators (L3 near-node storage tiers):
-S "#DW ..."); stage hot data onto a local flash tier to relieve network-Lustre bottlenecksCitation: Yildirim, Izzet, Hariharan Devarajan, Anthony Kougkas, Xian-He Sun, and Kathryn Mohror. "WisIO: Automated I/O Bottleneck Detection with Multi-Perspective Views for HPC Workflows." In Proceedings of the 39th ACM International Conference on Supercomputing, pp. 749–763. 2025.
What it provides:
Bottleneck → optimization mapping from WisIO:
| WisIO category | Metric in dfdiagnoser | Primary fix (L1→L3) |
|---|---|---|
| small-io | small_io_pct | L1: buffer reads; L2: collective I/O; L3: stripe |
| sequentiality | rand_pct / seq_pct | L1: sort access; L2: prefetch hint; L3: readahead |
| read-time | read_time_pct | L1: async I/O; L2: cb_buffer; L3: blockdev setra |
| write-time | write_time_pct | L1: async write; L2: dirty tuning; L3: vm.dirty |
| metadata | metadata_time_pct | L1: cache stats; L2: HDF5 metadata opt; L3: MDT |
| fetch-pressure | fetch_pressure | L1: DataLoader workers; L2: prefetch_factor |
| stragglers | epoch_straggler | L1: sort by size; L2: persistent_workers |
Citation: Bez, Jean Luca, Hammad Ather, and Suren Byna. "Drishti: Guiding end-users in the I/O optimization journey." In 2022 IEEE/ACM International Parallel Data Systems Workshop (PDSW), pp. 1–6. IEEE, 2022.
What it provides:
Drishti suggestion model:
| Drishti category | L1 (app) | L2 (software) | L3 (system) |
|---|---|---|---|
| small-io | buffer reads, batch writes | ROMIO cb_buffer, HDF5 chunks | lfs setstripe -S 4m |
| metadata | cache stat(), open once | H5Pset_coll_metadata_write | lfs mkdir -c N (DNE) |
| sequentiality | sort indices, posix_fadvise | romio_ds_read=enable | blockdev --setra 4096 |
| shared-file | independent file per rank | romio_cb_read=enable | increase OST count |
Use this when the agent must suggest which metric to optimize:
| Dominant bottleneck(s) | Recommended metric | Rationale |
|---|---|---|
| read_time_pct or write_time_pct ≥4 | time | Latency is the binding constraint |
| read_bw_mean or write_bw_mean low | bandwidth | Throughput is the binding limit |
| small_io_pct or rand_pct ≥4 | iops | Operation rate limits throughput |
| metadata_time_pct ≥4 | metadata_ops | Metadata is the bottleneck |
| multiple ≥4 | time (default) | Most general; addresses all |
cb_buffer_size=67108864)blockdev --setra 4096 to 16384)cpu_count // 2 with prefetch_factor=4| File | Contents |
|---|---|
optimization_context.json | Metric, bottlenecks, papers — shared context |
opt_l1_results.json | L1 proposals, status, files modified |
opt_l2_results.json | L2 proposals, env vars applied |
opt_l2_env.sh | Source before each run to keep L2 settings |
opt_l3_results.json | L3 changes + rollback commands |
optimization_papers.json | Full literature search results |
All artifact files above live under the current iteration's own opt<n>/
run directory (opt<n>/source/, opt<n>/patches/, opt<n>/scripts/,
opt<n>/traces/{raw,compact}/) — never a hand-built path like
ws/opt_results/. Strict rule (see dftracer-cheatsheet S0): before
starting a new iteration and again after applying L1/L2/L3 changes, call
session_validate_structure(run_id=RUN_ID); if clean=false, call
session_reorganize_structure(run_id=RUN_ID, dry_run=False) before
re-running the smoke test or collecting traces for that iteration.
These references are always available and must be used when their coverage matches the bottleneck being addressed.
resources/papers/HPDC26_GLANCED_IO.pdf (search via session_search_local_papers)Use this table to translate the chosen metric into the dftracer analysis fields that must improve:
| Metric | Goal | Target dftracer fields |
|---|---|---|
time | Minimize I/O latency | read_time_pct, write_time_pct, metadata_time_pct |
bandwidth | Maximize throughput | read_bw_mean, write_bw_mean, seq_pct |
iops | Maximize I/O operations/sec | small_io_pct, rand_pct, intensity_mean |
metadata_ops | Reduce metadata overhead | metadata_time_pct, metadata_time_frac_parent |
When multiple bottleneck scores are >= 4, default to time as the most general metric that addresses all dimensions.
These are source-code-level changes (applied under <WS>/annotated/). No system or middleware config changes at this layer.
Do not propose flipping a benchmark/app's access-pattern config (e.g. MEM_PATTERN/
FILE_PATTERN INTERLEAVED→CONTIG, random→sequential, etc.) just because the alternate pattern
measured faster elsewhere in the same session. That changes what the workload actually does —
it is not an optimization of the system's ability to serve the REAL pattern, and it can silently
break application logic/correctness (the access pattern is often load-bearing for what the app
computes, not an arbitrary benchmark knob). A finding like "write (INTERLEAVED) is slower than
read (CONTIG)" is a valid DIAGNOSIS, not license to rewrite write's config to be CONTIG.
Instead, keep the access pattern fixed and find ways to make THAT pattern faster:
H5Pset_cache/H5Pset_chunk_cache) so repeated/overlapping accesses under the same
pattern don't re-fetch from the backing store.posix_fadvise(POSIX_FADV_WILLNEED), background threads) so the same
pattern overlaps with compute instead of blocking it./dev/shm) before/after the real I/O phase, without altering the
pattern the app itself issues against that staged copy.cb_nodes, romio_cb_write/romio_ds_write,
CRAY_CB_NODES_MULTIPLIER — changes HOW the same logical accesses are physically aggregated
and dispatched, not what the app logically does.H5Pset_meta_block_size, chunked vs. contiguous HDF5 storage
layout (when layout, not access pattern, is the tunable), paged file-space strategy.lfs setstripe) matched to the (unchanged) access
pattern's transfer size and concurrency.Before proposing an L1 lever, search the literature specifically for buffering/caching/
prefetching/stage-in optimization techniques for the OBSERVED pattern (see dftracer-optimizer
agent's paper-search tools) rather than reaching for a pattern swap. See
[[dftracer-optimization-kb]] Rule 5 for the enforcement of this at record time.
This explicitly includes transfer-size / request-size sweeps (re-affirmed 2026-07-10).
Re-running a benchmark with a larger -t/transfer-size/block-size than the app or user actually
issues is an access-pattern change, not a system optimization — the same category as an
INTERLEAVED→CONTIG flip. "4 MB transfers get more bandwidth than 4 KB transfers" is a valid
DIAGNOSTIC characterization (it bounds how much headroom the small-request pattern is leaving on
the table), but it must never be reported as the "best config" or "the optimization" — the actual
best config must keep the workload's real request size and instead apply one of the techniques
above (ROMIO two-phase/collective buffering to coalesce many small requests into fewer physical
I/Os, HDF5 chunk/page caching, OS readahead, burst-buffer/node-local stage-in) to serve THAT
request size faster.
readv/writev (scatter/gather) instead of loops of read/writechunk_size in file iteration; use np.fromfile over loopsposix_fadvise(fd, 0, 0, POSIX_FADV_SEQUENTIAL) before sequential readsmmap for random access patterns to let the OS handle page faultsO_DIRECT for cache-bypassing when data reuse is lowio_uring (C/C++) or aiofiles/asyncio (Python)fallocate before writing to avoid fragmentationstat() results; avoid repeated lstat/stat on the same pathsreaddir() loops with pre-built file-list cachesos.path.exists() in hot loops with try/exceptnum_workers (recommend: cpu_count // 2)prefetch_factor=4 (PyTorch >= 1.7) or use tf.data.AUTOTUNEpersistent_workers=True to avoid worker respawn overheadThese are configuration-level changes: environment variables, library hint files, and runtime config. No source code changes. All changes must be reversible (document the original value).
ROMIO_HINTS=cb_buffer_size=67108864;romio_cb_read=enableromio_ds_read=enable;romio_ds_write=enablecb_nodes=<num_nodes><WS>/romio_hints.txt and set the ROMIO_HINTS env var to its pathH5Pset_coll_metadata_write, H5Pset_all_coll_metadata_opsH5Pset_cache(fapl, 0, 521, 64*1024*1024, 1.0)H5Pset_alignment) to stripe sizeH5Pset_object_track_times(False)nc_set_default_format(NC_FORMAT_NETCDF4_CLASSIC) for better performancenc_var_par_access(ncid, varid, NC_COLLECTIVE)num_workers = cpu_count // 2, prefetch_factor = 4pin_memory = True (when GPU is present)persistent_workers = True (avoids respawn per epoch)worker_init_fn to pre-open file handles per workerPYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:128 (reduces alloc fragmentation)DistributedSampler with shuffle=False; pre-shuffle indices oncetorch.save with _use_new_zipfile_serialization=False for faster writestorch.save in a separate threadTORCH_HOME to a fast local scratch directorytf.data.AUTOTUNE for prefetch buffer sizestf.data.experimental.AUTOTUNE for interleave cycle_lengthTF_CUDNN_USE_AUTOTUNE=1, TF_GPU_THREAD_MODE=gpu_privatePOSIX_FADV_SEQUENTIAL via posix_fadvise (env-level wrapper)OMP_NUM_THREADS, MKL_NUM_THREADS: set to physical core count for I/O threadingMALLOC_ARENA_MAX=2: reduce glibc arena fragmentation for I/O-heavy appsHBWMALLOC_POLICY=transparent: use high-bandwidth NUMA memory when available<WS>/opt_l2_env.sh; source before every runThese are OS- and filesystem-level tuning changes. Many require sudo or storage-admin access. Every proposal MUST include a rollback command and a privilege classification (no-sudo | sudo | admin-only).
MANDATORY: Before proposing any L3 strategy, confirm the detected FS_TYPE
(see Step 8-PRE of [[dftracer-pipeline]]). Only propose strategies whose
"Valid FS_TYPE" column matches the detected filesystem. Strategies for the
wrong filesystem are silently harmful — e.g., romio_ds_write=disable causes
a 4× write regression on VAST NVMe but is safe on Lustre spinning disk.
lfs setstripe -c <N> <WS>/traces/
Rollback: lfs setstripe -c 1 <WS>/traces/
Recommendation: N = OST count, capped at 8 for files < 1 GB; 16+ for very large fileslfs setstripe -S 4m <WS>/traces/ (4 MB aligns to ROMIO collective buffer)
Rollback: lfs setstripe -S 1m <WS>/traces/lfs setstripe --pool flash <WS>/lfs setstripe -c 4 -S 4m <WS>/traces/<file> before writinglfs setstripe --mdt-count 1 (for small dirs)lfs mkdir -c <N> (admin-only)-o localflock to reduce lock traffic (requires remount — admin)On systems with node-local NVMe accelerators, stage hot data onto a per-node or per-chassis flash tier instead of accessing network Lustre directly. No sudo — provisioned per job via the scheduler.
-S "#DW jobdw type=<xfs|gfs2|lustre> ...".
Pick the tier by data-sharing scope (SHM ≤20% node mem → XFS single-node ≤1 TB
→ GFS2 across ≤16 nodes on one chassis with --coral2-chassis=1 → Lustre for
16 nodes). Stage input onto
$DW_JOB_*once, run against it, copy persistent outputs back to Lustre at job end. The#DWdirective is an allocation-time flag — ask the user toflux allocwith it and report the JOBID before you can proxy in. Full guide: [[system-tuolumne-rabbit]].
sudo blockdev --setra <KB> /dev/<dev>
Default: 128 (64 KB); recommended: 4096–16384 (2–8 MB)
Rollback: sudo blockdev --setra 128 /dev/<dev>
Side effect: affects all processes reading from this deviceposix_fadvise(fd, 0, 0, POSIX_FADV_SEQUENTIAL) in source code/etc/udev/rules.d/60-readahead.rulessudo sysctl -w vm.dirty_ratio=20
(flush dirty pages when 20% of RAM is dirty; tune to 10–40)
Rollback: sudo sysctl -w vm.dirty_ratio=20sudo sysctl -w vm.dirty_background_ratio=5
(start background flush at 5%; reduces write stalls; default 10)sudo sysctl -w vm.dirty_expire_centisecs=3000 (30 s dirty page lifetime)sudo sysctl -w vm.dirty_writeback_centisecs=500 (flush every 5 s)sudo sysctl -w vm.vfs_cache_pressure=50
(reduce kernel tendency to reclaim page cache; default 100)
Rollback: sudo sysctl -w vm.vfs_cache_pressure=100cat /sys/block/<dev>/queue/schedulerecho none | sudo tee /sys/block/<dev>/queue/schedulerecho mq-deadline | sudo tee /sys/block/<dev>/queue/schedulerecho <original_scheduler> | sudo tee /sys/block/<dev>/queue/schedulernumactl --hardwarenumactl --cpunodebind=0 --membind=0 <run_command>mpirun --map-by numa:pe=<cores_per_node> (OpenMPI)These typically require storage admin action — surface as MANUAL RECOMMENDATIONS only. Do NOT apply these to lustre, vast, or local filesystems.
rsize=1048576,wsize=1048576,asyncmmchattr -r pagePool=<size> (admin tool)beegfs-ctl --settuning --clientNumWorkerThreads=16 (admin)Every optimization proposal presented to the user MUST include a verifiable citation with a URL (arXiv link, DOI, or stable webpage). Follow this search order for each unique bottleneck type:
PAPERS from optimization_context.json first (already fetched during pipeline setup).context_opportunities.json from session_search_optimization_context — exhaustive,
stack-wide, local-first search already run once per iteration (see "Start from what is
already known" above); its opportunities/benchmark_targets are already citation-backed.dftracer__rag_search(query="<bottleneck_type> optimization", bottleneck=..., system_config=...)
— semantic + lexical ranking over .dftracer_agents/resources/, free, no network.dftracer__search_arxiv(query="<bottleneck_type> optimization HPC parallel I/O MPI", max_results=5, category="cs.DC")dftracer__search_papers_combined(query="<bottleneck_type> I/O tuning parallel filesystem", max_results_each=5)dftracer__get_arxiv_paper(paper_id="<arxiv_id>")For every citation record:
https://arxiv.org/abs/XXXX.XXXXX)If no verifiable citation with a URL can be found for a proposal, that proposal MUST be skipped. Do not present uncited proposals to the user. Instead, write "no L<N> fix — no citation available" for that bottleneck.
Before proposing anything, load [[dftracer-optimization-kb]] and call:
opt_kb_lookup(system=<system>, workload=<app>, software="hdf5,mpi-io,lustre")
session_search_optimization_context(run_id=<run_id>, system=<system>, workload=<app>)
opt_kb_lookup returns MEASURED cross-session results, partitioned into
system-centric (L3), software-centric (L2) and workload-centric (L1) findings,
each with its citation, before/after numbers and caveats. Scope decides
transferability: a system finding does not leave its machine; a workload finding
does not leave its app; software findings travel across both.
session_search_optimization_context goes further than the per-bottleneck
searches below: it searches every software/system layer this session actually
detected (MPI impl, HDF5, ROMIO, Lustre, ML frameworks — not just the metric
tied to the current diagnosis), local-first (opt_kb_lookup + rag_search over
.dftracer_agents/resources/, free, no network) before any remote fan-out across
7 paper sources. It also runs a query class nothing else in this pipeline does:
benchmark-target search — published achieved bandwidth/throughput numbers at
this scale/filesystem, written to context_opportunities.json as
benchmark_targets. Read the snippets yourself to pull out the actual number —
the tool deliberately does not parse numbers out of free text; that judgment call
belongs to you.
Then render proposals with opt_proposal_table (uncited proposals are rejected),
apply one at a time, measure, and opt_kb_record every result — including
no-ops and regressions, and including metric_scope/paired app-metric for any
system-level result (see the non-degradation guard above). Finish with
opt_kb_render to publish into the KB skill.
Before any step that would open source files, use the graphify knowledge graph
(project dependency graphifyy, CLI graphify):
graphify query "<target>" --budget 1200 # locate: NODE <sym> [src=file loc=Lnn]
graphify explain <symbol> # definition + callers/callees
graphify affected <symbol> --depth 2 # blast radius before you change it
graphify update . # refresh after edits (~4s, no LLM)
Measured on this repo: locating cost 986 tokens vs 29,456 to read the three
relevant files (3.3%). Run affected before editing any shared function and
state the blast radius. Use the CLI, never graphify-mcp — its extra tool
schemas would sit in context permanently. See [[dftracer-context-economy]].
dataloader_num_workers>0, persistent_workers=True,
prefetch_factor, async checkpointing. Cheapest, usually the biggest win.
(Mohan et al., Analyzing and Mitigating Data Stalls in DNN Training, VLDB 2021,
https://arxiv.org/abs/2007.06775)pin_memory=True only pays off when
each rank is bound to all cores of its GPU's die. Pinned to a single core, the copy thread
contends with dataloader workers and the benefit inverts. On an APU (e.g. AMD MI300A) CPU and
GPU share the die and HBM, so affinity determines memory locality, not just scheduling.
(PyTorch memory-pinning docs, https://docs.pytorch.org/docs/stable/data.html#memory-pinning)Async checkpointing is only a win when checkpoint write time is a real fraction of epoch time — verify first. (Mohan et al., CheckFreq, USENIX FAST 2021, https://www.usenix.org/conference/fast21/presentation/mohan; Eisenman et al., Check-N-Run, USENIX NSDI 2022, https://www.usenix.org/conference/nsdi22/presentation/eisenman)
Guard rail. A wall-clock gain from writing fewer checkpoints, reading less data, or running fewer epochs is doing less, not going faster. Check event and byte counts before crediting it.