autofl-nvflare-report

name

autofl-nvflare-report

description

Generate and commit a markdown report after an Auto-FL NVFlare autoresearch experiment has been manually stopped. Use when the user asks to summarize a stopped campaign, report achieved improvements, explain implemented literature-derived ideas and sources, refresh progress plots, capture pasted agent model/effort/cost context when available, or commit the final report and progress plot to the current experiment branch.

Auto-FL NVFlare Report

Use this skill after the human has manually stopped an autoresearch campaign.

Do not launch new experiments. The job is to summarize the completed branch from results.tsv, existing artifacts, source notes, and any available pasted agent model/effort/cost context, then commit the report markdown and refreshed progress plot.

Workflow

1. Verify context:

   • Confirm the current branch with git status -sb.
   • If the branch is main, do not auto-commit unless the user explicitly asked to commit to main.
   • Confirm results.tsv exists and has rows.
   • Use the human-provided PYTHON interpreter when one is specified; do not create virtual environments or install dependencies unless the user explicitly asks.
   • Preserve unrelated dirty files; stage only the generated report and refreshed progress plot unless the user asks otherwise.

2. Refresh the progress plot:

   "${PYTHON:-python3}" scripts/plot_progress.py results.tsv --output progress.png
   

   If plotting fails because of local font/cache paths, retry with writable cache env vars, for example:

   MPLCONFIGDIR=/tmp/mpl-cache XDG_CACHE_HOME=/tmp/xdg-cache \
     "${PYTHON:-python3}" scripts/plot_progress.py results.tsv --output progress.png
   

3. Capture agent model/effort/cost context when available:

   • Claude Code: /model, /effort, and /cost are interactive slash commands and cannot be invoked from a shell or tool call. If the human pasted those outputs into the prompt or saved them to text files, pass the exact text via --agent-settings, --agent-settings-file, --agent-cost, or --agent-cost-file.
   • Claude Code without pasted model/effort output: pass --agent-settings "Agent model/effort telemetry unavailable in this Claude Code runtime; /model and /effort are interactive and were not provided to the reporting agent."
   • Claude Code without pasted cost output: pass --agent-cost "Agent cost telemetry unavailable in this Claude Code runtime; /cost is interactive and was not provided to the reporting agent. Experiment runtime cost is reported from results.tsv."
   • Codex: if no runtime model/effort/cost command is exposed, pass explicit unavailable notes with --agent-settings and --agent-cost.
   • Keep this separate from experiment runtime cost. Runtime cost comes from results.tsv and measures aggregate candidate execution time, not agent token spend.

4. Generate the report:

   "${PYTHON:-python3}" skills/autofl-nvflare-report/scripts/generate_report.py \
     --results results.tsv \
     --plot progress.png \
     --agent-settings-file /tmp/autofl-agent-settings.txt \
     --agent-cost-file /tmp/autofl-agent-cost.txt
   

   If no context files exist, use --agent-settings "<summary>" and --agent-cost "<summary>" instead. The script prints the report path. By default it writes to reports/<branch>-autoresearch-report.md.

5. Review the generated report for obvious parsing issues:

   • best score and baseline are present;
   • progress plot is embedded as a markdown image and points to the refreshed plot path;
   • final recommended stack includes exact budget/args;
   • runtime section includes total aggregate runtime and average runtime per candidate;
   • agent/tooling context section is present and either contains pasted model/effort/cost output or explains why it was unavailable;
   • literature-derived ideas include source refs from [src: ...] markers when present;
   • the report explicitly distinguishes candidates from reproduced/kept results.

6. Commit the report and progress plot automatically to the current experiment branch:

   git add <report-path> progress.png
   git commit -m "Add autoresearch campaign report"
   

   If a non-default plot path was used, stage that plot path instead of progress.png. Do not stage run_logs/, NVFlare result directories, generated local caches, or unrelated edits.

Report Expectations

The report should lead with an executive summary, then include a technical appendix. It must cover:

• baseline score, best score, absolute and relative lift;
• embedded progress plot image;
• best run index, description, commit, budget, runtime, status, and artifact path;
• major running-best milestones and likely improvement mechanisms;
• final recommended stack and whether it preserves the current contract or uses an explicit protocol mode such as SCAFFOLD;
• total aggregate candidate runtime, average runtime per candidate, crash count, and failure notes;
• agent model/effort settings and agent/tooling cost if provided by the human, or explicit unavailable notes;
• literature-derived ideas, source refs, and whether each helped;
• null/worse/unstable ideas that should not be retried;
• recommended reproduction and next experiment directions.