phase3-persistent-reporting

name

phase3-persistent-reporting

description

Use when working on MathcoderMvp phase-3 style tasks that require long-running execution, periodic supervision, and final reporting. Trigger this skill when Codex must keep work moving for hours, summarize progress without drifting, and produce a report that clearly states what was done, what evidence was collected, whether the work followed the user's intended line, and what conclusion is actually supported.

Phase 3 Persistent Reporting

Use this skill for long-running MathcoderMvp work where the user cares about two things at once:

• the job keeps moving instead of stalling silently
• the report explains the work in a way the user can audit

This skill is for phase-3 style validation work, especially RL/SFT/eval sweeps, remote training checks, checkpoint comparison, and watchdog-supervised experiment review.

Mission First

Do not launch a watchdog until the mission is written down in a core task file.

A long-running worker must always have one explicit mission document, preferably Markdown, for example:

• watchdog_mission.md
• task_brief.md
• mission.md

This file is the source of truth for direction. Even if the mission is only one sentence, it must be recorded before launch so the worker and coach have a stable target and later drift can be judged against something concrete.

Minimum required content:

• one-sentence core goal
• what "good output" looks like
• what kind of real evidence the user expects to see
• what should be prioritized first

Good example for a GitHub research scout:

• go to GitHub and find similar AI research / autonomous research projects
• download and run them locally
• identify which components or outputs are genuinely impressive in practice
• bring back real evidence that proves the result is good enough to reuse

If the user gives a rough goal verbally, Codex should convert it into the mission file before starting the watchdog.

If the goal is still ambiguous, do not guess and do not launch first. Pause and clarify the mission with the user. The clarification loop should be short and practical:

1. restate the current intended goal in plain language
2. point out what is still underspecified
3. ask only for the minimum clarification needed to avoid drift
4. write the clarified result into the mission file
5. only then start the worker and watchdog

The standard is not "perfectly specified". The standard is:

• clear enough that a later coach can judge whether the worker is still on-mission
• concrete enough that the worker knows what evidence to collect

Qualified Watchdog Startup

If this skill is used together with the watchdog supervisor, do not start a "bare" watchdog that only watches pane text.

A qualified watchdog must satisfy all of the following before it is launched:

• create or update the mission file first, and make sure the startup prompt explicitly tells the worker to read it before doing anything else
• set a real max_supervision_hours, and enable stop_after_max_hours
• for long-running external-flow tasks, enable coach.enabled=true, prefer coach.message_mode=replace, and run coach as an independent codex exec reviewer instead of a fixed-text template
• set reply_stale_seconds, so a worker that keeps waiting on external jobs must still produce a human-readable checkpoint
• configure progress_watch_paths to point at the actual run artifacts, not just the repo root
• configure progress_file_globs so the watchdog can see stage files, eval summaries, logs, checkpoints, or generated reports
• keep repeat_nudge_min_gap_seconds nonzero, so the same watchdog text is not injected over and over
• keep max_session_boots_per_window and boot_window_seconds set, so session_missing cannot loop forever
• run only one watchdog process per runtime dir/config

Good progress_watch_paths examples:

• RL/eval sweep: run directories, checkpoint folders, eval summary folders, training logs
• paper/agent pipeline: artifacts/, stage_health.json, pipeline_summary.json, heartbeat.json, run logs
• document/demo project: progress.md, next_actions.md, generated JSON/CSV/DB outputs, core scripts

Bad startup pattern:

• start a watchdog with only idle_seconds and no artifact watch paths
• enable coach but still leave it in append mode, so worker keeps receiving the same watchdog template plus extra text
• let it run forever with no max duration
• let multiple watch --interval ... loops point at the same runtime directory

Recommended launch checklist:

1. Write the mission file and keep it short but explicit.
2. Make the startup prompt tell the worker to read that mission file first.
3. Confirm the worker's real artifact paths.
4. Put those paths into progress_watch_paths.
5. Add globs for the files that actually prove progress.
6. Set reply_stale_seconds to force checkpoint-style reporting when external jobs run quietly.
7. For long-running tasks, wire coach.command and set coach.message_mode to replace.
8. Start exactly one watchdog loop for that config.
9. Verify the first summary shows recent artifact change time and artifact paths, not only pane status.

Recommended launch commands:

python3 watchdog-b-mvp/bin/supervisor.py --config /abs/path/to/watchdog.json start-session
python3 watchdog-b-mvp/bin/supervisor.py --config /abs/path/to/watchdog.json watch --interval 60

Preflight before watch --interval 60:

python3 watchdog-b-mvp/bin/supervisor.py --config /abs/path/to/watchdog.json status

The preflight is only acceptable if:

• last_artifact_change_at is populated for artifact-driven tasks
• halted_reason is empty
• boot_count is not already growing abnormally
• reply_stale_seconds and expected_finish_at match the intended run contract

Core rule

Do not treat "working for a long time" and "writing a report" as separate tasks.

The work must be run in a way that leaves behind an auditable trail:

• what question is being answered
• what commands or checks were actually run
• what evidence landed
• what did and did not improve
• whether the work still follows the user's intended line

What the user actually wants

In this project, the user usually does not just want a generic progress update. They want a report that answers:

1. What did you do during this period?
2. What came out of it?
3. Are you still executing the intended plan, or did you drift?
4. What evidence supports the current conclusion?
5. What should happen next?

If the report cannot answer those five questions, it is not done.

Operating mode during long runs

When the experiment is still running, do not spam long essays. Give short updates that preserve decision context.

Each periodic update should contain only the minimum needed:

• current phase
• which jobs are still running
• what new artifact landed, if any
• newest checkpoint or eval result
• comparison to the control or prior best
• whether human action is needed

If nothing new landed, say so directly. A good "still waiting" update is:

• no new checkpoint or eval has landed
• currently still waiting on X jobs
• current best result remains Y
• next check will focus on Z

Do not pad these updates with repeated background explanation.

Non-negotiable reporting rules

1. Answer the real research question first

State the exact question being verified. For this project that often means something like:

• can RL improve target capability beyond SFT on independent eval

Do not open with implementation trivia.

2. Keep the comparison chain explicit

Always make the ladder clear:

• base
• SFT
• RL checkpoint(s)

If the user is trying to prove RL helps, the report must explicitly show whether RL exceeds SFT and where.

3. Explain why the evidence is credible

Do not only cite file paths. Explain:

• where the data came from
• how train/test were split
• whether overlap was checked
• why the eval is independent enough for the current claim

Include at least one concrete train example and one concrete test example when useful.

4. Explain the mechanism, not just the label

If the method uses reward terms, GRPO, LoRA, verifier checks, or other machinery, explain:

• what each component does
• how the total score is formed
• how that score affects model updates

Do not stop at "used GRPO" or "used three rewards".

5. Give the actual settings

Include the real parameters that matter, not vague prose.

Typical examples:

• checkpoint source
• LoRA rank and alpha
• generation count
• learning rate
• temperature
• length budget
• save interval
• total training steps

6. Show the result table, not only the best number

Report the matrix that lets the reader judge stability:

• checkpoint-20 / 40 / 80
• multiple runs or sweep settings
• base and SFT baseline

If only one best point is shown, the reader cannot tell whether the gain is robust or accidental.

7. Use item-level evidence for capability claims

If claiming "model ability improved", include concrete examples where:

• RL fixed a reasoning error that SFT got wrong
• RL completed an answer that SFT left unfinished
• RL also broke something SFT previously handled

At least one positive case and one regression case should be discussed when available.

8. Distinguish current conclusion from overclaim

Say both:

• what the evidence already supports
• what it does not support yet

For example:

• "RL is shown effective on this setup"
• not "RL is now fully stable and generally solved"

9. Explain the execution tradeoffs

If infrastructure issues forced a route change, report the decision plainly.

Example:

• 8-card distributed job failed on NCCL/CUDA init
• switched to 8 single-card parallel runs
• reason: first prove the RL path is valid instead of burning the night on infra

This matters because the user wants to know whether the work followed the intended strategy.

10. Always include handoff state

End with:

• current best conclusion
• remaining uncertainty
• next highest-value actions
• whether the user needs to intervene now

Preferred report shape

For final or milestone reports, prefer article-style writing over shallow bullet dumping.

The report should usually cover these sections:

1. What question this round was validating
2. What data and split were used
3. How the training/eval setup works
4. Why the chosen execution path was taken
5. What the independent eval actually measures
6. Full result table
7. Case-based evidence of gains and regressions
8. What problems were solved during the run
9. The strongest conclusion justified today
10. What to do next

Use tables where they compress comparisons well. Use prose where the reader needs causal explanation.

Style constraints

• Write so an outsider can follow the logic without reading the whole repo first.
• Prefer plain, exact language over hype.
• Avoid pure bullet forests when a short explanatory paragraph is clearer.
• Do not hide uncertainty.
• Do not write like a detached benchmark bot; make the experimental reasoning legible.
• Do not confuse "more formatted output" with "stronger capability".

Quick self-check before sending a report

Before handing the report to the user, verify that it answers all of these:

• What was the core question?
• What work was actually done in this interval?
• What new evidence landed?
• What is the best current result and what is it compared against?
• Why is the evidence credible?
• What examples show real improvement or regression?
• Did the execution stay aligned with the user's plan?
• What remains uncertain?
• What should happen next?

If any answer is missing, the report is incomplete.

Default behavior when resuming after a long unattended period

If you are dropped back into a long-running MathcoderMvp thread after hours of unattended work:

1. reconstruct what happened from the session, logs, eval artifacts, and summaries
2. identify the current best supported conclusion
3. state whether the work stayed on-plan or drifted
4. write a concise handoff first
5. only then expand into the full report if needed

The user should be able to return after a night away and immediately know:

• what the system spent time on
• whether that time was well spent
• what the evidence now says