archive-experiment

name	archive-experiment
description	Archive a completed experiment, preserving all experiment files while deleting bulk checkpoint artifacts. Use after summarize-experiment or explore-experiment when an experiment is complete and results have been reviewed.

Archive Experiment

Archive a completed experiment by copying the entire experiment directory and deleting only the large checkpoint directories (*/artifacts/).

Your Task

Archive an experiment by:

1. Locating the experiment and validating completeness
2. Running a dry-run inventory showing what will be kept vs deleted
3. Resolving findings.md (prompting user if needed)
4. Getting user confirmation
5. Executing the archive (copy → verify → delete)
6. Logging the process

Prerequisites

• experiment_summary.yaml exists in the experiment directory
• Experiment is complete (all runs evaluated) — or user explicitly uses --force
• Conda environment activated — the script requires PyYAML

Workflow

1. Locate Experiment

Find the experiment directory:

• If user provided a path: use it
• If in an experiment directory (contains experiment_summary.yaml): use current directory
• Otherwise: ask user for the path

Read experiment_summary.yaml to determine:

• experiment.name — the experiment name
• experiment.dir — where model checkpoints live
• runs — list of run names
• evaluation.matrix — expected evaluations

2. Dry Run Preview

Run the archive script in dry-run mode:

python src/tools/experiment/archive_experiment.py <experiment_dir> --dry-run --pretty

If the default archive location is wrong, the user can specify --archive-base <path>.

Default archive location: ck-archive/{project}/{experiment_name}/ as sibling of the experiment's grandparent directory. For an experiment at __SCRATCH__/ck-projects/{project}/my_experiment/, the archive lands at __SCRATCH__/ck-archive/{project}/my_experiment/. The project layer is required — experiment.project must be set in experiment_summary.yaml.

Show the user:

• Number of files and total size to archive (entire experiment directory)
• Number of checkpoint directories and total size to delete
• Archive destination path
• Any incomplete runs (if --force would be needed)

3. Resolve findings.md

Check the dry-run output for findings_source:

1. If a source was found — tell the user which file will be copied as findings.md
2. If no source was found (findings_source: null) — ask the user:
   • "Would you like to write a brief findings.md before archiving? This captures what you learned from the experiment."
   • If yes: create {experiment_dir}/findings.md with user's content, then re-run dry-run
   • If no: proceed without findings.md

4. Confirm with User

Present the plan clearly:

Archive plan for: {experiment_name}
  Archive: {N} files ({X} MB) → {archive_dir}
  Delete:  {M} checkpoint dirs ({Y} MB)

Proceed? (y/n)

Use AskUserQuestion to get explicit confirmation before proceeding.

5. Execute Archive

Run the archive script without dry-run:

python src/tools/experiment/archive_experiment.py <experiment_dir> [--force] [--archive-base <path>] --pretty

The script will:

1. Copy the entire experiment directory to the archive
2. Verify the archive (file existence + size check)
3. Delete originals: checkpoint directories (*/artifacts/) and experiment directory
4. Report results

6. Report Results

Show the user:

• Archive location
• Files preserved
• Storage freed
• Any warnings (incomplete runs, copy errors)

7. Create Log

Write archive-experiment.log to the archive directory:

{archive_dir}/archive.log

See logging.md for action types and format.

What Gets Kept

The entire experiment directory is copied to the archive, preserving its original structure. This includes configs, SLURM scripts, eval logs, exploration, logs, and any other files. Since these are all small (KB-scale), there's no reason to selectively delete them.

What Gets Deleted

Artifact	Why Safe
Model checkpoints (*/artifacts/)	Reproducible via fine-tuning; these are the only large artifacts

Error Handling

Incomplete experiment (no --force)

• Show which runs are missing eval logs
• Refuse to archive
• Suggest: complete the runs, or use --force if results aren't needed

Archive directory already exists

• Refuse to overwrite
• Suggest: remove the existing archive first, or choose a different --archive-base

Verification failure

• Do NOT delete originals
• Report which files are missing or mismatched
• The archive directory may be in a partial state — user should investigate

experiment_summary.yaml missing or invalid

• Refuse to proceed
• This is the one file we absolutely need

Output Files

The archive mirrors the original experiment directory structure, nested under the project name:

{archive_base}/{project}/{experiment_name}/
├── experiment_summary.yaml
├── findings.md
├── summary.md
├── {dataset}.json
├── logs/
│   ├── design-experiment.log
│   ├── scaffold-torchtune.log
│   └── ...
├── {run_name}/
│   ├── finetune.yaml
│   ├── finetune.slurm
│   ├── setup_finetune.yaml
│   └── eval/
│       └── {task}_epoch{N}/       # One cell per (task, epoch) — issue #498
│           ├── eval.yaml
│           ├── cell.slurm
│           └── logs/*.eval
├── exploration/
│   ├── report.md
│   └── *.html
└── archive.log

Per-run artifacts/ directories are deleted, not archived — those are the only large items, and they can be regenerated by re-running fine-tuning.

Relationship to Other Skills

• After: summarize-experiment, explore-experiment (when experiment is complete)
• Reproducing: Feed experiment_summary.yaml back through scaffold-experiment + run-experiment
• Related: #398 (round-trip unarchive test), #399 (batch archiving)