doe-suite-experiments

name	doe-suite-experiments
description	Run remote experiments (and process results) using the doe-suite framework. Use when running experiments remotely or when processing experiment results to create plots or other diagrams.

This skill orchestrates the complete workflow for running DoE-Suite experiments on remote infrastructure (AWS EC2, ETHZ Euler, or existing machines).

Quick Start

Working Directory: All commands must be run from the doe-suite directory.

cd doe-suite
source .envrc

# Launch experiment
make run suite=<suite-name> id=new

# Monitor progress
make run suite=<suite-name> id=last

# Process aggregated results
make etl-super config=<config-name> out=<output-dir> pipelines="<pipeline1>,<pipeline2>"

# Cleanup (automatic after success, manual if needed)
make clean

Core Workflow

1. Design & Validate - Validate suite configuration before running
2. Launch Experiment - Start remote infrastructure and execute jobs
3. Monitor Progress - Track execution and download results
4. Review Results - Examine raw experiment outputs
5. Process with ETL - Aggregate and visualize results
6. Cleanup - Terminate cloud resources

Key Concepts

DoE-Suite manages remote experiments following Design of Experiments methodology:

• Suite: Collection of experiments (YAML config in doe-suite-config/designs/)
• Experiment: Set of runs with different configurations
• Factor: Parameter that varies (e.g., model, algorithm)
• Level: Specific value a factor takes (e.g., mnist, resnet18)
• ETL Pipeline: Extract-Transform-Load for processing results
• Super-ETL: Aggregates results across multiple suite runs

→ Detailed concepts and terminology

Prerequisites

Required environment variables in .envrc:

• DOES_PROJECT_DIR, DOES_PROJECT_ID_SUFFIX, DOES_SSH_KEY_NAME
• DOES_CLOUD, AWS_PROFILE, AWS_REGION

→ Complete prerequisites and setup guide

Available Suites

Discover suites:

ls doe-suite-config/designs/*.yml  # List all designs
make info                           # Show configured suites
make design suite=<suite-name>      # View specific suite details

→ Understanding suite configurations

Common Commands

# Validate design
make design-validate suite=<suite-name>

# Launch new run
make run suite=<suite-name> id=new

# Continue monitoring
make run suite=<suite-name> id=last

# Keep instances alive for debugging
make run-keep suite=<suite-name> id=new

# Process results across suites
make etl-super config=<config-name> out=<output-dir> pipelines="overview"

# Cleanup
make clean          # All resources
make clean-cloud    # Only cloud (keep local results)

→ All available commands

Need Help?

• Troubleshooting Guide - Common issues and solutions
• ETL Guide - Processing and visualizing results
• Example Workflows - Complete end-to-end examples
• DoE-Suite Docs - Framework documentation

Important Notes

⚠️ Cost Warning: AWS EC2 charges per hour. Set up billing alerts and verify cleanup.

💡 Parallel Execution: Run multiple suites in parallel using different sub-processes or agents.

🐛 Debugging: Use make run-keep to keep instances alive (remember to clean up!).

📊 Monitoring: Use make run (not bash sleep loops) for proper progress monitoring.

🔄 Retrying: Fix code issues, commit, push, then re-run with id=new or continue with id=last.

→ Detailed notes and best practices