experiment-sandbox

name	experiment-sandbox
description	Experiment sandbox execution for Research Loop. Supports four modes: local (venv), Docker (isolated containers), SSH remote (GPU compute on servers), and Colab (Google Drive bridge). Provides experiment harness templates, code validation, metric collection, deterministic seeding, and compute budget enforcement. Use before running experiments generated by the paper-pipeline.

If you were dispatched as a subagent to run a specific experiment in a sandbox, skip this skill. Execute the task and return structured results immediately. NEVER execute untrusted code outside a sandbox. If the experiment code contains os.system, subprocess, eval, exec, or network calls, REJECT it before execution. No exceptions.

Experiment Sandbox Skill

Sandboxed experiment execution for research pipelines. Runs generated experiment code in isolated environments with resource limits, metric collection, and reproducibility guarantees.

Sandbox Modes

Mode	When to Use	Isolation	GPU Support
local (venv)	Quick experiments, development, no sensitive data	Process-level	Limited
docker	Reproducible environments, dependency isolation	Container	Yes (with nvidia-docker)
ssh_remote	GPU compute on lab servers, cloud instances	Remote process	Yes
colab	Google Colab via Drive bridge	Remote session	Yes (T4/V100)

Default mode: local. Switch to docker or ssh_remote when GPU or strict reproducibility is required.

Prerequisites

Local Mode

python3 -m venv .venv
source .venv/bin/activate
pip install numpy

Docker Mode

# Build the experiment image
docker build -t research-loop/experiment:latest -f - <<'DOCKERFILE'
FROM python:3.11-slim
RUN pip install numpy
WORKDIR /experiment
DOCKERFILE

# For GPU support (Linux only):
# docker build -t research-loop/experiment:latest -f Dockerfile.gpu .

SSH Remote Mode

# SSH config
Host research-loop-worker
    HostName gpu-server.lab.edu
    User researcher
    IdentityFile ~/.ssh/id_rsa

# Verify connection
ssh research-loop-worker "python3 --version"

Colab Mode

# Mount Google Drive
# Create folder: ~/Library/CloudStorage/GoogleDrive-*/My Drive/research-loop
# In Colab: mount Drive, set WORKDIR, run the experiment

Experiment Harness Template

Every experiment must follow this structure for reliable metric collection:

import numpy as np
import json, time, sys

def main():
    # === SETUP ===
    np.random.seed(42)  # deterministic
    start_time = time.time()
    time_budget = 300   # seconds, from config

    # === TIME ESTIMATE ===
    # Run one condition to estimate
    # Print: TIME_ESTIMATE: Xs

    # === MAIN LOOP ===
    results = {}
    for condition in conditions:
        # Time guard
        elapsed = time.time() - start_time
        if elapsed > 0.8 * time_budget:
            print(f"TIME_GUARD: {elapsed:.1f}s — saving partial results")
            break

        # Run experiment
        metrics = run_condition(condition)

        # Log metrics
        print(f"condition: {condition}")
        for key, val in metrics.items():
            print(f"{key}: {val}")
        results[condition] = metrics

    # === SAVE ===
    with open("results.json", "w") as f:
        json.dump(results, f, indent=2, default=str)
    print(f"Total time: {time.time() - start_time:.1f}s")

if __name__ == "__main__":
    main()

Mandatory outputs:

• stdout: metric lines as name: value (one per line)
• results.json: structured JSON with all metrics
• TIME_ESTIMATE: Xs line before main loop
• Time guard at 80% of budget

Code Validation

Before any execution, validate the experiment code:

1. Syntax Check

python3 -c "import py_compile; py_compile.compile('main.py', doraise=True)"

2. Security Scan

Reject code containing:

• import os or from os import (except os.path)
• import subprocess
• import shutil
• import socket
• eval(, exec(
• __import__ (dynamic imports)
• open( with write mode (w, a, wb) outside allowed output paths

3. Import Check

python3 -c "
import sys
allowed = {'numpy', 'math', 'random', 'statistics', 'json', 'time', 'collections',
           'itertools', 'functools', 'typing', 'dataclasses'}
for mod in sys.modules:
    if mod not in allowed and mod.startswith('experiment_'):
        print(f'WARNING: unknown import {mod}')
"

4. Determinism Check

Verify that np.random.seed() or random.seed() is called before any random operations. If seeds are missing, the experiment is not reproducible.

Auto-Repair

If validation fails:

1. Parse the error message
2. Trace to root cause (not surface symptom)
3. Fix and re-validate (max 3 attempts)
4. If still failing after 3 attempts, log the failure and escalate

Running an Experiment

Local Mode

# Setup
cd /tmp/research-loop-experiment-$(uuidgen)
cp -r <experiment_dir>/* .
python3 -m venv .venv
source .venv/bin/activate
pip install numpy 2>&1 | tail -1

# Run with timeout
timeout 300 python3 main.py 2>&1 | tee run.log

# Check results
python3 -c "
import json
with open('results.json') as f:
    data = json.load(f)
print(f'Conditions: {len(data)}')
print(json.dumps(data, indent=2)[:500])
"

Docker Mode

docker run --rm \
  -v $(pwd)/<experiment_dir>:/experiment \
  -w /experiment \
  research-loop/experiment:latest \
  timeout 300 python3 main.py 2>&1 | tee run.log

SSH Remote Mode

# Copy experiment to remote
rsync -avz <experiment_dir>/ research-loop-worker:~/experiments/current/

# Run remotely
ssh research-loop-worker \
  "cd ~/experiments/current && timeout 300 python3 main.py" \
  2>&1 | tee run.log

Colab Mode

# Copy experiment to Drive
cp -r <experiment_dir>/* ~/Library/CloudStorage/GoogleDrive-*/My\ Drive/research-loop/experiments/current/

# In Colab:
# from google.colab import drive
# drive.mount('/content/drive')
# %cd /content/drive/MyDrive/research-loop/experiments/current
# !timeout 300 python3 main.py

Metric Collection

After the experiment runs, collect and validate metrics:

import json

def collect_metrics(run_dir):
    """Parse stdout and results.json into structured metrics dict."""
    # Parse stdout for "name: value" lines
    metrics = {}
    with open(f"{run_dir}/run.log") as f:
        for line in f:
            line = line.strip()
            if ": " in line and not line.startswith("#"):
                parts = line.split(": ", 1)
                try:
                    metrics[parts[0]] = float(parts[1])
                except ValueError:
                    metrics[parts[0]] = parts[1]

    # Merge with results.json
    try:
        with open(f"{run_dir}/results.json") as f:
            rj = json.load(f)
            metrics["_results_json"] = rj
    except (FileNotFoundError, json.JSONDecodeError):
        pass

    return metrics

Validation checks:

• All declared metrics from the experiment plan are present
• No NaN or Inf values in numeric metrics
• TIME_ESTIMATE was accurate within 20%
• Time guard did not fire prematurely (< 50% of budget)

Reproducibility Requirements

Every experiment MUST have:

1. Deterministic seeds — np.random.seed(42) or equivalent
2. Full config capture — save config/params used for the run
3. Environment snapshot — Python version, package versions:

   python3 --version > environment.txt
   pip freeze >> environment.txt
   

4. Timestamp and duration — start time, end time, total seconds
5. Git commit hash (if in a repo) — git rev-parse HEAD

Compute Budget Enforcement

Budget	Max Conditions	Max Seeds	Max Iterations/Condition
< 120s	1-2	1-2	≤ 1,000
120-300s	2-5	3-5	≤ 5,000
300-900s	5-10	5-10	≤ 20,000
> 900s	10-50	10-20	No limit (but guard at 80%)

Always print TIME_ESTIMATE: Xs based on a 1-condition pilot run before the main experiment loop. Always implement a time guard that saves partial results.

Anti-patterns

• Running without validation. Always validate code syntax and security before the first execution. Never skip this step.
• Ignoring NaN/Inf. If the experiment produces NaN or Inf values, treat this as a bug. Trace to root cause and fix, do not suppress.
• No partial results. If the experiment crashes at 90%, all work is lost. Always write intermediate results periodically.
• Non-deterministic runs. Without a fixed seed, the experiment cannot be reproduced. Always set seeds explicitly.
• Missing environment capture. Without knowing the Python version and package versions, results are not reproducible. Always run pip freeze.

Verification Checklist

• Code passes syntax check
• Security scan passes (no dangerous imports)
• Deterministic seeds are set before any random operations
• TIME_ESTIMATE line is present in the code
• Time guard is implemented at 80% of budget
• results.json contains all declared metrics
• stdout contains "{metric_name}: {value}" lines
• Environment is captured (python version, pip freeze)
• Experiment ran to completion or saved partial results