authoring-experiments

name	authoring-experiments
description	Use when creating or editing a GitHub Copilot experiment in this repository: defining Harbor/Pier task directories and Pier JobConfig YAML.

Authoring experiments

A task is a Harbor/Pier task directory. An experiment is a Pier JobConfig YAML file that combines tasks, agents, models, and attempts.

Task

• instruction.md — the instruction handed to the agent.
• task.toml — task metadata, environment, verifier, and artifacts.
• environment/ — Dockerfile or other environment definition.
• tests/test.sh — verifier script. It should write 1 or 0 to /logs/verifier/reward.txt.
• solution/solve.sh — optional oracle/reference solution.

Job config

job_name: my-task
jobs_dir: jobs
n_attempts: 3
n_concurrent_trials: 2
agents:
  - import_path: copilot_experiments.pier_agents.copilot_cli:CopilotCli
    model_name: claude-opus-4.7
    kwargs:
      reasoning_effort: medium
tasks:
  - path: ../tasks/my-task

Use more agents rows for model/agent comparisons and more tasks rows or datasets for suites.

For DeepSWE, do not rewrite tasks. Clone datacurve-ai/deep-swe and generate a Pier dataset job:

copilot-experiments deepswe-import vendor/deep-swe --n-tasks 3 --sample-seed 0

Validate

copilot-experiments validate   # validates configs, paths, auth, and backend setup