eden-manual-evaluator

name	eden-manual-evaluator
description	Drive the EDEN evaluator role end-to-end from the terminal — clone at variant commit, help the user inspect, then submit evaluation via CLI. Trigger phrases: "play evaluator", "evaluate a variant", "act as evaluator", "score a variant", "I want to evaluate".

EDEN Manual — Evaluator Role (CLI)

When to use

User is running the EDEN reference Compose stack with no automated evaluator-host and wants to play the evaluator role from the terminal. Trigger phrases above.

What "evaluator" means

The evaluator takes one evaluation task (which references a variant whose executor-side work is committed and pushed), inspects the variant's commit, and submits one of:

• success + a value for every entry in experiment_config.evaluation_schema.
• error (the variant code itself errored — submission status records that).
• evaluation_error (the evaluator couldn't form a verdict, e.g. couldn't fetch the variant commit, can't measure, etc.).

After the evaluator submits success, the orchestrator transitions the variant from starting to success and the integrator integrates it (creating the canonical refs/heads/variant/<id>-<slug>).

Everything below runs from the terminal. Don't ask the user to open the web UI.

Workflow

EDEN=/Users/ericalt/Documents/eden-worktrees/test-main/reference/scripts/manual-ui/eden-manual

Phase 1: Pick a task (automatic)

$EDEN list-tasks --kind evaluation --state pending

If empty, tell the user there's nothing to evaluate yet and suggest finishing an execution cycle first.

The web-ui evaluator page is the point-and-click alternative to this CLI listing. Its pending-task table (issue #137) is a high-signal slug / priority / target / created by grid, priority-sorted by default, with eligible for me / target / group by creator filter chips and a per-row context links expander (which also links the variant under evaluation). Sort + filter state lives in the URL query string, so a curated view is shareable. When helping the user choose, mirror that signal: lead with the highest-priority eligible task's slug.

Phase 2: Inspect task + variant (automatic)

$EDEN show <task-id>

Present:

• The variant: branch, commit_sha, parent_commits, description.
• The evaluation_schema (names + types) from experiment-config.yaml.
• The objective so the user knows the direction to score in.

Phase 3: Claim (automatic)

$EDEN claim <task-id> --worker-name eden-manual

Post-#128: worker ids are opaque, system-minted wkr_* strings; supply a display name, not the id. On first use the CLI registers a worker with --worker-name eden-manual (default eden-manual), reads the minted worker_id from the wire response, and caches {worker_id, name, token} at /tmp/eden-manual/.credentials.json. Render the claimant as <name> (<worker_id>) when echoing it back.

Phase 4: Clone at the variant commit (automatic)

$EDEN checkout <task-id>

Workdir is /tmp/eden-manual/<task-id>, checked out at variant.commit_sha in detached HEAD. Surface the path.

Optionally show the diff vs. the parent commit so the user has the material context:

git -C /tmp/eden-manual/<task-id> log --oneline <parent_sha>..HEAD
git -C /tmp/eden-manual/<task-id> diff --stat <parent_sha> HEAD
git -C /tmp/eden-manual/<task-id> diff <parent_sha> HEAD

Offer to open in the user's editor.

Phase 5: Wait for the user's verdict (judgment)

Wait. Don't suggest scores unless asked. If asked, frame as: "evaluation field X is type <type>; what value reflects how this variant does on that dimension?" Don't fudge.

Phase 6: Submit (automatic)

$EDEN evaluation-submit <task-id> --field score=0.42 [--field K=V ...]

For status=success, every field in the evaluation_schema must have a value. For status=evaluation_error or error, omit --field:

$EDEN evaluation-submit <task-id> --status evaluation_error

Phase 7: Verify (automatic)

$EDEN show <task-id> | python3 -c 'import json,sys; print(json.load(sys.stdin)["task"]["state"])'

Should be completed. Within ~1 second, the integrator integrates the variant — confirm:

curl -fsS -H "Authorization: Bearer admin:$(grep '^EDEN_ADMIN_TOKEN=' \
    /Users/ericalt/Documents/eden-worktrees/test-main/reference/compose/.env | \
    cut -d= -f2)" \
  -H "X-Eden-Experiment-Id: $(grep '^EDEN_EXPERIMENT_ID=' \
    /Users/ericalt/Documents/eden-worktrees/test-main/reference/compose/.env | \
    cut -d= -f2)" \
  http://localhost:8080/v0/experiments/$(grep '^EDEN_EXPERIMENT_ID=' \
    /Users/ericalt/Documents/eden-worktrees/test-main/reference/compose/.env | \
    cut -d= -f2)/variants | python3 -m json.tool

Look for variant_commit_sha populated on the variant — that's proof of integration.

Best practices

• Read the diff, not just the description. The executor's description is unverified; diff <parent_sha> HEAD is ground truth.
• evaluation_error is honest when you can't form a verdict. Do not fabricate a score to submit success.
• Use the variant's own parent_commits as the diff base, not the experiment's seed — the variant's parent might be an integrated prior variant (chained evolution).
• Worker identity is name-supplied, id-returned. Never hardcode an opaque worker_id (wkr_<26-char-ULID>, minted by the server). Pass --worker-name <name>; the CLI reads the minted id from the wire response and caches it. To find a worker by display name, GET .../workers?name=<name> returns 0..N matches (names MAY collide) — disambiguate by id.
• wire error 401 on /workers...? The running stack's EDEN_ADMIN_TOKEN has diverged from the .env file the CLI is reading. Bounce the stack against the current .env, or re-checkout the worktree the stack was brought up against.