e2e-run

name	e2e-run
description	Run Line 2 E2E test — submit engineering task, wait for completion, verify, write report. Use when user wants to test the engineering pipeline end-to-end.
allowed-tools	Bash, Read, Write, Edit, Grep, Glob
argument-hint	<test> [--feature] [--no-cleanup] [--no-nuke]

E2E Engineering Test Runner

⚠️ Requires running orchestrator. This skill tests the live pipeline and needs all services up. Before starting, verify: curl -sf http://localhost:8000/api/projects/ > /dev/null && echo "API OK" || echo "API NOT RUNNING — run 'make up' first"

Run one or more E2E tests end-to-end: create project via the full pipeline (scaffold → architect → engineering → deploy), monitor progress, verify results, collect reports, cleanup.

Arguments

• $0 — test selector (REQUIRED):
  • Project name: todo_api, weather_bot
  • Test number: 1, 2
  • Comma-separated: 1,2 or todo_api,weather_bot
  • all — run both tests sequentially
• --feature — after create+deploy succeeds, trigger a feature-add on the same project
• --no-cleanup — skip cleanup (keep repo, containers, DB records)
• --no-nuke — skip make nuke in Step 0 (assume stack is already clean)

Test Matrix

#	Name	Modules	Description
1	todo_api	backend	REST API for TODO items. GET/POST/PATCH/DELETE /todos. Fields: id, title, description, is_completed, created_at.
2	weather_bot	backend,tg_bot	/weather <city> returns mock data. Backend caches in PG 30min. GET /api/weather/{city} also available.

Feature Add Matrix (for --feature mode)

#	Name	Feature Description
1	todo_api	Add GET /todos/stats endpoint that returns {"total": N, "completed": N, "pending": N} counting all todos.
2	weather_bot	Add /forecast <city> command that returns mock 3-day forecast (today, tomorrow, day after). Backend endpoint GET /api/forecast/{city}.

Worker Audit

Workers already write /workspace/REPORT.md (per INSTRUCTIONS.md) with Issues Encountered and Suggestions sections. This IS the audit report — no separate AUDIT_REPORT.md needed. Worker reports are collected via task events API (step 7a).

QA worker writes QA_REPORT.md in the project dir on the server and logs the content as qa_report_content event. QA reports are collected via qa-worker logs (step 7c).

Key References

• docs/PIPELINE_V2.md — full pipeline architecture and status transitions
• docs/parallel-workers.md — worker containers, networks, bind-mounts
• docs/DEPLOY.md — deploy workflow, GitHub Actions, server setup
• docs/SECRETS.md — secret levels and handling

Key containers (each is separate in docker-compose):

• langgraph — PO agent
• architect — story decomposition (NOT inside scheduler!)
• scheduler — scaffold trigger, task dispatcher, story completion
• engineering-worker — engineering consumer
• deploy-worker — deploy consumer
• qa-worker — QA consumer (post-deploy testing via Claude Code on prod server)
• worker-manager — spawns worker containers
• scaffolder — project scaffolding

Status flow: DRAFT → scaffold → ACTIVE → architect → tasks (TODO→IN_DEV→DONE) → PR_REVIEW → DEPLOYING → TESTING → COMPLETED

E2E Secrets (for tg_bot tests)

Tests with tg_bot module need a TELEGRAM_BOT_TOKEN for deploy. Secrets are read from .claude/e2e-secrets.env (gitignored).

Token is given to PO in conversation — PO validates and stores it itself. Do NOT inject secrets manually via API. When PO asks for the bot token during the conversation flow, read it from the env file and send it as a chat message:

TG_TOKEN=$(grep -E '^TELEGRAM_BOT_TOKEN=' .claude/e2e-secrets.env 2>/dev/null | cut -d= -f2-)

if [ -z "$TG_TOKEN" ]; then
  echo "ERROR: TELEGRAM_BOT_TOKEN not found in .claude/e2e-secrets.env"
  # STOP this test — cannot deploy without token
fi

Then use the standard po:input / po:response mechanism to send the token as MESSAGE_TEXT="$TG_TOKEN" when PO asks for it.

GitHub & Server Access

See "GitHub Access" and "Server Access" in .claude/skills/shared/pipeline-recipes.md. Key point: local gh CLI has NO access — always use GitHubAppClient via docker compose exec.

Execution Flow

MANDATORY STEPS: Every test run MUST execute ALL steps in order: Steps 0–8.5 (test + report + commit), then Step 9 (Cleanup). Cleanup is NOT optional — skip it ONLY if --no-cleanup was explicitly passed. After Step 8.5, proceed to Step 9 immediately. Do NOT stop, summarize, or wait for user input between commit and cleanup.

Run tests sequentially (one at a time).

Repo naming: PO may create the GitHub repo with either underscores (weather_bot) or hyphens (weather-bot) — you cannot predict which. Always define both variants early:

REPO_SLUG=$(echo "$PROJECT_NAME" | tr '_' '-')   # hyphenated
REPO_UNDER=$(echo "$PROJECT_NAME" | tr '-' '_')   # underscored

Use both $REPO_SLUG and $REPO_UNDER when searching GitHub repos, server dirs, and docker containers. After Step 1d (Extract IDs), read the actual repo name from the API and set REPO_NAME to the real value — use that for all subsequent GitHub operations.

Step 0: Health check + pre-flight cleanup

Skip make nuke if --no-nuke is set.

make nuke

Then verify the stack:

curl -sf http://localhost:8000/health | jq .
docker compose ps --format "{{.Name}} {{.Status}}" | grep -v "Up"

If API is not healthy, STOP.

Worker image staleness check:

CURRENT_HASH=$(find shared packages/worker-wrapper \
  services/worker-manager/images -type f \
  -not -path '*/__pycache__/*' -not -name '*.pyc' \
  | LC_ALL=C sort | xargs sha256sum 2>/dev/null | sha256sum | cut -c1-16)

STORED_HASH=$(docker inspect worker-base-common:latest \
  --format '{{index .Config.Labels "org.codegen.worker_source_hash"}}' 2>/dev/null || echo "none")

if [ "$CURRENT_HASH" != "$STORED_HASH" ]; then
  echo "Worker images stale — rebuilding..."
  make rebuild-worker-images
else
  echo "Worker images up to date"
fi

Pre-flight cleanup (run for every test):

ORG="project-factory-organization"
REPO_SLUG=$(echo "$PROJECT_NAME" | tr '_' '-')
REPO_UNDER=$(echo "$PROJECT_NAME" | tr '-' '_')

# 1. Delete leftover GitHub repos (check BOTH underscore and hyphen variants)
# NOTE: Use org-level token, NOT GitHubAppClient.delete_repo() — the repo-scoped
# token fails with 404 on /repos/{repo}/installation for repos where the GitHub App
# installation is not bound yet.
docker compose exec -T api python -c "
import asyncio, httpx
from shared.clients.github import GitHubAppClient
async def main():
    gh = GitHubAppClient()
    token = await gh.get_org_token('$ORG')
    async with httpx.AsyncClient() as client:
        for repo in ['$REPO_SLUG', '$REPO_UNDER']:
            resp = await client.delete(
                f'https://api.github.com/repos/$ORG/{repo}',
                headers={'Authorization': f'token {token}', 'Accept': 'application/vnd.github+json'}
            )
            if resp.status_code == 204:
                print(f'Deleted leftover repo {repo}')
            elif resp.status_code == 404:
                print(f'{repo}: clean')
            else:
                print(f'{repo}: unexpected {resp.status_code}')
asyncio.run(main())
"

# 2. Kill leftover worker containers (by label AND by name pattern)
docker ps --filter "label=com.codegen.type=worker" --format "{{.Names}}" | xargs -r docker rm -f
docker ps -a --format "{{.Names}}" | grep -iE "${REPO_SLUG}|${REPO_UNDER}" | xargs -r docker rm -f

# 2.5. Clean scaffolder workspace (INSIDE the scaffolder container, not just host volume).
# Stale workspace with local git commits causes "nothing to commit" on re-scaffold.
REPO_ID_CLEANUP=$(curl -s "http://localhost:8000/api/projects/" \
  | jq -r --arg name "$PROJECT_NAME" --arg slug "$REPO_SLUG" \
    '.[] | select(.name == $name or .name == $slug) | .repositories[0].id // empty' 2>/dev/null | head -1)
if [ -n "$REPO_ID_CLEANUP" ]; then
  docker compose exec -T scaffolder rm -rf "/data/workspaces/$REPO_ID_CLEANUP" 2>/dev/null && \
    echo "Cleaned scaffolder workspace: $REPO_ID_CLEANUP" || true
  docker run --rm -v /data/workspaces:/workspaces alpine sh -c "rm -rf /workspaces/$REPO_ID_CLEANUP" 2>/dev/null || true
fi

# 3. Clean stale deployments on servers (check BOTH underscore and hyphen variants)
#    After compose down, also force-remove containers by name and verify ports are freed.
for SERVER_IP in $(curl -s "http://localhost:8000/api/servers/?is_managed=true" | jq -r '.[].public_ip'); do
  for DIR_NAME in "$REPO_UNDER" "$REPO_SLUG"; do
    HAS_DIR=$(bash infra/scripts/ssh-to-server.sh $SERVER_IP \
      "[ -d /opt/services/$DIR_NAME ] && echo EXISTS || echo CLEAN" 2>&1 | grep -xE 'EXISTS|CLEAN' || echo "SSH_FAIL")
    if [ "$HAS_DIR" = "EXISTS" ]; then
      echo "WARNING: Stale deployment $DIR_NAME on $SERVER_IP — cleaning"
      bash infra/scripts/ssh-to-server.sh $SERVER_IP "
        cd /opt/services/$DIR_NAME/infra 2>/dev/null && \
          docker compose --env-file ../.env -f compose.base.yml -f compose.prod.yml down -v --remove-orphans 2>/dev/null || true
        # Force-remove containers by name pattern (catches orphans after compose down)
        docker ps -a --format '{{.Names}}' | grep -i '$DIR_NAME' | xargs -r docker rm -f 2>/dev/null || true
        rm -rf /opt/services/$DIR_NAME
      "
    fi
  done
  # Verify no ports are still occupied by stale containers
  # NOTE: ssh-to-server.sh may emit SSH warnings (known_hosts updates) to stdout/stderr.
  # Filter to only lines that look like docker ps output (contain a container name).
  STALE_PORTS=$(bash infra/scripts/ssh-to-server.sh $SERVER_IP \
    "docker ps --format '{{.Names}} {{.Ports}}' | grep -iE '${REPO_SLUG}|${REPO_UNDER}'" 2>&1 \
    | grep -ivE '^#|known_hosts|Warning:|^$' || true)
  if [ -n "$STALE_PORTS" ]; then
    echo "WARNING: Stale containers still running on $SERVER_IP after cleanup: $STALE_PORTS"
    echo "  Force-removing..."
    bash infra/scripts/ssh-to-server.sh $SERVER_IP \
      "docker ps --format '{{.Names}}' | grep -iE '${REPO_SLUG}|${REPO_UNDER}' | xargs -r docker rm -f" 2>/dev/null || true
  fi
done

Step 0.5: Queue Health Check

Before starting, check queues for stale messages. Stale messages from previous runs can clog the architect for hours.

Recipes: See "Queue Health Check" in .claude/skills/shared/pipeline-recipes.md for bash commands (Debug API, raw Redis cross-check, stale message cleanup).

Step 1: Create via PO agent

1a. Upsert test user:

curl -s -X POST http://localhost:8000/api/users/upsert \
  -H "Content-Type: application/json" \
  -d '{
    "telegram_id": 999000001,
    "username": "e2e_test_user",
    "first_name": "E2E",
    "last_name": "Test",
    "is_admin": false
  }' | jq .

1b. Send to PO via po:input.

Compose a natural-language message with project name, modules, description, and audit instructions.

REQUEST_ID=$(python3 -c 'import uuid; print(uuid.uuid4())')
E2E_USER_ID="999000001"

docker compose exec -T \
  -e "REQUEST_ID=$REQUEST_ID" \
  -e "E2E_USER_ID=$E2E_USER_ID" \
  -e "MESSAGE_TEXT=$MESSAGE_TEXT" \
  api python -c "
import os, asyncio
from shared.contracts.queues.po import POUserMessage, to_flat_fields
from shared.redis.client import RedisStreamClient
from shared.queues import PO_INPUT_QUEUE

async def main():
    client = RedisStreamClient()
    await client.connect()
    msg = POUserMessage(
        text=os.environ['MESSAGE_TEXT'],
        user_id=os.environ['E2E_USER_ID'],
        request_id=os.environ['REQUEST_ID'],
    )
    mid = await client.publish_flat(PO_INPUT_QUEUE, to_flat_fields(msg))
    print(f'Published to po:input: mid={mid}')
    await client.close()

asyncio.run(main())
"

1c. Wait for PO response (timeout 120s):

docker compose exec -T -e "REQUEST_ID=$REQUEST_ID" api python -c "
import os, asyncio

async def main():
    import redis.asyncio as redis
    r = redis.from_url('redis://redis:6379')
    request_id = os.environ['REQUEST_ID']
    stream = f'po:response:{request_id}'
    for attempt in range(24):
        result = await r.xread({stream: '0'}, block=5000, count=1)
        if result:
            for stream_name, messages in result:
                for mid, fields in messages:
                    text = fields.get(b'text', b'').decode()
                    error = fields.get(b'error', b'').decode()
                    if error:
                        print(f'PO ERROR: {error}')
                    else:
                        print(f'PO RESPONSE: {text}')
                    await r.delete(stream)
                    await r.aclose()
                    return
    print('TIMEOUT: PO did not respond within 120s')
    await r.aclose()

asyncio.run(main())
"

1d. Extract IDs:

PO may create the project with a hyphenated name (weather-bot) even if you said weather_bot. Search by both variants:

REPO_SLUG=$(echo "$PROJECT_NAME" | tr '_' '-')

PROJECT_ID=$(curl -s "http://localhost:8000/api/projects/" \
  | jq -r --arg name "$PROJECT_NAME" --arg slug "$REPO_SLUG" \
    '.[] | select(.name == $name or .name == $slug) | .id' | head -1)

STORY_ID=$(curl -s "http://localhost:8000/api/stories/?sort=-created_at" \
  | jq -r --arg pid "$PROJECT_ID" '.[] | select(.project_id == $pid) | .id' | head -1)

REPO_ID=$(curl -s "http://localhost:8000/api/repositories/?project_id=$PROJECT_ID" \
  | jq -r '.[0].id // empty')

REPO_NAME=$(curl -s "http://localhost:8000/api/repositories/?project_id=$PROJECT_ID" \
  | jq -r '.[0].name // empty')

echo "PROJECT_ID=$PROJECT_ID STORY_ID=$STORY_ID REPO_ID=$REPO_ID REPO_NAME=$REPO_NAME"

If either is empty, send a follow-up to PO. If PO fails after 2 attempts, note "PO failed to create project" in report and stop the test.

1e. If test includes tg_bot — PO will ask for the bot token during the conversation (Step 1b/1c loop). When it does, read the token from .claude/e2e-secrets.env (see "E2E Secrets") and send it as a regular chat message via po:input. PO validates and stores the token itself — do NOT inject secrets manually via API.

Step 2: Monitor Scaffold

Scaffold is triggered automatically by scaffold_trigger (runs every 30s in scheduler). It checks for DRAFT projects with stories and repositories.

# Poll project status — scaffold transitions DRAFT → ACTIVE
# Timeout: 90s (6 × 15s). If scaffold doesn't complete in 90s, it's a bug — investigate immediately.
for i in $(seq 1 6); do
  STATUS=$(curl -s "http://localhost:8000/api/projects/$PROJECT_ID" | jq -r '.status')
  WORKSPACE=$(curl -s "http://localhost:8000/api/projects/$PROJECT_ID" | jq -r '.config.workspace_ready // false')
  echo "[$i/6] Project: status=$STATUS workspace_ready=$WORKSPACE"
  if [ "$STATUS" = "active" ] && [ "$WORKSPACE" = "true" ]; then
    echo "Scaffold complete"
    break
  fi
  sleep 15
done

If stuck after 90 seconds: This is a bug. Check scaffolder and scheduler logs:

docker compose logs scaffolder --tail=30 --since=5m 2>/dev/null
docker compose logs scheduler --tail=30 --since=5m 2>/dev/null | grep -i scaffold

Light intervention: If scaffold_trigger hasn't fired, check the scaffold:queue and inflight key. If there's a stale scaffold:inflight:{project_id} Redis key, clear it:

docker compose exec -T api python3 -c "
import asyncio, redis.asyncio as redis
async def main():
    r = redis.from_url('redis://redis:6379')
    key = 'scaffold:inflight:$PROJECT_ID'
    val = await r.get(key)
    if val:
        await r.delete(key)
        print(f'Cleared stale inflight key: {key}')
    else:
        print('No inflight key found')
    await r.aclose()
asyncio.run(main())
"

Step 3: Monitor Architect

The architect runs in its own container (not scheduler!).

# Poll for tasks created by architect
for i in $(seq 1 30); do
  TASKS=$(curl -s "http://localhost:8000/api/tasks/?story_id=$STORY_ID&sort=created_at")
  COUNT=$(echo "$TASKS" | jq 'length')
  echo "[$i/30] Tasks: $COUNT"
  if [ "$COUNT" -gt "0" ]; then
    echo "$TASKS" | jq -r '.[] | "\(.id)  \(.status)  \(.title)"'
    break
  fi
  sleep 10
done

If no tasks after 5 minutes: Check architect logs:

docker compose logs architect --tail=50 --since=5m 2>/dev/null | grep -v "HTTP Request" | tail -20

Known architect issues:

• blocked_by_task_id="None" string bug: Architect LLM returns Python "None" instead of null. Causes 500 FK violation. Check API logs:

  docker compose logs api --tail=200 --since=5m 2>/dev/null | grep -A5 "500\|ForeignKey"
  

  If this happens: note in report as a finding. Do NOT create tasks manually — just document it.

• Scaffold timeout: If project is still DRAFT, architect waits up to 5 min. Check if scaffold actually completed (see Step 2).

Verify: Task chain should have sensible descriptions and correct blocking order.

Post-architect check: After tasks appear, verify the first unblocked task transitions to in_dev within 2 minutes. If it stays todo:

# Check dispatcher is running and picking up tasks
docker compose logs scheduler --since=2m 2>/dev/null | grep -i dispatch | tail -10

# Check engineering:queue — was a message published?
curl -s "http://localhost:8000/debug/queues/engineering:queue/messages?count=10" | python3 -c "
import json, sys
data = json.load(sys.stdin)
print(f'Engineering queue: {data[\"total\"]} messages')
for m in data['messages']:
    print(f\"  {m['id']}  task={m['data'].get('task_id','?')}\")
"

If no dispatch after 2 min, the dispatcher may be stuck or the task isn't in todo status.

Step 4: Monitor Engineering

Track each task through its lifecycle. This is the longest phase.

Recipes: See "Task Status Polling", "Worker Monitoring", and "CI Status Check" in .claude/skills/shared/pipeline-recipes.md for bash commands.

What to check at each status:

todo → in_dev: Task dispatcher picks up unblocked TODO tasks every 30s.

• If stuck > 2 min and not blocked: check scheduler logs

in_dev: Worker is running. Use Docker ps (primary) and WM API (cross-check) from recipes.

in_ci: Code pushed, CI running. Use CI Status Check recipe.

done: Collect worker report via task events API (event_type=worker_report).

failed: Read failure_metadata from task API. If retriable:

curl -X POST "http://localhost:8000/api/tasks/$TASK_ID/transition?to_status=backlog"
curl -X POST "http://localhost:8000/api/tasks/$TASK_ID/transition?to_status=todo"

Document the retry in the timeline. If it fails again, record and move on.

Polling loop

Timeouts by phase (only workers take a long time — everything else should be fast):

• Scaffold: 90s max (poll every 15s) — if not done, it's a bug
• Architect: 5 min max (poll every 10s)
• Engineering worker: 30 min per task (poll every 30s) — this is the only long phase
• PR merge / auto-merge: 2 min max (poll every 15s). If stuck, check and intervene
• Deploy: 5 min max (poll every 15s). Deploy-worker triggers GH Actions then waits

If a non-worker phase exceeds its timeout, don't keep waiting — investigate immediately.

Keep a timeline log:

HH:MM  task-xxx  todo → in_dev (worker started)
HH:MM  task-xxx  in_dev → in_ci (commit pushed)
HH:MM  task-xxx  CI passed → done

Step 5: Monitor PR Review & Deploy

When all tasks are done, the dispatcher creates a PR from story/{story_id} → main, enables auto-merge, and transitions the story to pr_review. Deploy is triggered by poll_merged_prs() in the scheduler (runs every 30s) — it detects merged PRs for stories in pr_review and publishes a deploy message.

Flow: in_progress → pr_review → (PR merged, poller detects) → deploying → completed

IMPORTANT: The dispatcher's complete_stories only checks stories in in_progress status. If the story is stuck in created after all tasks are done (shouldn't happen in normal flow but can with race conditions):

# Light intervention: transition story to in_progress
curl -s -X POST "http://localhost:8000/api/stories/$STORY_ID/start" \
  -H "Content-Type: application/json" \
  -d '{"actor": "e2e-test"}'

Story API: Action-based endpoints

See "Story API — Action-Based Transitions" in .claude/skills/shared/pipeline-recipes.md.

Monitoring deploy

# Watch story status
curl -s http://localhost:8000/api/stories/$STORY_ID | python3 -c "
import json, sys
s = json.load(sys.stdin)
print(f\"Story: {s['status']}\")
"

# Deploy worker logs
docker compose logs deploy-worker --tail=50 --since=5m 2>/dev/null | grep -v "HTTP Request" | tail -20

Poll story status every 15s, timeout after 5 minutes (deploy itself runs on GitHub Actions, the deploy-worker just triggers it and waits). If no progress after 5 min, check deploy-worker logs. Story goes through: pr_review → deploying → testing → completed (or failed).

Step 5.5: Monitor QA Phase

After deploy succeeds, the deploy-worker publishes a QAMessage to qa:queue and transitions the story to testing. The QA consumer (qa-worker container) SSHes to the prod server and runs Claude Code CLI to test the deployed project as a real user would.

# Check story entered testing
curl -s http://localhost:8000/api/stories/$STORY_ID | python3 -c "
import json, sys
s = json.load(sys.stdin)
print(f\"Story: {s['status']}\")
"

# QA worker logs
docker compose logs qa-worker --tail=50 --since=5m 2>/dev/null | grep -v "HTTP Request" | tail -20

# Check qa:queue
curl -s "http://localhost:8000/debug/queues/qa:queue/messages?count=10" | python3 -m json.tool

What to watch:

• QA consumer picks up the message from qa:queue
• SSH to prod server succeeds
• Claude Code runs the QA prompt (tests endpoints, checks responses)
• QA result is parsed (JSON with pass, checks, summary)
• Story transitions to completed (if QA passed) or back to in_progress (if failed, creates fix task)

Timeouts: QA has a 20-minute timeout per run. Poll story status every 30s.

If QA fails: Check qa-worker logs for the reason. Common issues:

• SSH connection failed (server unreachable, credentials expired)
• Claude Code not installed on server (run qa_runner Ansible role)
• Claude Code session expired (re-copy .credentials.json)
• QA prompt produced unparseable output (non-JSON response)

If QA is stuck: Check if the qa:queue message was consumed:

curl -s "http://localhost:8000/debug/queues/qa:queue/qa-consumers/pending" | python3 -m json.tool

Step 6: Verify Deployment

6a. CI status:

docker compose exec -T api python -c "
import asyncio
from shared.clients.github import GitHubAppClient
async def main():
    gh = GitHubAppClient()
    run = await gh.get_latest_workflow_run('project-factory-organization', '$REPO_SLUG', 'ci.yml', 'main')
    if run:
        print(f\"CI: {run['status']} / {run.get('conclusion')}\")
        print(f\"URL: {run['html_url']}\")
asyncio.run(main())
"

6b. Find server and verify:

# Find deployed URL from service-deployments
curl -s "http://localhost:8000/api/service-deployments/?project_id=$PROJECT_ID" | python3 -c "
import json, sys
deps = json.load(sys.stdin)
for d in deps:
    print(f\"URL: http://{d.get('server_ip')}:{d.get('port')}\")
    print(f\"Server: {d.get('server_ip')}\")
"

If deploy succeeded:

bash infra/scripts/ssh-to-server.sh $SERVER_IP "
  cd /opt/services/$REPO_SLUG/infra
  COMPOSE='docker compose --env-file ../.env -f compose.base.yml -f compose.prod.yml'
  echo '=== Container status ==='
  \$COMPOSE ps -a
  echo '=== Restart counts ==='
  for cid in \$(\$COMPOSE ps -q 2>/dev/null); do
    restarts=\$(docker inspect --format '{{.RestartCount}}' \"\$cid\")
    name=\$(docker inspect --format '{{.Name}}' \"\$cid\")
    echo \"\$name: restarts=\$restarts\"
  done
"

# Health check
curl -sf "http://$SERVER_IP:$DEPLOY_PORT/health" | jq . || echo "Health endpoint not responding"

If deploy failed — collect crash diagnostics:

bash infra/scripts/ssh-to-server.sh $SERVER_IP "
  PROJECT_DIR=/opt/services/$REPO_SLUG
  if [ ! -d \"\$PROJECT_DIR\" ]; then
    echo 'No deployment directory found'
    exit 0
  fi

  echo '=== .env contents ==='
  cat \$PROJECT_DIR/.env 2>/dev/null || echo 'NO .env FILE'

  cd \$PROJECT_DIR/infra
  COMPOSE='docker compose --env-file ../.env -f compose.base.yml -f compose.prod.yml'

  echo '=== Container status ==='
  \$COMPOSE ps -a 2>/dev/null || echo 'No containers'

  echo '=== Backend logs ==='
  \$COMPOSE logs backend --tail=50 2>/dev/null || echo 'No backend logs'

  echo '=== DB logs ==='
  \$COMPOSE logs db --tail=20 2>/dev/null || echo 'No db logs'
"

Step 7: Collect Reports

IMPORTANT: This step MUST complete and save files BEFORE Step 9 cleanup. Cleanup deletes task_events from DB — if reports aren't saved to disk first, they're lost.

7a. Collect worker reports (primary + fallback):

See "Worker Report Collection" in .claude/skills/shared/pipeline-recipes.md for the full bash script (task events API + workspace archive fallback).

7c. Collect QA report (from qa-worker logs):

The QA worker logs the full QA_REPORT.md content as a structured log event (qa_report_content). Collect it the same way as worker reports.

QA_REPORT="docs/e2e_results/worker_reports/${PROJECT_NAME}-${DATE}-qa.md"

# Extract QA report from qa-worker logs (logged as qa_report_content event)
docker compose logs qa-worker --since=30m 2>/dev/null \
  | grep "qa_report_content" \
  | grep "$STORY_ID" \
  | tail -1 \
  | python3 -c "
import json, sys
line = sys.stdin.read().strip()
idx = line.find('{')
if idx >= 0:
    data = json.loads(line[idx:])
    report = data.get('report', '')
    if report:
        print(report)
" > "$QA_REPORT" 2>/dev/null

if [ -s "$QA_REPORT" ]; then
  echo "QA report saved to $QA_REPORT"
else
  echo "WARNING: No QA report found in logs"
  echo "(no QA report collected)" > "$QA_REPORT"
fi

Fallback: If logs are empty, try reading QA_REPORT.md directly from the server (QA runner writes it to the project dir before collecting):

if [ ! -s "$QA_REPORT" ] || grep -q "no QA report" "$QA_REPORT"; then
  REPORT_FROM_SERVER=$(bash infra/scripts/ssh-to-server.sh $SERVER_IP \
    "cat /opt/services/$REPO_NAME/QA_REPORT.md 2>/dev/null" 2>&1 \
    | grep -v "^#.*known_hosts\|Warning:")
  if [ -n "$REPORT_FROM_SERVER" ]; then
    echo "$REPORT_FROM_SERVER" > "$QA_REPORT"
    echo "QA report collected from server"
  fi
fi

CRITICAL: Do NOT proceed to Step 9 cleanup (especially workspace deletion in step 9.9) until worker reports AND QA report are successfully saved to disk. The workspace archive is the last fallback — once deleted, reports are gone forever.

Step 8: Write E2E Report

File: docs/e2e_results/<project_name>-<date>.md

Never overwrite existing reports — append suffix: -2, -3, etc.

Classify each problem by type:

Type	Meaning
orchestrator	Bug in codegen_orchestrator
template	Bug in service-template
meta	Error in this skill's instructions
other	Network, transient, hardware

# E2E Report: <project_name> — <brief summary>

> **Date**: YYYY-MM-DD
> **Project**: <project_name> (project_id: `...`)
> **Story**: <story_id>
> **Status**: Passed / Failed
> **Feature phase**: passed / failed / skipped (only if --feature)
> **Smoke**: pass / fail / none
> **Worker reports**: collected (N) | none
> **QA report**: collected | none

---

## Timeline
(chronological log — key timestamps and events)

## PO Interaction

## Problems Found

### Problem 1: <title>
- **Type**: orchestrator | template | meta | other
- **Severity**: critical / major / minor
- **Backlog**: `#XX` | `new` | `template` | `—`
- **Description**: ...
- **Root cause**: ...
- **Suggested fix**: ...

Steps F1-F4: Feature Add Phase (only if --feature)

Skip unless --feature is set AND the initial create+deploy passed.

Step F1: Create feature story via PO

Send a feature request message to PO: "Добавь в мой $PROJECT_NAME: $FEATURE_DESCRIPTION"

Use the same po:input / po:response mechanism from Step 1b/1c.

Then extract the new story ID:

FEATURE_STORY_ID=$(curl -s "http://localhost:8000/api/stories/?sort=-created_at" \
  | jq -r --arg pid "$PROJECT_ID" '[.[] | select(.project_id == $pid)] | .[0].id')

echo "FEATURE_STORY_ID=$FEATURE_STORY_ID"

Step F2: Monitor feature pipeline

Same as Steps 3-5, but for $FEATURE_STORY_ID. Scaffold runs in ensure mode (workspace already exists — no copier, just verify).

Verify NO full scaffold ran:

docker compose logs scaffolder --tail=20 --since=5m 2>/dev/null | grep -E "mode|copier"

Step F3: Verify feature

Same as Step 6, plus verify the specific feature from Feature Add Matrix:

• todo_api: curl -sf http://$SERVER_IP:$DEPLOY_PORT/todos/stats | jq .
• weather_bot: curl -sf http://$SERVER_IP:$DEPLOY_PORT/api/forecast/moscow | jq .

Step F4: Collect feature reports

Same as Step 7 — fetch worker reports from task events for the feature story. Save to docs/e2e_results/worker_reports/${PROJECT_NAME}-${DATE}-feature-worker.md.

Step 8.5: Commit reports

git add docs/e2e_results/
git commit -m "e2e: $PROJECT_NAME — <pass/fail>"

Do NOT push.

Proceed to Step 9 immediately — do NOT stop here.

Step 9: Cleanup (skip ONLY if --no-cleanup)

# 1. Kill worker containers
docker ps --filter "label=com.codegen.type=worker" --format "{{.Names}}" | xargs -r docker rm -f

# 2. Delete GitHub repo
docker compose exec -T api python -c "
import asyncio
from shared.clients.github import GitHubAppClient
async def main():
    gh = GitHubAppClient()
    await gh.delete_repo('project-factory-organization', '$REPO_SLUG')
    print('Repo deleted')
asyncio.run(main())
"

# 3. Clean server deployment (check both underscore and hyphen variants)
if [ -n "$SERVER_IP" ]; then
  for DIR_NAME in "$PROJECT_NAME" "$REPO_SLUG"; do
    bash infra/scripts/ssh-to-server.sh $SERVER_IP "
      if [ -d /opt/services/$DIR_NAME/infra ]; then
        cd /opt/services/$DIR_NAME/infra
        docker compose --env-file ../.env -f compose.base.yml -f compose.prod.yml down -v --remove-orphans 2>/dev/null || true
      fi
      rm -rf /opt/services/$DIR_NAME
    " 2>/dev/null || true
  done
  echo "Server cleanup done"
fi

# 4. Delete deployment records
curl -s "http://localhost:8000/api/service-deployments/?project_id=$PROJECT_ID" \
  | jq -r '.[].id' | while read ID; do
    curl -s -X DELETE "http://localhost:8000/api/service-deployments/$ID"
  done

# 5. Delete project from DB via SQL (single transaction — partial deletes cause FK issues)
# Tasks have project_id directly. application_health_history must go before applications.
docker compose exec -T db psql -U postgres -d orchestrator -c "
  BEGIN;
  DELETE FROM task_events WHERE task_id IN (SELECT id FROM tasks WHERE project_id = '$PROJECT_ID');
  DELETE FROM runs WHERE project_id = '$PROJECT_ID';
  DELETE FROM tasks WHERE project_id = '$PROJECT_ID';
  DELETE FROM stories WHERE project_id = '$PROJECT_ID';
  DELETE FROM application_health_history WHERE application_id IN (SELECT id FROM applications WHERE repo_id IN (SELECT id FROM repositories WHERE project_id = '$PROJECT_ID'));
  DELETE FROM port_allocations WHERE application_id IN (SELECT id FROM applications WHERE repo_id IN (SELECT id FROM repositories WHERE project_id = '$PROJECT_ID'));
  DELETE FROM applications WHERE repo_id IN (SELECT id FROM repositories WHERE project_id = '$PROJECT_ID');
  DELETE FROM service_deployments WHERE project_id = '$PROJECT_ID';
  DELETE FROM repositories WHERE project_id = '$PROJECT_ID';
  DELETE FROM projects WHERE id = '$PROJECT_ID';
  COMMIT;
"

# 6. Trim stale messages from all queues
docker compose exec -T api python3 -c "
import asyncio
import redis.asyncio as redis

async def main():
    r = redis.from_url('redis://redis:6379')
    for q in ['scaffold:queue', 'architect:queue', 'engineering:queue',
              'deploy:queue', 'worker:commands', 'po:input', 'po:proactive']:
        length = await r.xlen(q)
        if length > 0:
            await r.xtrim(q, maxlen=0)
            print(f'{q}: trimmed {length} messages')
    # Clean stale po:response streams
    async for key in r.scan_iter('po:response:*'):
        await r.delete(key)
        print(f'Deleted {key.decode()}')
    # Clean scaffold inflight keys
    async for key in r.scan_iter('scaffold:inflight:*'):
        await r.delete(key)
        print(f'Deleted {key.decode()}')
    await r.aclose()

asyncio.run(main())
"

# 7. Clean PO checkpoint for e2e test user
docker compose exec -T db psql -U postgres -d orchestrator -c "
  DELETE FROM langgraph.checkpoint_writes WHERE thread_id = 'po-user-999000001';
  DELETE FROM langgraph.checkpoint_blobs WHERE thread_id = 'po-user-999000001';
  DELETE FROM langgraph.checkpoints WHERE thread_id = 'po-user-999000001';
"

# 8. Delete e2e test user from DB
docker compose exec -T db psql -U postgres -d orchestrator -c "
  DELETE FROM users WHERE telegram_id = 999000001;
"

# 9. Delete local workspaces for this project's repos
# $REPO_ID was captured in step 1d (before DB cleanup in step 5).
docker run --rm -v /data/workspaces:/workspaces alpine sh -c "rm -rf /workspaces/$REPO_ID"

# 10. Clean up worker sidecar containers and dev networks
docker ps -a --filter "name=worker_" --format "{{.Names}}" | xargs -r docker rm -f 2>/dev/null || true
docker network ls --filter "name=dev_proj" --format "{{.Name}}" | xargs -r docker network rm 2>/dev/null || true

Final Summary

After all tests complete, print a summary table:

## E2E Test Results

| # | Project | Create | Feature | Duration | Problems |
|---|---------|--------|---------|----------|----------|
| 1 | todo_api | PASS | PASS | 25min | 0 |
| 2 | weather_bot | FAIL | SKIP | 18min | 2 |

Total: X passed, Y failed out of Z tests

Error Handling & Light Interventions

What you CAN do (light interventions):

• Transition stuck story/task statuses (e.g., POST .../start)
• Clean stale queue messages
• Clear stale Redis keys (inflight markers)
• Retry failed tasks (transition back to todo)

What you should NOT do:

• Write or fix code
• Clone repos and push commits
• Create tasks manually
• Restart orchestrator services

If something needs a heavy intervention to proceed, record it as a finding and move on.

General rules:

• If a step fails, document and continue to the next step
• Do NOT stop the entire run on a single failure — collect as much data as possible
• If the API is unreachable, STOP — the stack is down
• Always attempt cleanup even if the test failed (unless --no-cleanup)

Abort & Collect

If the user asks to stop early:

1. Kill worker containers
2. Collect whatever data is available (Steps 6-7)
3. Write report with "Failed (aborted)"
4. Cleanup (unless --no-cleanup)

Common Gotchas

See "Common Gotchas" in .claude/skills/shared/pipeline-recipes.md for the full list.

E2E-specific additions:

• Repo deletion in pre-flight: uses get_org_token() + httpx.delete() directly (belt-and-suspenders, not just GitHubAppClient.delete_repo())
• Do NOT manually trigger deploys — poll_merged_prs() in scheduler handles deploy triggering automatically every 30s. Manual intervention breaks the flow
• Deploy Run record uses type field (not run_type) — POST /api/runs/ with {"type": "deploy"}

Self-Feedback (Mandatory)

Before generating your final response, check: did you encounter wrong commands, missing info, or unexpected errors? If yes, append to docs/skill-feedback.md:

## [e2e-run] — <today's date>
- **Type**: bug | missing-info | optimization
- **Quote**: "<exact line from this skill>"
- **Problem**: <what went wrong>
- **Suggested fix**: <concrete change>