github-handoff-evals

name	github-handoff-evals
description	Run VibeTeam GitHub/Slack handoff validation with unit tests, Slack evals, GitHub webhook evals, and permission checks. Use when validating multi-agent GitHub communication (issues, discussions, PR comments) or when asked to prove changes via tests/evals and record status.

GitHub Handoff Evals

Checklist

1. Export env once:
   • export $( < ~/.env.d/codex.env )
   • export $( < .env )
2. Requirement for cross-channel handoff evals: use native role mentions only (@SoftwareEngineer, @SupportEngineer) in trigger text. Do not use slash mentions.
3. Pause rollouts before any eval:
   • kubectl rollout pause deployment/vibeteam-gateway -n vibeteam
   • kubectl rollout pause deployment/openhands-svc -n vibeteam
4. Run unit tests with rerunfailures disabled:
   • .venv/bin/python -m pytest tests/ -v -p no:rerunfailures
5. Run at least one Slack eval:
   • Preferred: .venv/bin/python scripts/eval_slack_e2e.py --scenario github_issue_pr_handoff_slack --channel C0ALG01DLJV --timeout 600
   • Fallback: .venv/bin/python scripts/eval_slack_e2e.py --scenario support_400_errors --channel C0ALG01DLJV --timeout 600
6. Run GitHub webhook evals (assignment-first, role-bot required):
   • Code-writing task (SoftwareEngineer bot assignee):
     • .venv/bin/python scripts/eval_github_e2e.py --scenario github_issue_handoff --repo VibeTechnologies/vibeteam-eval-hello-world --actor-login OpenCodeEngineer --issue-role software_engineer --issue-assignee 'vibeteam-swe-bot-260301[bot]' --timeout 600
   • Support task (SupportEngineer bot assignee):