メニュー
Evaluate VibeTeam agents by running tasks directly via scripts/run_agent.py and comparing responses
デフォルトでは、最初にソースを確認する Prompt が選択されています。直接コマンドに切り替えるか、ローカルコピーをダウンロードすることもできます。
インストールを決める前に、SKILL.md と SkillsMP に表示されている付属ファイルをお読みください。
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
直接コマンドでは確認用 Prompt が省略されます。実行前にソースを確認してください。
npx skills add https://github.com/VibeTechnologies/VibeTeam --skill agent-evaluation-directコマンドは1行のまま表示されます。コピー前に横へスクロールして全体を確認してください。
ローカルで確認しますか？SkillsMP が現在取得できるファイルをダウンロードできます。
SOC 職業分類に基づく
SKILL.md を表示中
| name | agent-evaluation-direct |
| description | Evaluate VibeTeam agents by running tasks directly via scripts/run_agent.py and comparing responses |
| license | MIT |
| compatibility | opencode |
| metadata | {"audience":"developers","workflow":"evaluation"} |
Evaluate agents by running tasks directly and comparing responses. OpenCode submits tasks to each agent using scripts/run_agent.py, then evaluates results with agents/benchmark.py.
scripts/run_agent.py for each frameworkComparativeEvaluator to score responsesRun single agent:
python scripts/run_agent.py autogen "List 3 GitHub issues"
python scripts/run_agent.py crewai
python scripts/run_agent.py openhands
Run all agents:
python scripts/run_agent.py all "List 3 GitHub issues"
JSON output (for parsing):
python scripts/run_agent.py autogen "List 3 GitHub issues" --json
Options:
--role - Agent role: software_engineer, support_engineer, release_engineer--json - Output as JSON--timeout - Timeout in seconds (default: 180)For each agent run, capture:
| Field | Description |
|---|---|
| Framework | autogen, crewai, openhands |
| Task | The input task |
| Response | Agent's output |
| Latency | Time in ms |
| Success | true/false |
| Framework | Input | Output | Score | Feedback | Recommendations |
|---|---|---|---|---|---|
| AutoGen | {task} | {truncated}... | 4/5 | {feedback} | {improvements} |
| CrewAI | {task} | {truncated}... | 3/5 | {feedback} | {improvements} |
| OpenHands | {task} | {truncated}... | 5/5 | {feedback} | {improvements} |
| Metric | Value |
|---|---|
| Winner | {framework} |
| Reasoning | {why} |
| Judge Model | {model} |
| Eval Time | {ms} |
# Run and capture output
python scripts/run_agent.py autogen "YOUR_TASK" --json > /tmp/autogen.json
python scripts/run_agent.py crewai "YOUR_TASK" --json > /tmp/crewai.json
python scripts/run_agent.py openhands "YOUR_TASK" --json > /tmp/openhands.json
Or run all at once:
python scripts/run_agent.py all "YOUR_TASK" --json
Use ComparativeEvaluator from agents/benchmark.py:
response field from each agent's outputevaluator.evaluate(task, responses)| Score | Meaning |
|---|---|
| 0 | Failed/error |
| 1 | Mostly wrong |
| 2 | Partial |
| 3 | Acceptable |
| 4 | Good |
| 5 | Excellent |
| Task | Role |
|---|---|
| List 3 recent GitHub issues | software_engineer |
| Summarize Sentry errors this week | support_engineer |
| Generate release notes for v1.2.0 | release_engineer |
| Triage open PRs | software_engineer |
| Check CI status | release_engineer |
| File | Purpose |
|---|---|
scripts/run_agent.py | CLI to run agents with tasks |
agents/benchmark.py | ComparativeEvaluator for scoring |
agents/autogen/*.py | AutoGen agents |
agents/crewai/*.py | CrewAI agents |
agents/openhands/*.py | OpenHands agents |
Create and configure role-scoped GitHub Apps for VibeTeam, map credentials to agents placeholders, and validate installation permissions/identity.
Run VibeTeam GitHub/Slack handoff validation with unit tests, Slack evals, GitHub webhook evals, and permission checks. Use when validating multi-agent GitHub communication (issues, discussions, PR comments) or when asked to prove changes via tests/evals and record status.
Create and configure VibeTeam Slack apps (one ingress app plus role-scoped responder apps), wire role tokens/secrets, and validate routing/identity behavior.