Point at any repo (GitHub, GitLab, Bitbucket, Azure DevOps, self-hosted, or local) and get eval tasks that compare baseline coding agents vs MCP-augmented agents. Mines merged PRs/MRs for real code-change tasks, auto-generates ground truth from patches, produces runnable Docker environments for both configs. Works with private repos on any host. Triggers on mine tasks, propose tasks, discover tasks, find tasks, analyze repo for tasks, eval my repo, benchmark my repo.
Point at any repo (GitHub, GitLab, Bitbucket, Azure DevOps, self-hosted, or local) and get eval tasks that compare baseline coding agents vs MCP-augmented agents. Mines merged PRs/MRs for real code-change tasks, auto-generates ground truth from patches, produces runnable Docker environments for both configs. Works with private repos on any host. Triggers on mine tasks, propose tasks, discover tasks, find tasks, analyze repo for tasks, eval my repo, benchmark my repo.
user-invocable
true
Mine Tasks
Point at a codebase — on any code host or locally — and get a baseline-vs-MCP comparison for AI coding agents. Mines real merged PRs/MRs to create eval tasks where agents must reproduce known fixes/features, then measures whether MCP tools (code search, semantic indexing) help agents find the right code faster.
Works with GitHub, GitLab, Bitbucket, Azure DevOps, self-hosted Gitea/Forgejo, or plain local git repos.
Two workflows:
Quick eval (default): Mine 5-10 SDLC tasks from merged PRs/MRs, auto-generate ground truth, produce runnable Dockerfiles, print docker run commands. ~10 minutes to a working eval.
Full mining: Deep analysis for contributing tasks to CodeScaleBench. Includes org-scale tasks, scoring, reviewer extraction, and /scaffold-task integration.
Host Adapter Reference
All host-specific operations go through a detected adapter. The skill auto-detects the host from the URL and uses the appropriate CLI/API. When no API is available, it falls back to git-only mode.
Host Detection
URL Pattern
Host
CLI
Adapter
github.com/*
GitHub
gh
github
gitlab.com/* or self-hosted with /api/v4/
GitLab
glab
gitlab
bitbucket.org/*
Bitbucket
curl (REST v2)
bitbucket
dev.azure.com/* or *.visualstudio.com/*
Azure DevOps
az repos
azure
*.gitea.* or *.forgejo.* or user-specified
Gitea/Forgejo
curl (REST)
gitea
Local path (no remote)
—
—
git_only
Any other remote
—
—
git_only
Merge Request Operations
The skill needs these operations from each host. "MR" is used generically below (= PR on GitHub/Bitbucket, MR on GitLab, PR on Azure DevOps).
The Dockerfile {CLONE_COMMAND} uses the detected host's pattern. The secret name is always repo_token (host-agnostic):
# Public
RUN git clone --filter=blob:none {PUBLIC_CLONE_URL} /workspace && \
cd /workspace && git checkout {BASE_COMMIT}
# Private (any host)
# Requires: docker build --secret id=repo_token,env=REPO_TOKEN .
RUN --mount=type=secret,id=repo_token \
REPO_TOKEN=$(cat /run/secrets/repo_token) && \
git clone --filter=blob:none {PRIVATE_CLONE_URL_TEMPLATE} /workspace && \
cd /workspace && git checkout {BASE_COMMIT}
Repo Validation
# Universal (works with any host)
git ls-remote {CLONE_URL} HEAD 2>/dev/null
# GitHub
gh repo view {REPO} --json name,isPrivate 2>/dev/null
# GitLab
glab repo view {REPO} --output json 2>/dev/null
# Others: git ls-remote is sufficient
Contributor Discovery
# Universal (works with any host, no API needed)
# Top contributors for a path — this is the primary method
git -C {LOCAL_REPO} log --format='%an' --since="12 months ago" -- {PATH} \
| sort | uniq -c | sort -rn | head -10
# GitHub (richer data: PR reviewers)
gh api repos/{REPO}/pulls/{NUMBER}/reviews --jq '.[].user.login'
gh api "repos/{REPO}/commits?path={PATH}&per_page=30" --jq '.[].author.login'
# GitLab (MR approvers)
glab api "projects/{PROJECT_ID}/merge_requests/{NUMBER}/approvals" --jq '.approved_by[].user.username'
# Azure DevOps (PR reviewers)
az repos pr reviewer list --id {NUMBER} --organization {ORG_URL} --output json
# Bitbucket (PR participants)
curl -s -u "${BB_USER}:${BB_APP_PASSWORD}" \
"https://api.bitbucket.org/2.0/repositories/{REPO}/pullrequests/{NUMBER}" \
| jq '.participants[] | select(.role == "REVIEWER") | .user.display_name'
Phase 0: Eval Goals
Ask the user:
Question 1 — Header: "What are you evaluating?"
Question: "What do you want to measure?"
Options:
Quick eval: baseline vs MCP — "Mine 5-10 tasks from my repo, auto-generate ground truth, get a runnable comparison in minutes"
Full benchmark mining — "Deep analysis for CodeScaleBench contribution — SDLC tasks, org-scale tasks, reviewer extraction, the works"
If Quick eval, set QUICK_MODE=true. Skip detailed questions — use sensible defaults (SDLC only, auto-discover, 5-10 tasks). Proceed to Phase 1 with just the repo source question.
If Full mining, set QUICK_MODE=false. Proceed to Phase 1 with all questions.
Question 2 — Header: "MCP provider"
Question: "Which MCP provider will the augmented agent use?"
Options:
Sourcegraph — "Sourcegraph code search (keyword + semantic search via MCP server). Works with any code host."
Custom MCP server — "I'll provide my own MCP config JSON"
Not sure yet — "Generate both Dockerfiles, I'll configure MCP later"
Record the selection as {MCP_PROVIDER}. If "Custom MCP server", prompt for the MCP config JSON (or path to a .mcp.json file). If "Not sure yet", generate Dockerfiles with a placeholder MCP config and clear instructions for how to fill it in.
Note: source.type is "merge_request" for MR/PR-based tasks, "commit" for git-only tasks (where number and url may be null).
Generate test.sh that verifies the agent's changes:
#!/bin/bash
set -e
cd /workspace
mkdir -p /logs/verifier
git config --global --add safe.directory /workspace 2>/dev/null || true
# Check if any changes were made
CHANGES=$(git diff --stat 2>/dev/null | wc -l)
STAGED=$(git diff --cached --stat 2>/dev/null | wc -l)
UNTRACKED=$(git ls-files --others --exclude-standard 2>/dev/null | wc -l)
if [ "$CHANGES" -eq 0 ] && [ "$STAGED" -eq 0 ] && [ "$UNTRACKED" -eq 0 ]; then
echo "No changes detected"
echo "0.0" > /logs/verifier/reward.txt
exit 0
fi
SCORE=0
MAX_SCORE=10
# Check 1: Expected files were modified (5 points)
EXPECTED_FILES=({EXPECTED_FILE_LIST})
MODIFIED=0
for f in "${EXPECTED_FILES[@]}"; do
if git diff --name-only | grep -q "$f" || \
git diff --cached --name-only | grep -q "$f" || \
git log --oneline HEAD~1..HEAD --name-only 2>/dev/null | grep -q "$f"; then
MODIFIED=$((MODIFIED + 1))
fi
done
FILE_SCORE=$(awk "BEGIN {printf \"%.0f\", 5 * $MODIFIED / ${#EXPECTED_FILES[@]}}")
SCORE=$((SCORE + FILE_SCORE))
echo "Files modified: $MODIFIED/${#EXPECTED_FILES[@]} (${FILE_SCORE}/5 points)"
# Check 2: Tests pass (5 points)
# {LANGUAGE_SPECIFIC_TEST_COMMAND}
if {TEST_COMMAND} 2>&1; then
echo "PASS: Tests pass"
SCORE=$((SCORE + 5))
else
echo "FAIL: Tests failed"
fi
FINAL_SCORE=$(awk "BEGIN {printf \"%.1f\", $SCORE / $MAX_SCORE}")
echo "$FINAL_SCORE" > /logs/verifier/reward.txt
echo "Score: $FINAL_SCORE (${SCORE}/${MAX_SCORE})"
Fill in {EXPECTED_FILE_LIST} from the changed files and {TEST_COMMAND} based on the language:
Language
Test Command
Go
go test ./... -count=1 -timeout 120s
Python
python -m pytest -x --timeout=120
TypeScript
npm test or npx jest --forceExit
Java
./gradlew test or mvn test
Rust
cargo test
C++
cmake --build build && ctest --test-dir build
C#
dotnet test
Ruby
bundle exec rake test
Step 3e: Generate Dockerfiles (Baseline + MCP)
For each task, generate two Dockerfiles. Clone URLs and auth are determined by the detected host — see "Clone URLs and Auth" in the Host Adapter Reference.
Dockerfile.baseline — Full code, no MCP
FROM {BASE_IMAGE}
RUN apt-get update && apt-get install -y git curl ripgrep && rm -rf /var/lib/apt/lists/*
# Install Node.js (for Claude Code CLI)
RUN if ! command -v node &> /dev/null; then \
curl -fsSL https://deb.nodesource.com/setup_20.x | bash - && \
apt-get install -y nodejs; \
fi
RUN npm install -g @anthropic-ai/claude-code
# Clone repo at the pre-fix commit
{CLONE_COMMAND}
RUN mkdir -p /workspace/tests /logs/verifier
COPY tests/ /workspace/tests/
RUN chmod +x /workspace/tests/test.sh
WORKDIR /workspace
{CLONE_COMMAND} uses the host-specific URL pattern:
Public (any host):
RUN git clone --filter=blob:none {CLONE_URL} /workspace && \
cd /workspace && git checkout {BASE_COMMIT}
Where {AUTHENTICATED_CLONE_URL} is the host-specific pattern from the Clone URLs table (e.g., https://oauth2:${REPO_TOKEN}@gitlab.com/... for GitLab, https://x-access-token:${REPO_TOKEN}@github.com/... for GitHub).
Dockerfile.mcp — Truncated code + MCP tools
FROM {BASE_IMAGE}
RUN apt-get update && apt-get install -y git curl ripgrep jq && rm -rf /var/lib/apt/lists/*
RUN if ! command -v node &> /dev/null; then \
curl -fsSL https://deb.nodesource.com/setup_20.x | bash - && \
apt-get install -y nodejs; \
fi
RUN npm install -g @anthropic-ai/claude-code
# Clone repo (will be truncated below)
{CLONE_COMMAND}
# Truncate source files to stubs (signatures only, no bodies)
# This forces the agent to use MCP tools to understand the code
RUN find /workspace -type f \( -name '*.go' -o -name '*.py' -o -name '*.ts' -o -name '*.java' \
-o -name '*.cpp' -o -name '*.rs' -o -name '*.c' -o -name '*.h' -o -name '*.cs' -o -name '*.rb' \) \
-exec sh -c 'head -50 "$1" > "$1.stub" && mv "$1.stub" "$1"' _ {} \;
# MCP configuration
{MCP_CONFIG_BLOCK}
RUN mkdir -p /workspace/tests /logs/verifier
COPY tests/ /workspace/tests/
RUN chmod +x /workspace/tests/test.sh
WORKDIR /workspace
{MCP_CONFIG_BLOCK} depends on the provider selected in Phase 0:
# TODO: Add your MCP configuration here
# Create /root/.config/claude/mcp.json with your MCP server config
# See https://docs.anthropic.com/en/docs/claude-code/mcp for format
RUN mkdir -p /root/.config/claude && \
echo '{"mcpServers":{}}' > /root/.config/claude/mcp.json
Step 3f: Extract Reviewer Information (full mining only)
Use the detected host adapter for reviewer data. See "Contributor Discovery" in the Host Adapter Reference.
The git log method works universally regardless of host:
# Top contributors for changed paths (works everywhere)
for path in {CHANGED_DIRS}; do
git -C {REPO_PATH} log --format='%an' --since="12 months ago" -- "$path" \
| sort | uniq -c | sort -rn | head -5
done
For hosts with reviewer APIs (GitHub, GitLab, Azure DevOps), also pull MR reviewer data for richer results.
Filter out bot accounts (dependabot, renovate, bors, k8s-ci-robot, gitlab-bot, etc.).
Difficulty Estimation
Files Changed
Lines Changed
Cross-Package
Difficulty
1-2
<50
No
medium
1-3
50-200
No
hard
3-8
50-500
Yes
hard
4-10
200-1000
Yes
very_hard
10+
500+
Yes
expert
Phase 4: Org-Scale Task Mining (full mining only)
Skip this phase in quick eval mode or if user selected "SDLC tasks" only.
Step 4a: Cross-Repo Dependency Analysis
For multi-repo inputs, identify cross-repo relationships:
For single repos, analyze internal module boundaries:
# Go: list internal packages and their importers
go list ./... 2>/dev/null | head -50
rg 'import.*".*internal/' --type go -l | head -20
# Find interface definitions (potential tracing targets)
rg '(type|interface|trait|abstract class)\s+\w+' --type go --type java --type py -l | head -30
Find plugin registries, factory patterns, hook systems
Domain model relationships
domain-lineage
Find entity types and their relationships
Step 4c: Generate Org-Scale Proposals
For each identified pattern, propose a task with a natural-language question:
Cross-repo dep trace: "Which {language} source files in the {package}/ tree of {repo} directly import {dependency}?"
Config trace: "Trace how the {ConfigType} configuration defined in {repo1} gets consumed by {repo2}."
Compliance audit: "Find all files in {repo} that configure TLS/SSL settings."
Migration inventory: "Find all uses of the deprecated {API} across {repo1} and {repo2}."
Incident debug: "Given that {ErrorType} is being thrown at runtime, trace the error origin across {repo1} and {repo2}."
Onboarding comprehension: "Explain the architecture of the {subsystem} in {repo}."
Step 4d: Extract Reviewer Information for Org Tasks
Use git log frequency analysis (works for any host):
for repo_path in {REPO_PATHS}; do
for area in {CODE_AREAS}; do
git -C "$repo_path" log --format='%an' --since="12 months ago" -- "$area" \
| sort | uniq -c | sort -rn | head -5
done
done
Phase 5: Feasibility Checks
SDLC Feasibility
For each SDLC proposal:
Commit exists: git ls-remote {CLONE_URL} {SHA} 2>/dev/null or, for local repos, git cat-file -t {SHA}
Repo is cloneable: git ls-remote {CLONE_URL} HEAD 2>/dev/null
Patch is self-contained: No external service deps, no DB migrations, no CI artifacts required.
No secrets in patch: Diff doesn't contain API keys, tokens, or credentials.
Org-Scale Feasibility (full mining only)
Repos accessible: All referenced repos exist and user has access.
Oracle is deterministic: The question has a concrete, verifiable answer (file lists, symbol names).
Task is non-trivial: Requires searching at least 3 files.
Phase 6: Generate Output
Quick Eval Mode
Write all task files to a single output directory:
{REPO_NAME}-eval/
tasks/
{task-id-1}/
instruction.md # Task description from issue/MR/commit
ground_truth.json # Auto-generated from patch
environment/
Dockerfile.baseline # Full code, no MCP
Dockerfile.mcp # Truncated code + MCP tools
tests/
test.sh # Verifier script
run_eval.sh # One-command runner (see below)
README.md # How to interpret results
Generate run_eval.sh
This is a standalone runner that doesn't require Harbor or Daytona:
#!/bin/bash
# Run baseline-vs-MCP eval for {REPO_NAME}
# Usage: ./run_eval.sh [--tasks N] [--timeout SECONDS]
#
# Required environment variables:
# ANTHROPIC_API_KEY — Claude API key
# For private repos:
# REPO_TOKEN — Git host access token (GitHub PAT, GitLab token, etc.)
# Optional (for MCP variant):
# {MCP_ENV_VARS} — Provider-specific tokens
set -euo pipefail
TASKS_DIR="$(cd "$(dirname "$0")/tasks" && pwd)"
RESULTS_DIR="$(cd "$(dirname "$0")" && pwd)/results"
TIMEOUT="${2:-900}"
mkdir -p "$RESULTS_DIR"
TASK_DIRS=($(ls -d "$TASKS_DIR"/*/))
echo "=== {REPO_NAME} Eval ==="
echo "Tasks: ${#TASK_DIRS[@]}"
echo "Timeout per task: ${TIMEOUT}s"
echo ""
# Build args for private repos
BUILD_SECRET_ARGS=""
if [ -n "${REPO_TOKEN:-}" ]; then
BUILD_SECRET_ARGS="--secret id=repo_token,env=REPO_TOKEN"
fi
for task_dir in "${TASK_DIRS[@]}"; do
task_id=$(basename "$task_dir")
echo "--- Task: $task_id ---"
for config in baseline mcp; do
dockerfile="$task_dir/environment/Dockerfile.$config"
if [ ! -f "$dockerfile" ]; then
echo " SKIP $config (no Dockerfile)"
continue
fi
result_dir="$RESULTS_DIR/$task_id/$config"
mkdir -p "$result_dir"
echo " Building $config image..."
docker build -f "$dockerfile" -t "eval-${task_id}-${config}" \
$BUILD_SECRET_ARGS \
"$task_dir/environment/" 2>"$result_dir/build.log" || {
echo " FAIL: build error (see $result_dir/build.log)"
continue
}
echo " Running $config agent (timeout: ${TIMEOUT}s)..."
timeout "$TIMEOUT" docker run --rm \
-e ANTHROPIC_API_KEY="$ANTHROPIC_API_KEY" \
-v "$result_dir:/logs" \
"eval-${task_id}-${config}" \
bash -c "
claude --print-only-result \
'Read /workspace/tests/instruction.md and complete the task. When done, run: bash /workspace/tests/test.sh' \
2>/logs/agent.log || true
bash /workspace/tests/test.sh 2>&1 | tee /logs/test_output.log
" 2>"$result_dir/run.log" || {
echo " TIMEOUT or error"
echo "0.0" > "$result_dir/reward.txt"
}
# Extract score
if [ -f "$result_dir/reward.txt" ]; then
score=$(cat "$result_dir/reward.txt")
else
score="0.0"
fi
echo " $config score: $score"
# Cleanup image
docker rmi "eval-${task_id}-${config}" 2>/dev/null || true
done
echo ""
done
# Print summary
echo "=== Results Summary ==="
echo ""
printf "%-40s %-10s %-10s %-10s\n" "Task" "Baseline" "MCP" "Delta"
printf "%-40s %-10s %-10s %-10s\n" "----" "--------" "---" "-----"
total_bl=0
total_mcp=0
count=0
for task_dir in "${TASK_DIRS[@]}"; do
task_id=$(basename "$task_dir")
bl_score=$(cat "$RESULTS_DIR/$task_id/baseline/reward.txt" 2>/dev/null || echo "N/A")
mcp_score=$(cat "$RESULTS_DIR/$task_id/mcp/reward.txt" 2>/dev/null || echo "N/A")
if [ "$bl_score" != "N/A" ] && [ "$mcp_score" != "N/A" ]; then
delta=$(awk "BEGIN {printf \"%.1f\", $mcp_score - $bl_score}")
total_bl=$(awk "BEGIN {print $total_bl + $bl_score}")
total_mcp=$(awk "BEGIN {print $total_mcp + $mcp_score}")
count=$((count + 1))
else
delta="N/A"
fi
printf "%-40s %-10s %-10s %-10s\n" "$task_id" "$bl_score" "$mcp_score" "$delta"
done
if [ "$count" -gt 0 ]; then
avg_bl=$(awk "BEGIN {printf \"%.2f\", $total_bl / $count}")
avg_mcp=$(awk "BEGIN {printf \"%.2f\", $total_mcp / $count}")
avg_delta=$(awk "BEGIN {printf \"%.2f\", ($total_mcp - $total_bl) / $count}")
echo ""
printf "%-40s %-10s %-10s %-10s\n" "AVERAGE ($count tasks)" "$avg_bl" "$avg_mcp" "$avg_delta"
fi
Present the output directory and print the commands:
Quick Eval Ready!
Output: {REPO_NAME}-eval/
Tasks: {N} tasks mined from {REPO}
Host: {DETECTED_HOST}
To run the eval:
export ANTHROPIC_API_KEY=sk-...
# For private repos:
export REPO_TOKEN=... # Your {HOST_NAME} access token
# For Sourcegraph MCP:
export SOURCEGRAPH_ACCESS_TOKEN=sgp_...
export SOURCEGRAPH_URL=https://sourcegraph.com
cd {REPO_NAME}-eval
bash run_eval.sh
Results will be written to {REPO_NAME}-eval/results/
Explain which token type is needed for their host (GitHub PAT, GitLab token, Bitbucket app password, Azure PAT).
Alternatively, suggest cloning locally and using the "Local path" input.
Mixed hosts (e.g., GitHub + GitLab repos)
Detect host per-repo independently — each repo gets its own adapter.
Clone URLs and auth patterns are per-repo, not global.
The REPO_TOKEN secret in Dockerfiles works if both hosts use the same token (unlikely). For mixed-auth scenarios, generate separate secrets: --secret id=github_token --secret id=gitlab_token.
No MCP provider configured
Generate Dockerfile.baseline only and placeholder Dockerfile.mcp.
Print instructions for adding MCP config later.
The baseline eval still works standalone.
Very large repo (>5GB)
Use --depth 1 shallow clone for analysis.
Limit MR scanning to last 3 months.
Focus on specific subsystems if user provided a focus area.
No merged MRs found (or git-only with no merge commits)
Fall back to analyzing recent commits directly.
Look for commits with messages containing "fix", "bug", "resolve".
Propose tasks based on commit diffs instead of MR metadata.
Single repo for org-scale
Analyze internal module boundaries as "cross-package" tasks.
Propose comprehension, audit, and incident-debug tasks within a single repo.
Suggest additional repos in the same ecosystem for cross-repo tasks.
