troubleshooter

name	troubleshooter
description	Diagnoses and resolves evaluation failures, patch application errors, and Salesforce issues. The agent invokes this skill when debugging problems, investigating errors, or troubleshooting failures.

Troubleshooter

Overview

This skill provides systematic troubleshooting methodology for SF-Bench issues. It covers evaluation failures, patch application errors, scratch org problems, and debugging strategies.

When This Skill Applies

• Evaluation failures or unexpected errors
• Patch application issues
• Scratch org creation or deployment problems
• Debugging test failures
• Investigating root causes

Troubleshooting Methodology

Step 1: Categorize the Error

Category	Indicators	Approach
Tool Bug	Affects all tasks, reproducible	Fix in codebase
Model Limitation	Affects specific task types	Document, adjust prompts
Platform Constraint	Salesforce-specific errors	Work within limits
Configuration	Missing env vars, bad config	Fix configuration
Transient	Intermittent, network-related	Retry with backoff

Step 2: Gather Evidence

1. Check Logs

   # View recent logs
   ls -la logs/
   
   # Search for errors
   grep -r "ERROR" logs/
   
   # View specific task log
   cat logs/<run_id>/<model>/<task_id>.log
   

2. Check Git Status

   # Verify no manual tweaks
   git status
   
   # View applied patch
   git diff
   

3. Check Scratch Org

   # List orgs
   sf org list --all
   
   # View org details
   sf org display -o <alias>
   
   # Check deployment status
   sf project deploy report -o <alias>
   

Step 3: Identify Root Cause

Use the error message to narrow down:

Error Message → Category → Root Cause → Fix

Step 4: Apply Fix and Verify

1. Apply the fix
2. Run relevant tests
3. Verify the issue is resolved
4. Document if pattern-based

Common Error Patterns

Patch Application Failures

Error: "patch does not apply"

Cause: Patch context doesn't match file content

Diagnosis:

# Check file content
cat <file-path>

# Check patch preview
head -50 solution.patch

Solutions:

1. Verify correct repository version
2. Use fuzzy matching: git apply --3way
3. Regenerate solution with updated context

Error: "corrupt patch at line X"

Cause: Malformed patch format

Diagnosis:

# Validate patch syntax
git apply --check solution.patch

Solutions:

1. Check for markdown fences in patch
2. Remove standalone +/- lines
3. Verify patch header format

Scratch Org Failures

Error: "SCRATCH_ORG_LIMIT_EXCEEDED"

Cause: Daily scratch org creation limit reached

Diagnosis:

# Check org limits
sf limits api display -o <devhub>

Solutions:

1. Delete unused scratch orgs
2. Wait for daily limit reset
3. Use different DevHub

Error: "InvalidOrgForCreate"

Cause: DevHub not authenticated or expired

Diagnosis:

# Check DevHub status
sf org list --all | grep DevHub

Solutions:

1. Re-authenticate: sf org login web -d
2. Verify DevHub features enabled
3. Check org expiration

Deployment Failures

Error: "CANNOT_ENABLE_DEVELOPER_MODE"

Cause: Scratch org config issue

Diagnosis:

# Check scratch def
cat config/project-scratch-def.json

Solutions:

1. Verify features in scratch def
2. Check for conflicting settings
3. Use minimal scratch def

Error: "Test coverage is XX%, at least 75% required"

Cause: Insufficient test coverage

Diagnosis:

# Run tests with coverage
sf apex run test -o <alias> -c -r human

Solutions:

1. Add more test methods
2. Cover edge cases
3. Check for uncovered branches

API Failures

Error: "rate_limit_exceeded"

Cause: AI provider rate limit

Diagnosis:

• Check request frequency
• Review provider dashboard

Solutions:

1. Implement rate limiting
2. Use exponential backoff
3. Switch to different provider

Error: "invalid_api_key"

Cause: Missing or incorrect API key

Diagnosis:

# Check environment
echo $ROUTELLM_API_KEY | head -c 10

Solutions:

1. Verify key in .env
2. Source environment: source .env
3. Regenerate API key

Diagnostic Commands

Preflight Diagnostics

# Run preflight checks
python -c "
from sfbench.utils.preflight import PreflightValidator
v = PreflightValidator()
r = v.run_all_checks('model-name', 'provider')
print('PASSED' if r.passed else 'FAILED')
for c in r.checks:
    print(f'{c.name}: {c.status}')
"

Scratch Org Diagnostics

# Create debug org
sf org create scratch -f config/project-scratch-def.json -a debug-org -d 1

# Deploy and check
sf project deploy start -o debug-org

# Run specific test
sf apex run test -o debug-org -n TestClassName -r human -c

# Cleanup
sf org delete scratch -o debug-org -p

Patch Diagnostics

# Validate patch
git apply --check solution.patch

# Apply with verbose output
git apply -v solution.patch

# Apply with fallback strategies
git apply --3way solution.patch
git apply --reject solution.patch

Escalation Path

If issue persists after troubleshooting:

1. Document the Issue

   • Error message
   • Steps to reproduce
   • Environment details
   • Attempted solutions

2. Update activeContext.md

   • Add to "Blockers" section
   • Note impact on current work

3. Create Issue

   • Use bug report template
   • Include all diagnostic information
   • Tag appropriately

Prevention Strategies

For Patch Issues

• Validate patches before applying
• Use multi-strategy fallback
• Clean patches before application

For Scratch Org Issues

• Monitor org limits
• Always cleanup in finally blocks
• Use retry with exponential backoff

For API Issues

• Implement rate limiting
• Cache responses where appropriate
• Use circuit breaker pattern for repeated failures