visa-vulnerability-agentic-harness

# Visa Vulnerability Agentic Harness (VVAH) Skill > Skill by [ara.so](https://ara.so) — AI Agent Skills collection. VVAH is Visa's open-source harness for autonomous vulnerability discovery using frontier AI models. It implements a 9-stage pipeline: attack surface mapping → threat modeling → multi-lens research → adversarial verification → exploit chaining → SARIF reporting. Designed to reduce Mean Time to Adapt (MTTA) from AI-discovered weakness to validated fix. ## Installation **Prerequisites:** - Python ≥ 3.10 - Claude Code session (`claude login`) OR API key (Anthropic/OpenAI) **Install into virtual environment (recommended):** ```bash python3 -m venv .venv source .venv/bin/activate # On Windows: .\.venv\Scripts\Activate.ps1 pip install . ``` **Or install globally with pipx:** ```bash pipx install . ``` This installs the `vvaharness` CLI command. ## Configuration **Initial setup:** ```bash # Copy environment template cp .env.example .env # Edit .env with your credentials # For Claude Code (default): set CLAUDE_CODE_OAUTH_TOKEN or run `claude login` # For Anthropic SDK: set ANTHROPIC_SDK_API_KEY # For OpenAI: set OPENAI_API_KEY and OPENAI_BASE_URL ``` **Backend selection (via profiles):** The default profile uses `via: cli` (Claude Code) for all stages. To customize: ```bash # Copy a profile template cp vvaharness/config/profiles/full.yaml ./config.yaml # Edit config.yaml to select backends per stage ``` **Profile structure:** ```yaml roles: surface_explorer: via: cli # Options: cli, sdk, openai model: claude-sonnet-4-6 computer_use: false threat_modeler: via: sdk model: claude-opus-4 temperature: 1.0 vulnerability_researcher: via: openai model: gpt-4 base_url: ${OPENAI_BASE_URL} ``` **Environment variables:** ```bash # Claude Code (cli backend) CLAUDE_CODE_OAUTH_TOKEN=your_token_here CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1 # For enterprise gateways # Anthropic SDK (sdk backend) ANTHROPIC_SDK_API_KEY=sk-ant-xxxxx ANTHROPIC_SDK_BASE_URL=https://your-gateway.com # Optional ANTHROPIC_SDK_CA_CERT=/path/to/ca.pem # For mTLS ANTHROPIC_SDK_CLIENT_CERT=/path/to/client.pem # For mTLS # OpenAI (openai backend) OPENAI_API_KEY=sk-xxxxx OPENAI_BASE_URL=https://api.openai.com/v1 # Or compatible endpoint # Enterprise gateway NODE_EXTRA_CA_CERTS=/path/to/ca-bundle.pem ``` ## Key Commands ### Setup and Validation ```bash # Verify configuration and credentials vvaharness doctor # Check what's needed for your setup vvaharness setup # Install agent instructions (Claude/Copilot/Gemini) vvaharness setup --install-agents ``` ### Cost Estimation ```bash # Estimate scan cost without spending vvaharness estimate --repo /path/to/target # Estimate for remote repository vvaharness estimate --repo https://github.com/org/repo --application-id 12345 ``` ### Single Repository Scan ```bash # Basic scan vvaharness scan --repo /path/to/target --application-id 12345 # Scan with custom config vvaharness scan \ --repo /path/to/target \ --application-id 12345 \ --config ./custom-config.yaml # Scan specific module vvaharness scan \ --repo /path/to/target \ --application-id 12345 \ --module backend # Dry run (no LLM calls) vvaharness scan --repo /path/to/target --dry-run ``` ### Batch Scanning ```bash # Scan multiple repos from CSV vvaharness scan \ --repo-file repos.csv \ --workspace ./scans \ --group-by-app \ --keep-clones # repos.csv format: # repo_url,application_id,module_name # https://github.com/org/app1,101,frontend # https://github.com/org/app2,102,backend ``` ### Output Locations Scan outputs are written to `<target>/security-scan/`: ``` <target>/ └── security-scan/ ├── <module>_<timestamp>_report.md # Human-readable findings ├── <module>_<timestamp>_report.sarif # SARIF 2.1.0 format ├── <module>_<timestamp>_errors.jsonl # Non-fatal errors └── run_manifest.json # Metadata (version, models, git SHA) ``` ## Pipeline Stages VVAH runs a 9-stage pipeline across three phases: **Discovery & Modeling (S1-S3):** - S1: Attack surface mapping (code, CMDB, CVE, controls) - S2: Threat modeling (STRIDE, OWASP, trust boundaries) - S3: Vulnerability research strategy (taint, API boundaries) **Deep Dive & Verification (S4-S6):** - S4: Multi-lens research (Language, Crypto, Logic, Access Control, IaC) - S5: Policy gates - S6: Adversarial verification (exploit chains, trust boundaries) **Synthesis & Reporting (S7-S9):** - S7: Deduplication - S8: Exploit chain construction (CWE, attack paths) - S9: SARIF emission ## Configuration Examples ### Budget Controls ```yaml step4: max_budget_usd: 5.0 # Per-stage spending cap max_budget_usd_per_finding: 0.50 # Per-finding cap step6: max_budget_usd: 3.0 verification_passes: 3 # Majority-vote rounds (sdk/openai only) ``` ### Language-Specific Research ```yaml lenses: - name: language_specific languages: [python, java, javascript] focus_areas: - injection vulnerabilities - unsafe deserialization - path traversal - name: crypto_specific focus_areas: - weak algorithms - hardcoded keys - IV reuse - name: access_control focus_areas: - broken authorization - privilege escalation - IDOR ``` ### Model Selection Strategy ```yaml # High-reasoning stages use Opus roles: threat_modeler: via: sdk model: claude-opus-4 adversarial_reviewer: via: sdk model: claude-opus-4 # Volume stages use Sonnet vulnerability_researcher: via: cli model: claude-sonnet-4-6 exploit_strategist: via: sdk model: claude-sonnet-4-6 ``` ## Common Patterns ### Enterprise Gateway Setup ```python # .env configuration for enterprise ANTHROPIC_SDK_BASE_URL=https://ai-gateway.corp.com ANTHROPIC_SDK_CA_CERT=/etc/ssl/certs/corp-ca.pem ANTHROPIC_SDK_CLIENT_CERT=/etc/ssl/certs/client.pem ANTHROPIC_SDK_CLIENT_KEY=/etc/ssl/private/client-key.pem NODE_EXTRA_CA_CERTS=/etc/ssl/certs/ca-bundle.pem CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1 ``` ### CI/CD Integration ```yaml # .github/workflows/security-scan.yml name: VVAH Security Scan on: pull_request: branches: [main] schedule: - cron: '0 2 * * 1' # Weekly Monday 2am jobs: scan: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Set up Python uses: actions/setup-python@v4 with: python-version: '3.10' - name: Install VVAH run: | python -m pip install --upgrade pip pip install . - name: Run scan env: ANTHROPIC_SDK_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} run: | vvaharness scan \ --repo . \ --application-id ${{ github.repository_id }} \ --config .vvah-config.yaml - name: Upload SARIF uses: github/codeql-action/upload-sarif@v2 with: sarif_file: security-scan/*_report.sarif ``` ### Programmatic Integration ```python import subprocess import json from pathlib import Path def run_vvah_scan(repo_path: str, app_id: str) -> dict: """Run VVAH scan and return manifest.""" result = subprocess.run( [ "vvaharness", "scan", "--repo", repo_path, "--application-id", app_id, "--config", "config.yaml" ], capture_output=True, text=True, check=True ) # Load manifest manifest_path = Path(repo_path) / "security-scan" / "run_manifest.json" with open(manifest_path) as f: return json.load(f) def estimate_scan_cost(repo_path: str) -> float: """Get cost estimate before running.""" result = subprocess.run( ["vvaharness", "estimate", "--repo", repo_path], capture_output=True, text=True, check=True ) # Parse estimate from stdout # Format: "Estimated cost: $X.XX" for line in result.stdout.splitlines(): if "Estimated cost:" in line: return float(line.split("$")[1]) return 0.0 ``` ### Multi-Model Strategy ```yaml # config.yaml - Mix backends for cost/quality tradeoff roles: # Fast exploration with CLI (no API cost if using Claude Code) surface_explorer: via: cli model: claude-sonnet-4-6 # Deep reasoning with SDK Opus threat_modeler: via: sdk model: claude-opus-4 temperature: 1.0 # Bulk research with OpenAI (if cheaper in your region) vulnerability_researcher: via: openai model: gpt-4-turbo # Final verification with SDK Opus adversarial_reviewer: via: sdk model: claude-opus-4 ``` ## Troubleshooting ### Authentication Issues ```bash # Check credential status vvaharness doctor # For Claude Code claude login # Or claude setup-token # Get token for CLAUDE_CODE_OAUTH_TOKEN # For SDK echo $ANTHROPIC_SDK_API_KEY # Should start with sk-ant- # For OpenAI echo $OPENAI_API_KEY # Should start with sk- ``` ### Budget Exceeded ```bash # Review per-stage budgets in config cat config.yaml | grep max_budget_usd # Reduce budgets step4: max_budget_usd: 2.0 # Down from 5.0 max_budget_usd_per_finding: 0.25 # Down from 0.50 ``` ### Gateway/Proxy Issues ```bash # Test connectivity curl -H "x-api-key: $ANTHROPIC_SDK_API_KEY" \ $ANTHROPIC_SDK_BASE_URL/v1/messages \ -d '{"model":"claude-sonnet-4","max_tokens":10,"messages":[{"role":"user","content":"hi"}]}' # Check CA cert openssl verify -CAfile $NODE_EXTRA_CA_CERTS $ANTHROPIC_SDK_CA_CERT # Disable betas for enterprise gateways export CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1 ``` ### No Findings Generated ```bash # Check errors log cat <repo>/security-scan/*_errors.jsonl # Increase verbosity (if supported in future version) vvaharness scan --repo . --application-id 123 --verbose # Verify lenses are enabled grep -A5 "lenses:" config.yaml ``` ### SARIF Validation ```bash # Validate SARIF output npm install -g @microsoft/sarif-multitool sarif-multitool validate security-scan/*_report.sarif # View in GitHub Code Scanning format gh api repos/:owner/:repo/code-scanning/sarifs \ -F sarif=@security-scan/*_report.sarif ``` ### Performance Optimization ```python # Limit to specific file patterns (custom config) discovery: include_patterns: - "**/*.py" - "**/*.js" exclude_patterns: - "**/node_modules/**" - "**/venv/**" - "**/*.test.js" # Reduce verification passes step6: verification_passes: 1 # Single-pass (faster, less FP filtering) ``` ## Best Practices 1. **Always estimate first:** Run `vvaharness estimate` before full scans 2. **Use profiles:** Copy and customize `profiles/full.yaml` rather than editing defaults 3. **Stage budgets:** Set `max_budget_usd` per stage to prevent runaway costs 4. **Batch scans:** Use `--repo-file` + `--group-by-app` for multi-repo workflows 5. **Human review required:** VVAH findings are triage candidates, not confirmed CVEs 6. **Keep manifests:** `run_manifest.json` tracks model versions for reproducibility 7. **Authorized use only:** Scan only code you own or have permission to test ## Output Format **Markdown Report Structure:** ```markdown # Security Scan Report ## Executive Summary - Total findings: X - Critical: X | High: X | Medium: X | Low: X ## Findings ### F-001: SQL Injection in User Authentication **Severity:** Critical **CWE:** CWE-89 **Location:** src/auth/login.py:45-52