judge-pentest

name	judge-pentest
description	Penetration testing evaluation checklist for the decision Agent. Evaluates whether a pentest has sufficiently covered all attack surfaces, determines task completion, and provides specific feedback on uncovered areas.
metadata	{"tags":"judge,evaluation,pentest,coverage,decision","category":"general"}

Penetration Testing Evaluation Checklist

Web Application Vulnerability Coverage Check

Check each attack surface below for whether it has been tested; mark untested ones as gaps:

Injection

• SQL Injection (login forms, search, API params, cookies)
• XPath Injection
• LDAP Injection
• Command Injection (OS Command Injection)
• SSTI (Server-Side Template Injection)
• XXE (XML External Entity Injection)

Cross-Site

• Reflected XSS (search box, URL params, error pages)
• Stored XSS (comments, feedback, user profiles)
• DOM XSS
• CSRF (transfers, password changes, critical operations)

Authentication & Authorization

• Default/weak credentials
• SQL injection auth bypass
• Brute force protection (account lockout mechanism)
• Username enumeration (error message differences)
• Session management (Session Fixation, Cookie security attributes)
• JWT/Token security (signature verification, algorithm confusion, plaintext encoding)
• Vertical privilege escalation (regular user → admin functions)
• Horizontal privilege escalation / IDOR (accessing other users' resources)

Business Logic

• IDOR — account info viewing
• IDOR — transfer/transaction operations
• IDOR — password change
• Negative/zero amount transactions
• Concurrency/race conditions
• Business flow bypass

Information Disclosure

• Error page info leaks (stack traces, paths)
• API documentation exposure (Swagger, WSDL)
• Backup file disclosure
• Sensitive config exposure
• HTTP response headers (Server version, X-Powered-By)

Server-Side

• SSRF (Server-Side Request Forgery)
• File upload vulnerabilities
• Path traversal / LFI / RFI
• Deserialization vulnerabilities

Configuration

• Directory listing

API-Specific

• REST API auth bypass
• API IDOR
• API parameter tampering
• API rate limiting

Known CVE/CNVD

• Known CVEs for target product/tech stack
• Known CNVDs for target product/tech stack

Evaluation Decision Logic

The goal of penetration testing is to discover as many vulnerabilities as possible; do NOT end prematurely.

# In pentest scenarios, complete is advisory only and does NOT trigger early exit
# The judge's core value is providing precise "what to test next round" feedback

if tested_categories >= 90% of total && two consecutive rounds with no new vulns:
    complete = true, confidence >= 0.8
else:
    complete = false
    feedback = explicitly list untested attack surfaces with specific testing suggestions
    missing_areas = names of untested categories

Important: Better to run one extra round than to miss one direction. Even if many vulnerabilities have been found, if there are still untested attack surfaces, return complete=false.

If a target product/tech stack was identified in this round but search_vulndb was never called, MUST return complete=false and require using search_vulndb(query="product name") to query known vulnerabilities.

Feedback Template

When complete=false, feedback should include:

1. Completed work (acknowledge positively, avoid repetition)
2. Specific missing directions (do NOT say "keep testing" vaguely — specify concrete endpoints + vulnerability types)
3. Suggested test steps (e.g., "use sqlmap for deep injection testing on the query parameter of /api/search")