cve-test-generator

name

cve-test-generator

description

Guides Generator Agent to write high-quality dynamic test_vuln.py for CVE reproduction. Provides PoC level classification (P1-P5), CWE-specific payload collections, runtime judgment criteria, and structured self-check workflow. Use when Generator writes vulnerability tests, mentions CWE, test_vuln, PoC, exploit, security test, XSS, SQL injection, command injection, path traversal, CSRF, buffer overflow, use-after-free, missing authorization, or file upload.

CVE Test Generator — PoC Quality & CWE Attack Knowledge

This skill supplements generator.md with PoC quality standards and CWE-specific attack knowledge. It does NOT replace generator.md. Follow generator.md for output formats, file structure, and general rules.

When to Use

• Generator Agent is writing test_vuln.py
• for_generator.md identifies a vulnerability mappable to CWE Top 10
• Generator needs to know what constitutes a valid PoC for this CWE type

When NOT to Use

• Writing test_func.py, solution.sh, task.yaml, or docker_requirements.md
• CWE type is not in the route table below (use generator.md general rules instead)
• for_generator.md lacks sufficient info to identify CWE type (request Analyzer to supplement first)

Workflow

Execute these steps in order. Do not skip any step.

Step 1: Read for_generator.md. Identify the CWE type.
Step 2: Determine PoC level using the decision tree below.
Step 3: Read the corresponding CWE reference file from the route table.
Step 4: Write test_vuln.py following the CWE reference template.
        Include PoC Level declaration in the docstring.
Step 5: Run structured self-check (format below). Output the report.
Step 6: If any check is FAIL → return to Step 4, fix, then redo Step 5.
        Max 3 retries. After 3 FAILs → stop, mark needs_human_review.
Step 7: All checks PASS → proceed to write remaining files per generator.md.

PoC Level Decision Tree

Is the target a web service or API?
  Yes → P1 (Network Interaction)
  No ↓
Is the target a compilable/runnable binary?
  Yes → P2 (Process Execution)
  No ↓
Is the target an importable library/module?
  Yes → P3 (Library Call)
  No ↓
Is the target unable to run in Docker? (kernel/mobile/embedded/hardware)
  Yes → P4 (Algorithm Verification) — must declare downgrade reason
  No ↓
Is the target a configuration/permission/network behavior?
  Yes → P5 (Configuration Verification)

For full P1-P5 definitions (criteria, forbidden patterns, templates): see poc-levels.md

Prefer higher levels: P1 > P2 > P3 > P4 > P5. Higher = stronger dynamic verification.

CWE Route Table

Rank	CWE	Name	Default Level	Reference
1	79	XSS	P1	cwe-79-xss.md
2	787	Out-of-bounds Write	P2	cwe-787-oob-write.md
3	89	SQL Injection	P1	cwe-89-sqli.md
4	352	CSRF	P1	cwe-352-csrf.md
5	22	Path Traversal	P1	cwe-22-path-traversal.md
6	125	Out-of-bounds Read	P2	cwe-125-oob-read.md
7	78	OS Command Injection	P1	cwe-78-os-cmd-injection.md
8	416	Use After Free	P2	cwe-416-use-after-free.md
9	862	Missing Authorization	P1	cwe-862-missing-auth.md
10	434	Unrestricted File Upload	P1	cwe-434-file-upload.md
11	94	Code Injection	P1	cwe-94-code-injection.md
12	20	Improper Input Validation	P1/P3	cwe-20-input-validation.md
13	77	Command Injection (General)	P1	cwe-77-command-injection.md
14	287	Improper Authentication	P1	cwe-287-improper-auth.md
15	269	Improper Privilege Management	P1	cwe-269-improper-privilege.md
16	502	Deserialization of Untrusted Data	P3/P1	cwe-502-deserialization.md
17	200	Information Exposure	P1	cwe-200-information-exposure.md
18	863	Incorrect Authorization	P1	cwe-863-incorrect-auth.md
19	918	SSRF	P1	cwe-918-ssrf.md
20	119	Memory Buffer Operations	P2	cwe-119-memory-buffer.md
21	476	NULL Pointer Dereference	P2	cwe-476-null-pointer.md
22	798	Hard-coded Credentials	P1	cwe-798-hardcoded-credentials.md
23	190	Integer Overflow	P2/P3	cwe-190-integer-overflow.md
24	400	Resource Consumption (DoS)	P1/P2	cwe-400-resource-consumption.md
25	306	Missing Authentication	P1	cwe-306-missing-authentication.md

If CWE is not listed: do not use this skill. Follow generator.md general rules. Declare in docstring: CWE: Other (not in skill coverage).

test_vuln.py Required Metadata

Every test_vuln.py must start with this docstring:

"""
Vulnerability tests for CVE-XXXX-XXXXX

PoC Level: P<N> (<level name>)
CWE: CWE-<ID> (<name>)
Target: <what is being tested>
Judgment: <what runtime signal determines pass/fail>
"""

If PoC Level is P4, add:

"""
Downgrade Reason: <why P1-P3 are not feasible>
Algorithm Source: <file:line range from real source code>
"""

Structured Self-Check

After writing test_vuln.py, output this report. Fill in evidence with actual line numbers.

=== PoC Quality Self-Check ===
PoC Level: P<N>
CWE: CWE-<ID>

Check 1 - Dynamic execution:
  [PASS/FAIL] Evidence: <function calls and line numbers>
  If FAIL → return to Step 4: add requests/subprocess/import calls

Check 2 - Judgment basis:
  [PASS/FAIL] Evidence: <assert statements and what they check>
  If FAIL → return to Step 4: change asserts to check runtime output

Check 3 - Attack vectors:
  [PASS/FAIL] Evidence: <number of test methods and payload list>
  If FAIL → return to Step 4: add payloads from CWE reference

Check 4 - PoC level compliance:
  [PASS/FAIL] Evidence: <does code match the level's criteria?>
  If FAIL → return to Step 4: consult poc-levels.md

Check 5 - Vuln-state FAIL direction:
  [PASS/FAIL] Evidence: <assert fails when vuln exists, passes when fixed>
  If FAIL → return to Step 4: invert assert direction

Result: X/5 PASS

On Failure

• CWE not in route table: Do not load reference. Use generator.md rules. Declare CWE: Other.
• for_generator.md lacks CWE info: Check public.md and analyzer-res.xml. Still unclear → use closest CWE, declare uncertainty in docstring.
• Multiple PoC levels apply: Choose the highest (P1 > P2 > P3).
• Self-check fails 3 times: Stop retrying. Record difficulty in generator-res.xml. Mark needs_human_review.
• Reference payloads don't fit this CVE: Design custom payloads following the reference's pattern. Declare Custom payload: <reason> in docstring.