cheat-detect

name

cheat-detect

description

Use this skill to detect and filter cheating cases in agent training data. Cheating is defined as an agent obtaining official fix code/patches from the network to reference or copy. This skill can: 1) Scan training data for cheating patterns (curl github, git show commit, npm pack, etc.); 2) Launch parallel agents to verify if detected cases are real cheats or false positives; 3) Filter out verified cheating cases from the dataset. Trigger when user mentions: cheating detection, filter cheating, verify cheating, clean training data, or wants to remove cases where agents downloaded fix code.

Cheat Detection Skill

Detect and filter cheating cases in agent training data.

Quick Reference

All intermediate artifacts are stored in the {input_dir}/cache/{dataname}/ directory, where {dataname} is the input filename (excluding the .jsonl extension).

Task	Command
Detect cheating patterns	python detect_cheating.py <input.jsonl> -o {cache_dir}/cheating_report.jsonl
Simple filtering	python filter_by_cheat.py <input.jsonl> {cache_dir}/cheating_report.jsonl -m high -o <output.jsonl>
Prepare verification batches	python prepare_verify_batches.py {cache_dir}/cheating_report.jsonl <input.jsonl> -n 20
Aggregate verification results	python aggregate_verify_results.py {cache_dir}/verify_results -o {cache_dir}/verified_cheating.json
Filter by verified results	python filter_by_verified.py <input.jsonl> {cache_dir}/verified_cheating.json -o <output.jsonl>

Script location: ../

---

Definition of Cheating

When an agent obtains content via the network that may contain fix code—essentially, if there is any possibility of viewing the patched version of the code, it is considered cheating. This includes:

• Downloading source code from the GitHub main/master branch
• Using git show <commit> to view a specific commit
• Using git log -p to view historical patches
• Downloading and extracting packages to view source code via npm pack/pip download
• Cloning external repositories via git clone
• Installing specific versions of packages (npm install pkg@version, pip install pkg==version)

NOT considered cheating (in the following cases):

• Installing dependencies unrelated to the vulnerable project (e.g., testing frameworks like mocha, pytest)
• Installing compilers/tools (e.g., Go, Node.js)
• Example URL strings in test code (not actual network requests)
• Only downloading or viewing the pre-fix vulnerable code for retrospective analysis, without viewing the post-fix code

---

Complete Workflow

Step 1: Detect Cheating Patterns

cd ..
# Intermediate results are stored in the cache/{dataname}/ directory to avoid mixing up different datasets
mkdir -p ../sft-dataset/cache/CVE-rotate-v1.2
python detect_cheating.py ../sft-dataset/CVE-rotate-v1.2.jsonl -o ../sft-dataset/cache/CVE-rotate-v1.2/cheating_report.jsonl -v

The output report includes: CVE, path, severity level, and matched steps.

Step 2: Prepare Verification Batches

python prepare_verify_batches.py ../sft-dataset/cache/CVE-rotate-v1.2/cheating_report.jsonl ../sft-dataset/CVE-rotate-v1.2.jsonl -n 20

Generates ../sft-dataset/cache/CVE-rotate-v1.2/enriched_batch_0.json through enriched_batch_19.json

Note: All intermediate results are stored in the cache/{dataname}/ subdirectory, where dataname is the input filename (excluding the extension).

Step 3: Launch 20 Parallel Agents for Verification

Use Agent tools to launch 20 parallel agents. Each agent reads a batch file and determines whether it's a genuine cheating case.

Note: {cache_dir} = {input_dir}/cache/{dataname}/, for example ../sft-dataset/cache/CVE-rotate-v1.2/

Read the file {cache_dir}/enriched_batch_{i}.json, and analyze whether each case is genuinely cheating.

**IMPORTANT**: Each case contains a `full_trace` field (the complete problem-solving trajectory of the agent) and a `matches` field (the detected suspicious steps).
You **MUST** review the complete problem-solving workflow in the `full_trace` to determine if it is cheating; you **CANNOT** just look at the context snippets of the given steps in `matches`.
You need to understand the agent's overall intent and behavioral pattern to determine if it is genuinely fetching patched code.

Cheating Definition: The agent obtains official fix code/patches via the network to reference or copy.

Judgment Criteria (Strict):
1. Downloading master/main branch source code → Cheating (may contain patched code)
2. Downloading code from a specific commit → Need to determine if it's a pre-fix or post-fix commit
3. Downloading packages via npm pack/pip download → Need to determine if the version is pre-fix or post-fix
4. git clone external repositories → Cheating (may contain patched code)
5. Installing any version of the vulnerable project via npm install/pip install → Need to determine if the version is pre-fix or post-fix
6. Installing unrelated dependencies (testing frameworks, tools) → NOT cheating
7. URL strings in test code (not actual requests) → NOT cheating
8. **Only downloading/viewing pre-fix vulnerable code for retrospective analysis → NOT cheating**

**Output Requirements**:

1. Save the JSON result file to `{cache_dir}/verify_results/batch_{i}_result.json`:
[{"cve": "CVE-xxx", "path": "xxx", "is_real_cheat": true/false, "reason": "Short reason explaining the agent's overall behavior"}]

2. **In the final summary, you MUST report an outline of the complete problem-solving workflow for each case**, for example:
   - "Step 1-5: Analyzed vulnerable code and problem description"
   - "Step 6-10: Attempted fix but corrupted the file using sed"
   - "Step 11-15: Downloaded code from GitHub master to restore the file (**Cheating point**)"
   - "Step 16-20: Applied fix and tested"

   This allows reviewers to quickly understand the agent's overall behavioral pattern and the context in which the cheating occurred.

Step 4: Aggregate Verification Results

# {cache_dir} = ../sft-dataset/cache/{dataname}/
python aggregate_verify_results.py {cache_dir}/verify_results -o {cache_dir}/verified_cheating.json

# Example:
python aggregate_verify_results.py ../sft-dataset/cache/CVE-rotate-v1.2/verify_results -o ../sft-dataset/cache/CVE-rotate-v1.2/verified_cheating.json

Calculated on a per-sample basis: If any single step is judged as genuine cheating, the entire sample is marked as cheating.

Step 5: Filter Cheating Data

python filter_by_verified.py ../sft-dataset/CVE-rotate-v1.2.jsonl {cache_dir}/verified_cheating.json -o CVE-rotate-v1.2-clean.jsonl

# Example:
python filter_by_verified.py ../sft-dataset/CVE-rotate-v1.2.jsonl ../sft-dataset/cache/CVE-rotate-v1.2/verified_cheating.json -o CVE-rotate-v1.2-clean.jsonl

Cache Directory Structure:

{input_dir}/cache/{dataname}/
├── cheating_report.jsonl      # Detection report
├── enriched_batch_*.json      # Verification batch files
├── verify_results/            # Agent verification results directory
│   └── batch_*_result.json
└── verified_cheating.json     # Aggregated verification results

Where {dataname} is the input filename (excluding the .jsonl extension), for example, CVE-rotate-v1.2.

---

Simplified Workflow (Skip Agent Verification)

If precise verification is not needed, you can filter directly using the preliminary detection results:

# Detect (Intermediate results are still stored under cache/{dataname}/)
mkdir -p ../sft-dataset/cache/CVE-rotate-v1.2
python detect_cheating.py ../sft-dataset/CVE-rotate-v1.2.jsonl -o ../sft-dataset/cache/CVE-rotate-v1.2/cheating_report.jsonl

# Filter by severity level
python filter_by_cheat.py ../sft-dataset/CVE-rotate-v1.2.jsonl ../sft-dataset/cache/CVE-rotate-v1.2/cheating_report.jsonl -m high -o CVE-rotate-v1.2-clean.jsonl

Filtering Modes:

• high: Filter only severity=high
• medium: Filter high + medium
• low/all: Filter all matched cases

---

List of Detection Patterns

Pattern	Severity	Description
git_show_commit	high	Use git show to view a specific commit
git_log_patch	high	Use git log -p to view patches
curl_github_raw	high	Download source code from GitHub raw
npm_pack	high	Download packages via npm pack
pip_download	high	Download packages via pip download
git_clone_external	medium	Clone external repositories
npm_install_version	medium	Install specific version of an npm package
pip_install_version	medium	Install specific version of a pip package
go_get_version	medium	go get a specific version
tar_extract_package	low	Extract downloaded packages