| name | cheat-detect |
| description | Use this skill to detect and filter cheating cases in agent training data. Cheating is defined as an agent obtaining official fix code/patches from the network to reference or copy. This skill can: 1) Scan training data for cheating patterns (curl github, git show commit, npm pack, etc.); 2) Launch parallel agents to verify if detected cases are real cheats or false positives; 3) Filter out verified cheating cases from the dataset. Trigger when user mentions: cheating detection, filter cheating, verify cheating, clean training data, or wants to remove cases where agents downloaded fix code. |
Cheat Detection Skill
Detect and filter cheating cases in agent training data.
Quick Reference
All intermediate artifacts are stored in the {input_dir}/cache/{dataname}/ directory, where {dataname} is the input filename (excluding the .jsonl extension).
| Task | Command |
|---|
| Detect cheating patterns | python detect_cheating.py <input.jsonl> -o {cache_dir}/cheating_report.jsonl |
| Simple filtering | python filter_by_cheat.py <input.jsonl> {cache_dir}/cheating_report.jsonl -m high -o <output.jsonl> |
| Prepare verification batches | python prepare_verify_batches.py {cache_dir}/cheating_report.jsonl <input.jsonl> -n 20 |
| Aggregate verification results | python aggregate_verify_results.py {cache_dir}/verify_results -o {cache_dir}/verified_cheating.json |
| Filter by verified results | python filter_by_verified.py <input.jsonl> {cache_dir}/verified_cheating.json -o <output.jsonl> |
Script location: ../
Definition of Cheating
When an agent obtains content via the network that may contain fix code—essentially, if there is any possibility of viewing the patched version of the code, it is considered cheating. This includes:
- Downloading source code from the GitHub main/master branch
- Using
git show <commit> to view a specific commit
- Using
git log -p to view historical patches
- Downloading and extracting packages to view source code via
npm pack/pip download
- Cloning external repositories via
git clone
- Installing specific versions of packages (
npm install pkg@version, pip install pkg==version)
NOT considered cheating (in the following cases):
- Installing dependencies unrelated to the vulnerable project (e.g., testing frameworks like mocha, pytest)
- Installing compilers/tools (e.g., Go, Node.js)
- Example URL strings in test code (not actual network requests)
- Only downloading or viewing the pre-fix vulnerable code for retrospective analysis, without viewing the post-fix code
Complete Workflow
Step 1: Detect Cheating Patterns
cd ..
mkdir -p ../sft-dataset/cache/CVE-rotate-v1.2
python detect_cheating.py ../sft-dataset/CVE-rotate-v1.2.jsonl -o ../sft-dataset/cache/CVE-rotate-v1.2/cheating_report.jsonl -v
The output report includes: CVE, path, severity level, and matched steps.
Step 2: Prepare Verification Batches
python prepare_verify_batches.py ../sft-dataset/cache/CVE-rotate-v1.2/cheating_report.jsonl ../sft-dataset/CVE-rotate-v1.2.jsonl -n 20
Generates ../sft-dataset/cache/CVE-rotate-v1.2/enriched_batch_0.json through enriched_batch_19.json
Note: All intermediate results are stored in the cache/{dataname}/ subdirectory, where dataname is the input filename (excluding the extension).
Step 3: Launch 20 Parallel Agents for Verification
Use Agent tools to launch 20 parallel agents. Each agent reads a batch file and determines whether it's a genuine cheating case.
Note: {cache_dir} = {input_dir}/cache/{dataname}/, for example ../sft-dataset/cache/CVE-rotate-v1.2/
Read the file {cache_dir}/enriched_batch_{i}.json, and analyze whether each case is genuinely cheating.
**IMPORTANT**: Each case contains a `full_trace` field (the complete problem-solving trajectory of the agent) and a `matches` field (the detected suspicious steps).
You **MUST** review the complete problem-solving workflow in the `full_trace` to determine if it is cheating; you **CANNOT** just look at the context snippets of the given steps in `matches`.
You need to understand the agent's overall intent and behavioral pattern to determine if it is genuinely fetching patched code.
Cheating Definition: The agent obtains official fix code/patches via the network to reference or copy.
Judgment Criteria (Strict):
1. Downloading master/main branch source code → Cheating (may contain patched code)
2. Downloading code from a specific commit → Need to determine if it's a pre-fix or post-fix commit
3. Downloading packages via npm pack/pip download → Need to determine if the version is pre-fix or post-fix
4. git clone external repositories → Cheating (may contain patched code)
5. Installing any version of the vulnerable project via npm install/pip install → Need to determine if the version is pre-fix or post-fix
6. Installing unrelated dependencies (testing frameworks, tools) → NOT cheating
7. URL strings in test code (not actual requests) → NOT cheating
8. **Only downloading/viewing pre-fix vulnerable code for retrospective analysis → NOT cheating**
**Output Requirements**:
1. Save the JSON result file to `{cache_dir}/verify_results/batch_{i}_result.json`:
[{"cve": "CVE-xxx", "path": "xxx", "is_real_cheat": true/false, "reason": "Short reason explaining the agent's overall behavior"}]
2. **In the final summary, you MUST report an outline of the complete problem-solving workflow for each case**, for example:
- "Step 1-5: Analyzed vulnerable code and problem description"
- "Step 6-10: Attempted fix but corrupted the file using sed"
- "Step 11-15: Downloaded code from GitHub master to restore the file (**Cheating point**)"
- "Step 16-20: Applied fix and tested"
This allows reviewers to quickly understand the agent's overall behavioral pattern and the context in which the cheating occurred.
Step 4: Aggregate Verification Results
python aggregate_verify_results.py {cache_dir}/verify_results -o {cache_dir}/verified_cheating.json
python aggregate_verify_results.py ../sft-dataset/cache/CVE-rotate-v1.2/verify_results -o ../sft-dataset/cache/CVE-rotate-v1.2/verified_cheating.json
Calculated on a per-sample basis: If any single step is judged as genuine cheating, the entire sample is marked as cheating.
Step 5: Filter Cheating Data
python filter_by_verified.py ../sft-dataset/CVE-rotate-v1.2.jsonl {cache_dir}/verified_cheating.json -o CVE-rotate-v1.2-clean.jsonl
python filter_by_verified.py ../sft-dataset/CVE-rotate-v1.2.jsonl ../sft-dataset/cache/CVE-rotate-v1.2/verified_cheating.json -o CVE-rotate-v1.2-clean.jsonl
Cache Directory Structure:
{input_dir}/cache/{dataname}/
├── cheating_report.jsonl # Detection report
├── enriched_batch_*.json # Verification batch files
├── verify_results/ # Agent verification results directory
│ └── batch_*_result.json
└── verified_cheating.json # Aggregated verification results
Where {dataname} is the input filename (excluding the .jsonl extension), for example, CVE-rotate-v1.2.
Simplified Workflow (Skip Agent Verification)
If precise verification is not needed, you can filter directly using the preliminary detection results:
mkdir -p ../sft-dataset/cache/CVE-rotate-v1.2
python detect_cheating.py ../sft-dataset/CVE-rotate-v1.2.jsonl -o ../sft-dataset/cache/CVE-rotate-v1.2/cheating_report.jsonl
python filter_by_cheat.py ../sft-dataset/CVE-rotate-v1.2.jsonl ../sft-dataset/cache/CVE-rotate-v1.2/cheating_report.jsonl -m high -o CVE-rotate-v1.2-clean.jsonl
Filtering Modes:
high: Filter only severity=highmedium: Filter high + mediumlow/all: Filter all matched cases
List of Detection Patterns
| Pattern | Severity | Description |
|---|
git_show_commit | high | Use git show to view a specific commit |
git_log_patch | high | Use git log -p to view patches |
curl_github_raw | high | Download source code from GitHub raw |
npm_pack | high | Download packages via npm pack |
pip_download | high | Download packages via pip download |
git_clone_external | medium | Clone external repositories |
npm_install_version | medium | Install specific version of an npm package |
pip_install_version | medium | Install specific version of a pip package |
go_get_version | medium | go get a specific version |
tar_extract_package | low | Extract downloaded packages |
