finding-audit

name	finding-audit
description	LOAD when reviewing flawed scan results to determine true/false positive rates, when auditing detection output against actual repo source code, when assessing engine quality after a corpus evaluation run, or when a previous scan produced findings that need manual verification before acting on them.

Manually verify whether flawed detection findings are real by cross-referencing each finding against the target repository's source code. This skill encodes the audit methodology that discovered flawed's 94% FP rate on flaskbb — and identified that ONE engine gap caused 90% of those false positives.

Before You Start

Gather your inputs

What	Where	Verify
Findings JSON	local/v1-assessment/sandbox-results*/	jq '.finding_count' < output.json — non-zero?
Target repo	rbr path <owner/name> (opaque managed path)	Resolve by name — the source is whatever ribrarian has materialized (consistent by construction; no hash check)
Rule source	src/flawed/_rules/<tier>/<rule>.py (or the finding's rule_path)	Read the rule to understand what it claims to detect
Previous audits	local/v1-assessment/L3/	Check if this rule/repo was audited before

Understand the finding structure

{
  "rule_id": "g090-check-coverage-report",
  "rule_path": "src/flawed/_rules/L0_gadgets/g090_check_coverage_report.py",
  "severity": "unknown",
  "summary": "Route handler 'forum.manage' lacks security check X",
  "location": {"file": "flaskbb/forum/views.py", "line": 42},
  "route_endpoint": "forum.manage",
  "evidence": [
    {"description": "Route decorator", "location": {"file": "...", "line": 30}},
    {"description": "Missing check", "location": {"file": "...", "line": 42}}
  ],
  "gaps": [
    {"kind": "symbol_unresolved", "message": "Cannot resolve ..."}
  ]
}

Critical: The gaps array is the engine's self-assessment of what it COULDN'T analyze. Every gap is a potential source of FPs or FNs. Read them.

The Audit Protocol

Step 1: Group findings by rule_id

cat output.json | python3 -c "
import json, sys
from collections import Counter
data = json.load(sys.stdin)
for rule, n in Counter(f['rule_id'] for f in data['findings']).most_common():
    print(f'  {rule}: {n}')
"

Step 2: For each rule_id, read the rule source first

Open src/flawed/_rules/<tier>/<rule>.py (the finding's rule_path). Understand:

• What confusion/vulnerability pattern does it detect?
• What engine APIs does it use? (routes, inputs, effects, checks, flows)
• What are its documented limitations? (read the docstring)
• What is its expected FP rate? (many rules document this)

Step 3: Verify findings against source code

For each finding (or sample — see strategy below):

1. Open the file at the referenced location in the repo source (rbr path <owner/name>)
2. Read the function/class — not just the line, the whole context
3. Trace the data flow manually — does user input really reach the sink?
4. Check for guards the engine might have missed:
   • Application-level guards: CSRFProtect(app), limiter.init_app(app)
   • Blueprint-level guards: @bp.before_request, limiter.limit()(bp)
   • Class-level guards: decorators = [login_required] on MethodView
   • Indirect guards: is_safe_url() called 2 hops away
5. Assign verdict:

Verdict	Criteria	Action
TRUE_POSITIVE	Real vulnerability — no guard exists, data flows from source to sink	Report as engine success
FALSE_POSITIVE	Guard exists but engine missed it	Identify WHICH gap caused the miss
UNCERTAIN	Guard might exist but you can't confirm without runtime data	Document what additional evidence would resolve it
STRUCTURAL_NOISE	Pattern matches technically but isn't security-relevant	Note the pattern for future rule tuning

Step 4: Identify the dominant gap

After reviewing findings, look for the pattern:

• Are 70%+ of FPs caused by the SAME engine gap?
• Is it an L1 gap (missing extraction) or L2 gap (missing provider/conversion)?
• Can you trace it to a specific DISC-XXX issue?

This is the most valuable output of the audit. In the flaskbb audit, discovering that DISC-047 (class decorators attribute) caused 90% of FPs meant we knew exactly where to invest engineering effort.

Step 5: Check for false negatives (what the engine SHOULD have found)

This is harder but equally important:

1. Read the target repo's security-relevant code manually
2. Look for patterns the rules claim to detect
3. If you find a real issue that the rules missed: WHY?
   • Was the rule not run? (wrong --rules-dir?)
   • Did a gap prevent detection? (check gaps in other findings)
   • Is the pattern outside the rule's scope?

Step 6: Capture confirmed real-world vulns

When a finding is confirmed (TRUE_POSITIVE) or strongly suspected to be a real vulnerability in a real app, capture it under examples/cve/<slug>/ following examples/cve/README.md ("How to add a finding"): a FINDING.md with the vuln location, mechanism×subject classification, and a TESTED exploration-API block, plus rules/ symlinks to the detecting production rules. This turns an audit result into a durable, reproducible regression fixture instead of a one-off note.

Sampling Strategy

Finding count	Strategy	Rationale
< 20	Review EVERY finding	Small enough to be thorough
20-50	Review all, but spend less time on obvious FP patterns	After 5 identical FPs, categorize the rest
50-100	Review 15+ thoroughly, categorize rest by rule_id pattern	Sample at least 5 from EACH rule_id
> 100	Review 5+ from each rule_id, then extrapolate	Focus depth on rules with highest finding counts

NEVER skip a rule_id entirely — FP patterns differ by rule. A rule with 2 findings might have 100% TP while a rule with 74 findings might have 0% TP. You won't know until you check.

Flaskbb Baseline (May 2025)

This is the measured quality bar. Future audits should compare against this.

Tier	Findings	TP	FP	FP Rate	Dominant root cause
L0	123	0	113	94%	DISC-047: decorators = [...] class attribute invisible to L1. Global CSRFProtect/rate limiter not recognized. url_for() redirects classified as user-input redirects.
L1	0	—	—	—	Correct result — flaskbb is well-maintained, no confusion patterns
L2	2	0	1	~50%	Flow tracing works (traces request.args.get("next") → redirect()), but misses is_safe_url() sanitizer 2 hops away
L3	39	0	39	100%	Same DISC-047 — comparative analysis inherits L0's blind spot

Key insight: Fixing ONE gap (DISC-047: extract MethodView decorators class attribute in L1) is projected to drop the FP rate from 94% to ~20% across L0+L3. This is the highest-leverage fix in the entire engine.

Report Template

Write your audit to local/v1-assessment/L3/<number>-<rule_ids>-audit.md:

# Audit: <rule_ids> against <repo>

## Methodology
How findings were verified. Repo version confirmed.

## Per-Rule Results

### <rule_id> (<N> findings)

| # | File:Line | Summary | Verdict | Reasoning |
|---|-----------|---------|---------|-----------|
| 1 | auth/views.py:42 | Missing CSRF on POST | FP | CSRFProtect(app) covers all routes |

### Summary Statistics

| Rule | Total | TP | FP | UNCERTAIN | NOISE |
|------|-------|----|----|-----------|-------|

## Dominant Gap Analysis

Gap X caused N/M FPs (X%). Root cause: ... Fix: ...

## Engine Capability Assessment

**What flawed detects reliably**: ...
**What it misses**: ...
**What it detects but with high FP rate**: ...

## Priority Fixes

1. Fix DISC-XXX → eliminates N FPs (X% reduction)

Anti-Patterns

Mistake	Why it's wrong	Correct approach
Running flawed during audit	Wastes compute, risks resource conflicts	Audit is READ-ONLY — JSON + source code
Trusting findings without checking source	You're not auditing, you're just counting	Open the actual file at the actual line
"FP" without saying WHY	Useless for fixing the engine	Identify the specific gap that caused the FP
Checking one finding per rule	Different findings have different patterns	Sample at least 5 per rule_id
Ignoring gaps array	That's the engine's own diagnostic output	Read it — it tells you what the engine couldn't do
Reporting "0 TP" as failure	0 TP on a secure repo IS the correct result	Check if the repo actually HAS the vulnerabilities