track-findings

name	track-findings
description	Use when the user wants to record human triage feedback or machine validation results against the findings of a *-security-audit report and produce a cumulative status report — ingesting merge-request comments, commits, Jira tickets, interactive human triage, live-validation reports, and remediation-verification reports into an append-only disposition layer, then generating a schema-conformant *-findings-current.{json,md} showing the current validity and resolution of every finding.
user-invocable	true

Track Findings

An audit report is a snapshot; the truth about its findings keeps evolving afterward. This skill maintains a disposition layer — an append-only ledger of who or what said what about each finding since the audit — and regenerates a cumulative report showing every finding's current state.

Two artifacts, per audited repo, living next to the original audit report (e.g. analysis-results/findings/<product>/<repo>/):

Artifact	Schema	Nature
<repo>-findings-layer.json	schema/layer.schema.json	Append-only ledger. Events are never edited or deleted; corrections are new events.
<repo>-findings-current.{json,md}	schema/report.schema.json (with disposition blocks)	Derived. Regenerated from scratch every run by build_cumulative.py.

The original *-security-audit.json is never modified — and since harness 0.39.0 that is machine-enforced: scripts/baseline_claims.py records a sha256 of every finding's claim fields into metadata.claim_hashes, validate_report.py verifies them on every layer validation, and build_cumulative.py refuses to rebuild from a baseline whose claims drifted. Sanctioned appends (follow-up findings) are pinned on the next run; in-place revision requires validation_status: corrected plus an explicit --rebaseline.

Naming matters: the cumulative file is deliberately named -findings-current, NOT -security-audit-*, so the portfolio roll-ups (executive-summary-findings, insecure-patterns) that glob *security-audit.{json,md} never double-count it.

When to Use

• User has triage feedback on findings from an MR/PR discussion, a commit, or a Jira ticket and wants it recorded
• User wants to run an interactive triage session over a report's findings
• User wants machine results (*-validation.json from validate-findings, *-remediation-verification.json from verify-remediation) folded into a single current-status view
• User asks "what's the current state of the findings for ?", "mark finding X false positive", "ingest the MR feedback", or "build the cumulative report"

Inputs

$ARGUMENTS contains the audit report reference followed by zero or more disposition sources:

1. The audit report — same resolution as verify-remediation:

   • Path to a *-security-audit.json
   • <product>/<repo> → analysis-results/findings/<product>/<repo>/<repo>-security-audit.json
   • <slug>-findings/<repo> → progress-tracker/processed-results/<slug>-findings/<repo>/...

2. Sources (each classified by shape):

   • MR/PR URL (/-/merge_requests/N or /pull/N) — ingest discussion comments
   • Commit (7–40 hex chars, or a commit URL, or --commit <sha>) — ingest the commit message
   • Jira key ([A-Z][A-Z0-9]+-\d+) — ingest ticket status + comments
   • Report path (*.json) — a validation or remediation-verification report
   • --interactive — run a triage session with the human
   • No sources at all — auto-discover: machine reports in the audit report's directory and analysis-results/validations/*/, plus any Jira keys recorded by the file-security-defect skill

Examples:

/track-findings openstack-operator/sg-core https://gitlab.cee.redhat.com/g/sg-core/-/merge_requests/17
/track-findings openstack-operator/sg-core OSPRH-1234 OSPRH-1235
/track-findings openstack-operator/sg-core --commit feedface12
/track-findings openstack-operator/sg-core path/to/sg-core-remediation-verification.json
/track-findings openstack-operator/sg-core --interactive
/track-findings openstack-operator/sg-core

--auto skips interactive confirmations; only tier-1 events are recorded and everything else queues (see the trust policy below).

---

The Two-Axis Disposition Model

Every event answers one or both of two independent questions:

Axis	Question	Values
validity	Is the finding real?	confirmed, false_positive, corrected, hardening (not_verified is the default, never set by an event)
resolution	Is it dealt with?	open, fix_in_progress, resolved, partially_resolved, risk_accepted, regression_introduced

hardening (harness ≥ 0.27.0) = accurately-described defense-in-depth or benchmark gap with no concrete exploit path (triage exclusion rule 13) — real, risk-bearing at a category-aware λ weight in findings-trends, never a false positive, and never counted as a confirmed vulnerability.

The cumulative report keeps validation_status equal to the validity axis, so all existing consumers keep working; the resolution axis lives in the new per-finding disposition block.

Source vocabulary mapping

Source	Feeds	Mapping
*-validation.json verdicts (via scripts/emit_validation_ledger_events.py)	validity	confirmed → confirmed (class-1 execution-verified — outranks every static determination); refuted → machine false_positive event (see countersign rule) + *-refuted-register.json merge-append; inconclusive/blocked_by_scope/not_attempted → no event. One validation report fans out across every repo ledger its source_reports cover; baselines resolve by direct path, content sha256, or claim-hash match (claim-identical alias trees all receive the event), and a direct-path baseline whose sha changed since validation queues a stale_baseline review item. --manifest batch mode replays a whole validation campaign idempotently.
*-triage.json verdicts (via scripts/emit_triage_ledger_events.py)	validity	true_positive → confirmed (skipped when needs_manual_test); hardening → hardening with category-aware λ recorded at emission; false_positive → machine FP event — auto-accept tier (auto_accept_tier: true: unanimous + lint-clean + confidence ≥ 8 + low/informational-claimed) sets validity directly with a 10% audit-valve sample queued, everything else countersign-gated; undetermined → needs_review item (undetermined_finding); duplicate → nothing. Every FP also lands in *-refuted-register.json (fuzzing/live-validation targeting input). Auto-discover *-triage.json next to the audit report.
*-remediation-verification.json verdicts	both	resolved → resolved; partially_resolved → partially_resolved; unresolved → open; new_approach → resolved; regression → regression_introduced; false_positive → validity per its disposition_rationale (human event if it names a verified human, else machine); risk_accepted → risk_accepted
Jira	both	Done/Fixed → resolved; Won't Do with risk language → risk_accepted; Not a Bug → false_positive (attributed to the resolving user from the ticket changelog, never the bare status); In Progress → fix_in_progress. Ambiguous resolutions are queued, never guessed.
MR/PR comment	either	Tier policy below
Commit message	resolution	Finding ID + fix trailer → fix_in_progress (never resolved — that claim belongs to verify-remediation, which actually checks the code)
Interactive	either	Human states it directly

The human-countersign rule (hard requirement)

validation_status: false_positive requires an LDAP-verified human determination — with one carve-out. A machine refuted verdict enters the ledger as evidence but the cumulative report shows the finding as "refuted — awaiting human sign-off" (disposition.refuted_awaiting_signoff: true) until a human countersigns via any tier-1 source or an interactive session. The carve-out (harness ≥ 0.27.0): auto-accept-tier triage FPs (auto_accept_tier: true — unanimous, verdict-citation-lint clean, mean confidence ≥ 8, claimed severity low/informational) set validity directly, with a deterministic 10% sample queued for human audit (fp_audit_valve). Both the layer validator and build_cumulative.py enforce this.

Evidence-class precedence and the falsifiability of FPs (harness ≥ 0.27.0)

Validity conflicts resolve by evidence strength, not actor identity: 1 execution-verified (validation_report, verification_report — reproducing exploits/crashes) > 2 human static determinations > 3 machine static (triage_report and other machine events). Recency breaks ties only within a class. Consequences, all enforced by build_cumulative.py:

• A class-1 confirmed overrides a human false-positive assertion — validity flips to confirmed and disposition.fp_overridden: true surfaces loudly with the original countersigner attributed. A human FP is a falsifiable claim, never a shield against assessment.
• Two-person rule: after execution-verified confirmation, re-asserting false positive requires TWO distinct LDAP-verified humans post-dating the proof; a single attempt sets disposition.fp_reassertion_blocked.
• Every FP (auto-accepted or countersigned) stays in the refuted register, feeding fuzz-harness target selection and live-validation scoping — nothing leaves verification scope because someone said "false positive".
• Each disposition carries assurance (claimed | machine_verified | human_reviewed | execution_proven) — the highest evidence class that has spoken; findings-trends reports the claimed/verified/proven views, the triage-compression and validation-gap deltas, and per-identity FP-override rates.

---

The Three-Tier Trust Policy for Human Statements

MR comments (and free-text Jira comments) are conversation, not workflow state. The skill only auto-records what needs zero interpretation.

Tier 1 — auto-record (all three required):

1. Explicit grammar: the statement names a canonical finding ID and makes a declarative disposition — either the structured convention /glasswing <disposition> <FINDING-ID> <rationale> (dispositions: false-positive, confirmed, risk-accepted, fix-in-progress, resolved) or an unambiguous declarative sentence ("SG_CORE-abcdef0-001 is a false positive: the value is validated in the CRD webhook before this path").
2. Verified identity: the author's forge username maps to an employee via python ai-security-harness/scripts/validate_employee.py.
3. Authority: the author is one of the findings package's assigned owners (owners.csv / assign-findings-owners output) or a project Maintainer+ (forge members API).

Tier 2 — needs-review queue: anything else that mentions a finding ID or reads disposition-like. Recorded in the layer's needs_review[] with verbatim quote, permalink, author, queue_reason (ambiguous_statement / unverified_identity / insufficient_authority / no_finding_id), and the skill's suggested interpretation — but no state change. Pending items surface at the top of the cumulative Markdown and get resolved in the next --interactive session: on confirmation, a real event is appended attributed to the confirming human, with the original commenter preserved in source.reported_by.

Never: model-inferred intent from vague text silently changing state. --auto does not lower the bar — it only skips confirmation for tier-1 events, which need none.

---

Procedure

Phase 1 — Resolve and load

1. Resolve the audit report; load findings and metadata.commit.
2. Load <repo>-findings-layer.json if present; otherwise initialize it:

   {"metadata": {"audit_report": "<relative path>", "audit_commit": "...",
     "repository": "...", "created": "<now>", "harness_version": "<VERSION>-<sha>"},
    "events": [], "needs_review": []}
   

3. Build the set of valid finding IDs. Every event must reference one.
4. Baseline the claims (tamper-evidence for the audit report):

   python3 <harness>/scripts/baseline_claims.py record <audit.json> <layer.json>
   

   This pins each finding's canonical claim fields into metadata.claim_hashes — new findings (sanctioned appends from /fuzz-harnesses or /vuln-scan) get added; already-baselined hashes are never overwritten. If record refuses (hash mismatch), the audit report was edited in place: stop and surface it to the user — either the report must be restored from git, or, for a finding legitimately revised under validation_status: corrected, re-run with --rebaseline <id>. Do not proceed to ingestion over a tampered baseline; build_cumulative.py would refuse anyway.

Phase 2 — Ingest each source

Classify each argument and run the matching adapter. For every candidate event, compute the idempotency key before appending:

event_id = sha256("<source.ref>|<finding_ref>|<validity or ''>|<resolution or ''>")

If an event with that event_id already exists, skip it silently — re-running the skill on the same MR/ticket/report is a no-op.

Set occurred_at on every event to the source's own timestamp — commit author date, Jira transition date, MR comment creation date, machine report metadata.date — so the findings-trends skill buckets the event into the period it actually happened, not the period it was ingested. recorded_at stays the append time (it must remain chronological across the ledger). If no source timestamp is available, omit occurred_at and tooling falls back to recorded_at.

MR/PR comments — GitLab: mr_discussions / list_merge_request_notes (MCP); GitHub: PR review comments. For each comment: scan for finding IDs and disposition statements; apply the tier policy; for tier-1, resolve the author (username → LDAP via validate_employee.py; authority via owners.csv and the members API); event source.ref is the comment permalink; rationale is the verbatim quote.

Commit — git show -s --format='%H%n%an <%ae>%n%aI%n%B' <sha> (or the forge commit API). Look for finding IDs in the message/trailers. Map to fix_in_progress. The commit author is the actor (LDAP-verify the email).

Jira — jira_get_issue with changelog + comments. Match the finding: tickets filed by file-security-defect carry the finding ID in the description; otherwise match by summary and confirm interactively. Map status/resolution per the table; attribute to the user who made the transition (from the changelog). Ambiguous → queue.

Machine reports — validate them first (validate_report.py --schema validation.schema.json / verification.schema.json); refuse unvalidated input. One event per finding verdict, actor.kind: machine, actor.identity: "<skill-name>", source.ref = report path, rationale = the report's explanation for that finding.

Interactive — use the countersign workbench (scripts/countersign.py); never hand-assemble interactive events.

1. Build the inbox: python3 <harness>/scripts/countersign.py queue --root <findings-tree> (or --repo-dir <one repo>). This derives every refuted_awaiting_signoff finding and pending needs_review item and renders one self-contained DECISION CARD each — the audit claim, the machine refutation with lint-verified evidence, and what signing records — so the reviewer never needs the original report open.
2. Present ONE card per question. Show the card as the question's preview; options: Confirm false positive / Keep open (reject the refutation) / Defer. A blank rationale on false_positive adopts the machine rationale, explicitly framed as the signer's adopted determination; keep_open REQUIRES the human's own rationale — "false positive" (or its rejection) without a why is not recordable.
3. Record through the tool: countersign.py record --layer <ledger> --finding <ref> --decision ... --identity <kerberos-id> [--rationale ...] — it LDAP-verifies the identity once (validate_employee.py; recording is REFUSED for an unverifiable identity), appends canonical events (source.ref = interactive:<ISO date>, ldap_verified: true), rebuilds the cumulative pair, and prints a receipt.
4. Asynchronous alternative: hand the reviewer the annotatable countersign-queue.md; they mark [x] decisions and rationales in an editor, then countersign.py apply countersign-queue.md --identity <id> ingests the whole batch (still LDAP-verified).
5. Pending needs_review items appear at the end of the queue file — confirm/reject each interactively (rejections get a resolution_note).

Append events in recorded_at order (the layer must stay chronological); update metadata.updated.

Phase 3 — Merge deterministically

Never hand-compute the cumulative state. Run the merge engine (from the workspace root):

python ai-security-harness/harnessing/track-findings/build_cumulative.py \
  <audit.json> <layer.json>

It applies the precedence rules (human > machine on validity; verification_report > jira > other on resolution; latest within a tier), the countersign gate, and conflict detection (confirmed + false_positive both present → conflict: true, surfaced in a "needs re-review" section, never silently resolved), and writes <repo>-findings-current.{json,md}.

Phase 4 — Validate and deliver

python ai-security-harness/scripts/validate_report.py --schema layer.schema.json <repo>-findings-layer.json
python ai-security-harness/scripts/validate_report.py <repo>-findings-current.json

Both must pass (the default schema validates the cumulative report's disposition blocks and summary counts). Then present the delta:

## Findings Status Updated: <repo>

**This run:** 3 events appended, 2 statements queued for review, 1 duplicate skipped.

| Finding | Was | Now | Via |
|---|---|---|---|
| SG_CORE-abcdef0-001 | not_verified / open | not_verified / resolved | verify-remediation report |
| SG_CORE-abcdef0-002 | not_verified / open | false_positive / open | MR !17 comment (jdoe, LDAP-verified) |

**Awaiting human sign-off:** SG_CORE-abcdef0-003 (refuted by live validation)
**Needs review:** 2 pending — resolve with `/track-findings <report> --interactive`

---

Gotchas

• The layer is the authority; the cumulative report is a cache. Never edit *-findings-current.json directly, and never edit or delete a ledger event — append a correcting event instead. The validator hard-errors on non-chronological or hash-mismatched events.

• Idempotency lives in the event_id. If you change how a source maps to a disposition, the same comment produces a different event_id and would double-record. Skip ingestion-time remapping of already-ingested sources; append a correction event instead.

• A machine can confirm, only a human can refute permanently. Execution evidence proving exploitability sets validity=confirmed on its own. Execution evidence failing to exploit sets refuted_awaiting_signoff — absence of proof isn't proof of absence, so a named human owns every false_positive call.

• Jira statuses lie about who decided. Always attribute to the user in the changelog transition, not the assignee or reporter. If the changelog is inaccessible, queue instead of recording.

• Don't let the cumulative filename drift. Anything matching *security-audit.{json,md} gets swept into the portfolio dashboards and double-counts the repo.

• Bot comments never tier-1. CI bots and integration accounts fail the LDAP check by design; their statements queue at most.

• Conflicts are a feature. When a human and the machine disagree, the report is supposed to look unresolved — that's the signal for re-validation or a triage conversation, not something to normalize away.

• This skill dispositions findings; it doesn't verify fixes. "The MR fixing this merged" is fix_in_progress until /verify-remediation says resolved. Point users there for code-level verification.