false-confidence-test-audit

Étoiles11

Forks0

Mis à jour17 juillet 2026 à 17:38

Audit tests that may pass without proving the claimed behavior. Use for periodic test-quality reviews or when coverage looks healthy but regressions still escape.

Installation

Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.

Source

aryaminus

aryaminus/controlkeel

Ouvrir le dépôt GitHub Voir les dépôts du créateur

Téléchargement

Métiers associésSOC

Basé sur la classification professionnelle SOC

Analystes en assurance qualité des logiciels et testeursProfessions informatiques et mathématiques·SOC 15-1253

SKILL.md

readonly

name	false-confidence-test-audit
description	Audit tests that may pass without proving the claimed behavior. Use for periodic test-quality reviews or when coverage looks healthy but regressions still escape.
when_to_use	Activate only when the user requests a test-quality, false-confidence, or assertion-strength audit. Do not rewrite tests until each weakness is reproduced or supported by concrete evidence.
argument-hint	[test path, suite, or commit range]
disable-model-invocation	true
license	Apache-2.0
compatibility	["codex","claude-standalone","claude-plugin","copilot-plugin","github-repo","open-standard","cursor-native","opencode-native"]
result-schema	{"type":"object","required":["scope","findings","commands_run","limitations"],"properties":{"scope":{"type":"string"},"findings":{"type":"array","items":{"type":"object","required":["path","claim","evidence","severity","recommended_probe"],"properties":{"path":{"type":"string"},"claim":{"type":"string"},"evidence":{"type":"string"},"severity":{"type":"string","enum":["high","medium","low"]},"recommended_probe":{"type":"string"}}}},"commands_run":{"type":"array","items":{"type":"string"}},"limitations":{"type":"array","items":{"type":"string"}}}}
metadata	{"author":"controlkeel","version":"1.0","category":"quality","ck_mcp_tools":["ck_context","ck_validate","ck_finding","ck_regression_result"]}

False-Confidence Test Audit

Find tests whose green result overstates what they prove. This is an evidence audit, not a request to maximize coverage or replace outcome tests with implementation checks.

Workflow

State the suite's claimed behavior and identify the production boundary that should make the claim observable.
Run the narrow test unchanged and record its command and result.
Inspect for assertions that cannot fail, status-only assertions, permissive schemas, broad truthiness checks, over-mocked boundaries, implementation mirroring, skipped CI lanes, missing negative cases, and fixtures that bypass the behavior under test.
For each suspected weakness, propose the smallest probe that would make the test fail if the production behavior were broken. Prefer a temporary mutation, boundary substitution, or explicit counterexample when safe.
Separate confirmed weaknesses from hypotheses. Record limitations when a probe cannot be run.
Fix only confirmed weaknesses, then demonstrate that the strengthened test fails against the broken behavior and passes against the correct behavior.

Boundaries

Never weaken production behavior to make a test easier to write.
Do not delete a test merely because it overlaps another; identify the distinct claim first.
Do not introduce sleeps, network dependence, random timing, or broad HTML snapshots as substitutes for behavioral evidence.
Full mutation testing is optional and must be budgeted separately.

Completion

Return output matching the declared result schema. A clean audit states what was inspected and which claims were actually challenged; “no findings” without commands and limitations is incomplete.

Plus depuis ce dépôt

même dépôt

bounded-loop

aryaminus/controlkeel

Run objective, repeatable improvement work under an immutable verifier and hard iteration, cost, time, and no-progress limits. Use for approved experiments with an automated metric and sandboxed worker.

2026-07-1711

communication-style

aryaminus/controlkeel

Write concise, calm technical responses whose structure matches the decision, evidence, or sequence being communicated. Use when drafting substantial plans, reviews, findings, documentation, or completion summaries.

2026-07-1711

controlkeel-governance

aryaminus/controlkeel

Operate inside a ControlKeel-governed session. Use this before code edits, shell execution, delegation, deploy work, or any task that needs CK validation, findings, budget, proof, or routing context.

2026-07-1711

cost-optimization

aryaminus/controlkeel

Keep a governed session within budget. Use this before long-running agent work, bulk processing, or any task where spend pressure could change the plan.

2026-07-1711

end-of-shift

aryaminus/controlkeel

Close a governed work session with validation, proof, findings, budget, digest, learning, and handoff checks. Use when the user asks to wrap up, stop for the day, or leave work ready for another agent.

2026-07-1711

handoff

aryaminus/controlkeel

Persist session state and hand off in-progress work to a background agent or delegated execution. Use when work outgrows the current session, context is near limit, or a task needs to continue unattended.

2026-07-1711

False-Confidence Test Audit

Find tests whose green result overstates what they prove. This is an evidence audit, not a request to maximize coverage or replace outcome tests with implementation checks.

Workflow

State the suite's claimed behavior and identify the production boundary that should make the claim observable.

Run the narrow test unchanged and record its command and result.

Inspect for assertions that cannot fail, status-only assertions, permissive schemas, broad truthiness checks, over-mocked boundaries, implementation mirroring, skipped CI lanes, missing negative cases, and fixtures that bypass the behavior under test.

For each suspected weakness, propose the smallest probe that would make the test fail if the production behavior were broken. Prefer a temporary mutation, boundary substitution, or explicit counterexample when safe.

Separate confirmed weaknesses from hypotheses. Record limitations when a probe cannot be run.

Fix only confirmed weaknesses, then demonstrate that the strengthened test fails against the broken behavior and passes against the correct behavior.

Boundaries

Never weaken production behavior to make a test easier to write.

Do not delete a test merely because it overlaps another; identify the distinct claim first.

Do not introduce sleeps, network dependence, random timing, or broad HTML snapshots as substitutes for behavioral evidence.

Full mutation testing is optional and must be budgeted separately.

Completion

Return output matching the declared result schema. A clean audit states what was inspected and which claims were actually challenged; “no findings” without commands and limitations is incomplete.