false-confidence-test-audit

스타11

포크0

업데이트2026년 7월 17일 17:38

Audit tests that may pass without proving the claimed behavior. Use for periodic test-quality reviews or when coverage looks healthy but regressions still escape.

설치

Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.

출처

aryaminus

aryaminus/controlkeel

GitHub 저장소 열기 Creator 저장소 보기

다운로드

관련 직업SOC

SOC 직업 분류 기준

소프트웨어 품질 보증 분석가·테스터컴퓨터 및 수학직·SOC 15-1253

SKILL.md

readonly

name	false-confidence-test-audit
description	Audit tests that may pass without proving the claimed behavior. Use for periodic test-quality reviews or when coverage looks healthy but regressions still escape.
when_to_use	Activate only when the user requests a test-quality, false-confidence, or assertion-strength audit. Do not rewrite tests until each weakness is reproduced or supported by concrete evidence.
argument-hint	[test path, suite, or commit range]
disable-model-invocation	true
license	Apache-2.0
compatibility	["codex","claude-standalone","claude-plugin","copilot-plugin","github-repo","open-standard","cursor-native","opencode-native"]
result-schema	{"type":"object","required":["scope","findings","commands_run","limitations"],"properties":{"scope":{"type":"string"},"findings":{"type":"array","items":{"type":"object","required":["path","claim","evidence","severity","recommended_probe"],"properties":{"path":{"type":"string"},"claim":{"type":"string"},"evidence":{"type":"string"},"severity":{"type":"string","enum":["high","medium","low"]},"recommended_probe":{"type":"string"}}}},"commands_run":{"type":"array","items":{"type":"string"}},"limitations":{"type":"array","items":{"type":"string"}}}}
metadata	{"author":"controlkeel","version":"1.0","category":"quality","ck_mcp_tools":["ck_context","ck_validate","ck_finding","ck_regression_result"]}

False-Confidence Test Audit

Find tests whose green result overstates what they prove. This is an evidence audit, not a request to maximize coverage or replace outcome tests with implementation checks.

Workflow

State the suite's claimed behavior and identify the production boundary that should make the claim observable.
Run the narrow test unchanged and record its command and result.
Inspect for assertions that cannot fail, status-only assertions, permissive schemas, broad truthiness checks, over-mocked boundaries, implementation mirroring, skipped CI lanes, missing negative cases, and fixtures that bypass the behavior under test.
For each suspected weakness, propose the smallest probe that would make the test fail if the production behavior were broken. Prefer a temporary mutation, boundary substitution, or explicit counterexample when safe.
Separate confirmed weaknesses from hypotheses. Record limitations when a probe cannot be run.
Fix only confirmed weaknesses, then demonstrate that the strengthened test fails against the broken behavior and passes against the correct behavior.

Boundaries

Never weaken production behavior to make a test easier to write.
Do not delete a test merely because it overlaps another; identify the distinct claim first.
Do not introduce sleeps, network dependence, random timing, or broad HTML snapshots as substitutes for behavioral evidence.
Full mutation testing is optional and must be budgeted separately.

Completion

Return output matching the declared result schema. A clean audit states what was inspected and which claims were actually challenged; “no findings” without commands and limitations is incomplete.

이 저장소의 다른 Skills

같은 저장소

bounded-loop

aryaminus/controlkeel

Run objective, repeatable improvement work under an immutable verifier and hard iteration, cost, time, and no-progress limits. Use for approved experiments with an automated metric and sandboxed worker.

2026-07-1711

communication-style

aryaminus/controlkeel

Write concise, calm technical responses whose structure matches the decision, evidence, or sequence being communicated. Use when drafting substantial plans, reviews, findings, documentation, or completion summaries.

2026-07-1711

controlkeel-governance

aryaminus/controlkeel

Operate inside a ControlKeel-governed session. Use this before code edits, shell execution, delegation, deploy work, or any task that needs CK validation, findings, budget, proof, or routing context.

2026-07-1711

cost-optimization

aryaminus/controlkeel

Keep a governed session within budget. Use this before long-running agent work, bulk processing, or any task where spend pressure could change the plan.

2026-07-1711

end-of-shift

aryaminus/controlkeel

Close a governed work session with validation, proof, findings, budget, digest, learning, and handoff checks. Use when the user asks to wrap up, stop for the day, or leave work ready for another agent.

2026-07-1711

handoff

aryaminus/controlkeel

Persist session state and hand off in-progress work to a background agent or delegated execution. Use when work outgrows the current session, context is near limit, or a task needs to continue unattended.

2026-07-1711

False-Confidence Test Audit

Find tests whose green result overstates what they prove. This is an evidence audit, not a request to maximize coverage or replace outcome tests with implementation checks.

Workflow

State the suite's claimed behavior and identify the production boundary that should make the claim observable.

Run the narrow test unchanged and record its command and result.

Inspect for assertions that cannot fail, status-only assertions, permissive schemas, broad truthiness checks, over-mocked boundaries, implementation mirroring, skipped CI lanes, missing negative cases, and fixtures that bypass the behavior under test.

For each suspected weakness, propose the smallest probe that would make the test fail if the production behavior were broken. Prefer a temporary mutation, boundary substitution, or explicit counterexample when safe.

Separate confirmed weaknesses from hypotheses. Record limitations when a probe cannot be run.

Fix only confirmed weaknesses, then demonstrate that the strengthened test fails against the broken behavior and passes against the correct behavior.

Boundaries

Never weaken production behavior to make a test easier to write.

Do not delete a test merely because it overlaps another; identify the distinct claim first.

Do not introduce sleeps, network dependence, random timing, or broad HTML snapshots as substitutes for behavioral evidence.

Full mutation testing is optional and must be budgeted separately.

Completion

Return output matching the declared result schema. A clean audit states what was inspected and which claims were actually challenged; “no findings” without commands and limitations is incomplete.