| name | rule-audit |
| description | Automated scan of a service/repo for compliance with AGENTS.md invariants D-01 through D-34 — produces a PASS/FAIL report with file:line evidence |
| allowed-tools | ["Read","Grep","Glob","Bash(grep:*)","Bash(rg:*)","Bash(python3:*)","Bash(conftest:*)","Bash(yq:*)"] |
| when_to_use | Invoke before a major release, after merging a large PR, during incident post-mortems to check whether a missing invariant contributed, or on a schedule (e.g. monthly) to catch drift from the template. Also useful when onboarding a new service created outside the scaffolder.
|
| argument-hint | [service-path] [--subset probes|pdb|security|closed-loop|all] |
| authorization_mode | {"scan":"AUTO","fix_findings":"CONSULT","patch_scaffolder":"STOP"} |
Rule Audit — Compliance scan against AGENTS.md invariants
This skill is READ-ONLY. It does not modify code. When findings are
identified, the agent produces a remediation plan and waits for human
approval (CONSULT) before any fix. Template-scaffolder patches require
STOP because they affect all future services.
When NOT to use this skill
- Before running tests — broken tests are a faster signal than
structural audits.
- During active incidents — use
/incident or /rollback first.
Come back to audit in the post-mortem.
- On third-party code — this skill is scoped to the template's own
conventions.
Invariant catalogue (D-01 → D-35)
The checks below map one-to-one onto the AGENTS.md anti-pattern table.
The agent runs the matching query for each ID and records PASS/FAIL +
evidence (file:line or metric) per service.
Serving (D-01 → D-05)
| ID | Check | Command |
|---|
| D-01 | uvicorn --workers N where N > 1 | rg -n "uvicorn.*--workers\s+[2-9]" --glob "**/Dockerfile*" --glob "**/*.sh" |
| D-02 | HPA uses memory metric | yq '.spec.metrics[] | select(.resource.name == "memory")' k8s/**/hpa.yaml |
| D-03 | model.predict( inside an async def without run_in_executor | rg -n "async def.*:" -A 30 app/ | rg "model\.(predict|predict_proba)\(" |
| D-04 | shap.TreeExplainer used with stacking/ensemble wrapper | rg -n "TreeExplainer" app/ |
| D-05 | == in requirements.txt for ML packages | rg -n "^(numpy|scipy|scikit-learn|pandas|xgboost|lightgbm)==" requirements*.txt |
Infrastructure + Quality (D-06 → D-12)
| ID | Check |
|---|
| D-06 | Any metric in artifacts/metrics.json ≥ 0.99 — requires ADR |
| D-07 | Fairness DIR < 0.80 or missing |
| D-08 | PSI CronJob reports but no alert routes configured |
| D-09 | Drift CronJob last_run > 48h → heartbeat missing |
| D-10 | terraform.tfstate* tracked in git |
| D-11 | Model artifact present in Dockerfile (COPY *.joblib / *.pkl) |
| D-12 | Promotion pipeline missing a quality-gate script |
Data + EDA (D-13 → D-16)
| ID | Check |
|---|
| D-13 | EDA notebooks writing under data/raw/ or data/processed/ |
| D-14 | Pandera schemas missing Check.in_range derived from EDA |
| D-15 | canonical baseline_distributions.parquet missing or never regenerated |
| D-16 | canonical feature_catalog.yaml without rationale: on every transform |
Security (D-17 → D-19)
| ID | Check |
|---|
| D-17 | `os.environ[.*API_KEY |
| D-18 | AWS_ACCESS_KEY_ID or raw JSON GCP SA key in committed files |
| D-19 | Deploy workflow missing cosign sign + syft step |
Closed-loop (D-20 → D-22)
| ID | Check |
|---|
| D-20 | PredictionEvent( without entity_id= or prediction_id= |
| D-21 | log_prediction( called synchronously in async def endpoint |
| D-22 | Missing except Exception + prediction_log_errors_total metric |
Lifecycle + Operations (D-23 → D-27 — v1.7.1)
| ID | Check | Automated? |
|---|
| D-23 | Liveness and readiness probes share httpGet.path | yes (Rego) |
| D-24 | KernelExplainer(...) constructed per request (not cached at startup) | rg -n "KernelExplainer\(" app/ |
| D-25 | terminationGracePeriodSeconds missing OR <= uvicorn --timeout-graceful-shutdown | yes (Rego) |
| D-26 | deploy-*.yml missing the 3-env chain (dev/staging/prod environments) | grep |
| D-27 | Deployment without matching PodDisruptionBudget | yes (Rego) |
Contracts + Supply chain + IAM (D-28 → D-32)
| ID | Check | Automated? |
|---|
| D-28 | tests/contract/openapi.snapshot.json changed without app.version bump in app/main.py | tests/contract/test_openapi_snapshot.py + CI Validate API contract |
| D-29 | Namespace manifest missing pod-security.kubernetes.io/enforce label | rg -n "pod-security.kubernetes.io/enforce" k8s/overlays/*/namespace.yaml (every overlay must hit) |
| D-30 | Deploy workflow missing cosign attest --type cyclonedx after the SBOM step | rg -n "cosign attest" .github/workflows/deploy-*.yml |
| D-31 | Single IAM identity reused across ci/deploy/runtime/drift/retrain | tests/test_iam_least_privilege.py (no wildcard principals, no Action: "*") |
| D-32 | K8s manifest references a Python path with kebab placeholder (src/{service-name}/...) | tests/policy/test_anti_patterns.py::test_d32_drift_cronjob_python_path |
Template lifecycle (D-33 → D-34)
| ID | Check | Command |
|---|
| D-33 | Manual cp -r + sed -i in the scaffolder instead of copier copy | rg -n "sed.*-i.*\{service\}|cp.*-r.*templates/service" templates/scripts/new-service.sh |
| {% raw %} | | |
| D-34 | Unquoted {@ @} Jinja tokens in YAML list items | rg -n '^\s*- \{@' templates/service/ --glob "*.yml" |
| {% endraw %} | | |
Execution flow
Step 1 — Scope (AUTO, 30 s)
Agent identifies services to audit:
find . -maxdepth 3 -type d -name "k8s" -prune | xargs dirname
If the user specifies [service-path], restrict to that directory.
Step 2 — Static invariants (AUTO, 2 min)
For each invariant in the catalogue, run the command, capture hits, and
write one row to ops/rule_audit.jsonl:
{"service":"fraud_detector","invariant":"D-03","status":"FAIL","evidence":"app/main.py:142: async def predict(...): prediction = model.predict(X)","timestamp":"..."}
Step 3 — Structural invariants via conftest (AUTO, 1 min)
The Rego policies in tests/infra/policies/ cover D-23/D-25/D-27 and
several D-02 / D-11 checks. Run:
conftest test --policy tests/infra/policies/ k8s/base/ k8s/overlays/<env>/ 2>&1
Record FAIL/WARN as additional rows.
Step 4 — Produce summary (AUTO, 30 s)
The agent emits:
# Rule Audit — {service-name} — {date}
Summary: 23 PASS / 2 FAIL / 1 WARN / 1 SKIPPED
## Failures (require CONSULT)
- D-04 app/explain.py:18 — uses shap.TreeExplainer with a StackingClassifier
Fix: swap to KernelExplainer via predict_proba_wrapper (rule 04a).
- D-26 .github/workflows/deploy-gcp.yml — single 'production-gcp' environment
Fix: adopt dev→staging→prod chain (docs/environment-promotion.md).
## Warnings
- D-25 k8s/base/deployment.yaml:45 — terminationGracePeriodSeconds (30)
equals uvicorn --timeout-graceful-shutdown (30). Best practice: strictly
greater (termGrace > uvicorn timeout).
## Skipped
- D-06 — no metrics.json available in this service.
Step 5 — Remediation plan (CONSULT)
For each FAIL the agent proposes the MINIMAL patch and waits for
operator approval. No code is written until the operator confirms.
Step 6 — Audit trail (AUTO)
Every run appends an AuditEntry tagged operation: rule_audit to
ops/audit.jsonl via common_utils.agent_context.AuditLog, carrying
the summary counts. A run with FAILs sets result: halted to flag the
issue in the ops log.
Subset flag
For large repos the agent can restrict scope:
--subset probes → D-23, D-25 only--subset pdb → D-27 only--subset security → D-17, D-18, D-19, D-30, D-31--subset closed-loop → D-20, D-21, D-22, D-24--subset contracts → D-28, D-32--subset template → D-33, D-34--subset all (default) → every invariant
What this skill is NOT
- NOT a replacement for unit tests or integration tests (different signal)
- NOT a code formatter (black/isort handle that)
- NOT a security scanner (trivy/gitleaks handle that — this skill CHECKS
that they are wired, not the findings themselves)
- NOT a migration tool — it identifies gaps but proposes patches as
CONSULT, letting the human decide whether each fix is worth it
Invariants
- Audit is READ-ONLY by default. The agent MUST NOT write fixes in AUTO.
- Every FAIL is evidence-backed (
file:line or metric name) — never
"might be missing".
- Subset flag is idempotent: running
--subset probes and then --subset all produces a superset of the first run.
- The audit log entry is ALWAYS created, even when every check passes
(historical record).
Related
- AGENTS.md anti-pattern table (D-01..D-35)
agentic/skills/security-audit/SKILL.md — deeper security scanagentic/skills/debug-ml-inference/SKILL.md — uses D-01..D-35 as
diagnostic checklistcommon_utils/agent_context.py::AuditLog — audit trail integration
