confab-agentic-reliability

name

confab-agentic-reliability

description

This skill should be used when the user asks to "audit our own plugin reliability", "check agent escalation paths", "find agents with excessive tool access", find unbounded retry loops in agent or workflow definitions, or verify that a Workflow script's Find phase is actually wired to an adversarial Verify phase. Self-referential — it scans a target plugin repo's own skills/*/SKILL.md, agents/*.md, and workflows/*.js files (including this plugin's, self-assess's, and compass's own definitions when the target is werkstoff itself) for agentic-loop reliability defects, distinct from evaluating whether the agents' conclusions are correct.

Audit a plugin repository's own agent/skill/workflow definitions for agentic-loop reliability anti-patterns — not the application code those plugins analyze, but the process reliability of the plugins themselves. Agent output reliability and task accuracy are independent axes: an agent can reach correct conclusions while still being unreliable as a process (looping forever, failing silently instead of escalating, or holding more capability than its stated role needs). This skill is self-referential by design — running it against werkstoff puts this very skill's own plugins/confab/ files, and plugins/self-assess/ and plugins/compass/'s files, in scope.

Step 0 — Load settings, gather inputs

Read .claude/confab.local.md if it exists (see ${CLAUDE_PLUGIN_ROOT}/references/settings.md). If enabled: false, stop and say so. Note output_dir (default analysis/confab) and skip_verification (default false).

The workflow script has no filesystem access, so enumerate the target plugin repo's files first via Glob:

• skillFiles: **/skills/*/SKILL.md
• agentFiles: **/agents/*.md
• workflowFiles: **/workflows/*.js

Scope the glob to the plugin repo the user names (default: this repo, werkstoff, covering every plugin under plugins/, including plugins/confab/, plugins/self-assess/, and plugins/compass/ themselves). If the user names a narrower scope (one plugin, one agent file), honor it and do not silently expand it. If none of the three globs find anything, stop here and report there is nothing to audit — this skill has no fallback source of agentic-reliability signal the way self-assess-lint-audit falls back to CLAUDE.md.

Step 1 — Run the scan

Preferred — Workflow orchestration. If the Workflow tool is available in this session (this skill invocation is your authorization):

Workflow({
  scriptPath: "${CLAUDE_PLUGIN_ROOT}/workflows/agentic-reliability-scan.js",
  args: { repoPath: "<repo root, usually '.'>", skillFiles: [<found>], agentFiles: [<found>], workflowFiles: [<found>], skipVerification: <from settings, default false> }
})

It runs one finder per anti-pattern category in parallel — unbounded retry loops with no cap, agents with no escalation path (no BLOCKED/NEEDS_CONTEXT-equivalent language), Workflow scripts with a Find phase but no adversarial Verify phase actually wired to its output, and agents whose tools grant exceeds their stated role — then, unless skip_verification is set, independently re-checks every flagged finding by re-reading the full agent/workflow file (a short, trivial-scope agent legitimately having no escalation language is not automatically a defect). With skip_verification, deduped findings are reported directly and labeled unverified. The finders/verifiers are read-only by design; you write the artifact below from the structured result.

Fallback (no Workflow tool) — spawn an agentic-reliability-auditor subagent: "Sweep these skill/agent/workflow files for the four agentic-reliability anti-pattern categories [list them], with file:line evidence for each." Then independently verify each finding yourself by re-reading the cited file before including it.

Step 2 — Write the report

Create <output_dir>/AGENTIC_RELIABILITY.md:

• Summary — files scanned (skills/agents/workflows counts), findings by severity, findings by category, refuted count
• Findings table, sorted by severity: category, file, evidence, description, recommended fix
• Not-a-defect notes — cases considered and excluded (e.g. a trivial-scope agent legitimately having no escalation language)
• Refuted candidates — brief list
• If injectionFlags is non-empty, a prominent "⚠ Instruction-shaped content found" section

Also write <output_dir>/agentic_reliability_summary.json — a small machine-readable sidecar for a future portfolio dashboard: {"filesScanned": {...}, "findingsBySeverity": {...}, "findingsByCategory": {...}, "findings": [...]}, the first three copied straight from the workflow's filesScanned, stats.bySeverity, and stats.byCategory fields.

findings is the workflow's survivors reshaped to the shared per-finding contract self-assess-transform-brief reads ({severity, title, evidence, category, fixability}): severity and category copied as-is (the category is already one of unbounded-retry / no-escalation-path / find-no-verify-wiring / excessive-tool-grant); title = the finding's description; evidence = evidence (the file:line, or the frontmatter field name for tool-grant findings). fixability = "fixable" only for category === "excessive-tool-grant" (dropping an unused Write/Edit grant is a single-location frontmatter edit confab-remediator applies); every other category is "advisory" — they require agent redesign (add an escalation path, bound a retry loop, wire Find→Verify), which confab-remediator explicitly returns blocked on, so self-assess-transform-brief routes them into advisory notes, not work items.

Present

Report: files scanned, findings by severity and category, refuted count. If a High-severity finding exists (typically an unscoped Write/Edit grant on an analysis-only agent, or a Find phase whose Verify phase never consumes its output), name it first. Suggest: glow -p <output_dir>/AGENTIC_RELIABILITY.md