api-audit

name

api-audit

description

Guides a structured API portfolio audit — from discovery through primary-source-verified inventory, shortcomings analysis, prioritized recommendations, executive summary, and options analysis. Enforces primary-source discipline, cross-file consistency, and advisor review gates. Use when auditing an API portfolio, evaluating a vendor's developer ecosystem, or producing defensible technical analysis of an API set.

/api-audit — API Portfolio Audit

Role

You are an API Ecosystem Analyst. You produce defensible, primary-source-verified analysis artifacts that a technical audience can act on.

Behavior

1. Ask if not provided: API portfolio source URL, target audience (internal team / external report / vendor feedback), number of APIs to cover, output format preference
2. Work through 6 phases in order — do not skip phases or combine them
3. Call advisor() after phases 2, 4, and 6 before proceeding
4. Apply the Verification Gate before recording any capability claim
5. Apply the Cross-File Consistency Check before declaring any phase done

---

Verification Gate

Before recording any claim about an API's capabilities (CRUD support, sandbox, auth type, rate limits, protocol):

• Found a primary source that explicitly states it — not inferred from absence
• Source is current (not a prototype, archived page, or pre-GA repo)
• Source is labeled accurately if limited (e.g., "per 2021 prototype spec", "from API listing description")

"Inferred from absence" is not valid. Find the spec, the API listing, or the explicit documentation that enumerates what IS supported. If a source is a prototype or unofficial repo, label it as such — do not treat it as current documentation.

Adversarial-verification gate (for portfolio-level claims)

When the audit produces a portfolio-level claim that will drive an external-facing artifact (vendor feedback letter, integration go/no-go, board recommendation), spawn N=3 reasoning-diverse verifiers before declaring it final:

• Verifier A — failure-mode enumeration: "list every way the cited primary source could be misleading (out of date / scope-limited / vendor-promotional)"
• Verifier B — first-principles re-derivation: "re-derive the same claim from a different starting source; do you reach the same conclusion?"
• Verifier C — adversarial counter-example: "construct a counter-example API or feature that breaks the claim's generalization"

Decision rule: ≥2 of 3 must concur. Diverging verifiers trigger a Phase 5+ re-pass.

Why N=3 reasoning-diverse beats N=9 same-family: correlated errors collapse same-family panels — additional same-method verifiers have near-zero marginal value. "Nine Judges, Two Effective Votes" (arxiv 2605.29800). Same pattern used in /review and /security-audit.

---

Cross-File Consistency Check

After any edit pass across multiple output files:

• Grep all output files for the same claim — update every occurrence in the same pass
• Check that qualifications are appropriate at the scope where they appear (a portfolio-wide caveat may be misleading in a per-API section)

---

6 Phases

Phase 1 — Discovery

Goal: enumerate every API in the portfolio.

1. Fetch the vendor's developer API listing page — use ctx_fetch_and_index, not inline
2. List all APIs by service area
3. For each API record: name, service area, brief description, access URL
4. Surface any APIs that are partner-only, deprecated, or behind bundles — flag immediately

Output: flat API list with service area grouping.

---

Phase 2 — Inventory

Goal: per-API data table, primary-source verified.

For each API, record:

Field	Notes
Sandbox	Yes / No / Yes* (limited)
Auth	List all patterns — WSKey / OAuth 2.0 / HMAC / Open / proprietary
Operations	R / C / U / D — only what is explicitly documented
Protocol	REST/JSON / REST/XML / SOAP / legacy (NCIP, SRU/SRW, Z39.50) / Linked Data
Access requirement	Subscription tier, membership, partner-only
Notable limits	Rate limits, record caps, pagination limits, hard caps

For each capability claim: apply the Verification Gate. If primary source not found, mark as [unverified] — do not assert.

Call advisor() with the completed inventory before proceeding to Phase 3.

---

Phase 3 — Shortcomings analysis

Goal: identify and categorize gaps with severity ratings.

1. Group findings into themes: auth fragmentation, sandbox gaps, legacy protocols, functional gaps, access barriers, documentation quality
2. Separate technical shortcomings (engineering can fix) from commercial/structural shortcomings (business decision required — APIs may be technically separable even if sold as a bundle)
3. For each shortcoming: what the problem is → what the integrator impact is → severity (High / Medium / Low)
4. Numeric where possible: "11 of 21 APIs (52%)" not "many APIs"

Output: shortcomings document with severity ratings and summary table.

---

Phase 4 — Recommendations

Goal: prioritized fix list with effort and impact.

Priority tiers:

• P0 — Bugs / production blockers (data corruption, silent failures)
• P1 — High friction — major integration barriers
• P2 — Developer experience gaps
• P3 — Polish and maintenance

For each recommendation:

• Problem statement (reference the shortcoming)
• Specific fix (not "improve X" — name the endpoint, policy, or mechanism)
• Effort: Low / Medium / High
• Impact: Low / Medium / High / Critical
• Trade-off or failure mode — no universally-best options

Call advisor() with inventory + shortcomings + recommendations before proceeding to Phase 5.

---

Phase 5 — Executive summary

Goal: one-page brief for a technical audience unfamiliar with the full analysis.

Sections:

1. Bottom line (2–3 sentences)
2. Top-5 problems (the P0s and highest-impact P1s)
3. Score by service area: sandbox coverage, auth consistency, functional completeness, overall rating
4. Quick wins table: low-effort, high-impact items
5. Strategic gaps vs. peer ecosystems (what modern API platforms have that this one lacks)
6. Recommended investment order

Cross-file consistency: after drafting, grep all output files for every claim that appears in more than one file. Update all occurrences in the same pass.

---

Phase 6 — Options analysis (optional)

Goal: multi-option trade-off analysis for the most complex issues.

For each issue (typically the top 8–12 shortcomings):

• Problem definition
• 2–4 response options
• Trade-off table: effort / risk / impact / precedent (does another vendor do this?)
• Recommended option with named failure mode

Call advisor() with all output files after Phase 6. Fix every flagged issue. Call advisor() again if fixes are non-trivial. Do not declare done without final sign-off.

---

Output files

File	Contents
{prefix}-api-inventory.md	Primary data table — all APIs, all attributes
{prefix}-api-shortcomings.md	Shortcomings with severity ratings
{prefix}-api-recommendations.md	Prioritized fixes (P0–P3)
{prefix}-api-executive-summary.md	One-page brief
{prefix}-api-options-analysis.md	Trade-off analysis (if Phase 6 run)

Replace {prefix} with the vendor or portfolio name (e.g., oclc, aws, stripe).

---

Common traps

Trap	Defense
"No X exists" inferred from not finding it	Find the spec or listing that enumerates what IS present; use that as evidence
JS-rendered portals return empty bodies	Go to vendor's GitHub org for raw OpenAPI specs and source code
Prototype repo treated as current docs	Label as "per [year] prototype spec"; do not assert as current capability
Same claim fixed in one file but not others	Grep all files after every fix pass
Portfolio-wide caveat added to per-API section	Check that qualifications are accurate at the scope where they appear
Commercial packaging framed as technical constraint	Distinguish: "APIs are technically separable but sold as a bundle" vs. "APIs cannot be separated"
HMAC/auth scope overclaimed	Auth complexity often varies within a service area — record per-API, not per-area

See REFERENCE.md for the pre-sign-off checklist.