| name | syllago-provider-audit |
| description | Use when auditing AI coding tool providers OR researching provider capabilities OR updating provider docs OR checking what changed in a provider. Runs structured research against official docs and produces standardized reports for syllago's provider accuracy. |
Provider Audit
Structured research workflow for auditing AI coding tool providers supported by syllago. Takes a provider slug, researches official docs, and produces/updates standardized 4-file reports that drive syllago's converter accuracy.
Accuracy Standard
Every discrepancy matters. No exceptions.
Syllago's converter output is only as good as its provider knowledge. A wrong tool name, a stale directory path, a renamed config field — these produce silent failures for users. Content converts "successfully" but doesn't work in the target provider. The user blames their setup, not our data.
This means:
- Fix every finding, no matter how small. A renamed URL path is as important as a missing tool.
- Never dismiss a discrepancy as "minor". If the docs say one thing and our reports say another, our reports are wrong.
- Always update reports AND create beads for anything that affects the converter, toolmap, or provider definition. Don't just note it — fix it or track it.
- After a diff, the default action is "both" (update reports + create beads). Only skip if the user explicitly says otherwise.
Usage
/syllago-provider-audit <provider-slug> # Full audit of one provider
/syllago-provider-audit diff <provider-slug> # Diff against existing reports
/syllago-provider-audit --all # Audit all 12 providers
Supported Providers
amp, claude-code, gemini-cli, cursor, windsurf, codex, copilot-cli, cline, roo-code, opencode, kiro, zed
Report Format
Each provider gets 4 research files in the syllago repo at docs/providers/<slug>/:
| File | Covers |
|---|
tools.md | Built-in tools: exact names, purposes, parameters, cross-provider equivalents |
content-types.md | Rules, skills, agents, hooks, MCP, commands: file formats, directory structure, config schema, required fields |
hooks.md | Hook events, config format, matcher syntax, execution model, structured output support |
skills-agents.md | Skill/agent definition format, model selection, tool permissions, invocation patterns |
Report Metadata
Every report file MUST start with a metadata block:
<!-- provider-audit-meta
provider: zed
provider_version: "0.178.x"
report_format: 1
researched: 2026-03-21
researcher: claude-opus-4.6
changelog_checked: https://zed.dev/releases
-->
All fields are required. provider_version is the version you researched against. changelog_checked is the URL you verified for recent releases. See docs/providers/REPORT-FORMAT.md for full spec.
Changelog Sources (MUST CHECK)
Every audit and diff MUST check the provider's changelog for releases newer than the last research date. This is not optional.
Source Attribution
Every finding must be tagged:
[Official] — from provider's official docs or repo[Community] — blog posts, tutorials, forums[Inferred] — derived from source code or config examples[Unverified] — stated but not confirmed
Include URLs for all sources. Prefer specific pages, not docs root.
Official Documentation Sources
Claude Code
Gemini CLI
Cursor
Windsurf
Codex
Copilot CLI
Cline
Roo Code
OpenCode
Kiro
Zed
Amp
- Manual: https://ampcode.com/manual (single-page reference — sub-pages are auth-gated)
- Developer: Sourcegraph (uses Claude models but is NOT an Anthropic product)
- Note: Sub-pages under
/manual/ require authentication, but the main /manual page contains the full reference
Audit Workflow
Step 1: Validate the provider slug
Step 2: Check changelog FIRST
Fetch the provider's changelog URL (from the table above). Identify:
- The latest release version/date
- What changed since our last research (check metadata block in existing reports for
researched date)
- Any new features, renamed tools, moved paths, deprecated features
This gives you a targeted list of what to verify, instead of re-researching everything blind.
Step 3: Load existing reports
Check docs/providers/<slug>/. If reports exist, read them and their metadata blocks. Note the provider_version and researched date.
Step 4: Research each area
For each of the 4 report files, fetch official docs and extract findings. Use parallel subagents (one per report area) for speed. Prioritize areas flagged by changelog changes.
Research strategy:
- Official docs first — fetch key pages from the URLs above
- GitHub source code — for open-source providers, check for undocumented features
- Changelog/releases — search for recent changes
- Community sources — only to fill gaps in official docs
- Prefer Readability MCP for fetching; fall back to WebFetch for JS-heavy sites (Cursor, Windsurf, Cline)
Step 5: Write reports
Write all 4 files to docs/providers/<slug>/. Each file MUST start with the metadata block (see Report Metadata section). Set provider_version to the latest release found in Step 2, researched to today, and changelog_checked to the changelog URL.
Step 6: Summarize
Report: provider version audited, tools found, content types supported, hook events, key differences from previous reports, recommended syllago changes (toolmap updates, provider definition fixes, new beads needed).
Diff Workflow
Step 1: Load existing reports from docs/providers/<slug>/
Step 2: Research current state (same strategy as audit)
Step 3: Produce diff report to stdout:
## New
- [tools.md] New tool: `codebase_search` [Official](url)
## Changed
- [content-types.md] MCP path changed to mcp-config.json [Official](url)
## Removed
- [skills-agents.md] Legacy JSON agent format deprecated [Official](url)
## Recommended Actions
- Update toolmap.go: add codebase_search
- Create bead for MCP path fix
Step 4: Act on every finding
Default: update reports AND create beads. Don't ask — just do both unless the user says otherwise.
For each finding in the diff:
| Finding type | Action |
|---|
| New tool | Update tools.md + check if toolmap.go needs a mapping |
| Changed path/filename | Update content-types.md + check provider definition (InstallDir, DiscoveryPaths) |
| Changed config field | Update content-types.md + check converter (canonicalize/render) |
| Changed URL | Update all reports referencing old URL |
| New content type | Update content-types.md + create bead for provider expansion |
| Removed/deprecated feature | Update relevant report + check if converter handles gracefully |
| Tool name discrepancy | Update tools.md + fix toolmap.go entry |
After applying all changes:
- Update the report files in
docs/providers/<slug>/
- Create beads for any converter/toolmap/provider definition changes
- Run
make build && make test if code was changed
- Summarize what was updated and what beads were created
Gotchas
- JS-heavy doc sites: Cursor, Windsurf, and Cline docs are React SPAs — use WebFetch not Readability MCP or you get empty content
- DuckDuckGo rate limiting: After ~20 searches, fall back to WebSearch
- GitHub raw content: For open-source providers, fetch from
raw.githubusercontent.com — much cleaner than rendered pages
- Kiro dual format: Agents support both JSON and markdown. Markdown is primary as of 2026-03.
- Source attribution is critical: Without it, future audits can't distinguish verified facts from assumptions
- Check git dates: Before diffing, check
git log -1 --format=%ai -- docs/providers/<slug>/ to know how stale the reports are
- Never dismiss "minor" findings: A renamed URL, a slightly different tool name, a moved config path — these all cause silent converter failures. Every discrepancy gets fixed or tracked. The diff workflow defaults to "update + create beads" for this reason.
