api-coverage

name	api-coverage
description	Generate AWS API coverage reports comparing winterbaume implementations against official AWS API models, plus Terraform converter coverage at the resource-type and per-attribute levels.
user_invocable	true

API Coverage Report Generator

Three complementary reports live in this skill, each answering a different "how much do we cover?" question:

1. AWS API operation coverage — generate_coverage.py writes .agents/docs/API_COVERAGE.md. Compares winterbaume's implemented operations against the official Smithy models under vendor/api-models-aws/, with moto / floci / kumo / fakecloud numbers alongside.
2. Terraform resource-type coverage — generate_terraform_resource_coverage.py writes .agents/docs/TERRAFORM_RESOURCE_COVERAGE.md. For each winterbaume service, lists how many Terraform aws_* resource types the converter handles vs how many the AWS provider schema declares for that service, plus the missing types per service.
3. Terraform attribute coverage — generate_terraform_converter_coverage.py writes .agents/docs/TERRAFORM_CONVERTER_COVERAGE.md. For each handled resource type, measures inject% and extract% as the fraction of the resource's TF schema attributes the converter touches in each direction.

Pick the right report for the question:

• "Do we know about all the IAM resources Terraform supports?" → resource coverage.
• "Does our aws_iam_role converter handle all the role's fields?" → attribute coverage.
• "Does winterbaume implement the AWS IAM API operations moto does?" → API operation coverage.

Instructions

Run the report you need:

# 1. AWS API operation coverage
python3 .agents/skills/api-coverage/scripts/generate_coverage.py

# 2. Per-service Terraform resource-type coverage
python3 .agents/skills/api-coverage/scripts/generate_terraform_resource_coverage.py

# 3. Per-resource Terraform attribute coverage
python3 .agents/skills/api-coverage/scripts/generate_terraform_converter_coverage.py

Each writes to .agents/docs/<NAME>.md by default; use --output PATH to redirect.

Prerequisites

vendor/api-models-aws must be present as a submodule:

git submodule update --init vendor/api-models-aws

moto, floci, kumo, and fakecloud do not require submodules. When vendor/moto, vendor/floci, vendor/kumo, or vendor/fakecloud are absent the script fetches the necessary source files directly from GitHub and caches the parsed results under .agents/skills/api-coverage/cache/ for 7 days. fakecloud is fetched as a single repo tarball ( far cheaper than fetching its ~880 crate source files individually ) and parsed in memory. A GITHUB_TOKEN environment variable is optional but raises the unauthenticated rate limit (60 req/hr) to 5 000 req/hr.

To force a cache refresh (e.g. after a new moto/floci/kumo/fakecloud release):

python3 .agents/skills/api-coverage/scripts/generate_coverage.py --refresh-remote

If you prefer local clones (faster, works offline):

git submodule add https://github.com/getmoto/moto        vendor/moto
git submodule add https://github.com/floci-io/floci      vendor/floci
git submodule add https://github.com/sivchari/kumo       vendor/kumo
git submodule add https://github.com/faiscadev/fakecloud vendor/fakecloud

The script prefers a local clone over the cache or a remote fetch when the vendor directory is present.

What the script does

1. Parses AWS API models from vendor/api-models-aws/models/<model-dir>/service/<version>/<name>.json (Smithy JSON format, extracting shapes with "type": "operation")
2. Parses winterbaume implementations from crates/winterbaume-*/src/handlers.rs, detecting operations across all dispatch patterns:
   • awsQuery: match action { "CreateBucket" => ... }
   • awsJson: "ServicePrefix.Operation" => ... in X-Amz-Target matching
   • REST: comments like // POST /path - OperationName and handler method names like handle_create_email_identity
3. Parses moto coverage from moto's IMPLEMENTATION_COVERAGE.md (fetched from GitHub and cached when vendor/moto is absent), mapping snake_case operation names back to PascalCase Smithy names
4. Parses floci coverage from per-service Markdown docs (docs/services/*.md), reading Category | Operations tables where operations are comma- or middle-dot-separated PascalCase names. Source: vendor/floci if present, otherwise fetched from GitHub and cached for 7 days.
5. Parses kumo coverage from Go source files (internal/service/<name>/*.go): string literals in Actions() []string return values (Query/JSON services) and handle<Op> receiver method names (REST services). Source: vendor/kumo if present, otherwise fetched from GitHub and cached for 7 days.
6. Parses fakecloud coverage from Rust service crates (crates/fakecloud-*/src/**/*.rs). Each service impls the AwsService trait; the parser pairs each fn service_name(&self) -> &str ( the AWS endpoint prefix ) with the fn supported_actions(&self) -> &[&str] that follows it in the same file, resolving named const <NAME>_ACTIONS: &[&str] slices as needed. Keying on service_name rather than the crate name keeps services that share a crate in their own columns ( e.g. iam vs sts, cognito-idp vs cognito-identity ). Source: vendor/fakecloud if present, otherwise fetched from GitHub as a single tarball and cached for 7 days.
7. Parses integration-test coverage from crates/winterbaume-*/tests/integration_test.rs, detecting AWS SDK client method calls and mapping them back to implemented Smithy operations
8. Parses Terraform E2E coverage from crates/winterbaume-e2e-tests/tests/terraform/*.rs, counting success-oriented terraform tests and distinct terraform resource types per service
9. Generates the report with a side-by-side overview table (winterbaume / moto / floci / kumo / fakecloud), integration-test and Terraform E2E summary tables, unimplemented services list, and detailed per-service checklists with W[x] M[x] F[x] K[x] C[x] markers ( C = fakecloud )

Maintaining the service mappings

Five mappings are defined in .agents/skills/api-coverage/scripts/generate_coverage.py:

• CRATE_TO_MODEL: Maps winterbaume crate directory names to api-models-aws model directory names. Add an entry when creating a new service crate.
• MODEL_TO_MOTO: Maps api-models-aws model directory names to moto service names as they appear in moto's IMPLEMENTATION_COVERAGE.md (only where they differ, e.g., "config-service": "config").
• FLOCI_STEM_TO_MODEL: Maps floci service document filename stems to api-models-aws model directory names (only where they differ, e.g., "cognito": "cognito-identity-provider"). Add an entry when floci adds a service whose doc filename does not match the model dir name.
• KUMO_DIR_TO_MODEL: Maps kumo internal/service/<name> directory names to api-models-aws model directory names (only where they differ, e.g., "elbv2": "elastic-load-balancing-v2"). Add an entry when kumo adds a service whose directory name does not match the model dir name.
• FAKECLOUD_SERVICE_TO_MODEL: Maps fakecloud's service_name values ( the AWS endpoint prefix each service impl returns, e.g. "monitoring", "events", "states" ) to api-models-aws model directory names (only where they differ, e.g., "states": "sfn", "logs": "cloudwatch-logs"). Add an entry when fakecloud adds a service whose endpoint prefix does not match the model dir name. FAKECLOUD_SKIP_CRATES lists framework crates ( fakecloud-core, fakecloud-conformance, ... ) whose AwsService impls are empty test doubles and must be ignored.

Output format

The report is written as Markdown with:

• Overview table: service, model, winterbaume count, moto count, floci count, kumo count, fakecloud count, total, wb%, moto%, floci%, kumo%, fakecloud%
• Overall aggregate stats for winterbaume, moto, floci, kumo, and fakecloud
• Integration-test summary lines plus a per-service integration-test table
• Terraform E2E summary lines plus a per-service Terraform E2E table
• List of AWS services not yet implemented
• Detailed per-service sections with W[x] M[x] F[x] K[x] C[x] side-by-side checklists for every operation ( C = fakecloud ), plus integration-test and Terraform E2E notes

Terraform resource-type coverage

generate_terraform_resource_coverage.py answers: "for each winterbaume service, how many of the AWS Terraform provider's resource types do we actually handle?"

It reads the canonical converter manifest at crates/winterbaume-terraform/specs/<service>.toml (the same specs the tf-converter-codegen tool consumes), extracts each [[resource]] type declaration, and compares the set against the AWS provider schema obtained via terraform providers schema -json. The schema is cached at .agents-workspace/tmp/tf-schema/aws_provider_schema.json (auto-generated on first run; reused thereafter).

For each service:

• The "service prefix" is the longest common prefix of the handled resource types, capped at two underscore-separated segments after aws_ so a 1-handled service doesn't produce an over-narrow prefix.
• Services with heterogeneous resource names ( ec2 mixes aws_vpc, aws_subnet, aws_security_group, etc.; rds mixes aws_db_* and aws_rds_*; elbv2 mixes aws_lb* and aws_alb* ) use a manual override list defined in PREFIX_OVERRIDES at the top of the script. Add a service to that table if its resources don't share a clean common prefix.
• Coverage % = handled / candidates_in_schema, where candidates are every schema resource matching the service's prefix(es).

Output sections:

1. A summary header with totals ( schema resources, classified, handled, missing ).
2. A table sorted by absolute missing-count desc.
3. Per-service "missing resources" list, suitable for triage and converter expansion.
4. ( If applicable ) a list of resources declared in specs but absent from the schema — typically typos or resources removed in newer provider versions.

Terraform attribute coverage

generate_terraform_converter_coverage.py answers: "for each handled resource type, how many of its TF schema attributes does the converter actually touch?"

It greps each migrated converter and the TfModel spec for field-level references, then computes inject% and extract% per resource. The terraform-converter skill uses these numbers to gate "excellent" ( inject ≥ 60%, extract ≥ 50% ).

Important Notes

• winterbaume-ses implements SES v2 (sesv2 model), not SES v1
• Only operations that exist in the official model are counted; spurious handler methods are filtered out
• Handler methods preceded by a // STUB[category]: reason comment (see .agents/docs/QUALITY_GATE.md "Stub Handler Discipline") are excluded from the winterbaume count and surfaced in a separate Stubs column, because they return empty/default responses without real logic. They still satisfy API routing but are not functionally implemented. To restore a stub to the count, remove the STUB[...] comment once real logic is added.
• The script also performs heuristic stub detection for unannotated drift: a handler whose body is short, contains a default/empty response indicator (::default(), ..Default::default(), Vec::new(), success_empty, ...), and contains no state-access indicator (self.backend, with_state, ?;, request.body, ...) is also classified as a stub even if it is missing the canonical // STUB[<category>]: ... annotation. The annotated-vs-heuristic split is not reported separately in the table; both flavours appear in the Stubs column. If a handler is misclassified, the canonical fix is to either annotate it explicitly or to make its real logic obvious to the heuristic.
• Integration-test coverage is reported against implemented operations only. It is derived from SDK method calls found in tests/integration_test.rs, so it is a useful approximation rather than a proof that every behavioural path is covered.
• Terraform E2E coverage is reported separately from API operation coverage. One Terraform resource can exercise multiple API operations, so the E2E metrics are a complementary validation signal rather than an operation-count substitute.