schema-coverage

# Schema Coverage (Terraform resource vs acceptance tests) ## Goal Given a Terraform resource, compare its **schema attributes/blocks** to what the acceptance tests **configure** and **assert**, then highlight opportunities to improve coverage. Report findings in this order: 1. **Attributes with no coverage** 2. **Attributes with poor coverage** ## Inputs (infer if not provided) - Resource name (e.g. `elasticstack_elasticsearch_security_api_key`) - Or the schema file / acceptance test file path - Or the Go package directory containing the resource and `*_acc_test.go` If the user has an acceptance test open, infer the resource under test from: - `resource.Test(...)` names like `resourceName := "elasticstack_..."` - `resource.ParallelTest(...)` steps referencing a single resource - config builder function names like `testAcc...Resource...` ## Workflow ### 1) Locate and parse the schema Find the resource schema definition and capture **all schema keys**: - Top-level attributes in `Schema: map[string]*schema.Schema{ ... }` - Nested block schemas in: - `Elem: &schema.Resource{ Schema: ... }` - lists/sets/maps of resources and objects - nested blocks referenced via helpers For each attribute/block, record: - **Path**: terraform attribute path (top-level `foo`, nested `block.0.bar`, etc.) - **Schema metadata**: `Required`/`Optional`/`Computed`, `ForceNew`, type (`TypeString`, `TypeList`, etc.), `MaxItems/MinItems` ### 2) Locate acceptance tests and collect attribute usage Scan the acceptance tests for two independent signals: - **Configured attributes**: attributes explicitly set in HCL test configs (including nested blocks). - **Asserted attributes**: attributes referenced in checks, including: - value assertions (e.g. `TestCheckResourceAttr`) - set-only assertions (e.g. `TestCheckResourceAttrSet`) - absence assertions (e.g. `TestCheckNoResourceAttr`) - collection assertions (e.g. type-set element checks, list length checks) Also detect **update coverage** by identifying multiple `resource.TestStep{ Config: ... }` steps for the same resource and whether attribute values change between steps. ### 3) Build a coverage matrix For each schema attribute path, compute: - **Configured?** (ever set in any test config) - **Asserted?** (ever referenced by any check) - **Assertion quality**: - **value-specific**: asserts the exact value (preferred) - **set-only**: only checks “is set” (weaker) - **absence**: checks unset / removed (good for optional fields) - **Value diversity**: count distinct values across steps/configs (e.g. `name="a"` vs `name="b"`) - **Optional-unset coverage**: optional field has a test step where it is deliberately omitted, plus an assertion that it is absent (or defaults appropriately) - **Empty-collection coverage**: collection has a step where it is empty (or omitted if optional), plus assertions validating the empty state - **Update coverage**: attribute value changes across steps, and a post-update assertion verifies the new value ### 4) Produce the report (strict ordering) Use the template below. ## Report template ```markdown ## Schema coverage report: <resource> ### Scope - **Schema**: <file(s) or function(s)> - **Acceptance tests**: <test file(s)> ### 1) Attributes with no coverage These schema attributes/blocks are not referenced in acceptance tests (neither configured nor asserted): - `<attr_path>`: <Required/Optional/Computed>, <type>. **Gap**: not configured, not asserted. ### 2) Attributes with poor coverage These attributes appear in tests but the coverage is weak: - `<attr_path>`: <schema flags/type> - **Observed**: <how it’s currently used (configured/asserted), example values> - **Gaps**: - <one or more of: set-only assertion, single value only, no unset coverage, no empty collection coverage, no update coverage> - **Suggested improvements**: - <concrete test step or assertion to add> ### Suggested next steps (smallest diffs first) 1. Add value-specific assertions for set-only checks 2. Add an “unset optional” step + `TestCheckNoResourceAttr` 3. Add an “empty collection” step + collection assertions 4. Add an “update” step changing the attribute + post-update assertions ``` ## Rules of thumb (for prioritization) - Prefer **value-specific assertions** over “is set”. - For **Optional** attributes, include at least one step where the attribute is **omitted**, and assert absence/default behavior. - For **collections** (list/set/map), add a case for **empty** (or omitted if optional), and assert the expected empty state. - For **Update** behavior, ensure there is at least one multi-step test where the attribute changes and the test asserts the new state. - For **Computed-only** attributes, it’s acceptable to use set-only assertions when exact values are not deterministic, but prefer deterministic assertions when possible. ## Notes / limitations - Some schemas are built via helpers; follow helper references until all attribute keys are accounted for. - Attribute paths in checks often use `block.0.attr` indexing; normalize consistently when matching to schema blocks. - Don’t mark an attribute as “covered” solely because it appears in raw HCL—prefer it being **asserted**. Treat “configured but never asserted” as poor coverage, not good coverage.