| name | tgc-fix-handwritten-resources-tests-skill |
| description | Fix integration tests for handwritten resources in TGC. Use when you need to fix integration tests for TGC. |
tgc-fix-handwritten-resources-tests-skill
When you need to fix integration tests for handwritten resources in TGC, use this skill.
When to Use This Skill
- Use this when fixing integration tests for handwritten resources in TGC.
- This is helpful when you need to address build and testing failures for any Terraform Google Conversion (TGC) resource.
How to Use It
General Rules for Fixing Tests
When troubleshooting and resolving test failures, adhere to these constraints:
- DON'T modify
mmv1/api/resource.go to hardcode or manually append ignored fields for tests. If there is a field path mapping mismatch (e.g., due to singular/plural name differences), rename the property name in the resource YAML configuration (e.g., Instance.yaml) and use api_name to map it to the API field name.
- DON'T modify shared provider templates/files in
mmv1/third_party/terraform. All changes must be localized to TGC Next code under mmv1/third_party/tgc_next/.
- DON'T modify legacy TGC code in
mmv1/third_party/cai2hcl or mmv1/third_party/tgc. All changes must be made to TGC Next code in mmv1/third_party/tgc_next/.
- DO add a comment for each fix in the YAML file or other files to explain the root cause and the solution.
Steps
- Find the error message
- Fix the tests in magic-modules by following the guidelines and playbook below:
File Naming Conventions for Handwritten Tests
When adding or fixing tests for a resource, ensure that the handwritten test files in mmv1/third_party/terraform/services/<product>/ follow the expected naming convention: resource_<product>_<resource_name>_test.go.
If the file name does not match (e.g., it uses a legacy name or has extra suffixes), the generator function addTestsFromHandwrittenTests will fail to find it and will log a message like no handwritten test file found for <resource>.
Solution:
Rename the file to match the expected convention so the generator can discover it and include its tests in the generated TGC test suite.
Example: Fixing Omitempty JSON Tag Issue with ForceSendFields
Failing Test: TestAccContainerCluster_auto_ipam_config_enabled_step3_roundtrip.tf
The test might fail during the evaluation of the terraform plan phase with an error indicating a missing argument inside a generated block, such as:
Error: Missing required argument
The argument "enabled" is required, but no definition was found in auto_ipam_config.
If a boolean or zero-value field (e.g., enabled = false) is silently omitted during the API export process, it can cause Terraform to fail with missing required fields (e.g., returning an empty block auto_ipam_config {} instead of auto_ipam_config { enabled = false }).
Solution:
To fix issues where zero-value boolean fields are omitted:
- tfplan2cai (Expander): Ensure that
ForceSendFields is explicitly specified in the expander function (e.g., expandAutoIpamConfig), so that the converted CAI asset in roundtrip.json includes the zero value (e.g., enabled: false). For example, add ForceSendFields: []string{"Enabled"} to ensure the 0-value is not dropped by the Go omitempty JSON struct tag.
After adding these, rebuild TGC and run the respective integration test to verify the fix.
Example: Fixing Terraform Validation Error with Unrecognized CAI Enum Values
TestAccContainerCluster_withLoggingConfig.tf
The test might fail during the evaluation of terraform plan phase with the following error:
expected pod_autoscaling.0.hpa_profile to be one of ["NONE" "PERFORMANCE"], got HPA_PROFILE_UNSPECIFIED
The underlying issue is that the CAI export contains a default/unspecified enumeration value (HPA_PROFILE_UNSPECIFIED) that Terraform's internal schema explicitly forbids.
Solution:
To fix issues where an unspecified enum fails Terraform validation:
- cai2hcl (Flattener): Handle this explicitly in the
cai2hcl.go flattener (e.g., flattenPodAutoscaling). When the underlying map value matches an unspecified enum value (or is an empty string), discard the block entirely by returning nil. This correctly implies the default absent state back into the HCL config block mapping.
Example: Emitting Default Enums for Expected CAI Payload Values
When user provides no terraform configuration or supplies a completely empty block for a struct that expects a CAI enum (like HpaProfile expecting HPA_PROFILE_UNSPECIFIED rather than nil), it may produce missing field differences in the asset verification checks.
Solution:
- tfplan2cai (Expander): Explicitly check if the configured value is
nil or empty. If so, return a new struct with the correct default CAI enum explicitly assigned (e.g. &container.PodAutoscaling{HpaProfile: "HPA_PROFILE_UNSPECIFIED"}) rather than directly returning nil. This forces the proper default to be populated into the CAI export payload generated by TGC.
Example: Fixing Missing Required Fields inside Nested Blocks by Emitting Default Configurations
TestAccContainerCluster_withCostManagementConfig.tf
The test might fail during the evaluation of terraform plan phase with the following error:
Error: Missing required argument "enabled" in cost_management_config
The underlying issue is that the CAI-to-HCL flattener was generating an empty block (e.g., cost_management_config {}) instead of properly defaulting the required fields or omitting the block. Since enabled is a required attribute within cost_management_config in the Terraform schema, generating an empty block triggers schema validation errors.
Solution:
To fix issues where a nested block requires a specific field value to pass validation when generated without explicit data:
- cai2hcl (Flattener): Handle this explicitly in the
cai2hcl.go flattener (e.g., flattenManagementConfig). Instead of returning an empty map when the CAI asset lacks the fields, explicitly default the required attributes to their default values (e.g., set enabled to false). This ensures the emitted HCL block meets schema requirements natively.
- tfplan2cai (Expander): Ensure the expander generates the exact equivalent CAI API empty object (e.g.
costManagementConfig: {}) instead of omitting the field entirely. The expander (e.g., expandCostManagementConfig) must return a pointer to an empty struct (&container.CostManagementConfig{}) rather than nil when the user configuration block is absent or disabled. This ensures structural equality in roundtrip.json tests.
Example: Fixing Missing Required Fields for Empty Strings Dropped by API
Failing Test: TestAccContainerCluster_withAuthenticatorGroupsConfig.tf
The test might fail during the evaluation of terraform plan phase with the following error:
Error: Missing required argument
The argument "security_group" is required, but no definition was found.
The underlying issue is that the user specified an empty string for a required field (e.g. security_group = "") in a nested block. During the CAI export, this empty string may be dropped by the omitempty JSON tag, causing the flattener to interpret the field as missing completely rather than as an empty string. This generates an empty block {} which fails schema validation for the required field.
Solution:
To fix issues where required fields are dropped because they are empty strings:
- cai2hcl (Flattener): Handle this explicitly in the
cai2hcl.go flattener (e.g., flattenAuthenticatorGroupsConfig). Check if the payload contains the field; if not, explicitly fall back to an empty string "" instead of returning a missing value map. This correctly satisfies the Required: true HCL schema.
Example: Resolving cty.Value Unmarshaling Errors for Missing Schema Attributes
Failing Test: TestAccContainerCluster_withAddons
The test might fail during the test framework's asset assertions with an error:
error when converting the export assets into export config: &errors.errorString{s:"error unmarshaling JSON as cty.Value: unsupported attribute \"slice_controller_config\""}
This error occurs when the CAI-to-HCL flattener correctly exports a nested field (e.g., slice_controller_config), but the associated exact Terraform schema block is missing from the downstream terraform-google-conversion generated codebase. When the framework attempts to decode the output JSON into a cty.Value to verify matches against the TF plan, it fails to recognize the exported attribute because the schema is stale or incomplete.
Solution:
To fix unsupported attribute unmarshaling errors for newly mapped CAI fields:
- Schema Update: Locate the schema definition in the appropriate handwritten or generated file (e.g.
mmv1/third_party/tgc_next/pkg/services/container/resource_container_cluster.go or mmv1/products/...) and ensure the new attribute block is declared structurally within its parent schema type definition map.
- Rebuild TGC Generation Correctly: Ensure you rebuild the codebase by providing the exact absolute path to the downstream repository so your target schema file gets cleanly overwritten. A relative path from the Magic Modules root will write the generated files to the wrong place since generation runs inside the
mmv1 directory!
make tgc OUTPUT_PATH="/path/to/downstream/terraform-google-conversion"
- Re-build the conversion binaries downstream (
make mod-clean && make build) and re-run the integration test.
Example: Resolving Missing Fields Due to Map Type Assertion with Nil Values
Failing Test: TestAccContainerCluster_withReleaseChannelEnabled.tf
The test might fail during the evaluation phase with the following error:
TestAccContainerCluster_withReleaseChannelEnabled_step3: missing fields in resource google_container_cluster.with_release_channel after cai2hcl conversion:
[release_channel.channel]
The underlying issue is that the GKE API considers UNSPECIFIED as the zero state for releaseChannel.channel and omits it entirely from the JSON response. When the field is omitted, the CAI asset contains an empty map {}. If the flattener checks if c, ok := v.(map[string]interface{}); ok && c["channel"] != "", the missing key evaluates to nil. In Go, interface{}(nil) != interface{}("") evaluates to true. This incorrectly maps a nil back into the HCL configuration, which eventually drops during map serialization, causing a missing field discrepancy.
Solution:
To fix missing fields errors caused by nil map values bypassing string interface comparisons:
- cai2hcl (Flattener): Update the
cai2hcl.go flattener (e.g., flattenReleaseChannel) to explicitly perform a string type assertion (if ch, ok := c["channel"].(string); ok && ch != "") before assignment. This forces missing or non-string configurations to properly fallback to the default HCL value (e.g., "UNSPECIFIED").
- tfplan2cai (Expander): If the upstream API returns an empty object (
{}) for the zero-state (like releaseChannel: {} when "UNSPECIFIED"), ensure the expander matches this behavior. Configure the expander (e.g., expandReleaseChannel) to return an empty struct pointer (&container.ReleaseChannel{}) when the zero-state is detected. Because internal fields like Channel use the omitempty JSON tag, the empty struct will correctly serialize into an empty JSON object ({}), ensuring the tfplan2cai roundtrip payload perfectly matches the upstream cai2hcl conversion output.
Example: Resolving "Value for unconfigurable attribute" Validation Errors
Failing Test: TestAccContainerCluster_deleteExclusionWindow.tf
The test might fail during the evaluation of the terraform plan phase against the generated output with an error indicating an attribute cannot be explicitly set:
Error: Value for unconfigurable attribute
Can't configure a value for "maintenance_policy.0.daily_maintenance_window.0.duration": its value will be decided automatically based on the result of applying this configuration.
The underlying issue is that the Terraform schema defines the attribute (e.g. duration) strictly as Computed: true (read-only output from the API). Because it is generated dynamically by the upstream API, Terraform forbids explicitly defining or configuring it within .tf blocks. However, during the cai2hcl conversion mapping process, the flattener unconditionally exported the duration value from the CAI asset into the generated struct, populating an unconfigurable field and triggering schema validation rejection.
Solution:
To fix validation errors caused by unconfigurable computed attributes being populated in HCL maps:
- cai2hcl (Flattener): Locate the flattener associated with the failing block (e.g.
flattenMaintenancePolicy). Find where the nested properties are assigned from the API map into the transformed JSON equivalent. Surgically omit mapping the offending Computed attribute (e.g. duration). By mapping only the configurable fields (e.g., start_time), Terraform gracefully allows the implicitly computed value to pass without complaining about manually configured read-only attributes.
Example: Resolving Shared Parameter Collisions in CAI Resource Identification
Failing Test: Multiple generated resources mapping to the same API endpoints
The TGC code generation loops through all known CAI endpoints to match a converted asset back to a single unique HCL block name. Previously, it searched for a single strictly unique IdentityParam string segment (e.g. "global" or "projects"). When multiple resources within a group (e.g. compute.googleapis.com/FirewallPolicy) shared these single unique segments (_region_network_firewall_policy has "projects", and _firewall_policy has "global"), the deterministic identifier fallback misclassified the asset resource type.
Solution:
To fix CAI mapping collisions between similar resources:
- terraform_tgc_next (Generator): The generator natively supports an array-based
IdentityParams slice. Instead of identifying a single unique segment, removeSharedElements strips common URL components across a product, leaving all remaining distinct parameters. The generator natively evaluates all segments concurrently (e.g. contains(projects) && contains(global)), resulting in perfect multi-parameter exact-path checking for highly nested collision-prone resources.
Example: Fixing Missing Fields Not Present in CAI Asset via YAML
Debugging Order of Operations (Handling Missing Fields)
When a field is reported as missing or failing in a handwritten resource's integration test, you must follow this strict logical order of operations rather than immediately marking it as ignored:
- Verify Resource Type: Confirm the resource is indeed a handwritten/custom resource (e.g., GKE Cluster or GKE NodePool) and not generated from a standard MM YAML.
- Verify Go-Level Conversion Support: Check the
tgc_next handwritten Go files (e.g., node_config.go) to see if the schema, flattener, and expander for the missing field are already implemented.
- If missing:
- Check if the field is only mapped/set in the standard downstream Terraform Provider when the field exists in the Terraform state or configuration (e.g., via checks like
d.GetOk("field")).
- If so: do NOT convert it in TGC. Only implement Go-level flatteners in TGC for fields that are unconditionally mapped/set from the live API response.
- Otherwise: Implement the conversion code in Go first. Do NOT skip this step; doing so would mask a missing feature with a false-positive ignore rule.
- Verify Post-Code Test Behavior: Rebuild and re-run the integration test.
- Apply Ignored Metadata: If the field is still reported as missing during validation, check if it's because of a permanent API-level omission in the CAI asset payload. Only after proving a CAI-level omission, add
is_missing_in_cai: true to the MM YAML (e.g., NodePool.yaml) to gracefully bypass validation.
Solution for API-Level CAI Omissions:
- Add the field to the resource's YAML file under the appropriate property block.
- Set
is_missing_in_cai: true.
This ensures that the validation framework gracefully ignores the CAI-level API omission during testing, while still allowing your Go-level conversion code (schema, flattener, and expander) to execute normally when properties are converted.
[!TIP]
GKE CAI Parity Warning: When dealing with nested configuration blocks shared between parent resources (like GKE Cluster) and standalone child resources (like GKE NodePool), do not assume their API CAI exporters share the same level of parity. For example, while GKE Clusters return crashLoopBackOff in CAI, the standalone GKE NodePool CAI asset (container.googleapis.com/NodePool) has a permanent omission and never returns crashLoopBackOff. Always verify target CAI asset types specifically.
Example: Resolving Missing Fields by Implementing Go-Level Flatteners
Failing Test: TestAccComputeInstance_schedulingTerminationTime
The test fails during the CAI-to-HCL validation check because of a missing field:
TestAccComputeInstance_schedulingTerminationTime_step1: Failed after 5 attempts. First real error: retryable: missing fields: [scheduling.termination_time]
Analysis:
- Inspect the raw CAI asset (
TestAccComputeInstance_schedulingTerminationTime_step1.json). If the property (terminationTime) is present in the CAI payload, it is NOT missing from CAI. Thus, is_missing_in_cai: true must NOT be used.
- For handwritten resources (e.g.,
google_compute_instance), look at the custom conversion files (e.g., compute_instance_cai2hcl.go). If the property flattener is missing the mapping code, the property won't be written to the exported HCL.
Solution:
- cai2hcl (Flattener): Locate the flattener function mapping the nested block (e.g.,
flattenSchedulingTgcNext). Add Go code to extract the value from the CAI response map and write it to the HCL schema map:
if tt, ok := resp["terminationTime"].(string); ok && tt != "" {
schedulingMap["termination_time"] = tt
}
