test-monitor-workflow

# `test-monitor-workflow` This document guides the agent through the task of monitoring nightly acceptance test runs. ## Overview The `test-monitor` workflow provides a non-destructive monitoring lifecycle that tracks nightly acceptance test execution, identifies persistent failures across a 7-day window, correlates failing tests with open GitHub issues, retrieves debug logs, and outputs a comprehensive Markdown monitoring report. ## Available Skills Used in This Workflow * **`fetch-nightly-results`** (`.agents/skills/fetch-nightly-results/SKILL.md`): Downloads nightly test status JSON files for Beta and GA providers from GCS (`gs://nightly-test-data/test-metadata/`). * **`automate-test-triage`** (`.agents/skills/automate-test-triage/SKILL.md`): Triages 7-day failure trends, matches open GitHub `test-failure` issues, sanitizes dynamic error tokens for provider comparison, and outputs reports. ## Execution Steps ### 1. Fetch Test Results * Execute the `fetch-nightly-results` skill to retrieve test status JSON files for both GA and Beta providers from Google Cloud Storage (`gs://nightly-test-data/test-metadata/`) for the latest available run and past 7 days using `gcloud storage cat` (if GCS authentication or permission fails, verify `gcloud auth login` and `roles/storage.objectViewer` access). ### 2. Triage & Aggregate Failure Trends * Execute the `automate-test-triage` skill by running `python3 .agents/skills/automate-test-triage/scripts/triage.py` from the root workspace (or pass `--date YYYY-MM-DD` to specify a target end date for the 7-day window). * Group the target date's latest run failures by error signature and flag High-Impact errors based on: 1. 🚨 **Critical Severity**: Provider panic/crash (`panic:`, `runtime error:`, `SIGSEGV`) or API enablement errors in CI test environment projects, prioritized at the top of Section 1 regardless of test count. 2. ⚠️ **High Volume**: Error signatures affecting $\ge 3$ tests in the latest run. * Separate non-actionable errors requiring human intervention (Quota / Rate Limit / Stockout, Internal Error / Error Code 13, Tenant Project Creation) into Section 2. * Filter for persistent actionable test failures (failing in latest run AND $\ge 4$ out of the past 7 days). ### 3. Cross-Reference GitHub Issues * Fetch open issues labelled `test-failure` from `hashicorp/terraform-provider-google` using `gh issue list`. * Match test names against issue titles to correlate persistent failures with existing open tracking tickets. ### 4. Categorize Root Causes & Analyze Errors * Extract `error_message` and `log_link` directly from `test-metadata` entries. * Analyze API error payloads and debug logs to categorize the failure domain (e.g., Provider Panic, Quota Exceeded, API Permission / IAM, Model Availability, State Mismatch, Flakiness). ### 5. Generate Monitoring Report * Save the final Markdown report to `tmp/test-status/test-report-<date>.md` (e.g., `tmp/test-status/test-report-2026-07-28.md`). * Present the executive summary and 4 report sections (where every section table includes clickable GCS debug log links `[Log](url)`, expanded 500-character error summaries, and links to Section 4) to the user: Section 1 (High-Impact Actionable Errors: Panics & API Enablement in Test Env, High Volume), Section 2 (Test Failures Requiring Human Action), Section 3 (Persistent Actionable Failures Grouped by Error Signature), and Section 4 (Detailed Test Failures Grouped by Service Package, showing all latest-run failures with failed out of total tests, percentages, and a Human Action Required? column). --- ## Boundaries & Guardrails * **Monitoring Only:** Do NOT modify source code or template files in `magic-modules`. * **No Code Fixes / PRs:** Do NOT create fix branches or attempt to open pull requests.