release-workflow

# Konflux Release Flow Specification ## Overview This document defines the complete workflow for OpenShift z-stream release orchestration under the Konflux release platform. All releases from 4.12 to 4.20 follow this flow. **Target Audience:** AI agents (Claude Code) and human release engineers **Execution Method:** All `oar` commands are executed via the MCP (Model Context Protocol) server, NOT local CLI. The MCP server exposes OAR commands as structured tools with proper input validation and output parsing. ## Architecture Components ### StateBox (M2 - Primary Source of Truth) - **Primary source of truth** for AI workflow resumption and task execution state - Persists in GitHub: `_releases/{y-stream}/statebox/{release}.yaml` - Stores complete release context: - Metadata (advisories, builds, Jira ticket, release date, shipment MR) - Task execution history with AI-readable results (sensitive data masked) - Blocking issues with resolution tracking - Enables workflow resumption across multiple AI sessions (hours/days/weeks apart) - Supports concurrent access via SHA-based optimistic locking - Automatically updated by all OAR CLI commands (transparent to AI) - **AI MUST retrieve StateBox state at start of each session**: `oar_get_release_status(release)` ### Google Sheets (Legacy - Human Interface) - Still updated for backwards compatibility and human visibility - Each OAR command updates its corresponding task status directly - Task status: "Not Started" / "In Progress" / "Pass" / "Fail" - Overall status: "Green" / "Red" - **Used as fallback** if StateBox doesn't exist (abnormal situation) - **Note**: AI should always check StateBox first, Google Sheets second ### MCP Server - Exposes 27 OAR tools for command execution - Handles authentication, environment setup, error handling - Returns stdout/stderr for status detection ### Test Result Files - Stored in GitHub: `_releases/ocp-test-result-{build}-amd64.json` - Key attributes: - `aggregated: true/false` - All test jobs completed and results collected - `accepted: true/false` - BO3 verification passed ## Task Graph ``` create-test-report ↓ take-ownership ↓ check-cve-tracker-bug (always passes, notifies ART) ↓ check-rhcos-security-alerts (Konflux only - checks blocking security alerts) ↓ ├─→ push-to-cdn-staging (async - runs independently in parallel) └─→ [WAIT FOR BUILD PROMOTION - check API until phase == "Accepted"] ↓ ├─→ image-consistency-check (async - triggered immediately after promotion) ├─→ stage-testing (async - triggered immediately after promotion) └─→ [WAIT FOR TEST RESULT FILE - check GitHub until file exists] ↓ [WAIT FOR AGGREGATION - check until aggregated == true] ↓ analyze-promoted-build (conditionally - only if accepted == false) ↓ [GATE CHECK - promoted build must be acceptable] ↓ [WAIT FOR ALL 3 ASYNC TASKS TO COMPLETE] (push-to-cdn-staging, image-consistency-check, stage-testing) ↓ image-signed-check ↓ change-advisory-status (final approval) [PARALLEL TRACK - can start immediately] analyze-candidate-build (conditionally - only if accepted == false) ``` **Key Characteristics:** - **Sequential:** Most tasks run one after another - **Parallel Execution:** - `analyze-candidate-build` runs independently (tests already completed when flow starts) - `push-to-cdn-staging` starts immediately after check-rhcos-security-alerts (runs while waiting for build promotion) - **ENHANCED:** 2 async tasks (image-consistency-check, stage-testing) triggered immediately after build promotion is detected, running in parallel with test result analysis - **Build Promotion Checkpoint:** Critical decision point - once detected, async tasks trigger immediately - **Test Result Checkpoints:** Must wait for file existence and aggregation (runs in parallel with async tasks) - **Gate Check:** Promoted build must have acceptable test results before proceeding to final approval - **Final Sync Point:** image-signed-check waits for BOTH: 1. All 3 async tasks complete (push-to-cdn-staging, image-consistency-check, stage-testing) 2. Gate check passes (promoted build analysis acceptable) - **Default Architecture:** amd64 (x86_64) unless specified otherwise ## Build Promotion Checkpoint ### Build States **Candidate Build:** - Nightly build initially selected by ART - Format: `4.20.0-0.nightly-2025-01-28-123456` - Retrieved from: `oar_get_release_metadata(release)` → `candidate_builds.x86_64` - **Test status:** Tests already completed when release flow starts **Promoted Build:** - Stable build after ART promotion - Format: `X.Y.Z` (z-stream version, e.g., 4.20.1) - Checked via Release Controller API - **Test status:** Tests triggered after promotion, must wait for completion ### Promotion Detection **API Endpoint:** ``` https://amd64.ocp.releases.ci.openshift.org/api/v1/releasestream/{major_version}-stable/release/{release} # Where {major_version} is extracted from {release} (e.g., "4" from "4.20.1", "5" from "5.0.1") ``` **Success Criteria:** ```json { "phase": "Accepted" } ``` **MCP Execution:** Not a direct OAR command - AI must fetch URL and parse JSON response. **When to Check:** - User invokes `/release:drive {release}` command - AI checks current promotion status - If not yet promoted (`phase != "Accepted"`), AI reports status to user - User should re-invoke `/release:drive` periodically until promotion completes - Typical promotion time: 6-24 hours ## Test Result Evaluation ### Candidate Build Analysis **When to run:** Can start immediately when release flow begins (tests already completed) **Execution Flow:** ``` 1. Retrieve candidate build from oar_get_release_metadata(release).candidate_builds.x86_64 2. Fetch test result file from GitHub: _releases/ocp-test-result-{candidate_build}-amd64.json 3. Check attributes: IF aggregated == false: Report to user: "Candidate build tests still aggregating, check again later" (Should rarely happen - tests complete before flow starts) IF aggregated == true AND accepted == true: Mark analyze-candidate-build as "Pass" (all tests passed) No further action needed IF aggregated == true AND accepted == false: Trigger /ci:analyze-build-test-results {candidate_build} Parse AI recommendation: - ACCEPT: Mark task "Pass" (failures are waivable) - REJECT: Mark task "Fail" (blocking issues found) ``` **Slash Command:** ```bash /ci:analyze-build-test-results {candidate_build} ``` The slash command provides: - BO3 (Best of 3) verification status for blocking jobs - Acceptance recommendation (ACCEPT/REJECT) - Root cause analysis for failures - Waiver guidance ### Promoted Build Analysis **When to run:** After build promotion detected (`phase == "Accepted"`) **Execution Flow:** ``` 1. [CHECKPOINT 1] Check for test result file existence File: _releases/ocp-test-result-{release}-amd64.json When user invokes /release:drive: IF file does not exist: Report to user: "Test result file not yet created, check again in 5-10 minutes" Typical wait time: 10-120 minutes after promotion ELSE: Proceed to Checkpoint 2 2. [CHECKPOINT 2] Check for test aggregation Read file and check: aggregated == true When user invokes /release:drive: IF aggregated == false: Report to user: "Tests still running/aggregating, check again in 5-10 minutes" Typical wait time: 10-30 minutes after file creation ELSE: Proceed to Checkpoint 3 3. [CHECKPOINT 3] Check acceptance status IF accepted == true: Mark analyze-promoted-build as "Pass" (all tests passed) Proceed to gate check IF accepted == false: Trigger /ci:analyze-build-test-results {release} Parse AI recommendation: - ACCEPT: Mark task "Pass" (failures are waivable) - REJECT: Mark task "Fail" (blocking issues found) ``` **Slash Command:** ```bash /ci:analyze-build-test-results {release} ``` ### Gate Check Logic **ENHANCED LOGIC (Early Async Task Triggering):** The gate check has been optimized to trigger async tasks as soon as the stable build is promoted, without waiting for blocking tests analysis to complete. **Condition to Trigger Async Tasks:** ``` Build promotion detected (phase == "Accepted") ``` **Rationale:** Once the stable build is accepted by ART and promoted to the release stream, we can immediately start parallel async tasks (image-consistency-check, stage-testing) to save time. The blocking tests analysis happens independently and doesn't need to gate these operations. **Condition to PROCEED to Final Approval:** ``` promoted_build_analysis == "Pass" (either accepted == true OR AI recommendation == ACCEPT) AND all 3 async tasks complete successfully ``` **Note:** Candidate build analysis runs independently and doesn't block the gate check. It's informational for context. **If promoted build test analysis FAILS:** 1. Update overall status to "Red" 2. Mark analyze-promoted-build task as "Fail" 3. Notify owner via Slack with failure details from test analysis 4. STOP pipeline - manual intervention required 5. **Async tasks may still be running** - they will complete but won't proceed to final approval **If promoted build test analysis PASSES:** 1. Mark analyze-promoted-build task as "Pass" 2. **Async tasks already triggered** - wait for completion 3. Continue to final approval when all tasks complete ## Task Definitions ### 1. create-test-report **Purpose:** Initialize Google Sheets test report and ConfigStore entry **MCP Tool:** `oar_create_test_report(release)` **Input:** - `release`: Z-stream version (e.g., `4.20.1`) **Success Detection:** ``` stdout contains: "task [Create test report] status is changed to [Pass]" OR exiting report url ``` **Failure Detection:** ``` stdout contains: "task [Create test report] status is changed to [Fail]" ``` **Expected Duration:** 5 mins **Next Action:** Proceed to take-ownership --- ### 2. take-ownership **Purpose:** Assign release ownership to QE team member **MCP Tool:** `oar_take_ownership(release, email)` **Input:** - `release`: Z-stream version - `email`: Owner email (e.g., `user@redhat.com`) **AI Decision Logic:** - If user provided email in `/release:drive` command: Use that email - Otherwise: Query `oar_get_release_metadata(release)` for current owner - If no owner: Prompt user for email **Success Detection:** ``` stdout contains: "task [Take ownership] status is changed to [Pass]" ``` **Expected Duration:** 10 seconds **Next Action:** Proceed to check-cve-tracker-bug --- ### 3. check-cve-tracker-bug **Purpose:** Verify CVE tracker bug coverage for release **MCP Tool:** `oar_check_cve_tracker_bug(release)` **Input:** - `release`: Z-stream version **Behavior:** - Check if there is any missed CVE tracker bugs - Sends missed trackers to ART via Slack - Updates test report with all missed trackers - **ALWAYS PASSES** - does not block pipeline **Success Detection:** ``` stdout contains: "task [Check CVE tracker bugs] status is changed to [Pass]" ``` **Expected Duration:** 1 minute **Next Action:** Proceed to check-rhcos-security-alerts --- ### 4. check-rhcos-security-alerts **Purpose:** Check for blocking security alerts on RHCOS advisory (Konflux flow only) **When to run:** Konflux release flow only (releases with shipment_mr in metadata) **Prerequisites:** check-cve-tracker-bug completed **Implementation:** Uses curl with Kerberos authentication (no existing MCP tool) **Execution Steps:** **Step 1: Verify Kerberos ticket exists** ```bash klist ``` If no ticket or ticket expired: - Report to user: "No valid Kerberos ticket found. Please run: kinit $kid@$domain" - STOP task execution **Step 2: Get RHCOS advisory ID** ```python metadata = oar_get_release_metadata(release) rhcos_advisory_id = metadata.advisories.rhcos ``` **Step 3: Fetch security alerts from Errata Tool** ```bash curl -s -u : --negotiate 'https://errata.devel.redhat.com/api/v1/erratum/{rhcos_advisory_id}/security_alerts' ``` **Step 4: Parse response and check for blocking alerts** ```python response = json.loads(curl_output) # Filter blocking alerts from the alerts array blocking_alerts = [alert for alert in response.alerts.alerts if alert.blocking == true] IF len(blocking_alerts) > 0: Report to user with alert details and ask to email secalert@redhat.com # Continue pipeline - this is not a hard blocker, but requires follow-up ELSE: Report to user: "No blocking security alerts found on RHCOS advisory" ``` **Success Detection:** ``` Task always passes - this is an informational check Blocking alerts require manual follow-up but don't stop the pipeline ``` **Expected Duration:** 10 seconds **Errata Tool API Response Format:** ```json { "alerts": { "alerts": [ { "name": "erratum_missing_notes_link", "text": "...", "description": "...", "how_to_resolve": "...", "blocking": false } ], "blocking": false } } ``` **Key Fields:** - `.alerts.alerts[]` (array) - List of individual alerts - `.alerts.alerts[].blocking` (boolean) - Per-alert blocking status (THIS is what we check) - `.alerts.blocking` (boolean) - Top-level blocking status (informational only) **Next Action:** - Trigger push-to-cdn-staging (async) - Start checking for build promotion (parallel) --- ### 5. push-to-cdn-staging (Async Task) **Purpose:** Push release images to CDN staging environment **MCP Tool:** `oar_push_to_cdn_staging(release)` **Input:** - `release`: Z-stream version **Prerequisites:** check-cve-tracker-bug completed **Execution Phases:** **Phase 1 - Trigger:** ``` stdout contains: "task [Push to CDN staging] status is changed to [In Progress]" ```