hidden-pipeline-race-investigate

name	hidden-pipeline-race-investigate
description	Investigate a system that appears to work but silently fails under specific conditions. Demonstrates Tier 2 investigation with the 'no hypotheses survive' recovery pattern and hidden mechanism discovery.
metadata	{"author":"TGPSKI","version":"2.0"}
compatibility	Requires access to CI/CD logs, pipeline API, git history

Hidden Mechanism Investigation

Investigate a system optimization that appears to work in testing but silently fails in production. This example demonstrates the full two-tier approach where Tier 1 verifies coordinates quickly, and Tier 2 discovers a hidden platform behavior outside the developer's mental model.

Key Methodology Concepts Demonstrated

• Tier 1 fast-pass: Coordinates verified quickly (correct environment, correct feature flag, correct namespace)
• Hidden mechanism: The system interacts with a platform that produces unexpected behavior
• "No hypotheses survive" recovery: When all initial hypotheses are eliminated, the evidence points to something outside the current model
• Log-based timeline construction: Per-group timelines reveal the exact divergence point
• Diagnostic logging as remediation: Adding catch-all logging prevents future blind spots

---

Tier 1: Intake & Coordinate Resolution

Step 1: Intake

| Fact | Source | Tag |
|------|--------|-----|
| "Feature works in testing but groups are prematurely closed in production" | Reporter | STATED |
| Feature: {optimization name} | Reporter | STATED |
| "Prematurely closed" = members still processing when group terminated | Reporter | INFERRED |
| Affected environment: production | Reporter | STATED |
| Feature flag enabled in production | Config | STATED/MISSING |

Step 2: Adversarial Review

Inference	Simplest Alternative
"Works in testing" = same code path exercised	Testing may not exercise the production-specific platform behavior
"Prematurely closed" = close logic has a bug	Close logic is correct, but its input (pipeline status) is unexpected

Step 3: Coordinate Resolution

Coordinate	Verification
Environment	Reporter confirmed production (not staging/test)?
Feature flag	Feature is actually enabled for the affected groups?
Namespace	Correct namespace and deployment (not canary vs stable)?
Version	Running the expected code version?

Verdict: COORDINATES VERIFIED — confirmed production, feature enabled, correct deployment.

→ Proceed to Tier 2.

---

Tier 2: Investigation

Step 4: Timeline

Build a per-group timeline from production logs. The key insight is not the average behavior but the specific sequence for FAILED groups.

Source	What to Extract
Application logs	Group lifecycle events (create, add member, process, close)
Pipeline/CI API	Status of each member's pipeline at close time
Feature flag history	When the feature was enabled/disabled

Group {id} timeline:
| Time | Event | Details | Confidence |
|------|-------|---------|------------|
| T+0s | Group created | Lead: item A | DEFINITIVE |
| T+5s | Member added | Item B, pipeline pending | DEFINITIVE |
| T+10s | Lead processed | Pipeline SUCCESS | DEFINITIVE |
| T+15s | Member rebased (skip-CI) | Rebase completed | DEFINITIVE |
| T+20s → T+90s | ??? | GAP — what happened? | MISSING |
| T+90s | Group closed | "no active members" | DEFINITIVE |

The gap between T+15 and T+90 is where the investigation focuses.

Step 5: Hypothesize

Hypothesis	Prior	Supporting Evidence	Contradicting Evidence	Discriminating Check
Member pipeline timed out	Medium	Group eventually closed		Check pipeline status via API at close time
Member ejected for conflict	Medium	Group lost a member		Check for eject log events
Member's pipeline status unrecognized	Low	No familiar status in logs		Check what status API returns after skip-CI
Race between rebase and status check	Low	Timing is tight		Check interval between rebase and next loop

Step 6: Discriminate

Cheapest discriminating check: Query the pipeline API for the member's latest pipeline status at the time the group was closed.

curl -s "https://ci.example.com/api/v4/projects/{id}/pipelines?sha={sha}&per_page=5" | jq '.[].status'

API Returns	Interpretation	Confidence
"running" or "pending"	Pipeline active, close logic has a bug	DEFINITIVE
"success" or "failed"	Pipeline completed, close logic correct	DEFINITIVE
"skipped"	Unexpected — platform created pipeline with skipped after skip-CI	DEFINITIVE
Empty list	No pipeline for this SHA yet	DEFINITIVE

Coordinate system verification:

• Correct project and item?
• Correct SHA (post-rebase, not pre-rebase)?
• Correct time window in logs?

Step 7: Narrow

Discovery: The platform creates a pipeline with skipped status after a skip-CI operation. This status is not in the system's handling logic.

What happens:

1. Member is rebased with skip-CI flag
2. Platform creates a new pipeline with status skipped (undocumented behavior)
3. Processing loop encounters skipped status
4. skipped is not in the handled set (running, pending, success, failed)
5. Member falls through all status checks without setting the "active" flag
6. Close logic sees no active members → terminates the group

Evidence-hypothesis table update:

Hypothesis	Status	Evidence	Confidence
Pipeline timed out	Eliminated	API shows pipeline exists, not timed out	DEFINITIVE
Conflict eject	Eliminated	No eject events in logs	DEFINITIVE
Unrecognized status	Confirmed	API returns skipped, code doesn't handle it	DEFINITIVE
Race condition	Eliminated	Timing correct, status itself is the issue	DEFINITIVE

Step 8: Contain

Action	Rationale
Disable optimization via feature flag	Stop premature closures immediately
Add skipped status handling	Filter out skipped pipelines before processing
Add catch-all logging for unhandled statuses	Prevent future hidden mechanism blind spots
Handle empty pipeline list after filtering	skip-CI + filter can produce empty list

Artifact Checkpoint

# Investigation: Premature Group Closure

Date: {date}
Status: Mitigated (feature flag disabled)

## Executive Summary
The CI platform creates a pipeline with `skipped` status after a skip-CI
rebase. The group processing code doesn't handle this status, causing members
to fall through all checks without being marked active. This triggers premature
closure. Mitigated by disabling the feature; fix requires handling `skipped`
status and adding catch-all logging.

## Finding 1: Unhandled Pipeline Status (Hidden Mechanism)
**What happened**: skip-CI rebase → platform creates `skipped` pipeline →
status falls through all handler branches → `any_active` never set → group closed.
**Evidence**:
| Source | Observation | Confidence |
|--------|-------------|------------|
| Pipeline API | Status = `skipped` after skip-CI rebase | DEFINITIVE |
| Application logs | No status match for member in processing loop | DEFINITIVE |
| 5 test cases | All skip-CI rebases produce `skipped` pipeline | DEFINITIVE |
| Platform docs | `skipped` status not documented for this operation | CONFIG (absence of docs) |

**Remediation**: Filter `skipped` pipelines before processing. Handle empty
pipeline list. Add catch-all with logging for future unhandled statuses.

---

What This Example Teaches

1. Tier 1 is quick for systemic issues: When the reporter is looking at the correct environment and system, coordinates verify fast.
2. "No hypotheses survive" is a signal: When your initial hypothesis set is exhausted, the real cause is outside your mental model. Generate new hypotheses incorporating the unexpected evidence.
3. Hidden mechanisms are undocumented platform behaviors: The platform vendor didn't document that skip-CI creates a skipped pipeline. This behavior is invisible until it interacts with code that doesn't expect it.
4. Catch-all logging is preventive: Adding logging for unhandled cases turns future hidden mechanisms into quick discoveries instead of lengthy investigations.