vault-path-confusion-investigate

name	vault-path-confusion-investigate
description	Investigate reported secret mismatches between a cluster and a secret store. Demonstrates Tier 1 resolution — coordinate mismatch identified and resolved without deep analysis.
metadata	{"author":"TGPSKI","version":"2.0"}
compatibility	Requires vault CLI, kubectl/oc, access to cluster and secret store

Secret Mismatch Investigation

Investigate reports of "the secret on the cluster doesn't match the secret store." This example demonstrates the Tier 1 fast path — where the investigator resolves a coordinate mismatch without entering deep analysis.

Key Methodology Concepts Demonstrated

• Tier 1 resolution: Coordinate mismatch resolved in Steps 1-4, no timeline or hypothesis generation needed
• Reporter methodology ≠ system path: What the reporter checked and where the system writes are independent
• Evidence hierarchy in practice: Thread consensus (SOCIAL) can contradict deployment config (CONFIG)
• Ground truth validation: A single non-destructive read confirms the verdict
• Cheapest discriminating check: Reading two paths costs 10 seconds and resolves the majority of these reports

---

Step 1: Intake

Extract facts from the report:

| Fact | Source | Tag |
|------|--------|-----|
| "Secret on cluster doesn't match what's in the store" | Reporter | STATED |
| Namespace: {ns} | Reporter | STATED |
| Secret name: {name} | Reporter | STATED |
| Which path reporter checked | ? | MISSING |
| Which system manages this secret | Config | INFERRED |

Critical MISSING fact: What exact path/command did the reporter use to confirm the mismatch?

Step 2: Adversarial Review

Inference	Simplest Alternative
"Secret doesn't match" = sync failure	Reporter checked wrong path (stale data at legacy location)
System that manages this secret is X	Config may show a different manager than expected

Missing Fact Assessment:

Missing	Score	Reason
Reporter's check methodology	5	If they checked a legacy path, the case closes immediately

Evidence hierarchy check: If the thread has multiple people agreeing "sync is broken" (SOCIAL), but config shows the system is configured to write to a different path (CONFIG) — do not defer to social proof. Validate with ground truth.

Step 3: Coordinate Resolution

Resolve the Reader

Look for evidence of reporter's methodology in the report:

• CLI commands pasted (e.g., vault kv get secret/{some-path}/...)
• Screenshots showing a specific path
• Explicit statements ("I checked path X")

If not shown, ask:

"What exact vault path did you check to confirm the mismatch? This may close the case immediately."

Resolve the System Path

Read from deployment configuration:

• Namespace definition: which integration manages this resource?
• Integration config: what is the output_path or secret_path prefix?
• Migration status: is this resource managed by the old or new system?

SYSTEM uses: secret/{current-prefix}/{namespace}/{identifier}
Source: {namespace config file showing active manager + path}

Compare (Gate)

REPORTER checked: secret/{legacy-prefix}/{namespace}/{identifier}
SYSTEM uses:      secret/{current-prefix}/{namespace}/{identifier}

Verdict: COORDINATE MISMATCH — reporter checked the legacy path, system writes to the current path.

Step 4: Ground Truth Check

Validate with non-destructive reads:

# Check the system's path — should have recent data
vault kv metadata get secret/{current-prefix}/{namespace}/{identifier}
# → updated_time: recent timestamp confirms system is writing here

# Optionally check the reporter's path — should be stale
vault kv metadata get secret/{legacy-prefix}/{namespace}/{identifier}
# → updated_time: old timestamp confirms stale data at legacy location

GROUND TRUTH: vault kv metadata get at system path → updated_time is recent
Confirms: COORDINATE MISMATCH — system is writing to current path, reporter checked legacy path

Resolution: The system is working correctly. The reporter observed stale data at a legacy path that is no longer active.

Step 5: Contain

Outcome	Action
Coordinate mismatch (wrong path)	Inform the reporter of the correct path. No system action needed.
(If actual sync failure were found)	Check operator health, restart if needed.

Artifact Checkpoint

# Investigation: Secret Mismatch Report

Date: {date}
Status: Resolved — no incident (coordinate mismatch)

## Executive Summary
Reporter checked the legacy path prefix which contains stale data from before
the migration. The current path (managed by the active system) contains the
correct value matching the cluster secret. No sync failure occurred.

## Finding 1: Coordinate Mismatch
**What happened**: Reporter read from `legacy-prefix/` which contains a stale
snapshot from before migration. The active path is `current-prefix/`.
**Evidence**:
| Source | Observation | Confidence |
|--------|-------------|------------|
| Secret store (legacy path) | Value is stale, last written {date} | DEFINITIVE |
| Secret store (current path) | Value matches cluster, updated recently | DEFINITIVE |
| Deployment config | Manager: current-system, path: current-prefix/ | CONFIG |
| Thread consensus | "Sync is broken" | SOCIAL (contradicted by evidence) |

**Remediation**: Informed reporter of correct path. Consider cleanup of legacy path
to prevent future confusion.

---

What This Example Teaches

1. Tier 1 resolves the case: No timeline, hypotheses, or deep analysis needed. The investigation completes in Steps 1-4.
2. The reader path is the critical missing fact: Without knowing what the reporter actually checked, you cannot detect a coordinate mismatch.
3. Thread consensus is not evidence: Multiple engineers agreeing "sync is broken" is SOCIAL confidence. The system's actual behavior (CONFIG + DEFINITIVE ground truth) tells a different story.
4. Ground truth closes the loop: A single vault kv metadata get confirms the system is writing to the correct path with a recent timestamp.