| name | diagnose-sunbeam |
| description | Use when diagnosing failed Sunbeam CI runs, analyzing sosreport tarballs, juju status files, sunbeam CLI logs, or investigating multi-node OpenStack deployment failures on Canonical K8s |
Diagnose Sunbeam CI Failures
Core Principle
Prove less, qualify more. Symptoms are not causes. A diagnosis must separate what the artifacts show from what you infer. Name a root cause only when direct evidence supports it. When the artifacts prove only the failure surface, say so.
Mandatory Diagnosis Algorithm
Every diagnosis MUST follow these steps in order. Do not skip steps or reorder them.
1. Extract archives
2. Identify deployment phase from artifacts
3. Collect observed facts with file:line references
4. Run false-negative checklist (MANDATORY before declaring real failure)
5. Identify the immediate failure surface (what directly failed)
6. Evaluate candidate mechanisms (what might have caused it)
7. Check counter-evidence for each candidate
8. Classify each claim: Confirmed / Supported / Speculative
9. Write DIAGNOSTICS.md
Evidence Hierarchy
When artifacts conflict, higher-ranked evidence wins:
| Rank | Source | What it proves |
|---|
| 1 | Remote CLI completion log (sosreport-*/logs/sunbeam-*.log ending in ResultType.COMPLETED) | The remote operation succeeded, regardless of what the CI transport reports |
| 2 | Juju status + cluster list at collection time | Actual deployment state at a known timestamp |
| 3 | CI output log (generated-sunbeam-output.log) exit codes and stderr | What the CI runner observed — may not reflect remote reality |
| 4 | Juju debug logs | Agent lifecycle events — useful for sequencing, not for proving external causes |
| 5 | Pod logs, sosreport system logs | Supporting detail — confirms or weakens a hypothesis |
Key rule: A CI exit code 255 with "Broken pipe" is a Rank 3 observation. A remote CLI log showing ResultType.COMPLETED is Rank 1. Rank 1 overrides Rank 3. Always check before concluding the operation failed.
Partial log caveat: ResultType.COMPLETED on the last step of a CLI log confirms full completion. ResultType.COMPLETED on an intermediate step only confirms that step succeeded. Check which step is last before declaring the entire operation succeeded.
Red Herrings — Do Not Treat These as Root Causes
| Observation | Why it's a red herring | What to do instead |
|---|
Interactive prompt in SSH stdout (Configure endpoint services? [y/n]) | Often appears in the output buffer BEFORE the real work ran; does not prove the prompt caused the hang | Check remote CLI logs for what actually executed |
Post-failure SSH host key change (REMOTE HOST IDENTIFICATION HAS CHANGED) | Proves the host identity changed AFTER the failure was recorded; does not prove MAAS re-provisioned DURING the operation | State: "host key changed post-failure; cause of the original failure is not established by this artifact" |
No route to host after an SSH session dies | Proves the node was unreachable at cleanup time; does not prove when or why it became unreachable | Report it as a post-failure observation, not a cause |
| Downstream blocked/waiting units | A unit in blocked after another unit failed is a consequence, not a cause | Trace back to the first unit that left active/idle |
generated-monitor.log showing "No new models found" on single-node | Normal for single-node — controller is on the remote LXD, not visible to the CI monitor | Ignore for single-node topology |
Step 0: Extract Archives
cd /path/to/run-directory
for f in generated-sunbeam-sosreport-*.tar.xz; do
tar -xf "$f" 2>/dev/null
done
for f in generated-sunbeam-pods_*_logs.tgz; do
tar -xzf "$f" 2>/dev/null
done
Step 1: Identify Deployment Phase
Determine how far the deployment got:
| Artifacts Present | Phase |
|---|
| No juju status files, no sunbeam CLI logs in sosreports | Pre-bootstrap |
| Juju status files exist, only bootstrap node has CLI logs | Bootstrap (joins never started) |
| Juju status files exist, multiple nodes have CLI logs | Join phase |
Validation logs exist (generated-sunbeam-validation_*.log) | Post-deploy / test |
Plugin enable output visible in generated-sunbeam-output.log | Plugin enable |
Quick artifact inventory:
ls generated-sunbeam-output.log generated-sunbeam-juju_status_*.txt \
generated-sunbeam-validation_*.log generated-foundation.log 2>/dev/null
for sos in sosreport-*/; do
hostname=$(echo "$sos" | sed 's/sosreport-\(.*\)-20[0-9]*-.*/\1/')
count=$(ls "$sos/home/ubuntu/snap/openstack/common/logs"/*.log 2>/dev/null | wc -l)
echo "$hostname: $count CLI logs"
done
Topology: Check config-sunbeam.yaml or the run directory name for SKU:
single_node: one node, LXD-backed controller (monitor "No new models" is normal)external_juju: pre-existing controller, generated-foundation.log covers MAAS layer- Multi-node: 3-7+ bare-metal nodes
Step 2: Collect Observed Facts
Read the primary artifacts for the identified phase. Record only what the file directly shows, with file:line references.
For every phase — always check these first:
grep -n "client_loop: send disconnect: Broken pipe" generated-sunbeam-output.log
grep -n "Connection to .* closed by remote host" generated-sunbeam-output.log
grep -n "REMOTE HOST IDENTIFICATION HAS CHANGED" generated-sunbeam-output.log generated-github-runner-run.log
Phase-specific checks:
Pre-bootstrap
generated-sunbeam-output.log — check: wipefs errors, CalledProcessError, ERROR: No external connectivitygenerated-foundation.log — check: MAAS API errors (ServerError: 500, 400 Bad Request); does the error body name a specific issue or is it generic? Also check pg_dump failures, Python tracebacksgenerated-github-runner-run.log — check: ssh-keyscan failures (timeout vs connection refused?), Testflinger provisioning errors, Mirror sync in progress? (if multiple nodes fail the same curtin step simultaneously, mirror issue is a candidate)sosreport-*/sos_commands/block/lsblk — check: does the node have the expected disks, or only an OS drive?
Bootstrap
generated-sunbeam-output.log: exit codes, stdout/stderr of bootstrap command- Remote CLI log:
sosreport-<bootstrap>/home/ubuntu/snap/openstack/common/logs/sunbeam-*.log — search for final ResultType entry
pexpect.exceptions.TIMEOUT: juju add-machine connectivity issue
Join
generated-sunbeam-output.log: per-node join exit codes, stdout, stderr- Remote CLI logs for each failing node: search for final
ResultType entry
generated-sunbeam-juju_status_openstack-machines.txt: unit states, scalegenerated-sunbeam-sunbeam_cluster_list.txt: which nodes are present
Post-deploy / Plugin enable
generated-sunbeam-output.log: which command failed, exit code, timeout messagesgenerated-sunbeam-juju_status_openstack.txt: unit statesgenerated-sunbeam-validation_*.log: test results, HTTP errors- Pod logs (
generated/sunbeam/logs-openstack-*.txt): service-level errors
Step 3: False-Negative Check (MANDATORY)
You MUST run this checklist before declaring any failure real. This is not optional.
| Check | How | If true |
|---|
CI stderr shows client_loop: send disconnect: Broken pipe? | Grep output.log | SSH transport died; remote may have succeeded |
Remote CLI log ends with ResultType.COMPLETED? | Read last 20 lines of sosreport sunbeam-*.log for the failing node | Remote operation succeeded — CI failure is false negative |
sunbeam_cluster_list.txt shows all expected nodes? | Compare node count to config-nodes.yaml | Cluster formed successfully |
All juju units active/idle? | Scan both juju status files | Deployment is healthy |
All K8s pods Running with 0 restarts? | Check kubectl_get_pod.txt | K8s layer is healthy |
If all five checks pass: the failure is a false negative. State this directly. Do not speculate about what "might have" gone wrong.
If the remote CLI log is not available (no sosreport, or sosreport from wrong node): state that the false-negative check is inconclusive, not that the operation failed.
Step 4: Identify Immediate Failure Surface
State what directly failed using only the facts collected in Step 2. This is the narrowest true statement you can make.
Examples of good failure surface statements:
- "The CI recorded exit code 255 and
Broken pipe for the SSH session running sunbeam cluster join on node X at timestamp T."
- "
wipefs -a /dev/disk/by-dname/disk1 on node X returned No such file or directory."
- "
sunbeam enable loadbalancer exited with wait timed out after 900s at timestamp T."
Examples of over-claiming (avoid these):
- "MAAS re-provisioned the node" (unless you have direct MAAS logs proving it)
- "The node ran out of memory" (unless you have OOM killer logs)
- "Network connectivity was lost" (unless you have both endpoints' logs showing the drop)
Step 5: Evaluate Candidate Mechanisms
For each plausible explanation, apply this template:
Candidate: [mechanism name]
Suggests it: [what symptom points here]
Would confirm it: [what artifact would prove this]
Would contradict it: [what artifact would disprove this]
Artifact check result: [what you actually found]
Status: Confirmed / Supported / Speculative
Status definitions:
- Confirmed: Direct artifact evidence proves the mechanism (e.g., remote CLI log shows
ResultType.COMPLETED proving the operation succeeded; lsblk shows no disk1 proving the disk is missing)
- Supported: Artifacts are consistent with the mechanism but do not prove it exclusively (e.g., SSH exit 255 + empty stdout is consistent with PTY failure, but could also be a network drop)
- Speculative: No direct evidence; mechanism is plausible based on general knowledge but artifacts do not distinguish it from alternatives (e.g., "likely OOM" when no OOM killer log exists)
Counter-evidence requirement: For every candidate you evaluate, you MUST check for at least one artifact that would weaken the claim. If you skip this, the diagnosis is incomplete.
Missing remote evidence rule: If no sosreport exists for the failing node (node was down at collection time, or sosreport is from a different host like a Juju controller LXD), then no claim about what happened on that node can be Confirmed. You may list candidates as Supported or Speculative, but the diagnosis MUST state: "Remote-side evidence unavailable for node X; mechanism not established."
Diagnostic Heuristics
These are patterns observed in prior CI failures. Use them as hypotheses to test, not conclusions to assume. Each requires the evidence listed under "Confirms it" before you can claim it.
SSH Transport Failures
Broken Pipe (false negative candidate)
- Suggests it: exit 255,
client_loop: send disconnect: Broken pipe in stderr
- Confirms it: remote CLI log ends with
ResultType.COMPLETED
- Contradicts it: remote CLI log shows
ERROR/Exception, or no remote CLI log exists
- If unconfirmed: "CI SSH session died (exit 255, Broken pipe); remote outcome not established"
PTY Allocation Failure (false negative candidate)
- Suggests it:
Pseudo-terminal will not be allocated in stderr, empty stdout, exit 255
- Confirms it: remote CLI log shows the command ran and completed
- Contradicts it: remote CLI log absent or shows no execution
- If unconfirmed: "SSH PTY allocation failed; whether the remote command executed is not established"
Infrastructure Failures
Missing disk
- Suggests it:
wipefs: error: /dev/disk/by-dname/disk1: No such file or directory
- Confirms it:
lsblk in sosreport shows no secondary disk on that node
- Contradicts it:
lsblk shows the disk exists (maybe a symlink issue instead)
MAAS API failure
- Suggests it:
ServerError: 500 or 400 Bad Request in generated-foundation.log
- Confirms it: specific MAAS error message in the response body (e.g.,
maasserver_routable_pairs does not exist)
- Contradicts it: error is a client-side timeout, not a server error
- Caution: do not claim "MAAS DB corruption" from a 500 alone; 500 could be transient
VM boot timeout
- Suggests it:
ssh-keyscan retries exhausted in generated-github-runner-run.log
- Confirms it: timing shows VM boot took longer than retry window
- Contradicts it: keyscan failed for a different reason (connection refused vs timeout)
Juju / Charm Failures
Agent loss after model migration
- Suggests it: unit in
unknown/lost state
- Confirms it: juju debug log shows
QUIESCE + agent.conf left unchanged or invalid entity name or password for that specific unit
- Contradicts it: unit went lost at a time with no migration events in the log
- If unconfirmed: "unit X is in unknown/lost state; cause not established"
Terraform state lock contention
- Suggests it:
Error acquiring the state lock in CLI logs
- Confirms it: lock error followed by successful retry (60s later) — confirms contention, not failure
- Note: lock contention alone rarely causes deployment failure; check whether it contributed to a timeout
Timeout near-miss
- Suggests it:
wait timed out after N in output.log
- Confirms it: pod logs or juju debug log show the unit going active within minutes after the timeout timestamp
- Contradicts it: unit never reached active state
Post-Deploy Failures
Traefik hook backlog after TLS
- Suggests it: traefik unit stuck in
maintenance after sunbeam enable tls
- Confirms it: juju debug log shows continuous
ingress-relation-changed hook executions on that unit spanning the timeout window
- Contradicts it: traefik unit was idle; a different unit caused the timeout
MySQL connection exhaustion
- Suggests it:
MySQL Error (1040) in any pod log
- Confirms it: error 1040 appears across multiple mysql-router pod logs simultaneously
- Contradicts it: error appears in only one pod log (might be a single-service issue, not exhaustion)
Validation test failures
- Suggests it: non-zero failure count in
validation_*.log
- Confirms it: specific HTTP error codes and messages in the test log
- Note: if pre-TLS tests passed but post-TLS tests failed, the TLS enablement window is a candidate — but confirm the specific failing test endpoints use the TLS-affected ingress path. If failing tests are unrelated to TLS (e.g., Swift/RGW auth, metadata service), timing correlation alone does not establish TLS as the cause
Node Unavailability
Node became unreachable mid-run
- Suggests it:
Connection to ... closed by remote host followed by No route to host
- Confirms it: sosreport
uptime shows reboot during the run, OR sosreport shows fresh image (snap list = only snapd+core, missing juju dir)
- Contradicts it: sosreport uptime is consistent with continuous operation; node was available at collection time
- Caution: "closed by remote host" + post-failure host key change does NOT prove MAAS re-provisioned the node during the operation. It proves: (1) the remote SSH daemon terminated the session, and (2) the host key was different when checked later. The mechanism is not established. Use: "Node became unreachable at T; host key had changed by T+N. Cause not established by available artifacts."
Log File Reference
| File | Contents | Priority |
|---|
generated-sunbeam-output.log | CI execution log (SSH commands, exit codes, stdout/stderr per node) | 1 |
sosreport-*/home/ubuntu/snap/openstack/common/logs/sunbeam-*.log | Sunbeam CLI logs per node (remote-side execution — outranks CI transport) | 1 |
generated-sunbeam-juju_status_openstack.txt | K8s model: OpenStack service unit states | 2 |
generated-sunbeam-juju_status_openstack-machines.txt | Machine model: bare-metal unit states | 2 |
generated-sunbeam-sunbeam_cluster_list.txt | Cluster membership | 2 |
generated-foundation.log | MAAS/Terraform infra provisioning | 2 (pre-bootstrap) |
generated-sunbeam-validation_*.log | Validation test results | 2 (post-deploy) |
generated-sunbeam-juju_debug_log_openstack-machines.txt | Machine model agent logs | 3 |
generated-sunbeam-juju_debug_log_openstack.txt | K8s model agent logs | 3 |
generated/sunbeam/logs-openstack-*.txt | Extracted pod logs per service | 3 (post-deploy) |
generated-github-runner-run.log | CI orchestrator, Python tracebacks | 4 |
generated-sunbeam-show_units_openstack.txt | Unit relation data | 4 |
generated-sunbeam-juju_debug_log_controller.txt | Controller agent logs | 4 |
generated-monitor.log | Deployment monitor (normal="No new models" on single-node) | 5 |
generated-sunbeam-manifest.yaml | Manifest (database topology, channel) | 5 |
generated-sunbeam-kubectl_get_pod.txt | K8s pod status | 5 |
config-sunbeam.yaml | Deployment config (osd_devices, roles) | 5 |
generated-lastlines.txt | Last CI output lines + tracebacks | 5 |
sosreport-*/sos_commands/block/lsblk | Block devices per node | On-demand |
sosreport-*/sos_commands/snap/snap_list.txt | Installed snaps (detect fresh image) | On-demand |
Output Format
Produce a DIAGNOSTICS.md in the CI run directory:
# Diagnostics: <run-id>
## Verdict
**FALSE NEGATIVE / REAL FAILURE / INCONCLUSIVE** — one sentence stating what is proven.
## Failure Phase
Which phase: pre-bootstrap / bootstrap / join / plugin-enable / validation
## Immediate Failure Surface
What directly failed, stated narrowly with file:line references. No mechanism claims here.
## Observed Facts
Bullet list of what the artifacts show, each with a file:line reference.
Include facts that support AND facts that weaken any candidate explanation.
## Candidate Causes
### 1. [Name] — Status: CONFIRMED / SUPPORTED / SPECULATIVE
What suggests it, what confirms it, what was checked as counter-evidence.
**Evidence for:**
- `file:line` — what it shows
**Evidence against / not found:**
- What was checked and what it showed (or "not available")
### 2. ...
## What Is Not Established
Explicitly list plausible-sounding claims that the artifacts do NOT prove.
Example: "The artifacts do not establish whether the node was re-provisioned by MAAS
or crashed for another reason."
## Juju Status Summary
Units in error/blocked/lost, or "All healthy", or "N/A".
## Key Log Files Examined
Table of files checked with one-line findings.
Parallelization
When diagnosing multiple CI runs, dispatch one subagent per run directory. Each agent follows the full algorithm independently.
