Diagnose Sunbeam

Diagnose Sunbeam CI Failures

Core Principle

Prove less, qualify more. Symptoms are not causes. A diagnosis must separate what the artifacts show from what you infer. Name a root cause only when direct evidence supports it. When the artifacts prove only the failure surface, say so.

Mandatory Diagnosis Algorithm

Every diagnosis MUST follow these steps in order. Do not skip steps or reorder them.

1. Extract archives
2. Identify deployment phase from artifacts
3. Collect observed facts with file:line references
4. Run false-negative checklist (MANDATORY before declaring real failure)
5. Identify the immediate failure surface (what directly failed)
6. Evaluate candidate mechanisms (what might have caused it)
7. Check counter-evidence for each candidate
8. Classify each claim: Confirmed / Supported / Speculative
9. Write DIAGNOSTICS.md

Evidence Hierarchy

When artifacts conflict, higher-ranked evidence wins:

Key rule: A CI exit code 255 with "Broken pipe" is a Rank 3 observation. A remote CLI log showing ResultType.COMPLETED is Rank 1. Rank 1 overrides Rank 3. Always check before concluding the operation failed.

Partial log caveat: ResultType.COMPLETED on the last step of a CLI log confirms full completion. ResultType.COMPLETED on an intermediate step only confirms that step succeeded. Check which step is last before declaring the entire operation succeeded.

Red Herrings — Do Not Treat These as Root Causes

Step 0: Extract Archives

cd /path/to/run-directory

# Extract sosreport tarballs (mknod errors are expected and harmless)
for f in generated-sunbeam-sosreport-*.tar.xz; do
  tar -xf "$f" 2>/dev/null
done

# Extract pod log tarballs
for f in generated-sunbeam-pods_*_logs.tgz; do
  tar -xzf "$f" 2>/dev/null
done

Step 1: Identify Deployment Phase

Determine how far the deployment got:

Quick artifact inventory:

# Which key files exist?
ls generated-sunbeam-output.log generated-sunbeam-juju_status_*.txt \
   generated-sunbeam-validation_*.log generated-foundation.log 2>/dev/null

# Count CLI logs per node (most logs = likely bootstrap node)
for sos in sosreport-*/; do
  hostname=$(echo "$sos" | sed 's/sosreport-\(.*\)-20[0-9]*-.*/\1/')
  count=$(ls "$sos/home/ubuntu/snap/openstack/common/logs"/*.log 2>/dev/null | wc -l)
  echo "$hostname: $count CLI logs"
done

Topology: Check config-sunbeam.yaml or the run directory name for SKU:

• single_node: one node, LXD-backed controller (monitor "No new models" is normal)
• external_juju: pre-existing controller, generated-foundation.log covers MAAS layer
• Multi-node: 3-7+ bare-metal nodes

Step 2: Collect Observed Facts

Read the primary artifacts for the identified phase. Record only what the file directly shows, with file:line references.

For every phase — always check these first:

# SSH transport failures
grep -n "client_loop: send disconnect: Broken pipe" generated-sunbeam-output.log
grep -n "Connection to .* closed by remote host" generated-sunbeam-output.log
grep -n "REMOTE HOST IDENTIFICATION HAS CHANGED" generated-sunbeam-output.log generated-github-runner-run.log

Phase-specific checks:

Pre-bootstrap

• generated-sunbeam-output.log — check: wipefs errors, CalledProcessError, ERROR: No external connectivity
• generated-foundation.log — check: MAAS API errors (ServerError: 500, 400 Bad Request); does the error body name a specific issue or is it generic? Also check pg_dump failures, Python tracebacks
• generated-github-runner-run.log — check: ssh-keyscan failures (timeout vs connection refused?), Testflinger provisioning errors, Mirror sync in progress? (if multiple nodes fail the same curtin step simultaneously, mirror issue is a candidate)
• sosreport-*/sos_commands/block/lsblk — check: does the node have the expected disks, or only an OS drive?

Bootstrap

• generated-sunbeam-output.log: exit codes, stdout/stderr of bootstrap command
• Remote CLI log: sosreport-<bootstrap>/home/ubuntu/snap/openstack/common/logs/sunbeam-*.log — search for final ResultType entry
• pexpect.exceptions.TIMEOUT: juju add-machine connectivity issue

Join

• generated-sunbeam-output.log: per-node join exit codes, stdout, stderr
• Remote CLI logs for each failing node: search for final ResultType entry
• generated-sunbeam-juju_status_openstack-machines.txt: unit states, scale
• generated-sunbeam-sunbeam_cluster_list.txt: which nodes are present

Post-deploy / Plugin enable

• generated-sunbeam-output.log: which command failed, exit code, timeout messages
• generated-sunbeam-juju_status_openstack.txt: unit states
• generated-sunbeam-validation_*.log: test results, HTTP errors
• Pod logs (generated/sunbeam/logs-openstack-*.txt): service-level errors

Step 3: False-Negative Check (MANDATORY)

You MUST run this checklist before declaring any failure real. This is not optional.

If all five checks pass: the failure is a false negative. State this directly. Do not speculate about what "might have" gone wrong.

If the remote CLI log is not available (no sosreport, or sosreport from wrong node): state that the false-negative check is inconclusive, not that the operation failed.

Step 4: Identify Immediate Failure Surface

State what directly failed using only the facts collected in Step 2. This is the narrowest true statement you can make.

Examples of good failure surface statements:

• "The CI recorded exit code 255 and Broken pipe for the SSH session running sunbeam cluster join on node X at timestamp T."
• "wipefs -a /dev/disk/by-dname/disk1 on node X returned No such file or directory."
• "sunbeam enable loadbalancer exited with wait timed out after 900s at timestamp T."

Examples of over-claiming (avoid these):

• "MAAS re-provisioned the node" (unless you have direct MAAS logs proving it)
• "The node ran out of memory" (unless you have OOM killer logs)
• "Network connectivity was lost" (unless you have both endpoints' logs showing the drop)

Step 5: Evaluate Candidate Mechanisms

For each plausible explanation, apply this template:

Candidate: [mechanism name]
Suggests it: [what symptom points here]
Would confirm it: [what artifact would prove this]
Would contradict it: [what artifact would disprove this]
Artifact check result: [what you actually found]
Status: Confirmed / Supported / Speculative

Status definitions:

• Confirmed: Direct artifact evidence proves the mechanism (e.g., remote CLI log shows ResultType.COMPLETED proving the operation succeeded; lsblk shows no disk1 proving the disk is missing)
• Supported: Artifacts are consistent with the mechanism but do not prove it exclusively (e.g., SSH exit 255 + empty stdout is consistent with PTY failure, but could also be a network drop)
• Speculative: No direct evidence; mechanism is plausible based on general knowledge but artifacts do not distinguish it from alternatives (e.g., "likely OOM" when no OOM killer log exists)

Counter-evidence requirement: For every candidate you evaluate, you MUST check for at least one artifact that would weaken the claim. If you skip this, the diagnosis is incomplete.

Missing remote evidence rule: If no sosreport exists for the failing node (node was down at collection time, or sosreport is from a different host like a Juju controller LXD), then no claim about what happened on that node can be Confirmed. You may list candidates as Supported or Speculative, but the diagnosis MUST state: "Remote-side evidence unavailable for node X; mechanism not established."

Diagnostic Heuristics

These are patterns observed in prior CI failures. Use them as hypotheses to test, not conclusions to assume. Each requires the evidence listed under "Confirms it" before you can claim it.

SSH Transport Failures

Broken Pipe (false negative candidate)

• Suggests it: exit 255, client_loop: send disconnect: Broken pipe in stderr
• Confirms it: remote CLI log ends with ResultType.COMPLETED
• Contradicts it: remote CLI log shows ERROR/Exception, or no remote CLI log exists
• If unconfirmed: "CI SSH session died (exit 255, Broken pipe); remote outcome not established"

PTY Allocation Failure (false negative candidate)

• Suggests it: Pseudo-terminal will not be allocated in stderr, empty stdout, exit 255
• Confirms it: remote CLI log shows the command ran and completed
• Contradicts it: remote CLI log absent or shows no execution
• If unconfirmed: "SSH PTY allocation failed; whether the remote command executed is not established"

Infrastructure Failures

Missing disk

• Suggests it: wipefs: error: /dev/disk/by-dname/disk1: No such file or directory
• Confirms it: lsblk in sosreport shows no secondary disk on that node
• Contradicts it: lsblk shows the disk exists (maybe a symlink issue instead)

MAAS API failure

• Suggests it: ServerError: 500 or 400 Bad Request in generated-foundation.log
• Confirms it: specific MAAS error message in the response body (e.g., maasserver_routable_pairs does not exist)
• Contradicts it: error is a client-side timeout, not a server error
• Caution: do not claim "MAAS DB corruption" from a 500 alone; 500 could be transient

VM boot timeout

• Suggests it: ssh-keyscan retries exhausted in generated-github-runner-run.log
• Confirms it: timing shows VM boot took longer than retry window
• Contradicts it: keyscan failed for a different reason (connection refused vs timeout)

Juju / Charm Failures

Agent loss after model migration

• Suggests it: unit in unknown/lost state
• Confirms it: juju debug log shows QUIESCE + agent.conf left unchanged or invalid entity name or password for that specific unit
• Contradicts it: unit went lost at a time with no migration events in the log
• If unconfirmed: "unit X is in unknown/lost state; cause not established"

Terraform state lock contention

• Suggests it: Error acquiring the state lock in CLI logs
• Confirms it: lock error followed by successful retry (60s later) — confirms contention, not failure
• Note: lock contention alone rarely causes deployment failure; check whether it contributed to a timeout

Timeout near-miss

• Suggests it: wait timed out after N in output.log
• Confirms it: pod logs or juju debug log show the unit going active within minutes after the timeout timestamp
• Contradicts it: unit never reached active state

Post-Deploy Failures

Traefik hook backlog after TLS

• Suggests it: traefik unit stuck in maintenance after sunbeam enable tls
• Confirms it: juju debug log shows continuous ingress-relation-changed hook executions on that unit spanning the timeout window
• Contradicts it: traefik unit was idle; a different unit caused the timeout

MySQL connection exhaustion

• Suggests it: MySQL Error (1040) in any pod log
• Confirms it: error 1040 appears across multiple mysql-router pod logs simultaneously
• Contradicts it: error appears in only one pod log (might be a single-service issue, not exhaustion)

Validation test failures

• Suggests it: non-zero failure count in validation_*.log
• Confirms it: specific HTTP error codes and messages in the test log
• Note: if pre-TLS tests passed but post-TLS tests failed, the TLS enablement window is a candidate — but confirm the specific failing test endpoints use the TLS-affected ingress path. If failing tests are unrelated to TLS (e.g., Swift/RGW auth, metadata service), timing correlation alone does not establish TLS as the cause

Node Unavailability

Node became unreachable mid-run

• Suggests it: Connection to ... closed by remote host followed by No route to host
• Confirms it: sosreport uptime shows reboot during the run, OR sosreport shows fresh image (snap list = only snapd+core, missing juju dir)
• Contradicts it: sosreport uptime is consistent with continuous operation; node was available at collection time
• Caution: "closed by remote host" + post-failure host key change does NOT prove MAAS re-provisioned the node during the operation. It proves: (1) the remote SSH daemon terminated the session, and (2) the host key was different when checked later. The mechanism is not established. Use: "Node became unreachable at T; host key had changed by T+N. Cause not established by available artifacts."

Log File Reference

Output Format

Produce a DIAGNOSTICS.md in the CI run directory:

# Diagnostics: <run-id>

## Verdict
**FALSE NEGATIVE / REAL FAILURE / INCONCLUSIVE** — one sentence stating what is proven.

## Failure Phase
Which phase: pre-bootstrap / bootstrap / join / plugin-enable / validation

## Immediate Failure Surface
What directly failed, stated narrowly with file:line references. No mechanism claims here.

## Observed Facts
Bullet list of what the artifacts show, each with a file:line reference.
Include facts that support AND facts that weaken any candidate explanation.

## Candidate Causes

### 1. [Name] — Status: CONFIRMED / SUPPORTED / SPECULATIVE
What suggests it, what confirms it, what was checked as counter-evidence.

**Evidence for:**
- `file:line` — what it shows

**Evidence against / not found:**
- What was checked and what it showed (or "not available")

### 2. ...

## What Is Not Established
Explicitly list plausible-sounding claims that the artifacts do NOT prove.
Example: "The artifacts do not establish whether the node was re-provisioned by MAAS
or crashed for another reason."

## Juju Status Summary
Units in error/blocked/lost, or "All healthy", or "N/A".

## Key Log Files Examined
Table of files checked with one-line findings.

Parallelization

When diagnosing multiple CI runs, dispatch one subagent per run directory. Each agent follows the full algorithm independently.