Spectrocloud Troubleshooting

Spectro Cloud Troubleshooting

Debug cluster provisioning, upgrades, and operational issues using Palette's event streams and logs.

For the $PALETTE_API_KEY / $PROJECT_UID env vars used below, see the spectrocloud-common skill (auth + project UID lookup).

Key Concepts

• Event Stream: Live cluster events showing orchestration progress and errors
• Cluster Conditions: Milestones like "Creating Infrastructure", "Adding Control Plane Node"
• Log Bundle: Downloadable archive with Spectro logs, system logs, and manifests
• Reconciliation: Palette retries failed operations - intermittent errors may resolve automatically

Cluster Event Stream (API)

Events stream live and show orchestration progress. Watch these during provisioning.

Get Cluster Events

palette-axi events $CLUSTER_UID --project $PROJECT_UID --limit 50

Replaces the old GET /v1/spectroclusters/{uid}/events recipe: that endpoint (and /status/events) 404s in practice — confirmed at a 75% failure rate — because the working collection is /v1/events/components/spectrocluster/{uid}, which palette-axi events calls instead. Add --full for untruncated messages.

Get Cluster Status & Conditions

# Overview with conditions and upgrade history
curl -s "https://api.spectrocloud.com/v1/dashboard/spectroclusters/$CLUSTER_UID/overview" \
  -H "ApiKey: $PALETTE_API_KEY" \
  -H "ProjectUid: $PROJECT_UID" | \
  jq '{state: .status.state, conditions: .status.conditions, upgrades: .status.upgrades}'

Watch Events (Poll)

# Poll events every 10 seconds during provisioning
while true; do
  echo "=== $(date) ==="
  palette-axi events $CLUSTER_UID --project $PROJECT_UID --limit 10
  sleep 10
done

Log Bundle Download

Download comprehensive logs for deep debugging or support requests.

Download via API

# Download log bundle (returns zip file)
curl -s "https://api.spectrocloud.com/v1/spectroclusters/$CLUSTER_UID/features/logFetcher/logs" \
  -H "ApiKey: $PALETTE_API_KEY" \
  -H "ProjectUid: $PROJECT_UID" \
  -o spectro_logs.zip

# Extract and examine
unzip spectro_logs.zip -d cluster-logs/
ls cluster-logs/

Log Bundle Contents

Edge Host Logs (SSH)

For edge deployments, SSH to the host for live debugging.

# SSH to edge host (default creds: kairos/kairos)
ssh kairos@<edge-host-ip>

# Watch Palette agent logs (most useful)
sudo journalctl -u spectro-stylus-agent.service -f

# Check all Spectro services
sudo systemctl list-units 'spectro*'

# Key log files
cat /var/log/stylus-upgrade.log
cat /var/log/kube-init.log
cat /var/log/kube-join.log

# Check kubelet if K8s is initializing
sudo journalctl -u kubelet -f

Cluster State Reference

Known Errors: Noise vs. Actionable

Palette's reconciliation pattern means some errors are transient and resolve automatically.

Normal/Transient (Usually Safe to Ignore)

These often appear during provisioning but resolve:

Actionable Errors (Investigate)

These indicate real problems:

2-Node Specific Errors

Debugging Workflow

1. Check Cluster State

curl -s "https://api.spectrocloud.com/v1/spectroclusters/$CLUSTER_UID" \
  -H "ApiKey: $PALETTE_API_KEY" \
  -H "ProjectUid: $PROJECT_UID" | \
  jq '{name: .metadata.name, state: .status.state, health: .status.clusterHealth}'

2. Check Active Condition

curl -s "https://api.spectrocloud.com/v1/dashboard/spectroclusters/$CLUSTER_UID/overview" \
  -H "ApiKey: $PALETTE_API_KEY" \
  -H "ProjectUid: $PROJECT_UID" | \
  jq '.status.conditions[] | select(.status != "True") | {type, message, reason}'

3. Recent Events

palette-axi events $CLUSTER_UID --project $PROJECT_UID --limit 20

4. SSH to Edge Host (if applicable)

ssh kairos@<edge-host-ip>
sudo journalctl -u spectro-stylus-agent.service -n 100 --no-pager

5. Download Full Logs (for support)

curl -s "https://api.spectrocloud.com/v1/spectroclusters/$CLUSTER_UID/features/logFetcher/logs" \
  -H "ApiKey: $PALETTE_API_KEY" \
  -H "ProjectUid: $PROJECT_UID" \
  -o spectro_logs_$(date +%Y%m%d_%H%M).zip

Edge Host Registration Issues

# List edge hosts and their state
curl -s "https://api.spectrocloud.com/v1/edgehosts" \
  -H "ApiKey: $PALETTE_API_KEY" \
  -H "ProjectUid: $PROJECT_UID" | \
  jq '[.items[] | {name: .metadata.name, uid: .metadata.uid, state: .status.state,
      health: .status.health.state, cluster: .status.clusterUid}]'

# Check specific edge host
curl -s "https://api.spectrocloud.com/v1/edgehosts/$EDGEHOST_UID" \
  -H "ApiKey: $PALETTE_API_KEY" \
  -H "ProjectUid: $PROJECT_UID" | \
  jq '{name: .metadata.name, state: .status.state, health: .status.health,
      lastHeartbeat: .status.lastHeartbeat}'

Pack Deployment Issues

# Check pack status on cluster
curl -s "https://api.spectrocloud.com/v1/spectroclusters/$CLUSTER_UID/packs/status" \
  -H "ApiKey: $PALETTE_API_KEY" \
  -H "ProjectUid: $PROJECT_UID" | \
  jq '[.items[] | {name: .name, status: .status, message: .message}]'

Adding New Learnings

This skill is a living document. When you encounter new errors:

1. Determine if it's noise (transient, resolved by retry) or actionable
2. Add to the appropriate table above
3. Include the error pattern, cause, and recommended action
4. Commit with descriptive message

Quick Reference

Links

• Nodes and Clusters Troubleshooting
• Edge Troubleshooting
• Common Issues