Hydra Head Troubleshooter

hydra-head-troubleshooter

When to use

• Hydra Head doesn't start or no head observed
• Head exists but doesn't make progress
• Peers out of sync or disconnected
• Log messages: PeerConnected, AckSn, LogicOutcome errors

Operating rules (must follow)

• Confirm network and hydra-node version first
• Request logs from ALL participants for same time window
• Never request key contents (file paths OK)
• Output: (1) Root cause (2) Fix steps (3) Verification

Docker fallback mode

If hydra-node is not installed locally, use the wrapper script in this skill folder to run hydra-node inside Docker (Hydra upstream recommends Docker images for quickest start).

chmod +x {baseDir}/scripts/hydra-node.sh
{baseDir}/scripts/hydra-node.sh --help
{baseDir}/scripts/hydra-node.sh gen-hydra-key --output-file hydra

For full multi-node Head demos, prefer the hydra.family Docker Compose demo (it's the canonical "known-good" setup).

Quick diagnostic commands

# Check hydra-node version
hydra-node --version

# Check API health
curl -s localhost:4001/health

# Check peers
curl -s localhost:4001/peers

# Check head status
curl -s localhost:4001/status

# Check metrics (if enabled)
curl -s localhost:6001/metrics | grep hydra

Decision tree

A) "No head is observed from the chain"

Symptoms:

• Head never appears in logs
• No Init/Commit/Open progression
• Client shows no head state

Check 1: Cardano connection

# Verify cardano-node is ready
cardano-cli query tip --testnet-magic 1
# Should show current slot, not error

# Check socket exists
ls -la $CARDANO_NODE_SOCKET_PATH

Fix: Wait for cardano-node sync, verify socket path and network magic

Check 2: Scripts tx id

# Verify you're using correct scripts tx id for network
# Get from hydra-node release notes

# In hydra-node logs, look for:
grep -i "script" hydra-node.log | head -20

Fix: Use correct --hydra-scripts-tx-id for your network

Check 3: Key mismatch

# Verify cardano.sk matches what peers have as your vkey
cardano-cli key verification-key \
  --signing-key-file cardano.sk \
  --verification-key-file check.vkey

# Compare check.vkey with what you distributed

Fix: Re-exchange verification keys with all peers

B) "Head does not make progress"

Symptoms:

• Head exists but stuck at Init or Commit
• Snapshots not confirmed
• Commands hang or timeout

Check 1: Peer connectivity

# In logs, look for PeerConnected
grep "PeerConnected" hydra-node.log

# Check metrics
curl -s localhost:6001/metrics | grep peers_connected

Fix: Verify --peer host:port is correct and ports are reachable

Check 2: Hydra key mismatch

# Look for AckSn issues
grep -E "AckSn|LogicOutcome" hydra-node.log

# Verify hydra keys match
hydra-node verify-keys \
  --hydra-signing-key hydra.sk \
  --hydra-verification-key peer-hydra.vk

Fix: Re-exchange hydra verification keys

Check 3: Version mismatch

# All nodes must run compatible versions
hydra-node --version
# Compare across all participants

Fix: Align hydra-node versions

C) "Head stuck: peer out of sync"

Symptoms:

• One node accepts txs, others reject
• Snapshots stop being signed
• Ledger state diverged

Cause: Different ledger views due to config drift or missed txs

Fix: Snapshot side-loading

# 1. Get latest snapshot from healthy node
curl -s http://healthy-node:4001/snapshot > snapshot.json

# 2. Post to out-of-sync node
curl -X POST http://out-of-sync-node:4001/snapshot \
  -H "Content-Type: application/json" \
  -d @snapshot.json

Verify: Snapshots resume, all parties sign again

D) Mirror nodes / HA issues

Symptoms:

• SnapshotAlreadySigned in logs
• etcd quorum issues

Reality:

• SnapshotAlreadySigned is transient and harmless with mirrors
• Keep mirror count < floor(n/2) for quorum stability

Fix: Reduce mirror count or accept transient warnings

Log patterns to grep

Verification checklist

After any fix:

• All nodes same hydra-node version
• Scripts tx id correct for network
• cardano-node ready (query tip works)
• PeerConnected for all peers
• AckSn messages flowing
• No LogicOutcome errors

Examples

Example: Diagnose stuck head

User request: "My Hydra head won't open—stuck at commit"

Response:

# 1. Check all peers connected
grep "PeerConnected" hydra-node.log | tail -10
# Should see PeerConnected for each peer

# 2. If missing peers, check network
nc -zv peer-host 5001
# Should show "Connection succeeded"

# 3. Check all parties committed
curl -s localhost:4001/status | jq .
# Look for parties and their commit status

# 4. Check for key issues
grep -E "InvalidSignature|WrongParty" hydra-node.log

# Common fix: restart with correct peer addresses
# Verify --peer host:port matches actual peer locations

References

• hydra-head (operational guidance)
• hydra-head-operator (execution)
• reference/sources.md (doc provenance)
• hydra.family troubleshooting