Troubleshooting Devbox

Troubleshooting Devbox

Overview

Common issues and fixes for the Hetzner NixOS remote development environment.

Port Forwarding

The devbox connection uses several port forwards for development tools:

Local forwards (-L): devbox service → accessible on macOS localhost Remote forwards (-R): macOS service → accessible on devbox localhost

These should be configured in ~/.ssh/config (see below) so every ssh devbox includes them automatically.

Can't SSH After Recreate

Host key changed because server was recreated with new IP or reinstalled.

# Remove old host key
ssh-keygen -R devbox
ssh-keygen -R $(hcloud server ip devbox)

# Reconnect with new key acceptance
ssh -o StrictHostKeyChecking=accept-new devbox

Verify NixOS Installation

Check NixOS version:

ssh devbox 'nixos-version'
# Expected: 24.11... (Vicuna)

Check tools are installed:

ssh devbox 'tmux -V && nvim --version | head -1 && mise --version'

NixOS Configuration Issues

View current system generation:

ssh devbox 'sudo nix-env --list-generations --profile /nix/var/nix/profiles/system'

Rollback to previous generation:

ssh devbox 'sudo nixos-rebuild switch --rollback'

Check system journal for errors:

ssh devbox 'journalctl -b --priority=err'

nixos-anywhere Deployment Failed

If deployment fails mid-way, the server may be in an inconsistent state.

Check if server is accessible:

# Try root (during nixos-anywhere)
ssh root@$(hcloud server ip devbox) 'uname -a'

# Try dev (after completion)
ssh dev@$(hcloud server ip devbox) 'uname -a'

If stuck, use Hetzner rescue mode:

hcloud server enable-rescue devbox --type linux64
hcloud server reboot devbox
# Then SSH as root with rescue password from console

Or just destroy and recreate:

/rebuild

Verify SSH Hardening

# Should fail (root login disabled):
ssh root@$(hcloud server ip devbox) 'echo test'

# Should work:
ssh dev@$(hcloud server ip devbox) 'echo test'

Connection Timeout / Slow

Check latency:

ping -c 5 $(hcloud server ip devbox)

Helsinki datacenter latency from US East: ~100-150ms (acceptable for terminal work).

Server Not Responding

Check Hetzner console:

hcloud server list
hcloud server describe devbox

Power cycle if needed:

hcloud server reboot devbox

Disk Space Issues

NixOS can accumulate old generations. Clean up:

ssh devbox 'sudo nix-collect-garbage --delete-older-than 7d'
ssh devbox 'sudo nix-store --optimise'

Check disk usage:

ssh devbox 'df -h /'

Systemd User Timers

The devbox runs several user-level timers for automation.

Check Timer Status

# List all user timers
systemctl --user list-timers

# Specific timers
systemctl --user status pull-workstation.timer
systemctl --user status home-manager-auto-expire.timer

View Timer Logs

# Auto-update pull logs
journalctl --user -u pull-workstation -n 50

# Generation cleanup logs
journalctl --user -u home-manager-auto-expire -n 50

Timer Not Running

If a timer isn't active after home-manager switch:

# Reload systemd
systemctl --user daemon-reload

# Enable and start timer
systemctl --user enable --now pull-workstation.timer

See Automated Updates for full details on the update pipeline.