Tailscale Mosh Recover

Tailscale + Mosh Recovery (Headless Mac mini)

Problem

After a Tailscale update or daemon restart on a headless Mac mini, mosh sessions break. Common symptoms (any combination):

• tailscale status returns: failed to connect to local Tailscale service; is Tailscale running?
• mosh <host> returns: could not get canonical name for <host>: nodename nor servname provided, or not known, then Connection closed by UNKNOWN port 65535, then Did not find remote IP address (is SSH ProxyCommand disabled?).
• Tailscale GUI app CLI returns: Tailscale.CLIError error 1.
• which tailscale reports a path that does not actually exist on disk.
• tailscale up registers the machine as <hostname>-1 (a duplicate of the original node — IP changed).

The two underlying problems are causally linked: Tailscale daemon trouble forces a re-auth → new Tailscale IP → existing mosh-server processes are bound to the old IP and become unreachable.

Context / Triggers

Solution

Work in this order: Layer 1 (Tailscale daemon) then Layer 2 (mosh cleanup). If only mosh is broken and tailscale status is healthy, skip Layer 1.

Layer 1: Tailscale daemon health (headless Mac mini)

The headless constraint: the standalone Tailscale.app GUI bundle's daemon needs an active user session. On a headless Mac mini (no monitor, no auto-login), the GUI daemon never starts. The right setup for headless is Homebrew's tailscale package run as a launchd service — it's a system service that survives reboots without any logged-in user.

Step 1.1: Confirm the symptom + identify installs

tailscale status              # likely: "failed to connect to local Tailscale service"
which tailscale               # may report a path that doesn't exist
ls -la /Applications/Tailscale.app  # GUI app present?
brew list tailscale 2>/dev/null     # Homebrew install present?
ls -la $(brew --prefix)/Cellar/tailscale  # Cellar contents if Homebrew has it

If both installs exist, that's the dual-install conflict. The CLI symlink at /opt/homebrew/bin/tailscale may have been removed by a recent update — leaving the daemon running (or not) and no CLI to talk to it.

Step 1.2: Remove the GUI app's broken CLI shim (if present)

The GUI app installs a shim at /usr/local/bin/tailscale that errors with Tailscale.CLIError error 1 when the GUI app isn't running. Remove it to avoid PATH ambiguity:

sudo rm -f /usr/local/bin/tailscale

Step 1.3: Re-link the Homebrew CLI

The Homebrew binary still exists in $(brew --prefix)/Cellar/tailscale/<version>/bin/tailscale; it's just unlinked.

brew link tailscale
hash -r                       # clear shell PATH cache
which tailscale               # should now report /opt/homebrew/bin/tailscale

If brew link complains about a conflict, the conflicting path was likely the GUI shim from Step 1.2 — re-run Step 1.2 and try again. If you've decided to remove the GUI app entirely, brew link --overwrite tailscale is acceptable.

Variant: orphan-after-autoremove

If which tailscale returns nothing right after a brew uninstall of something else reported Autoremoving ... tailscale: the daemon is still running on the unlinked binary — brew link --overwrite tailscale restores the CLI without touching it. Full diagnosis + 2026-05-22 incident: references/extended-diagnostics.md §Variant.

Step 1.4: Confirm the Homebrew daemon is running as a system service

sudo brew services list | grep tailscale

Expected: tailscale started root /Library/LaunchDaemons/homebrew.mxcl.tailscale.plist.

If started and root are correct, the daemon is already healthy — skip to Step 1.5. If not started:

sudo brew services start tailscale

The sudo matters — without it, tailscaled runs as your user and lacks the privileges for subnet routing / exit node mode.

Step 1.5: Re-authenticate

tailscale up

Follow the auth URL on another device. If the prior config used flags (exit node, subnet routes, etc.), include them — for example:

sudo tailscale up --advertise-exit-node --advertise-routes=192.168.1.0/24 --accept-routes

Re-auth creates a fresh node identity, which is why the machine's Tailscale IP can change. Note the new IP — Layer 2 depends on it.

tailscale status              # confirm: tailnet visible, node "active"

Step 1.6: Clean up the GUI app (recommended for headless)

For a headless Mac mini, the GUI app is dead weight and risks dropping a broken CLI shim back on future updates:

sudo rm -rf /Applications/Tailscale.app
sudo rm -f /usr/local/bin/tailscale
# Optional: user-level support files (vary by App Store vs standalone build):
rm -rf ~/Library/Containers/io.tailscale.ipn.macos
rm -rf ~/Library/Group\ Containers/*.io.tailscale.ipn.macsys
rm -rf ~/Library/Application\ Support/Tailscale
rm -rf ~/Library/Preferences/io.tailscale.ipn.macos.plist
rm -rf ~/Library/Caches/io.tailscale.ipn.macos
# Verify nothing GUI-related in LaunchAgents (Homebrew daemon plist must remain):
ls -la /Library/LaunchDaemons/ | grep -i tailscale  # keep homebrew.mxcl.tailscale.plist
ls -la /Library/LaunchAgents/ ~/Library/LaunchAgents/ 2>/dev/null | grep -i tailscale

Step 1.7: Admin-console cleanup (browser, not CLI)

Visit https://login.tailscale.com/admin/machines:

• Delete the old offline node (the one with the original hostname before re-auth).
• Rename the new node back to the clean hostname if you ended up with <name>-1.
• Re-approve advertised routes / exit-node status — these need explicit approval per node.

Layer 2: Mosh-server cleanup (downstream of any Tailscale IP change)

After any Tailscale re-auth or daemon restart that changes the machine's IP, existing mosh-server processes are bound to the old IP and become unreachable. tmux sessions remain alive and untouched — only the mosh-server wrapper needs replacing.

Step 2.1: List mosh-servers and their UDP bindings

ps aux | grep mosh-server | grep -v grep
lsof -nP -iUDP 2>/dev/null | grep mosh

The lsof output shows each mosh-server's PID and bound IP/port (e.g. mosh-serv 94933 ... UDP 100.x.y.z:60005). Compare the bound IPs against the current Tailscale IP from tailscale status (e.g. 100.x.y.z). Any mosh-server bound to a different IP is stale.

Step 2.2: Verify which tmux sessions correspond to which mosh-servers

tmux ls
ps -ef | grep mosh-server | grep -v grep   # `mosh-server new ... -- tmux new -s <SESSION>` shows the session each was attached to

This lets you confirm a stale mosh-server is the wrapper for a specific tmux session — that session's work is preserved on disk by tmux regardless of mosh-server fate.

Step 2.3: Kill stale mosh-servers (preserve the healthy one)

kill <PID-list-of-stale-servers>

Do not include any mosh-server whose UDP binding matches the current Tailscale IP. tmux sessions survive.

Step 2.4: Verify

ps aux | grep mosh-server | grep -v grep   # only the healthy ones remain
lsof -nP -iUDP | grep mosh                 # bindings should all show the current IP
tmux ls                                    # sessions unchanged

Step 2.5: Reconnect from the remote machine

From the client (e.g. MacBook):

mosh <host> -- tmux a -t <session-name>

This spawns a fresh mosh-server bound to the current Tailscale IP and reattaches to the existing tmux session. If mosh <host> fails with the canonical-name error, fall back to the resolved hostname directly:

mosh <my-mini-or-actual-tailscale-name> -- tmux a -t <session-name>

Verification

After Layer 1 + Layer 2:

# On the Mac mini:
which tailscale                            # /opt/homebrew/bin/tailscale
tailscale status                           # shows tailnet; this node "active"
sudo brew services list | grep tailscale   # tailscale started root
ps aux | grep mosh-server | grep -v grep   # only mosh-servers bound to current IP

# From the client:
mosh <host> -- tmux a -t <session-name>    # connects; tmux session intact

Layer 3 — Downstream references to the (changed) Tailscale IP

Hostname-first principle: use MagicDNS hostnames (my-mini, my-vps) everywhere; never hardcode 100.x.y.z IPs. After a re-auth changes the IP, grep every peer machine for the OLD IP (health-check scripts, ~/.ssh/config HostName lines, known_hosts, monitoring targets) and hostname-ify each hit.

Full Layer 3 protocol — detection greps, per-location fix table, VPS health-check verification — plus three related failure modes (macOS resolver stuck Not Reachable after Tailscale restart → dscacheutil -flushcache + killall -HUP mDNSResponder; RustDesk/VNC relay blocked by MDM → connect via Tailscale direct-IP; mosh-server: command not found → Homebrew PATH in ~/.zshenv), the worked 2026-05-18 example, and edge-case notes: references/extended-diagnostics.md.

Layer 4 — Dropbox EPERM (FDA on mosh-server)

A distinct mosh-server failure, not an IP-change one: mid-session, every shell access to the Mini's configured Dropbox root returns "Operation not permitted" (EPERM) while the same paths work from the user's own SSH terminal, and disabling the client sandbox does not help. Cause: Dropbox File Provider re-adopted a folder (usually after a rename) → provider-managed files require Full Disk Access, and TCC attributes access to the mosh-server responsible process, not tmux. Fix: grant FDA to /opt/homebrew/bin/mosh-server (+ tmux), then start a new mosh-server (tmux kill-server, fully disconnect + reconnect — TCC grants only apply at process launch). Full mechanism, verification, wrong turns, and prevention: references/dropbox-fda-eperm.md.

Anti-Patterns

• Don't reach for tailscale up --reset first. It resets state on the coordination server, including all advertised routes / exit-node approvals. Use it only when nothing else recovers the daemon.
• Don't kill -9 mosh-servers. A regular SIGTERM (kill PID) lets them release UDP bindings cleanly. -9 skips cleanup; the OS will still release the socket but session state files in /tmp linger.
• Don't restart the whole Mac mini to fix this. Layer 1 + Layer 2 cumulatively take ~3 minutes; a restart adds nothing except risk of losing tmux sessions if anything in them wasn't saved to disk.
• Don't assume which tailscale reflects disk reality. Always cross-check with ls -la $(which tailscale). The PATH cache mismatch is the specific gotcha this skill flags.
• Don't remove /Library/LaunchDaemons/homebrew.mxcl.tailscale.plist during GUI-app cleanup. That's the Homebrew daemon's plist — removing it is what you're trying to avoid.
• Don't apply Layer 1 from a remote SSH session over Tailscale. The tailscale down / tailscale up cycle drops your SSH connection. Either work locally on the Mini's console (Screen Sharing over LAN works), or open a backup SSH session via the LAN IP (ssh user@<mini-LAN-IP>) before starting.

Cross-references

Canonical setup docs (read these first if Layer 1 is unfamiliar)

Related skills

Origin

Extracted 2026-05-18 from a real diagnostic session — full narrative in references/extended-diagnostics.md §Origin.

Origin

This skill was extracted on 2026-05-18 from a real diagnostic session: the Mac mini's Tailscale GUI app auto-updated, daemon disconnected, headless box couldn't run the GUI daemon, switched to Homebrew/launchd, re-auth gave a new Tailscale IP, existing mosh-servers stranded on the old IP. Original conversation: ~/Library/Mobile Documents/com~apple~CloudDocs/Cloud Downloads/mosh-tailscale.md (iCloud — TCC-gated; copy to /tmp/ to read from Claude Code).