# Monitor With Tmux

> Monitor training progress by reading tmux content with exponential backoff intervals (30s, 1min, 2min, 4min, 8min, 16min), analyze logs when anomalies occur, and provide fix suggestions

- Skill: `modelscope/monitor-with-tmux` (Agent Skill)
- Install (CLI): `npx skillmds@latest add modelscope/monitor-with-tmux`
- Raw SKILL.md: https://api.skillmd.com/api/skills/modelscope/monitor-with-tmux/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: See LICENSE.txt for full terms
- Author: modelscope (https://skillmd.com/u/modelscope)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/modelscope/monitor-with-tmux

---


# Monitor with Tmux

Monitor training progress in tmux, detect anomalies, analyze errors, provide fix suggestions.

## Step Zero

Create a sleep script for tmux monitoring:

1. Create `./tmp/wait_tmux.py`

```python
import argparse
import subprocess
import time

SHELLS = {"bash", "zsh", "sh", "fish", "csh", "tcsh", "ksh", "dash", "ash"}

def smart_sleep(session: str, seconds: float, check_every: float = 2.0) -> bool:
    """
    Alternative to time.sleep(), but returns early when commands finish.

    Returns:
        True  - Normal timeout (command still running)
        False - Early return (command finished or session gone)
    """
    end_time = time.time() + seconds
    while time.time() < end_time:
        try:
            r = subprocess.run(
                ["tmux", "list-panes", "-F", "#{pane_current_command}", "-t", session],
                capture_output=True, text=True, timeout=5
            )
            if r.returncode != 0:
                return False  # session gone
            cmds = [l.strip().lower() for l in r.stdout.splitlines() if l.strip()]
            if not any(c not in SHELLS for c in cmds):
                return False  # command finished, back to shell
        except Exception:
            return False

        time.sleep(min(check_every, end_time - time.time()))

    return True


def main():
    parser = argparse.ArgumentParser(description="Wait for a tmux session with smart early-exit.")
    parser.add_argument("session", help="tmux session name")
    parser.add_argument("seconds", type=float, help="total seconds to wait")
    args = parser.parse_args()

    timed_out = smart_sleep(args.session, args.seconds, 2)
    raise SystemExit(0 if timed_out else 1)


if __name__ == "__main__":
    main()
```

## Start Monitoring

When you need to monitor a tmux window, run:

```bash
python ./tmp/tmux_wait.py my_ajet_session_name 30
```

This means:
1. Monitor tmux session named `my_ajet_session_name`
2. Wait 30 seconds

- Exit code 0: Normal timeout (command still running)
- Exit code 1: Command ended early or session disappeared

## Using SSH

When using SSH, always use a local tmux window to establish the SSH connection. (Use tmux skill, refer to `ajet/copilot/monitor-with-tmux/SKILL.md` under the agentjet codebase)

## When You Want to Delay Before Reading tmux Window Again

You must have error immediate return functionality - do not use `sleep xxx`, instead use `python ./tmux_wait.py my_ajet_session_name xxx`

Don't use: `sleep 60 && tmux capture-pane -t my_ajet_session_name -p | tail -80`

You should use: `python ./tmux_wait.py my_ajet_session_name 30 && tmux capture-pane -t my_ajet_session_name -p | tail -80`

- 60 seconds is too long
- Always use `python ./tmux_wait.py` to wait



## Examples


### Examples Without SSH

0 examples available


### Examples With SSH

1 example available

```agent

--- example user instruction begin ---
login
ssh root@localhost -p 8824
goto
/mnt/data_cpfs/qingxu.fu/agentjet/good-luck-agentjet
run with venv and monitor
source .venv/bin/activate && python -m ajet.launcher --conf tests/bench/benchmark_math/benchmark_math.yaml --autokill
--- example user instruction end ---

--- example agent reaction begin ---
The user wants me to monitor a tmux session while running a training task. Let me break down what they're asking:
1. SSH to remote machine (root@localhost -p 8824)
2. Go to specific directory
3. Activate virtual environment and run Python command
4. Monitor tmux session

# Write ./tmp/tmux_wait.py
....

# Create tmux session for training (note: session name must contain keyword `ajet`)
$ tmux new-session -d -s ajet_session -c "/mnt/data_cpfs/qingxu.fu/agentjet/good-luck-agentjet"
# note: do not add commands after `tmux new-session`, otherwise the session may exit immediately
...

# Send SSH command to tmux session
$ tmux send-keys -t ajet_session "ssh root@localhost -p 8824" Enter
...

# Wait and check SSH connection
$ python3 /tmp/tmux_wait.py ajet_session 10 && tmux capture-pane -t ajet_session -p | tail -20
SSH connection successful. Now I need to navigate to the directory and run the training command. Let me send commands to the tmux session.
...

# Navigate to working directory
$ tmux send-keys -t ajet_session "cd /mnt/data_cpfs/qingxu.fu/agentjet/good-luck-agentjet" Enter
...

# Start training with venv and launcher
$ tmux send-keys -t ajet_session "source .venv/bin/activate && python -m ajet.launcher --conf tests/bench/benchmark_math/benchmark_math.yaml --autokill" Enter
...

# Capture initial training output
$ tmux capture-pane -t ajet_session -p | tail -40
...

# Wait 30 seconds and check training progress
$ python3 /tmp/tmux_wait.py ajet_session 30 && tmux capture-pane -t ajet_session -p | tail -50
...
Training is running. Commands are executing correctly. Let me continue monitoring at 1-minute intervals.

# Wait 1 minute and check training progress
$ python3 /tmp/tmux_wait.py ajet_session 60 && tmux capture-pane -t ajet_session -p | tail -50
...

# Wait 2 minutes and check training progress
$ python3 /tmp/tmux_wait.py ajet_session 120 && tmux capture-pane -t ajet_session -p | tail -60
...
Training is progressing well. Checkpoints loaded, vLLM server started, swanlab tracking the run. No errors so far, only warnings. Let me continue monitoring.

# Wait 4 minutes and check training progress
$ python3 /tmp/tmux_wait.py ajet_session 240 && tmux capture-pane -t ajet_session -p | tail -60
...

... multiple rounds of monitoring ...

--- example agent reaction end ---

# Destroy tmux session
tmux kill-session -t ajet_session

```


## For AgentJet Swarm

- You should create seperate tmux session for each agentjet swarm servers and each agentjet swarm clients
- When debugging, please do not restart agentjet swarm servers frequently, that waste too much time
- When you really having difficulty for clearing GPU memory, run `ajet --autokill` to automatically kill all python and ray processes (however, I still recommend using this as a last resort).
- For AgentJet, always use tmux session name that starts with `ajet-*`