Monitor With Tmux

Monitor with Tmux

Monitor training progress in tmux, detect anomalies, analyze errors, provide fix suggestions.

Step Zero

Create a sleep script for tmux monitoring:

1. Create ./tmp/wait_tmux.py

import argparse
import subprocess
import time

SHELLS = {"bash", "zsh", "sh", "fish", "csh", "tcsh", "ksh", "dash", "ash"}

def smart_sleep(session: str, seconds: float, check_every: float = 2.0) -> bool:
    """
    Alternative to time.sleep(), but returns early when commands finish.

    Returns:
        True  - Normal timeout (command still running)
        False - Early return (command finished or session gone)
    """
    end_time = time.time() + seconds
    while time.time() < end_time:
        try:
            r = subprocess.run(
                ["tmux", "list-panes", "-F", "#{pane_current_command}", "-t", session],
                capture_output=True, text=True, timeout=5
            )
            if r.returncode != 0:
                return False  # session gone
            cmds = [l.strip().lower() for l in r.stdout.splitlines() if l.strip()]
            if not any(c not in SHELLS for c in cmds):
                return False  # command finished, back to shell
        except Exception:
            return False

        time.sleep(min(check_every, end_time - time.time()))

    return True


def main():
    parser = argparse.ArgumentParser(description="Wait for a tmux session with smart early-exit.")
    parser.add_argument("session", help="tmux session name")
    parser.add_argument("seconds", type=float, help="total seconds to wait")
    args = parser.parse_args()

    timed_out = smart_sleep(args.session, args.seconds, 2)
    raise SystemExit(0 if timed_out else 1)


if __name__ == "__main__":
    main()

Start Monitoring

When you need to monitor a tmux window, run:

python ./tmp/tmux_wait.py my_ajet_session_name 30

This means:

1. Monitor tmux session named my_ajet_session_name
2. Wait 30 seconds

• Exit code 0: Normal timeout (command still running)
• Exit code 1: Command ended early or session disappeared

Using SSH

When using SSH, always use a local tmux window to establish the SSH connection. (Use tmux skill, refer to ajet/copilot/monitor-with-tmux/SKILL.md under the agentjet codebase)

When You Want to Delay Before Reading tmux Window Again

You must have error immediate return functionality - do not use sleep xxx, instead use python ./tmux_wait.py my_ajet_session_name xxx

Don't use: sleep 60 && tmux capture-pane -t my_ajet_session_name -p | tail -80

You should use: python ./tmux_wait.py my_ajet_session_name 30 && tmux capture-pane -t my_ajet_session_name -p | tail -80

• 60 seconds is too long
• Always use python ./tmux_wait.py to wait

Examples

Examples Without SSH

0 examples available

Examples With SSH

1 example available

--- example user instruction begin ---
login
ssh root@localhost -p 8824
goto
/mnt/data_cpfs/qingxu.fu/agentjet/good-luck-agentjet
run with venv and monitor
source .venv/bin/activate && python -m ajet.launcher --conf tests/bench/benchmark_math/benchmark_math.yaml --autokill
--- example user instruction end ---

--- example agent reaction begin ---
The user wants me to monitor a tmux session while running a training task. Let me break down what they're asking:
1. SSH to remote machine (root@localhost -p 8824)
2. Go to specific directory
3. Activate virtual environment and run Python command
4. Monitor tmux session

# Write ./tmp/tmux_wait.py
....

# Create tmux session for training (note: session name must contain keyword `ajet`)
$ tmux new-session -d -s ajet_session -c "/mnt/data_cpfs/qingxu.fu/agentjet/good-luck-agentjet"
# note: do not add commands after `tmux new-session`, otherwise the session may exit immediately
...

# Send SSH command to tmux session
$ tmux send-keys -t ajet_session "ssh root@localhost -p 8824" Enter
...

# Wait and check SSH connection
$ python3 /tmp/tmux_wait.py ajet_session 10 && tmux capture-pane -t ajet_session -p | tail -20
SSH connection successful. Now I need to navigate to the directory and run the training command. Let me send commands to the tmux session.
...

# Navigate to working directory
$ tmux send-keys -t ajet_session "cd /mnt/data_cpfs/qingxu.fu/agentjet/good-luck-agentjet" Enter
...

# Start training with venv and launcher
$ tmux send-keys -t ajet_session "source .venv/bin/activate && python -m ajet.launcher --conf tests/bench/benchmark_math/benchmark_math.yaml --autokill" Enter
...

# Capture initial training output
$ tmux capture-pane -t ajet_session -p | tail -40
...

# Wait 30 seconds and check training progress
$ python3 /tmp/tmux_wait.py ajet_session 30 && tmux capture-pane -t ajet_session -p | tail -50
...
Training is running. Commands are executing correctly. Let me continue monitoring at 1-minute intervals.

# Wait 1 minute and check training progress
$ python3 /tmp/tmux_wait.py ajet_session 60 && tmux capture-pane -t ajet_session -p | tail -50
...

# Wait 2 minutes and check training progress
$ python3 /tmp/tmux_wait.py ajet_session 120 && tmux capture-pane -t ajet_session -p | tail -60
...
Training is progressing well. Checkpoints loaded, vLLM server started, swanlab tracking the run. No errors so far, only warnings. Let me continue monitoring.

# Wait 4 minutes and check training progress
$ python3 /tmp/tmux_wait.py ajet_session 240 && tmux capture-pane -t ajet_session -p | tail -60
...

... multiple rounds of monitoring ...

--- example agent reaction end ---

# Destroy tmux session
tmux kill-session -t ajet_session

For AgentJet Swarm

• You should create seperate tmux session for each agentjet swarm servers and each agentjet swarm clients
• When debugging, please do not restart agentjet swarm servers frequently, that waste too much time
• When you really having difficulty for clearing GPU memory, run ajet --autokill to automatically kill all python and ray processes (however, I still recommend using this as a last resort).
• For AgentJet, always use tmux session name that starts with ajet-*