Julien Workflow Queuing Background Tasks

Idle Queue - Background Task Manager

Queue manager that runs jobs at low priority when user is active, and normal priority when idle (5min inactivity).

Prerequisites

• Python 3.10+
• Service must be running before adding jobs
• Windows only (uses Windows API for idle detection)

Quick Start

Verified behavior: When user is active, jobs run at IDLE priority (4) - your PC stays responsive. After 5 minutes idle, priority increases for faster execution. Only 1 job runs when active.

# 1. Start the service (keep this terminal open)
qm start --foreground

# 2. In another terminal, add a job
qm add "python my_script.py" --name "My Task"

# 3. Monitor
qm list
qm logs 1

Service Startup

Manual start

qm start                   # Background (daemon)
qm start --foreground      # With dashboard at http://127.0.0.1:8742

Auto-start at login (Windows)

# Create shortcut in startup folder
# Target: pythonw -m idle_queue.cli start
# Or add to Task Scheduler

Check if service is running

qm status                  # Shows service status
curl http://127.0.0.1:8742/api/status  # API check

When to Use

• Scripts taking > 30 seconds
• Batch processing (transcoding, conversions)
• API-heavy operations (translations, LLM calls)
• Any task that shouldn't block user work

CLI Commands

All commands via python -m idle_queue.cli or qm (if installed):

# Add a job to queue
qm add "python script.py" --name "My Task"
qm add "ffmpeg -i in.mp4 out.webm" --name "Transcode" --cwd "C:/videos"
qm add "node build.js" --force  # Run even when user active

# List jobs
qm list                    # All jobs
qm list --status pending   # Filter by status

# Job details
qm status <id>             # Full job info
qm logs <id>               # View stdout/stderr
qm logs <id> --follow      # Stream logs live

# Job management
qm cancel <id>             # Cancel pending/running job
qm retry <id>              # Retry failed job
qm force <id>              # Force pending job to run now

# Service control
qm start                   # Start as background service
qm start --foreground      # Start with dashboard
qm stop                    # Stop service

# Configuration
qm config show             # Show current config
qm config set workers.max_when_idle 4

REST API

Base URL: http://127.0.0.1:8742/api/

Job Statuses

• pending - Waiting in queue
• running - Currently executing
• completed - Finished successfully (exit_code=0)
• failed - Finished with error (exit_code!=0)
• cancelled - Manually cancelled

Usage Patterns

Queue a Python script

qm add "python process_data.py --input data.csv" --name "Process CSV"

Queue with working directory

qm add "npm run build" --name "Build Project" --cwd "C:/projects/myapp"

Force immediate execution

qm add "python urgent.py" --name "Urgent Task" --force

Check job and get logs

qm status 42
qm logs 42

API usage from Python

import requests

# Add job
r = requests.post("http://127.0.0.1:8742/api/jobs", json={
    "command": "python long_task.py",
    "name": "Background Task"
})
job_id = r.json()["id"]

# Check status
status = requests.get(f"http://127.0.0.1:8742/api/jobs/{job_id}").json()
print(status["status"])  # pending, running, completed, failed

Important Notes

1. Service must be running - Start with qm start before adding jobs
2. Dashboard available at http://127.0.0.1:8742/ when service runs
3. Logs stored in %APPDATA%\idle-queue\logs\
4. Config file at %APPDATA%\idle-queue\config.yaml
5. Duplicate detection - Same command+cwd rejected unless --force-duplicate

Priority Behavior (Verified)

Tested: Process shows Priority: 4 (IDLE_PRIORITY_CLASS) when user is active.

Troubleshooting

Special characters in paths (IMPORTANT)

Problem: Shell escaping issues with special characters in file paths.

Recommended approach by complexity:

# SIMPLE - paths without special characters
qm add "ffmpeg -i input.mp4 output.mp4"

# MEDIUM - use --cwd to avoid path issues
qm add "ffmpeg -i input.mp4 output.mp4" --cwd "C:/Movies/My Folder"

# COMPLEX - use API for any special characters
python -c "
import requests
requests.post('http://127.0.0.1:8742/api/jobs', json={
    'command': 'ffmpeg -i \"C:/Movies/What's Up!.mp4\" out.mp4',
    'name': 'Transcode'
})
"

Why API is safest: The API receives JSON directly without shell parsing, so special characters are preserved exactly as-is.

Service won't start

# Check if port is already in use
netstat -ano | findstr 8742

# Kill process using port
taskkill /F /PID <pid>

# Retry
qm start --foreground

Job stuck in "pending"

• Service not running → qm start
• User is active + job not forced → wait for idle or qm force <id>

Job fails immediately

# Check logs for error
qm logs <id>

# Common causes:
# - Command not found → check PATH or use full path
# - Working directory doesn't exist → verify --cwd path
# - Permission denied → run as admin if needed

Can't connect to API/Dashboard

• Service not running → qm start
• Wrong port → check qm config show for api.port
• Firewall blocking → allow port 8742

Logs not appearing

• Job hasn't started yet (pending)
• Log directory permissions → check %APPDATA%\idle-queue\logs\

Skill Chaining

Skills Required Before

• None

Input Expected

• Long-running command to execute
• Optional: name, working directory, force flag

Output Produced

• Format: Job ID and status
• Side effects: Creates job in queue, executes when appropriate
• Logs: Stored in %APPDATA%\idle-queue\logs/

Compatible Skills After

• Any skill that needs to wait for background processing

Tools Used

• Bash (usage: run qm commands)
• WebFetch (usage: API calls if needed)

Usage Example

Scenario: User asks to transcode videos

Command:

qm add "ffmpeg -i input.mp4 -c:v libx265 output.mp4" --name "Transcode to H265"

Result: Job queued, runs at low priority, notifications on completion