Parallel

Parallel Pool Management

Overview

Manage a pool of subagents with a precise concurrency limit, dispatching new tasks immediately as slots free up. Unlike fire-all-at-once approaches, this maintains steady parallelism without overwhelming systems.

When to Use

✅ Use when:

• Processing 5+ independent tasks (chapters, files, API calls, etc.)
• Need to limit concurrent execution (API rate limits, memory constraints, system overload)
• Want parallelism benefits without fire-all-at-once resource spikes
• Tasks can complete incrementally (don't need all results simultaneously)

❌ Don't use when:

• Tasks have sequential dependencies or shared state
• Only 1-4 tasks (just dispatch them directly)
• Need all results simultaneously before proceeding (use superpowers:dispatching-parallel-agents instead)

Core Pattern

Pool Management Algorithm

1. INITIALIZE: Set concurrency limit (default 3), create task queue
2. DISPATCH: Fire min(limit, queue_size) subagents initially
3. WAIT: Monitor for ANY completion (not polling - check notification)
4. REFILL: When slot frees, immediately dispatch next task from queue
5. REPEAT: Until queue empty AND all slots free

Critical: Refill slots immediately when ANY completes, not "wait for all then dispatch next batch."

Implementation Pattern

// Pool state management
const limit = 3;  // or custom limit from args
const queue = [...tasks];       // remaining tasks
const running = new Set();      // currently-executing agent IDs
const results = [];             // collected results

// Step 1: Initial dispatch (fill pool to limit)
for (let i = 0; i < Math.min(limit, queue.length); i++) {
  const task = queue.shift();
  const agent = dispatchAgent(task);
  running.add(agent.id);
}

// Step 2: Monitor and refill loop
while (running.size > 0) {
  // Wait for ANY completion to finish
  const completed = awaitNextCompletion(running);

  // Free the slot
  running.delete(completed.agentId);
  results.push(completed.result);

  // Refill immediately if queue has tasks
  if (queue.length > 0) {
    const next = queue.shift();
    const nextAgent = dispatchAgent(next);
    running.add(nextAgent.id);
  }
}

// running.size === 0 and queue.length === 0 means done

Error Handling

When a task fails, the slot must still free:

try {
  const result = await agentCompletion;
  results.push({success: true, data: result});
} catch (error) {
  results.push({success: false, error: error.message});
  // Slot still freed - running.delete() happens regardless
}

// Decide: retry (add back to queue) or log and skip?
if (shouldRetry(task, error)) {
  queue.push(task);  // will be re-dispatched
}

Principle: Failures are local. One task failing doesn't block the queue. Free the slot, decide retry/skip, continue.

Quick Reference

Common Mistakes

Self-Check Questions

Before using this pattern:

• Are tasks truly independent? (no shared state, no sequential dependencies)
• Do I need a concurrency limit? (rate limits, resource constraints, system stability)
• Do I have 5+ tasks? (otherwise just dispatch directly)
• Can I process incrementally? (don't need all results before proceeding)
• Will I track running state? (Set of agent IDs)

Comparison to Related Skills

• superpowers:dispatching-parallel-agents: Fire ALL agents at once in one block. Use when you need all results simultaneously and have no rate limits.
• superpowers:subagent-driven-development: Sequential execution (one task at a time). Use when tasks might conflict or need careful integration.
• parallel (this skill): Bounded pool with steady refill. Use when you need parallelism with resource constraints.

Why This Matters

Without pool management, agents either:

• Fire all at once → 50 concurrent agents overwhelm APIs, exhaust memory, trigger rate limits
• Process sequentially → 50 tasks take 50x longer than necessary

Pool management gives you controlled parallelism: steady throughput without resource spikes.