Hive Debugger

Hive Debugger

An interactive debugging companion that helps developers identify and fix runtime issues in Hive agents. The debugger analyzes runtime logs at three levels (L1/L2/L3), categorizes issues, and provides actionable fix recommendations.

When to Use This Skill

Use /hive-debugger when:

• Your agent is failing or producing unexpected results
• You need to understand why a specific node is retrying repeatedly
• Tool calls are failing and you need to identify the root cause
• Agent execution is stalled or taking too long
• You want to monitor agent behavior in real-time during development

This skill works alongside agents running in TUI mode and provides supervisor-level insights into execution behavior.

Prerequisites

Before using this skill, ensure:

1. You have an exported agent in exports/{agent_name}/
2. The agent has been run at least once (logs exist)
3. Runtime logging is enabled (default in Hive framework)
4. You have access to the agent's working directory at ~/.hive/agents/{agent_name}/

Workflow

Stage 1: Setup & Context Gathering

Objective: Understand the agent being debugged

What to do:

1. Ask the developer which agent needs debugging:

   • Get agent name (e.g., "twitter_outreach", "deep_research_agent")
   • Confirm the agent exists in exports/{agent_name}/

2. Determine agent working directory:

   • Calculate: ~/.hive/agents/{agent_name}/
   • Verify this directory exists and contains session logs

3. Read agent configuration:

   • Read file: exports/{agent_name}/agent.json
   • Extract goal information from the JSON:
     • goal.id - The goal identifier
     • goal.success_criteria - What success looks like
     • goal.constraints - Rules the agent must follow
   • Extract graph information:
     • List of node IDs from graph.nodes
     • List of edges from graph.edges

4. Store context for the debugging session:

   • agent_name
   • agent_work_dir (e.g., /home/user/.hive/twitter_outreach)
   • goal_id
   • success_criteria
   • constraints
   • node_ids

Example:

Developer: "My twitter_outreach agent keeps failing"

You: "I'll help debug the twitter_outreach agent. Let me gather context..."

[Read exports/twitter_outreach/agent.json]

Context gathered:
- Agent: twitter_outreach
- Goal: twitter-outreach-multi-loop
- Working Directory: /home/user/.hive/twitter_outreach
- Success Criteria: ["Successfully send 5 personalized outreach messages"]
- Constraints: ["Must verify handle exists", "Must personalize message"]
- Nodes: ["intake-collector", "profile-analyzer", "message-composer", "outreach-sender"]

Stage 2: Mode Selection

Objective: Choose the debugging approach that best fits the situation

What to do:

Ask the developer which debugging mode they want to use. Use AskUserQuestion with these options:

1. Real-time Monitoring Mode

   • Description: Monitor active TUI session continuously, poll logs every 5-10 seconds, alert on new issues immediately
   • Best for: Live debugging sessions where you want to catch issues as they happen
   • Note: Requires agent to be currently running

2. Post-Mortem Analysis Mode

   • Description: Analyze completed or failed runs in detail, deep dive into specific session
   • Best for: Understanding why a past execution failed
   • Note: Most common mode for debugging

3. Historical Trends Mode

   • Description: Analyze patterns across multiple runs, identify recurring issues
   • Best for: Finding systemic problems that happen repeatedly
   • Note: Useful for agents that have run many times

Implementation:

Use AskUserQuestion to present these options and let the developer choose.
Store the selected mode for the session.

Stage 3: Triage (L1 Analysis)

Objective: Identify which sessions need attention

What to do:

1. Query high-level run summaries using the MCP tool:

   query_runtime_logs(
       agent_work_dir="{agent_work_dir}",
       status="needs_attention",
       limit=20
   )
   

2. Analyze the results:

   • Look for runs with needs_attention: true
   • Check attention_summary.categories for issue types
   • Note the run_id of problematic sessions
   • Check status field: "degraded", "failure", "in_progress"

3. Attention flag triggers to understand: From runtime_logger.py, runs are flagged when:

   • retry_count > 3
   • escalate_count > 2
   • latency_ms > 60000
   • tokens_used > 100000
   • total_steps > 20

4. Present findings to developer:

   • Summarize how many runs need attention
   • List the most recent problematic runs
   • Show attention categories for each
   • Ask which run they want to investigate (if multiple)

Example Output:

Found 2 runs needing attention:

1. session_20260206_115718_e22339c5 (30 minutes ago)
   Status: degraded
   Categories: missing_outputs, retry_loops

2. session_20260206_103422_9f8d1b2a (2 hours ago)
   Status: failure
   Categories: tool_failures, high_latency

Which run would you like to investigate?

Stage 4: Diagnosis (L2 Analysis)

Objective: Identify which nodes failed and what patterns exist

What to do:

1. Query per-node details using the MCP tool:

   query_runtime_log_details(
       agent_work_dir="{agent_work_dir}",
       run_id="{selected_run_id}",
       needs_attention_only=True
   )
   

2. Categorize issues using the Issue Taxonomy:

   10 Issue Categories:

   Category	Detection Pattern	Meaning
   Missing Outputs	exit_status != "success", attention_reasons contains "missing_outputs"	Node didn't call set_output with required keys
   Tool Errors	tool_error_count > 0, attention_reasons contains "tool_failures"	Tool calls failed (API errors, timeouts, auth issues)
   Retry Loops	retry_count > 3, verdict_counts.RETRY > 5	Judge repeatedly rejecting outputs
   Guard Failures	guard_reject_count > 0	Output validation failed (wrong types, missing keys)
   Stalled Execution	total_steps > 20, verdict_counts.CONTINUE > 10	EventLoopNode not making progress
   High Latency	latency_ms > 60000, avg_step_latency > 5000	Slow tool calls or LLM responses
   Client-Facing Issues	client_input_requested but no user_input_received	Premature set_output before user input
   Edge Routing Errors	exit_status == "no_valid_edge", attention_reasons contains "routing_issue"	No edges match current state
   Memory/Context Issues	tokens_used > 100000, context_overflow_count > 0	Conversation history too long
   Constraint Violations	Compare output against goal constraints	Agent violated goal-level rules

3. Analyze each flagged node:

   • Node ID and name
   • Exit status
   • Retry count
   • Verdict distribution (ACCEPT/RETRY/ESCALATE/CONTINUE)
   • Attention reasons
   • Total steps executed

4. Present diagnosis to developer:

   • List problematic nodes
   • Categorize each issue
   • Highlight the most severe problems
   • Show evidence (retry counts, error types)

Example Output:

Diagnosis for session_20260206_115718_e22339c5:

Problem Node: intake-collector
├─ Exit Status: escalate
├─ Retry Count: 5 (HIGH)
├─ Verdict Counts: {RETRY: 5, ESCALATE: 1}
├─ Attention Reasons: ["high_retry_count", "missing_outputs"]
├─ Total Steps: 8
└─ Categories: Missing Outputs + Retry Loops

Root Issue: The intake-collector node is stuck in a retry loop because it's not setting required outputs.

Stage 5: Root Cause Analysis (L3 Analysis)

Objective: Understand exactly what went wrong by examining detailed logs

What to do:

1. Query detailed tool/LLM logs using the MCP tool:

   query_runtime_log_raw(
       agent_work_dir="{agent_work_dir}",
       run_id="{run_id}",
       node_id="{problem_node_id}"
   )
   

2. Analyze based on issue category:

   For Missing Outputs:

   • Check step.tool_calls for set_output usage
   • Look for conditional logic that skipped set_output
   • Check if LLM is calling other tools instead

   For Tool Errors:

   • Check step.tool_results for error messages
   • Identify error types: rate limits, auth failures, timeouts, network errors
   • Note which specific tool is failing

   For Retry Loops:

   • Check step.verdict_feedback from judge
   • Look for repeated failure reasons
   • Identify if it's the same issue every time

   For Guard Failures:

   • Check step.guard_results for validation errors
   • Identify missing keys or type mismatches
   • Compare actual output to expected schema

   For Stalled Execution:

   • Check step.llm_response_text for repetition
   • Look for LLM stuck in same action loop
   • Check if tool calls are succeeding but not progressing

3. Extract evidence:

   • Specific error messages
   • Tool call arguments and results
   • LLM response text
   • Judge feedback
   • Step-by-step progression

4. Formulate root cause explanation:

   • Clearly state what is happening
   • Explain why it's happening
   • Show evidence from logs

Example Output:

Root Cause Analysis for intake-collector:

Step-by-step breakdown:

Step 3:
- Tool Call: web_search(query="@RomuloNevesOf")
- Result: Found Twitter profile information
- Verdict: RETRY
- Feedback: "Missing required output 'twitter_handles'. You found the handle but didn't call set_output."

Step 4:
- Tool Call: web_search(query="@RomuloNevesOf twitter")
- Result: Found additional Twitter information
- Verdict: RETRY
- Feedback: "Still missing 'twitter_handles'. Use set_output to save your findings."

Steps 5-7: Similar pattern continues...

ROOT CAUSE: The node is successfully finding Twitter handles via web_search, but the LLM is not calling set_output to save the results. It keeps searching for more information instead of completing the task.

Stage 6: Fix Recommendations

Objective: Provide actionable solutions the developer can implement

What to do:

Based on the issue category identified, provide specific fix recommendations using these templates:

Template 1: Missing Outputs (Client-Facing Nodes)

## Issue: Premature set_output in Client-Facing Node

**Root Cause:** Node called set_output before receiving user input

**Fix:** Use STEP 1/STEP 2 prompt pattern

**File to edit:** `exports/{agent_name}/nodes/{node_name}.py`

**Changes:**
1. Update the system_prompt to include explicit step guidance:
   ```python
   system_prompt = """
   STEP 1: Analyze the user input and decide what action to take.
   DO NOT call set_output in this step.

   STEP 2: After receiving feedback or completing analysis,
   ONLY THEN call set_output with your results.
   """

2. If some inputs are optional (like feedback on retry edges), add nullable_output_keys:

   nullable_output_keys=["feedback"]
   

Verification:

• Run the agent with test input
• Verify the client-facing node waits for user input before calling set_output

#### Template 2: Retry Loops

```markdown
## Issue: Judge Repeatedly Rejecting Outputs

**Root Cause:** {Insert specific reason from verdict_feedback}

**Fix Options:**

**Option A - If outputs are actually correct:** Adjust judge evaluation rules
- File: `exports/{agent_name}/agent.json`
- Update `evaluation_rules` section to accept the current output format
- Example: If judge expects list but gets string, update rule to accept both

**Option B - If prompt is ambiguous:** Clarify node instructions
- File: `exports/{agent_name}/nodes/{node_name}.py`
- Make system_prompt more explicit about output format and requirements
- Add examples of correct outputs

**Option C - If tool is unreliable:** Add retry logic with fallback
- Consider using alternative tools
- Add manual fallback option
- Update prompt to handle tool failures gracefully

**Verification:**
- Run the node with test input
- Confirm judge accepts output on first try
- Check that retry_count stays at 0

Template 3: Tool Errors

## Issue: {tool_name} Failing with {error_type}

**Root Cause:** {Insert specific error message from logs}

**Fix Strategy:**

**If API rate limit:**
1. Add exponential backoff in tool retry logic
2. Reduce API call frequency
3. Consider caching results

**If auth failure:**
1. Check credentials using:
   ```bash
   /hive-credentials --agent {agent_name}

2. Verify API key environment variables
3. Update mcp_servers.json if needed

If timeout:

1. Increase timeout in mcp_servers.json:

   {
     "timeout_ms": 60000
   }
   

2. Consider using faster alternative tools
3. Break large requests into smaller chunks

Verification:

• Test tool call manually
• Confirm successful response
• Monitor for recurring errors

#### Template 4: Edge Routing Errors

```markdown
## Issue: No Valid Edge from Node {node_id}

**Root Cause:** No edge condition matched the current state

**File to edit:** `exports/{agent_name}/agent.json`

**Analysis:**
- Current node output: {show actual output keys}
- Existing edge conditions: {list edge conditions}
- Why no match: {explain the mismatch}

**Fix:**
Add the missing edge to the graph:
```json
{
  "edge_id": "{node_id}_to_{target_node}",
  "source": "{node_id}",
  "target": "{target_node}",
  "condition": "on_success"
}

Alternative: Update existing edge condition to cover this case

Verification:

• Run agent with same input
• Verify edge is traversed successfully
• Check that execution continues to next node

#### Template 5: Stalled Execution

```markdown
## Issue: EventLoopNode Not Making Progress

**Root Cause:** {Insert analysis - e.g., "LLM repeating same failed action"}

**File to edit:** `exports/{agent_name}/nodes/{node_name}.py`

**Fix:** Update system_prompt to guide LLM out of loops

**Add this guidance:**
```python
system_prompt = """
{existing prompt}

IMPORTANT: If a tool call fails multiple times:
1. Try an alternative approach or different tool
2. If no alternatives work, call set_output with partial results
3. DO NOT retry the same failed action more than 3 times

Progress is more important than perfection. Move forward even with incomplete data.
"""

Additional fix: Lower max_iterations to prevent infinite loops

# In node configuration
max_node_visits=3  # Prevent getting stuck

Verification:

• Run node with same input that caused stall
• Verify it exits after reasonable attempts (< 10 steps)
• Confirm it calls set_output eventually

#### Template 6: Checkpoint Recovery (Post-Fix Resume)

```markdown
## Recovery Strategy: Resume from Last Clean Checkpoint

**Situation:** You've fixed the issue, but the failed session is stuck mid-execution

**Solution:** Resume execution from a checkpoint before the failure

### Option A: Auto-Resume from Latest Checkpoint (Recommended)

Use CLI arguments to auto-resume when launching TUI:

```bash
PYTHONPATH=core:exports python -m {agent_name} --tui \
    --resume-session {session_id}

This will:

• Load session state from state.json
• Continue from where it paused/failed
• Apply your fixes immediately

Option B: Resume from Specific Checkpoint (Time-Travel)

If you need to go back to an earlier point:

PYTHONPATH=core:exports python -m {agent_name} --tui \
    --resume-session {session_id} \
    --checkpoint {checkpoint_id}

Example:

PYTHONPATH=core:exports python -m deep_research_agent --tui \
    --resume-session session_20260208_143022_abc12345 \
    --checkpoint cp_node_complete_intake_143030

Option C: Use TUI Commands

Alternatively, launch TUI normally and use commands:

# Launch TUI
PYTHONPATH=core:exports python -m {agent_name} --tui

# In TUI, use commands:
/resume {session_id}                    # Resume from session state
/recover {session_id} {checkpoint_id}   # Recover from specific checkpoint

When to Use Each Option:

Use /resume (or --resume-session) when:

• You fixed credentials and want to retry
• Agent paused and you want to continue
• Agent failed and you want to retry from last state

Use /recover (or --resume-session + --checkpoint) when:

• You need to go back to an earlier checkpoint
• You want to try a different path from a specific point
• Debugging requires time-travel to earlier state

Find Available Checkpoints:

# In TUI:
/sessions {session_id}

# This shows all checkpoints with timestamps:
Available Checkpoints: (3)
  1. cp_node_complete_intake_143030
  2. cp_node_complete_research_143115
  3. cp_pause_research_143130

Verification:

• Use --resume-session to test your fix immediately
• No need to re-run from the beginning
• Session continues with your code changes applied

**Selecting the right template:**
- Match the issue category from Stage 4
- Customize with specific details from Stage 5
- Include actual error messages and code snippets
- Provide file paths and line numbers when possible
- **Always include recovery commands** (Template 6) after providing fix recommendations

---

### Stage 7: Verification Support

**Objective:** Help the developer confirm their fixes work

**What to do:**

1. **Suggest appropriate tests based on fix type:**

   **For node-level fixes:**
   ```bash
   # Use hive-test to run goal-based tests
   /hive-test --agent {agent_name} --goal {goal_id}

   # Or run specific test scenarios
   /hive-test --agent {agent_name} --scenario {specific_input}

For quick manual tests:

# Launch the interactive TUI dashboard
hive tui

Then use arrow keys to select the agent from the list and press Enter to run it.

2. Provide MCP tool queries to validate the fix:

   Check if issue is resolved:

   query_runtime_logs(
       agent_work_dir="~/.hive/agents/{agent_name}",
       status="needs_attention",
       limit=5
   )
   # Should show 0 results if fully fixed
   

   Verify specific node behavior:

   query_runtime_log_details(
       agent_work_dir="~/.hive/agents/{agent_name}",
       run_id="{new_run_id}",
       node_id="{fixed_node_id}"
   )
   # Should show exit_status="success", retry_count=0
   

3. Monitor for regression:

   • Run the agent multiple times
   • Check for similar issues reappearing
   • Verify fix works across different inputs

4. Provide verification checklist:

   Verification Checklist:
   □ Applied recommended fix to code
   □ Ran agent with test input
   □ Checked runtime logs show no attention flags
   □ Verified specific node completes successfully
   □ Tested with multiple inputs
   □ No regression of original issue
   □ Agent meets success criteria
   

Example interaction:

Developer: "I applied the fix to intake-collector. How do I verify it works?"

You: "Great! Let's verify the fix with these steps:

1. Launch the TUI dashboard:
   hive tui
   Then select your agent from the list and press Enter to run it.

2. After it completes, check the logs:
   [Use query_runtime_logs to check for attention flags]

3. Verify the specific node:
   [Use query_runtime_log_details for intake-collector]

Expected results:
- No 'needs_attention' flags
- intake-collector shows exit_status='success'
- retry_count should be 0

Let me know when you've run it and I'll help check the logs!"

MCP Tool Usage Guide

Three Levels of Observability

L1: query_runtime_logs - Session-level summaries

• When to use: Initial triage, identifying problematic runs, monitoring trends
• Returns: List of runs with status, attention flags, timestamps
• Example:

  query_runtime_logs(
      agent_work_dir="/home/user/.hive/twitter_outreach",
      status="needs_attention",
      limit=20
  )
  

L2: query_runtime_log_details - Node-level details

• When to use: Diagnosing which nodes failed, understanding retry patterns
• Returns: Per-node completion details, retry counts, verdicts
• Example:

  query_runtime_log_details(
      agent_work_dir="/home/user/.hive/twitter_outreach",
      run_id="session_20260206_115718_e22339c5",
      needs_attention_only=True
  )
  

L3: query_runtime_log_raw - Step-level details

• When to use: Root cause analysis, understanding exact failures
• Returns: Full tool calls, LLM responses, judge feedback
• Example:

  query_runtime_log_raw(
      agent_work_dir="/home/user/.hive/twitter_outreach",
      run_id="session_20260206_115718_e22339c5",
      node_id="intake-collector"
  )
  

Query Patterns

Pattern 1: Top-Down Investigation (Most common)

1. L1: Find problematic runs
2. L2: Identify failing nodes
3. L3: Analyze specific failures

Pattern 2: Node-Specific Debugging

1. L2: Get details for specific node across all runs
2. L3: Deep dive into worst failures

Pattern 3: Real-time Monitoring

Loop every 10 seconds:
  1. L1: Check for new needs_attention runs
  2. If found: Alert and drill into L2

Complete Example Walkthrough

Scenario: twitter_outreach Agent Failing

Developer says: "My twitter_outreach agent keeps failing. It's not finding Twitter handles correctly."

Stage 1: Setup

You: "I'll help debug the twitter_outreach agent. Let me gather context..."

[Read exports/twitter_outreach/agent.json]

Context:
- Agent: twitter_outreach
- Goal: twitter-outreach-multi-loop
- Working Dir: ~/.hive/agents/twitter_outreach
- Success Criteria: ["Successfully send 5 personalized outreach messages"]
- Constraints: ["Must verify handle exists", "Must personalize message"]
- Nodes: intake-collector, profile-analyzer, message-composer, outreach-sender

Stage 2: Mode Selection

You: "Which debugging approach would you prefer?"

[Present 3 mode options using AskUserQuestion]

Developer selects: "Post-Mortem Analysis Mode"

Stage 3: Triage (L1)

You: "Querying runtime logs for problematic runs..."

[Call query_runtime_logs with status="needs_attention"]

Results:
- Found 1 run needing attention
- session_20260206_115718_e22339c5 (30 min ago)
- Status: degraded
- Categories: missing_outputs, retry_loops

Stage 4: Diagnosis (L2)

You: "Analyzing node-level details..."

[Call query_runtime_log_details for the session]

Results:
- Problem Node: intake-collector
- Exit Status: escalate
- Retry Count: 5
- Verdict Counts: {RETRY: 5, ESCALATE: 1}
- Attention Reasons: ["high_retry_count", "missing_outputs"]
- Total Steps: 8

Diagnosis: intake-collector has retry loop + missing outputs issue
Category: Missing Outputs + Retry Loops

Stage 5: Root Cause (L3)

You: "Drilling into detailed logs for intake-collector..."

[Call query_runtime_log_raw for node intake-collector]

Evidence:
Step 3: web_search("@RomuloNevesOf") → Found profile
  Verdict: RETRY
  Feedback: "Missing required output 'twitter_handles'. You found the handle but didn't call set_output."

Step 4: web_search("@RomuloNevesOf twitter") → Found more info
  Verdict: RETRY
  Feedback: "Still missing 'twitter_handles'. Use set_output to save your findings."

Steps 5-7: Same pattern repeats...

ROOT CAUSE: Node successfully finds handles but LLM doesn't call set_output. It keeps searching instead of completing.

Stage 6: Fix Recommendation

You: "Here's the fix for this issue:

## Issue: intake-collector Not Setting Outputs

**Root Cause:** Node successfully finds Twitter handles but doesn't call set_output

**Category:** Missing Outputs

**File to edit:** exports/twitter_outreach/nodes/intake_collector.py

**Changes:**
Update the system_prompt to emphasize the output requirement:

```python
system_prompt = """
You are the Intake Collector for the Twitter outreach agent.

Your task:
1. Extract Twitter handle from user input (URL or @mention)
2. Verify it's a valid Twitter handle format
3. **CRITICAL: Call set_output(twitter_handles=[handle]) immediately after finding valid handle**

IMPORTANT: You MUST call set_output even if you found the handle on the first try.
Do not continue searching once you have a valid handle - set output and finish.

Output format:
{
  "twitter_handles": ["@handle1", "@handle2", ...]
}
"""

Why this works: The explicit instruction to call set_output immediately prevents the LLM from continuing to search after finding the handle.

#### Stage 7: Verification

Developer: "I updated the prompt. How do I test it?"

You: "Let's verify the fix:

1. Launch the TUI dashboard: hive tui Then select your agent and press Enter to run it.

2. I'll check the logs after it runs..."

[Developer runs agent]

Developer: "It completed! Can you check if it worked?"

You: "Checking now..."

[Call query_runtime_log_details for the new run]

Results:

• Node: intake-collector
• Exit Status: success ✓
• Retry Count: 0 ✓
• Total Steps: 2 ✓
• No attention flags ✓

"Perfect! The fix worked. The intake-collector now:

• Completes successfully on first try
• No retries needed
• Calls set_output properly

Your agent should now work correctly!"

---

## Tips for Effective Debugging

1. **Always start with L1 logs** - Don't jump straight to detailed logs
2. **Focus on attention flags** - They highlight the real issues
3. **Compare verdict_feedback across steps** - Patterns reveal root causes
4. **Check tool error messages carefully** - They often contain the exact problem
5. **Consider the agent's goal** - Fixes should align with success criteria
6. **Test fixes immediately** - Quick verification prevents wasted effort
7. **Look for patterns across multiple runs** - One-time failures might be transient

## Common Pitfalls to Avoid

1. **Don't recommend code you haven't verified exists** - Always read files first
2. **Don't assume tool capabilities** - Check MCP server configs
3. **Don't ignore edge conditions** - Missing edges cause routing failures
4. **Don't overlook judge configuration** - Mismatched expectations cause retry loops
5. **Don't forget nullable_output_keys** - Optional inputs need explicit marking

---

## Storage Locations Reference

**New unified storage (default):**
- Logs: `~/.hive/agents/{agent_name}/sessions/session_YYYYMMDD_HHMMSS_{uuid}/logs/`
- State: `~/.hive/agents/{agent_name}/sessions/{session_id}/state.json`
- Conversations: `~/.hive/agents/{agent_name}/sessions/{session_id}/conversations/`

**Old storage (deprecated, still supported):**
- Logs: `~/.hive/agents/{agent_name}/runtime_logs/runs/{run_id}/`

The MCP tools automatically check both locations.

---

**Remember:** Your role is to be a debugging companion and thought partner. Guide the developer through the investigation, explain what you find, and provide actionable fixes. Don't just report errors - help understand and solve them.