Debug agent execution: $ARGUMENTS
Before Starting
- Read the tracing infrastructure:
src/agents/shared/agent-tracer.ts — AgentTracer class, event recording
src/agents/shared/hooks.ts — lifecycle hooks (onStart, onToolCall, onToolComplete, onComplete, onError)
src/lib/database/schema.ts — search for agent_traces and agent_metrics tables
Trace Event Types
The AgentTracer records these events during execution:
| Event |
When |
Data |
start |
Execution begins |
message length, context availability |
tool_call |
Tool invoked |
toolName, params preview (truncated 200 chars) |
tool_complete |
Tool returns |
toolName, duration, success flag |
handoff |
Agent delegates |
target agent, reason |
stream_delta |
First text token |
(recorded once only) |
complete |
Execution done |
output length, duration, tool count |
error |
Failure |
message, stack (truncated) |
Querying Traces
Recent traces for an agent
SELECT id, agent_name, trace_type, duration, tool_calls_count, error_message,
created_at
FROM agent_traces
WHERE agent_name = '<agent-name>'
ORDER BY created_at DESC
LIMIT 20;
Traces with errors
SELECT id, agent_name, error_message, trace_data, created_at
FROM agent_traces
WHERE error_message IS NOT NULL
ORDER BY created_at DESC
LIMIT 10;
Slow traces (over 10s)
SELECT id, agent_name, duration, tool_calls_count, trace_data, created_at
FROM agent_traces
WHERE duration > 10000
ORDER BY duration DESC
LIMIT 10;
Tool call breakdown
SELECT
trace_data->'events' AS events,
tool_calls_count,
duration
FROM agent_traces
WHERE agent_name = '<agent-name>'
AND tool_calls_count > 0
ORDER BY created_at DESC
LIMIT 5;
Agent metrics summary
SELECT agent_name, date,
usage_count, avg_response_time, success_rate,
cost_total, tokens_total, error_count
FROM agent_metrics
WHERE agent_name = '<agent-name>'
ORDER BY date DESC
LIMIT 30;
Debug Flags
Set these environment variables to enable verbose logging:
AI_LAB_DEBUG_TOOLS=true # Log tool selection & validation
AI_LAB_DEBUG_CONTEXT=true # Log context receipt & changes
AI_LAB_DEBUG_INSTRUCTIONS=true # Log assembled instructions (caution: large)
AI_LAB_DEBUG_RETRIEVAL=true # Log file_search queries, sources, scores
Analysis Patterns
Diagnosing slow responses
- Query traces with high duration
- Check
events array for tool_call → tool_complete durations
- Identify which tool is the bottleneck
- Check if the tool is hitting an external service (file_search, search API)
- Consider caching or timeout adjustments
Diagnosing tool failures
- Query traces where
error_message is not null
- Look at the
tool_call event immediately before the error event
- Check if the tool params were valid
- Check if the external service was available
Diagnosing guardrail rejections
- Guardrail rejections show as
error events with the guardrail name
- Check the
reason field for why the message was rejected
- Review the input message and guardrail logic
Comparing agent performance over time
- Query
agent_metrics table for daily aggregates
- Check trends in avg_response_time, success_rate, error_count
- Correlate spikes with deployment dates or config changes
Rules
- Use the Supabase MCP (
mcp__supabase__execute_sql) for querying traces directly
- Never delete traces — they're the audit trail for agent behavior
- When reporting findings, include specific trace IDs for reference
- Debug flags should only be enabled temporarily — they generate large log volumes
- If traces reveal a systemic issue, recommend a fix in the relevant layer (tool, instructions, guardrail)
1---2name: agent-trace3description: Debug agent execution with tracing — analyze traces, tool calls, durations, errors, and performance metrics. Use when investigating agent behavior.4user-invocable: true5allowed-tools: Read, Grep, Glob, Bash, Agent, mcp__supabase__execute_sql, mcp__openai-docs__search_openai_docs6metadata:7 source: https://skillmd.com/skills/joshuashepherd/agent-trace8---910Debug agent execution: $ARGUMENTS1112## Before Starting13141. Read the tracing infrastructure:15 - `src/agents/shared/agent-tracer.ts` — `AgentTracer` class, event recording16 - `src/agents/shared/hooks.ts` — lifecycle hooks (onStart, onToolCall, onToolComplete, onComplete, onError)17 - `src/lib/database/schema.ts` — search for `agent_traces` and `agent_metrics` tables1819## Trace Event Types2021The `AgentTracer` records these events during execution:2223| Event | When | Data |24|-------|------|------|25| `start` | Execution begins | message length, context availability |26| `tool_call` | Tool invoked | toolName, params preview (truncated 200 chars) |27| `tool_complete` | Tool returns | toolName, duration, success flag |28| `handoff` | Agent delegates | target agent, reason |29| `stream_delta` | First text token | (recorded once only) |30| `complete` | Execution done | output length, duration, tool count |31| `error` | Failure | message, stack (truncated) |3233## Querying Traces3435### Recent traces for an agent36```sql37SELECT id, agent_name, trace_type, duration, tool_calls_count, error_message,38 created_at39FROM agent_traces40WHERE agent_name = '<agent-name>'41ORDER BY created_at DESC42LIMIT 20;43```4445### Traces with errors46```sql47SELECT id, agent_name, error_message, trace_data, created_at48FROM agent_traces49WHERE error_message IS NOT NULL50ORDER BY created_at DESC51LIMIT 10;52```5354### Slow traces (over 10s)55```sql56SELECT id, agent_name, duration, tool_calls_count, trace_data, created_at57FROM agent_traces58WHERE duration > 1000059ORDER BY duration DESC60LIMIT 10;61```6263### Tool call breakdown64```sql65SELECT66 trace_data->'events' AS events,67 tool_calls_count,68 duration69FROM agent_traces70WHERE agent_name = '<agent-name>'71 AND tool_calls_count > 072ORDER BY created_at DESC73LIMIT 5;74```7576### Agent metrics summary77```sql78SELECT agent_name, date,79 usage_count, avg_response_time, success_rate,80 cost_total, tokens_total, error_count81FROM agent_metrics82WHERE agent_name = '<agent-name>'83ORDER BY date DESC84LIMIT 30;85```8687## Debug Flags8889Set these environment variables to enable verbose logging:9091```bash92AI_LAB_DEBUG_TOOLS=true # Log tool selection & validation93AI_LAB_DEBUG_CONTEXT=true # Log context receipt & changes94AI_LAB_DEBUG_INSTRUCTIONS=true # Log assembled instructions (caution: large)95AI_LAB_DEBUG_RETRIEVAL=true # Log file_search queries, sources, scores96```9798## Analysis Patterns99100### Diagnosing slow responses1011. Query traces with high duration1022. Check `events` array for tool_call → tool_complete durations1033. Identify which tool is the bottleneck1044. Check if the tool is hitting an external service (file_search, search API)1055. Consider caching or timeout adjustments106107### Diagnosing tool failures1081. Query traces where `error_message` is not null1092. Look at the `tool_call` event immediately before the `error` event1103. Check if the tool params were valid1114. Check if the external service was available112113### Diagnosing guardrail rejections1141. Guardrail rejections show as `error` events with the guardrail name1152. Check the `reason` field for why the message was rejected1163. Review the input message and guardrail logic117118### Comparing agent performance over time1191. Query `agent_metrics` table for daily aggregates1202. Check trends in avg_response_time, success_rate, error_count1213. Correlate spikes with deployment dates or config changes122123## Rules124125- Use the Supabase MCP (`mcp__supabase__execute_sql`) for querying traces directly126- Never delete traces — they're the audit trail for agent behavior127- When reporting findings, include specific trace IDs for reference128- Debug flags should only be enabled temporarily — they generate large log volumes129- If traces reveal a systemic issue, recommend a fix in the relevant layer (tool, instructions, guardrail)