fritzprix Jobs Trace & Harness Bottleneck Analyzer
Analyze benchmark run trajectories stored under jobs/<job_timestamp>/ to
pinpoint harness bottlenecks in LibrAgent (Builtin Tools, Prompts, Circuit
Breakers, Timeouts, Context Compaction) that degrade scores, and formulate
evidence-backed improvement plans.
Relationship to harbor-harness-improvement-loop
This skill is a lighter inventory / categorization pass over jobs/<run>/.
| Concern | Use |
|---|---|
| Quick pass-rate, failure buckets, tool frequency | jobs-trace-analyzer (this skill) |
| Evidence-backed BM → fix → rerun cycles | harbor-harness-improvement-loop |
Harbor’s analyzer is stricter: verifier/reward.txt as reward SSOT, ATIF-first
metrics, and explicit heuristic ≠ verified failure. Prefer that skill when
proposing product changes. Do not treat this script’s categories as causal proof.
Workflow
1. Job Inventory & Automatic Trace Analysis
python .agents/skills/jobs-trace-analyzer/scripts/analyze_jobs_trace.py \
jobs/<job_timestamp> \
--output .libragent/work/trace-analysis/<job_timestamp>-analysis.md
Optional: --verbose prints per-task tool name lists.
The script reports:
- Total trials, pass rate (%), failed trial counts
- Failure categories (see below)
- Tool call frequency (ATIF-first when metadata trajectory is empty)
heuristic_error_observations(keyword hits in ATIF observation text — not verified tool failures)
Failure categories
| Category | Meaning |
|---|---|
SUCCESS |
Reward ≥ 1.0 |
TIMEOUT_EXCEEDED |
Exception type contains Timeout |
NETWORK_API_ERROR |
Connection / API network exception |
TOOL_EXECUTION_ERROR |
Structured tool error fields only (rare; not observation heuristics) |
INITIALIZATION_OR_EARLY_ABORT |
Zero turns |
AGENT_LOOP_OR_STUCK |
≥3 repeated adjacent identical tool calls (loop detected) |
HIGH_TURN_COUNT |
Turns > 30 without repeated-call evidence (inspect manually) |
VERIFIER_FAILED_WRONG_STATE |
Default failed bucket |
Tool extraction priority
agent_result.metadata.trajectorytool calls when present- Else ATIF
agent/trajectory.json→steps[].tool_calls[].function_name - Else
metadata.tool_calls_count(count only; names unavailable)
Reward prefers verifier/reward.txt, then verifier_result.rewards.reward.
2. Deep-Dive Trajectory Inspection (ATIF & Task Logs)
For each failed trial, inspect jobs/<job_timestamp>/<task_name>/:
result.json: rewards,exception_info/exception.txt, metadata turnsagent/trajectory.json(ATIF): first divergence turnverifier/reward.txt&trial.log: why verification failed
3. Harness Layer Mapping
| Symptom | Primary Owning Layer | Target Code |
|---|---|---|
| Tool parameter / schema error | Builtin MCP tools | src-tauri/src/mcp/builtin/ |
| Repeated tool loop | Circuit breaker / natural recovery | src-tauri/src/agent/llm/circuit_breaker.rs, natural_recovery.rs |
| Timeout / unfinished | Timeout multipliers / shell | src-tauri/src/mcp/builtin/workspace/utils.rs |
| Missing verification / premature end | Prompts | src-tauri/src/agent/prompt/ |
| Context loss / truncation | Compaction | src-tauri/src/agent/llm/completion/request/compact.rs |
4. Formulate Evidence-Based Improvement Plan
For each issue: symptom + trajectory evidence → root cause → smallest
intervention → verification (pnpm refactor:validate, cargo test, re-run
slice). Escalate to harbor-harness-improvement-loop for controlled BM cycles.
5. Report Output Location
Save reports under:
.libragent/work/trace-analysis/<job_name>-analysis.md