# Instrumentation Planning

> Plan what to measure in AI agent systems using tiered approach

- Skill: `nexus-labs-automation/instrumentation-planning` (Agent Skill)
- Install (CLI): `npx skillmds@latest add nexus-labs-automation/instrumentation-planning`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nexus-labs-automation/instrumentation-planning/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML, Coding & Dev Tools
- Author: nexus-labs-automation (https://skillmd.com/u/nexus-labs-automation)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/nexus-labs-automation/instrumentation-planning

---


# Instrumentation Planning for Agents

Plan agent observability using a tiered, outcome-focused approach.

## Core Principle

Every metric and span should answer one of these questions:
1. **Did the agent complete its task?** (success/failure)
2. **How long did it take?** (latency)
3. **How much did it cost?** (tokens/money)
4. **Why did it fail?** (error context)
5. **What decisions did it make?** (reasoning trace)

## 5-Tier Implementation Framework

### Tier 0: Foundation (Day 1)
Essential observability to ship any agent:
- SDK initialization
- Root span for agent runs
- Unhandled error capture
- Basic success/failure status

### Tier 1: Core Tracing (Week 1)
Understand agent execution:
- LLM call spans (model, latency)
- Tool execution spans (name, result)
- Agent loop iterations
- Retry attempts

### Tier 2: Context & Attribution (Week 2)
Track costs and ownership:
- Token counts (input/output/total)
- Cost per call (USD)
- User/session context
- Feature/workflow attribution

### Tier 3: Multi-Agent Coordination (Week 3)
For multi-agent systems:
- Parent-child span relationships
- Agent handoff tracking
- Delegation reasoning
- Supervisor decisions

### Tier 4: Evaluation & Quality (Month 1)
Measure agent quality:
- Response quality scores
- Human feedback capture
- Automated eval results
- Hallucination detection signals

## What NOT to Instrument

- Full prompt/response content (PII, storage cost)
- Every intermediate thought (noise)
- Timestamps as attributes (use span timing)
- User-provided secrets

## Span Naming Convention

Use semantic, hierarchical names:
```
agent.run              # Root agent execution
agent.think            # Reasoning step
llm.call               # LLM API call
llm.stream             # Streaming LLM call
tool.execute           # Tool execution
tool.validate          # Tool input validation
retrieval.search       # RAG retrieval
retrieval.rerank       # Reranking step
memory.read            # Memory fetch
memory.write           # Memory store
handoff.delegate       # Agent delegation
handoff.receive        # Receiving delegation
human.request          # Human approval request
human.response         # Human response received
eval.score             # Evaluation scoring
```

## Attribute Naming Convention

Use dot-notation, consistent types:
```
# Agent context
agent.name             # string: "researcher"
agent.type             # string: "langgraph"
agent.run_id           # string: UUID

# LLM context
llm.model              # string: "claude-3-opus"
llm.provider           # string: "anthropic"
llm.temperature        # float: 0.7
llm.tokens.input       # int: 1500
llm.tokens.output      # int: 350
llm.tokens.total       # int: 1850
llm.cost_usd           # float: 0.025
llm.latency_ms         # int: 2340

# Tool context
tool.name              # string: "web_search"
tool.success           # bool: true
tool.error             # string: error message
tool.latency_ms        # int: 450

# User context
user.id                # string: hash or ID
session.id             # string: session UUID
```

## Sampling Strategy

For high-volume agents:
- **100%**: Errors, slow requests (>P95), human escalations
- **10-25%**: Normal successful runs
- **1%**: High-frequency background tasks

Configure sampling at SDK level, not in code.

## Related Skills
- `llm-call-tracing` - LLM instrumentation details
- `tool-call-tracking` - Tool execution patterns
- `token-cost-tracking` - Cost calculation methods