modbender Agent Observability Dashboard ๐
Unified observability for OpenClaw agents โ metrics, traces, and performance insights.
What It Does
OpenClaw agents need production-grade visibility. Multiple platforms exist (Langfuse, Langsmith, AgentOps) but no unified view.
Agent Observability Dashboard provides:
- Metrics tracking โ Latency, success rate, token usage, error counts
- Trace visualization โ Tool chains, decision flows, session timelines
- Cross-agent aggregation โ Compare performance across multiple agents/sessions
- Exportable reports โ JSON, CSV, markdown for human review
- Alert thresholds โ Notify when metrics exceed limits
Problem It Solves
- No centralized view of OpenClaw agent performance
- Hard to debug across multiple tool calls
- No way to compare agents or track regressions
- Production monitoring is enterprise-grade; agents need the same
Usage
# Start dashboard server
python3 scripts/observability.py --dashboard
# Record metrics from a session
python3 scripts/observability.py --record --session agent:main --latency 1.5 --success true
# View session trace
python3 scripts/observability.py --trace --session agent:main:12345
# Get performance report
python3 scripts/observability.py --report --period 24h
# Export to CSV
python3 scripts/observability.py --export metrics.csv
# Set alert thresholds
python3 scripts/observability.py --alert --metric latency --threshold 5.0
Metrics Tracked
| Category | Metric | Description |
|---|---|---|
| Performance | Latency | Tool call latency (ms) |
| Throughput | Calls per second | |
| Success | Success Rate | % of successful tool calls |
| Error Count | Failed operations | |
| Cost | Token Usage | Input + output tokens |
| API Cost | Estimated cost in USD | |
| Quality | Hallucinations | Detected false outputs |
| Corrections Needed | User corrections |
Trace Format
Each tool call is logged with:
- Timestamp
- Agent session ID
- Tool name + parameters
- Latency
- Success/failure
- Token usage
- Error details (if failed)
Example trace:
{
"session_id": "agent:main:12345",
"trace": [
{
"timestamp": "2026-01-31T14:00:00Z",
"tool": "web_search",
"params": {"query": "agent observability"},
"latency_ms": 1234,
"success": true,
"tokens_used": 150
},
{
"timestamp": "2026-01-31T14:00:02Z",
"tool": "memory_write",
"params": {"content": "..."},
"latency_ms": 45,
"success": true,
"tokens_used": 0
}
]
}
Architecture
โโโโโโโโโโโโโโโโโโโ
โ Instrumentationโ โ Auto-capture from OpenClaw logs
โโโโโโโโโโฌโโโโโโโโโ
โ
โผ
โโโโโโโโโโโโโโโโโโโ
โ Metrics Store โ โ SQLite/InfluxDB for time-series
โโโโโโโโโโฌโโโโโโโโโ
โ
โผ
โโโโโโโโโโโโโโโโโโโ
โ Analytics โ โ Aggregations, trends, anomalies
โโโโโโโโโโฌโโโโโโโโโ
โ
โผ
โโโโโโโโโโโโโโโโโโโ
โ Dashboard UI โ โ Web interface (Flask/FastAPI)
โโโโโโโโโโโโโโโโโโโ
Requirements
- Python 3.9+
- flask (for dashboard web UI)
- pandas (for analytics)
- influxdb-client (optional, for production storage)
Installation
# Clone repo
git clone https://github.com/orosha-ai/agent-observability-dashboard
# Install dependencies
pip install flask pandas influxdb-client
# Run dashboard
python3 scripts/observability.py --dashboard
# Open http://localhost:5000
Inspiration
- Dynatrace AI Observability App โ Enterprise-grade unified observability
- Langfuse vs AgentOps benchmarks โ Comparison of platforms
- Microsoft .NET tracing guide โ Practical implementation patterns
- OpenLLMetry โ OpenTelemetry integration for LLMs
Local-Only Promise
- Metrics stored locally (SQLite/InfluxDB)
- Dashboard runs locally
- No data sent to external services
Version History
- v0.1 โ MVP: Metrics tracking, trace visualization, dashboard UI
- Roadmap: InfluxDB integration, anomaly detection, multi-agent comparison