# Log Analyzer

> AI-driven log analysis for active bots, executors, and gateway — anomaly detection and failure pattern recognition over Hummingbot logs, for both real-time monitoring and retrospective diagnostics.

- Skill: `hummingbot/log-analyzer` (Agent Skill)
- Install (CLI): `npx skillmds@latest add hummingbot/log-analyzer`
- Raw SKILL.md: https://api.skillmd.com/api/skills/hummingbot/log-analyzer/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: hummingbot (https://skillmd.com/u/hummingbot)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/hummingbot/log-analyzer

---


You are doing **AI-driven log analysis** over Hummingbot logs surfaced by the
Backend API. Goal: turn a wall of raw log lines into a diagnosis — *what* is
failing, *how often*, *since when*, *which component*, and *what to do*. Two
modes: **retrospective** (a one-shot summary / root-cause triage) and
**real-time** (a continuous watch that flags new anomalies as they happen).

This is not grep. The value is in **clustering** noisy messages into a handful of
failure *patterns*, **ranking** them by frequency/recency/blast-radius, and
**reasoning** about likely cause — that's the "AI" part: NLP-style message
templating + anomaly detection on top of the structured logs.

## Where logs come from (data sources)

All via `client = await get_client(context._chat_id, context=context)`:

| Source | Call | What you get |
|--------|------|--------------|
| All active bots | `client.bot_orchestration.get_active_bots_status()` | `data{bot: {error_logs, general_logs, status, performance, ...}}` |
| One bot | `client.bot_orchestration.get_bot_status(bot_name)` | same shape for a single bot (perf + logs + activity) |
| One executor | `client.executors.get_executor_logs(executor_id, limit=100, level="ERROR")` | per-executor log entries |
| Gateway (DEX) | `client.gateway.get_logs(tail=100)` | gateway process logs |

**Log entry shape** (each item in `error_logs` / `general_logs`):
```python
{"level_name": "ERROR",        # INFO | WARNING | NETWORK | ERROR | CRITICAL
 "level_no": 40,                # >=40 is a failure
 "msg": "Open order failed x-nbQe1H39BSLUC6552... Retrying 0/10",
 "timestamp": 1782477690.27,    # unix epoch float (UTC)
 "logger_name": "hummingbot.strategy_v2.executors.position_executor.position_executor"}
```
`error_logs` is the curated failure stream; `general_logs` also carries
`WARNING`/`NETWORK` lines worth scanning.

## The default move: run the `logs_summary` routine

For "any errors?", "logs summary", "what's failing across my bots" — **run the
routine, don't hand-roll it.** It already does the clustering, incident
detection, and report generation, tested against live bots.

```
manage_routines(action="run", name="logs_summary",
                config={"bot_name": "", "include_warnings": True,
                        "top_patterns": 8, "recent_incident_min": 15})
```
- `bot_name=""` → all active bots; set a substring to focus one bot.
- It returns a per-bot table (errors, warns, last-error age, top failure, source
  logger), the top cross-bot failure patterns, and flags **active incidents**
  (last error within `recent_incident_min`). It also writes a persistent report.

Read its output, then **add the diagnosis the routine can't** — explain the
patterns, judge severity, and recommend a fix. The routine surfaces the *what*;
you provide the *why* and *what next*.

## Technique — how to analyze (when going deeper than the routine)

1. **Normalize → cluster.** Raw messages differ only by IDs/numbers. Collapse
   them to a template before counting: strip order ids (`x-…`), hashes (`0x…`),
   UUIDs, timestamps, and numbers to placeholders, then group identical
   templates. 200 lines usually collapse to 3–5 real patterns. (The routine's
   `_normalize()` is the reference implementation — reuse its regexes.)
2. **Rank patterns** by `count` (severity), `recency` (last seen — is it still
   happening?), and `blast radius` (how many bots/executors hit it).
3. **Anomaly detection** — flag what's abnormal, not just present:
   - **Recency spike**: errors in the last N minutes ⇒ *active incident*.
   - **Rate**: errors-per-hour vs the bot's baseline; a sudden jump matters more
     than a steady trickle.
   - **New pattern**: a template not seen in earlier windows ⇒ regression.
   - **Correlated failure**: the same pattern across many bots at once ⇒ a shared
     cause (exchange outage, key/permission issue, gateway down) — not the bot.
4. **Attribute** via `logger_name` — it names the failing component
   (`position_executor`, `mqtt`, `connector…`). Group by it to localize.
5. **Diagnose & recommend.** Map the pattern to a likely cause and a concrete
   next step (see triage table), then say it plainly.

## Real-time mode (continuous watch)

For "watch the logs" / live monitoring, build a **continuous routine**
(`CONTINUOUS = True`) that polls `get_active_bots_status()` on an interval,
keeps a `seen` set of pattern fingerprints, and alerts via
`context.bot.send_message` only on **new or spiking** patterns (don't re-report
the same steady error every tick). Use a `LiveReport` to keep one always-current
incident board. Hand the build to a background worker
(`delegate(action="start", agent="condor", task="...")`) — it follows the
`routine_cookbook` playbook — and tell it to reuse the `logs_summary`
normalization/clustering logic as the core.

## Triage reference (common Hummingbot patterns)

| Pattern (normalized) | Source logger | Likely cause | Action |
|----------------------|---------------|--------------|--------|
| `Open order failed … Retrying N/10` | `position_executor` | Exchange reject (insufficient margin, price band, rate limit) | Check balance/leverage; if retries exhaust, inspect connector & symbol filters |
| `Take profit order failed … Retrying` | `position_executor` | Same as above on the TP leg | Verify position still open; check min-notional/tick size |
| `NETWORK …` / connection lost | `connector` / `mqtt` | Transient exchange/network blip | Tolerable if isolated & self-recovers; alarming if sustained/correlated |
| Auth / key / permission errors | `connector` | Bad/expired API key, missing permission | Re-check `/keys`; confirm trade & futures perms |
| Same error across many bots | any | Shared infra (exchange outage, gateway down, key) | Treat as one incident at the source, not per-bot |

## Reporting rules (non-negotiable)

- **Always re-fetch** — never reuse a prior run's log counts; logs move every
  second. Re-run before answering.
- Lead with the verdict: healthy vs. how many errors / active incidents. Then the
  top patterns, then the recommendation. `key: value`, not prose.
- Quote the **normalized pattern** and a **count + last-seen age**, never a single
  raw line as if it were the whole story.
- Don't guess a runtime or invent a cause — if a pattern is unfamiliar, say so and
  show the raw exemplar.

## Rules

- Be direct and concise. Run `logs_summary` first for any summary/triage ask;
  only hand-roll analysis when the routine's output isn't enough.
- One routine per task; a real-time watcher must be tested before it is handed
  over — the background worker does that as part of the job, so wait for its
  report rather than announcing an untested watcher.
- Diagnosis is the deliverable — the counts are evidence, the cause + fix is the
  answer.