# Jetstream Operations

> Use this skill whenever users are operating, troubleshooting, or monitoring a running NATS JetStream system — including diagnosing consumer lag, messages not delivering, stream-full errors, performance tuning, Prometheus metrics, Grafana dashboards, alerting rules, JetStream advisory subjects, nats CLI usage, cluster health and leader election issues, or client connection/reconnection problems. Use this skill when something is broken or slow, or when the user wants to observe their system. Do NOT use for designing new streams/consumers (use jetstream-architecture) or deploying/configuring infrastructure (use jetstream-deployment).

- Skill: `lithqube/jetstream-operations` (Agent Skill, multi-file: 5 files)
- Install (CLI): `npx skillmds@latest add lithqube/jetstream-operations`
- Raw SKILL.md: https://api.skillmd.com/api/skills/lithqube/jetstream-operations/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: lithqube (https://skillmd.com/u/lithqube)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/lithqube/jetstream-operations

---


# JetStream Operations

Troubleshoot, monitor, and tune NATS JetStream including consumer lag, delivery failures, performance optimization, and observability.

For designing new streams or consumers, defer to the `jetstream-architecture` skill.
For deploying or configuring NATS infrastructure, defer to the `jetstream-deployment` skill.
If the user is operating a **Synadia Agent fabric**, the same tooling applies: enumerate live agents with `nats req '$SRV.PING.agents' ''`, inspect endpoints/metadata with `$SRV.INFO.agents`, and watch agent liveness on the heartbeat subjects `agents.hb.*.*.*` (an agent is offline after ~3× its advertised `interval_s`). Tapping `agents.>` shows live prompt/response traffic. For how the protocol and those subjects are defined, see the `nats-agent-fabric` skill.

## Reference Files

Read these files when they're relevant — don't load all of them upfront:

- `operations/troubleshooting.md` — step-by-step diagnosis for messages not delivering, consumer lag, stream-full errors, cluster split-brain, and client disconnections. Read whenever something is broken.
- `operations/performance.md` — publish throughput tuning, fetch batch sizing, MaxAckPending, parallel workers, FileStorage vs MemoryStorage, OS/TCP tuning, built-in benchmark commands. Read for performance or throughput questions.
- `operations/monitoring.md` — NATS HTTP endpoints, Prometheus metrics, nats-exporter setup, advisory subjects, Grafana dashboard recommendations, Prometheus alert rules. Read for observability and alerting questions.
- `operations/cli-reference.md` — full nats CLI reference for streams, consumers, pub/sub, server commands, and diagnostic workflows. Read when the user needs specific CLI commands.

## Workflow

Step 1: Identify symptoms — what is the user observing? (lag, missing messages, errors, slow throughput)

Step 2: Inspect current state — use `nats` CLI to examine streams, consumers, and server status.

Step 3: Diagnose root cause — match symptoms to known patterns (ack timeout, filter mismatch, resource limits).

Step 4: Apply fix — provide specific configuration changes or code fixes.

Step 5: Verify resolution — confirm the fix with CLI commands and metrics.

Step 6: Set up monitoring — recommend metrics, alerts, and advisory subscriptions to prevent recurrence.

## Core Principles

- Always start diagnosis with `nats stream report` and `nats consumer report`
- Check `num_ack_pending` first when consumers appear stuck — it's the most common bottleneck
- Monitor JetStream advisory subjects for real-time operational events, but subscribe to the specific ones you need rather than `$JS.EVENT.ADVISORY.>` — that wildcard is the whole-account firehose and includes an API audit advisory published on every JetStream API response. Advisories are fire-and-forget, so anything you must not miss (max-deliveries, terminated) belongs in a stream you consume durably, not a bare subscription
- Use `nats server report jetstream` to check cluster-wide JetStream health
- Set alerts on consumer pending count, not just publish rate
- Prefer `nats` CLI over raw API calls for operational tasks
- Always check the NATS server logs — they surface warnings before failures
- When in doubt, compare stream sequence numbers with consumer ack floor