NATS & JetStream Distributed Messaging AI Skill Guide
Overview & Engine Architecture
Core NATS is fire-and-forget pub/sub (at-most-once on lossy paths). JetStream adds persistence, replay, consumer acks, KV/Object stores, and Raft-replicated metadata. Publish deduplication (Nats-Msg-Id + stream duplicate_window) prevents duplicate records in the stream; consumer delivery remains at-least-once unless handlers are idempotent and ack timing is correct.
Publishers (TLS, creds/JWT)
-> NATS cluster (4222 client / 6222 route)
-> JetStream streams (limits/workqueue/interest retention)
-> consumers (pull preferred in prod)
Subscribers ack/nak/in-progress
When to use / when not to
Use when
- Lightweight cloud-native messaging, request-reply, edge telemetry fanout
- Work queues with JetStream WorkQueue retention (one consumer per filter subject)
- KV/Object config with TTL and history limits
Do not use when
- You need Kafka-grade log retention analytics ecosystem without ops appetite
- Global ordering across all messages without partition/key design
- Marketing “exactly-once delivery” without idempotent consumers—correct to at-least-once + dedupe
Operational Capabilities & Agent Directives
- Subject design: Hierarchical
app.region.entity.action; scope stream subjects narrowly—avoid catch-all > on high-throughput streams without filters.
- Publish dedupe: Set
Nats-Msg-Id to stable business IDs; size duplicate_window to retry horizon—not a substitute for consumer idempotency (streams docs).
- Pull consumers (prod): Durable pull with explicit batch/
max_bytes; keep MaxAckPending ≥ batch size; avoid huge batches if processing is sequential (pull consumers).
- Ack timing:
AckWait must exceed p99 processing or send in-progress heartbeats; short AckWait causes double delivery while work still runs (ack docs). duplicate_window ≠ AckWait (nats-server #6628).
- Backpressure: Core NATS slow consumer drops—move to JetStream pull; tune fetch batch vs memory; scale consumer workers.
- Auth/TLS: Operator mode/JWT or NKeys; TLS everywhere; never commit creds files to git.
- Backup/restore: JetStream file store snapshots / server backup procedures; restore drills on staging; WorkQueue streams delete messages on ack—design retention accordingly.
Production examples
Stream + deduped publish (TypeScript)
import { connect, JSONCodec, headers } from "nats";
const nc = await connect({ servers: ["nats://127.0.0.1:4222"], tls: { /* caFile */ } });
const jsm = await nc.jetstreamManager();
await jsm.streams.add({
name: "ORDERS",
subjects: ["orders.*"],
retention: "limits",
max_bytes: 1024 ** 3,
duplicate_window: 120 * 1e9, // 2m publish dedupe window
});
const hdrs = headers();
hdrs.set("Nats-Msg-Id", `order-${order.orderId}`);
await nc.jetstream().publish("orders.created", codec.encode(order), { headers: hdrs });
Durable pull consumer
const consumer = await js.consumers.get("ORDERS", "order-worker");
const iter = await consumer.fetch({ max_messages: 10, expires: 30_000 });
for await (const msg of iter) {
try {
await processIdempotent(msg);
msg.ack();
} catch (e) {
msg.nak(millis(2000));
}
}
Work queue constraints
- WorkQueue retention: one durable consumer per overlapping subject filter; message deleted after ack.
Technical troubleshooting matrix
| Failure signature |
Root cause |
Diagnostic & fix |
| Slow consumer (core) |
Push faster than handler |
JetStream pull; scale workers |
| Duplicate deliveries |
AckWait too short / no ack |
Raise AckWait; in-progress; idempotent handler |
| Duplicate publishes retried |
Missing Nats-Msg-Id |
Stable msg id + duplicate_window |
| Stalled pull |
MaxAckPending too low |
Raise pending ≥ batch; ack faster |
No storage / disk full |
Stream limits / volume |
nats stream info; expand PVC |
| Raft leader issues |
Even-sized cluster / partition |
Odd node count; check route mesh |
| Lost ack redelivery |
Network drop after process |
Double ack mode for critical ops (delivery doc) |
Best practices
- Use interest or limits retention for fanout analytics; workqueue for job processing.
- Terminate poison messages (
term()) after N deliveries with alert.
- Monitor stream bytes, consumer ack floor lag, and redelivery counts.
- Supercluster/disaster recovery: document RPO/RTO separately for meta vs file store.
Limitations
- Cross-region active/active is non-trivial; expect eventual consistency between sites.
- JetStream performance depends on disk fsync and batch sizes.
- Not a drop-in replacement for Kafka log compaction semantics in all cases.
Related skills
@kafka — heavier log-oriented streaming
@rabbitmq — classic broker queues
@temporal — durable orchestration over activities
Agent Operational Directive
MANDATORY: Do not equate duplicate_window with exactly-once processing. Use pull consumers with explicit ack after side effects. Set AckWait above processing p99 or use in-progress. Financial flows require idempotent handlers and consider double ack.
Source anchors (research)
1---2name: nats3description: "Design NATS subjects and JetStream streams; use pull consumers, AckWait, and Nats-Msg-Id dedupe; configure TLS/auth; debunk exactly-once—at-least-once with publish dedupe and idempotent handlers."4category: devops5risk: safe6source: self7source_type: self8date_added: "2026-09-13"9tags: ["nats", "jetstream", "messaging", "pull-consumer", "deduplication", "ack-wait", "workqueue", "tls", "backpressure"]10tools: ["claude", "cursor", "gemini", "codex"]11metadata:12 source: https://skillmd.com/skills/alivirgo/nats13---1415# NATS & JetStream Distributed Messaging AI Skill Guide1617## Overview & Engine Architecture1819**Core NATS** is fire-and-forget pub/sub (at-most-once on lossy paths). **JetStream** adds persistence, replay, consumer acks, KV/Object stores, and Raft-replicated metadata. **Publish deduplication** (`Nats-Msg-Id` + stream `duplicate_window`) prevents duplicate *records in the stream*; **consumer delivery** remains **at-least-once** unless handlers are idempotent and ack timing is correct.2021```22Publishers (TLS, creds/JWT)23 -> NATS cluster (4222 client / 6222 route)24 -> JetStream streams (limits/workqueue/interest retention)25 -> consumers (pull preferred in prod)26Subscribers ack/nak/in-progress27```2829---3031## When to use / when not to3233**Use when**3435- Lightweight cloud-native messaging, request-reply, edge telemetry fanout36- Work queues with JetStream **WorkQueue** retention (one consumer per filter subject)37- KV/Object config with TTL and history limits3839**Do not use when**4041- You need Kafka-grade log retention analytics ecosystem without ops appetite42- Global ordering across all messages without partition/key design43- Marketing “exactly-once delivery” without idempotent consumers—correct to **at-least-once + dedupe**4445---4647## Operational Capabilities & Agent Directives48491. **Subject design**: Hierarchical `app.region.entity.action`; scope stream `subjects` narrowly—avoid catch-all `>` on high-throughput streams without filters.502. **Publish dedupe**: Set `Nats-Msg-Id` to stable business IDs; size `duplicate_window` to retry horizon—not a substitute for consumer idempotency ([streams docs](https://github.com/nats-io/nats.docs/blob/master/nats-concepts/jetstream/streams.md)).513. **Pull consumers (prod)**: Durable pull with explicit batch/`max_bytes`; keep **`MaxAckPending` ≥ batch size**; avoid huge batches if processing is sequential ([pull consumers](https://docs.nats.io/learn/jetstream/pull-consumers)).524. **Ack timing**: `AckWait` must exceed p99 processing or send **in-progress** heartbeats; short AckWait causes double delivery while work still runs ([ack docs](https://docs.nats.io/learn/jetstream/acknowledgment)). **`duplicate_window` ≠ AckWait** ([nats-server #6628](https://github.com/nats-io/nats-server/discussions/6628)).535. **Backpressure**: Core NATS **slow consumer** drops—move to JetStream pull; tune fetch batch vs memory; scale consumer workers.546. **Auth/TLS**: Operator mode/JWT or NKeys; TLS everywhere; never commit creds files to git.557. **Backup/restore**: JetStream file store snapshots / server backup procedures; restore drills on staging; WorkQueue streams delete messages on ack—design retention accordingly.5657---5859## Production examples6061### Stream + deduped publish (TypeScript)6263```typescript64import { connect, JSONCodec, headers } from "nats";6566const nc = await connect({ servers: ["nats://127.0.0.1:4222"], tls: { /* caFile */ } });67const jsm = await nc.jetstreamManager();68await jsm.streams.add({69 name: "ORDERS",70 subjects: ["orders.*"],71 retention: "limits",72 max_bytes: 1024 ** 3,73 duplicate_window: 120 * 1e9, // 2m publish dedupe window74});7576const hdrs = headers();77hdrs.set("Nats-Msg-Id", `order-${order.orderId}`);78await nc.jetstream().publish("orders.created", codec.encode(order), { headers: hdrs });79```8081### Durable pull consumer8283```typescript84const consumer = await js.consumers.get("ORDERS", "order-worker");85const iter = await consumer.fetch({ max_messages: 10, expires: 30_000 });86for await (const msg of iter) {87 try {88 await processIdempotent(msg);89 msg.ack();90 } catch (e) {91 msg.nak(millis(2000));92 }93}94```9596### Work queue constraints9798- **WorkQueue** retention: one durable consumer per overlapping subject filter; message deleted after ack.99100---101102## Technical troubleshooting matrix103104| Failure signature | Root cause | Diagnostic & fix |105| :--- | :--- | :--- |106| Slow consumer (core) | Push faster than handler | JetStream pull; scale workers |107| Duplicate deliveries | AckWait too short / no ack | Raise AckWait; in-progress; idempotent handler |108| Duplicate publishes retried | Missing `Nats-Msg-Id` | Stable msg id + duplicate_window |109| Stalled pull | `MaxAckPending` too low | Raise pending ≥ batch; ack faster |110| `No storage` / disk full | Stream limits / volume | `nats stream info`; expand PVC |111| Raft leader issues | Even-sized cluster / partition | Odd node count; check route mesh |112| Lost ack redelivery | Network drop after process | Double ack mode for critical ops ([delivery doc](https://docs.nats.io/learn/jetstream/delivery-and-acknowledgment)) |113114---115116## Best practices117118- Use **interest** or **limits** retention for fanout analytics; **workqueue** for job processing.119- Terminate poison messages (`term()`) after N deliveries with alert.120- Monitor stream bytes, consumer ack floor lag, and redelivery counts.121- Supercluster/disaster recovery: document RPO/RTO separately for meta vs file store.122123---124125## Limitations126127- Cross-region active/active is non-trivial; expect eventual consistency between sites.128- JetStream performance depends on disk fsync and batch sizes.129- Not a drop-in replacement for Kafka log compaction semantics in all cases.130131---132133## Related skills134135- `@kafka` — heavier log-oriented streaming136- `@rabbitmq` — classic broker queues137- `@temporal` — durable orchestration over activities138139---140141## Agent Operational Directive142143> **MANDATORY**: Do not equate `duplicate_window` with exactly-once processing. Use pull consumers with explicit ack after side effects. Set AckWait above processing p99 or use in-progress. Financial flows require idempotent handlers and consider double ack.144145---146147## Source anchors (research)148149- [JetStream streams (dedupe window)](https://github.com/nats-io/nats.docs/blob/master/nats-concepts/jetstream/streams.md)150- [AckWait vs duplicate_window (#6628)](https://github.com/nats-io/nats-server/discussions/6628)151- [Pull consumers in depth](https://docs.nats.io/learn/jetstream/pull-consumers)152- [Acknowledgment & redelivery](https://docs.nats.io/learn/jetstream/acknowledgment)153- [Delivery and double ack](https://docs.nats.io/learn/jetstream/delivery-and-acknowledgment)