1---2name: pubsub-messaging3description: Pub/Sub Messaging4---56# Pub/Sub Messaging78## When to Use9- Designing event-driven or streaming ingestion on GCP10- Choosing between Pub/Sub and Pub/Sub Lite11- Troubleshooting message delivery, ordering, or acknowledgement issues12- Preparing for GCP Professional Data Engineer exam1314## Core Jobs1516### 1. Topic and Subscription Design17- One **topic** = one logical event stream (e.g., `orders-created`, `sensor-readings`)18- Multiple **subscriptions** = multiple independent consumers of the same topic19- Each subscription maintains its own offset — messages delivered to each independently20- Subscription **ack deadline** (default 10s, max 600s) — extend if processing takes longer2122### 2. Pull vs Push Subscriptions23| Factor | Pull | Push |24|--------|------|------|25| Consumer | Subscriber polls for messages | Pub/Sub pushes to HTTPS endpoint |26| Control | Consumer controls rate | Pub/Sub controls delivery rate |27| Best for | Dataflow, batch consumers | Cloud Run, webhooks, Cloud Functions |28| Auth | Service account key/WIF | OIDC token in Authorization header |2930### 3. Message Ordering31- By default, Pub/Sub does NOT guarantee ordering32- Enable **ordering keys** to guarantee ordered delivery within a key33- Ordering keys work only within a single region (cross-region = no ordering guarantee)34- Use case: ordered events per user_id, device_id, transaction_id3536### 4. Dead Letter Topics37- Configure a **dead letter topic** on a subscription for undeliverable messages38- Messages moved to dead letter after max delivery attempts (5–100, configurable)39- Monitor dead letter topics with Cloud Monitoring alerts40- Always process dead letters (don't ignore them)4142### 5. Pub/Sub Lite vs Pub/Sub43| Factor | Pub/Sub | Pub/Sub Lite |44|--------|---------|--------------|45| Management | Fully managed, global | Zone/region-specific, manual capacity |46| Ordering | Ordering keys | Partition-based ordering (like Kafka) |47| Cost | Per message/byte | Provisioned capacity (cheaper at scale) |48| Retention | 7 days | Configurable up to 31 days |49| Use case | Default choice | Cost-sensitive high-volume workloads |5051### 6. Integration Patterns52- **Pub/Sub → Dataflow → BigQuery** — standard streaming analytics pipeline53- **Pub/Sub → Cloud Functions** — lightweight event processing (push subscription)54- **Pub/Sub → Cloud Storage** — use Dataflow or Pub/Sub export for archiving55- **Cloud Scheduler → Pub/Sub → Cloud Functions** — scheduled event trigger pattern5657## Key Concepts58- **At-least-once delivery** — same message may be delivered multiple times → idempotent consumers59- **Exactly-once** — available with Dataflow (deduplication via message ID)60- **Message ID** — globally unique; use for deduplication61- **Seek** — replay messages from a timestamp or snapshot6263## Checklist64- [ ] Consumers designed to be idempotent (handle duplicate messages)?65- [ ] Ack deadline set longer than max processing time?66- [ ] Dead letter topic configured and monitored?67- [ ] Ordering keys used only where strict ordering is required?68- [ ] Push subscription endpoint uses HTTPS with OIDC auth?69- [ ] Pub/Sub Lite considered for high-volume cost reduction?7071## Output Format72- 🔴 **Critical** — no idempotency with at-least-once delivery (data duplication risk)73- 🟡 **Warning** — no dead letter topic, ack deadline too short for processing time74- 🟢 **Suggestion** — Pub/Sub Lite for high-volume cost savings, ordering keys opportunity7576## Exam Tips77- Pub/Sub does NOT guarantee ordering by default → use ordering keys (single region only)78- At-least-once → always build idempotent consumers79- **Dataflow is the standard bridge** between Pub/Sub and BigQuery for streaming80- Push subscriptions → Cloud Run / Cloud Functions (serverless, event-driven)81- Dead letter topic = where undeliverable messages go after max_delivery_attempts82- Pub/Sub Lite = cheaper but zone-specific, partition-based (Kafka-like model)