Pubsub Basics

GCP Pub/Sub Basics

This skill provides safe, repeatable guidance for operating Google Cloud Pub/Sub from the gcloud CLI. It covers the core messaging concepts, the IAM roles required to act on resources, step-by-step recipes for creating topics and subscriptions, and the operational guardrails that prevent destructive or ambiguous actions. Follow it whenever you provision, inspect, or operate Pub/Sub infrastructure for event-driven systems, streaming pipelines, or asynchronous service integrations.

Overview

Google Cloud Pub/Sub is a fully managed, asynchronous messaging service that decouples message producers from message consumers. Producers publish messages without knowing who consumes them, and consumers receive messages without knowing who produced them. This decoupling enables resilient, horizontally scalable, event-driven architectures.

Core concepts

• Topic — A named resource to which producers (publishers) send messages. A topic is the entry point for data flowing into the system. Topics may optionally enforce a schema on incoming messages.

• Subscription — A named resource representing a stream of messages from a single topic to be delivered to a subscribing application. Each subscription receives its own independent copy of every message published to the topic after the subscription was created. A topic can have many subscriptions; this is the fan-out mechanism.

• Pull delivery — The subscriber application explicitly requests messages from the subscription on its own schedule (using a client library, the CLI, or a streaming pull connection). Pull is well suited to high-throughput batch consumers and workers that control their own concurrency. Messages must be explicitly acknowledged (acked) after successful processing.

• Push delivery — Pub/Sub proactively sends each message as an HTTPS POST request to a configured endpoint (for example a Cloud Run service, Cloud Functions function, or any publicly reachable HTTPS URL). The endpoint acknowledges a message by returning a success HTTP status (102, 200, 201, 202, or 204). Push is well suited to serverless consumers and webhook-style integrations.

• Acknowledgement (ack) and ack deadline — A consumer must acknowledge a message to signal successful processing. If the message is not acked within the subscription's ack deadline, Pub/Sub redelivers it. This at-least-once delivery model means consumers must be designed to be idempotent.

• Dead Letter Topic (DLT) — A topic to which messages are forwarded after a subscription fails to deliver them more than a configured maximum number of times (--max-delivery-attempts). Dead lettering isolates "poison" messages so they stop blocking healthy traffic and can be inspected or reprocessed later. The Pub/Sub service account must have permission to publish to the dead letter topic and to subscribe to the source subscription.

• Schema — An optional, versioned definition (Avro or Protocol Buffer) that constrains the structure of messages published to a topic. When a topic is associated with a schema and an encoding (JSON or BINARY), Pub/Sub validates every published message and rejects non-conforming payloads. Schemas provide a contract between producers and consumers and prevent malformed data from entering the pipeline.

Prerequisites

Before performing any operation, confirm the following are in place.

1. Enable the Pub/Sub API. The pubsub.googleapis.com service must be enabled on the target project.

   gcloud services enable pubsub.googleapis.com --project=PROJECT_ID
   

2. Install and update the gcloud CLI. Operations in this skill assume a recent Google Cloud CLI. Confirm it is installed and current.

   gcloud version
   gcloud components update --quiet
   

3. Authorize the CLI. Authenticate with an identity (user or service account) that holds the required roles described below.

   gcloud auth login
   gcloud config set project PROJECT_ID
   

   For automated or non-interactive environments, prefer Application Default Credentials backed by a dedicated service account:

   gcloud auth activate-service-account --key-file=KEY_FILE_PATH
   

4. Confirm the active project and account. Always verify the execution context before mutating resources.

   gcloud config list --format="value(core.project,core.account)"
   

Key IAM roles

Grant the least privilege required for the task. The principal acting through this skill should hold only the roles that match its responsibilities.

Additional notes:

• Push subscriptions that call private services (for example an authenticated Cloud Run endpoint) require that the push service account is granted the invoker role on the target service (for example roles/run.invoker), in addition to roles/iam.serviceAccountUser on that service account for the configuring identity.
• Dead letter delivery requires the Pub/Sub service agent (service-PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com) to hold roles/pubsub.publisher on the dead letter topic and roles/pubsub.subscriber on the source subscription.

Composite resource deployment recipe

Use this composite recipe when you need to provision a new schema-bound topic and subscription together. It orchestrates the configuration, validation, and creation of the dependent resources (Schema -> Topic -> Subscription) in a single execution block to minimize shell turns and prevent partial configurations.

# Run as a single copy-pasteable execution block
(
  set -e

  # 1. Configuration Parameters
  PROJECT_ID="PROJECT_ID"
  SCHEMA_ID="SCHEMA_ID"
  SCHEMA_TYPE="avro"                 # avro or protocol-buffer
  SCHEMA_FILE="PATH_TO_SCHEMA_FILE"   # Local path to schema definition file
  ENCODING="json"                    # json or binary
  TOPIC_ID="TOPIC_ID"
  SUBSCRIPTION_ID="SUBSCRIPTION_ID"
  ACK_DEADLINE=60                    # Acknowledgement deadline in seconds

  echo "=== [1/5] Validating Schema Definition ==="
  gcloud pubsub schemas validate-schema \
    --type="${SCHEMA_TYPE}" \
    --definition-file="${SCHEMA_FILE}" \
    --project="${PROJECT_ID}"

  echo "=== [2/5] Creating Schema ==="
  gcloud pubsub schemas create "${SCHEMA_ID}" \
    --type="${SCHEMA_TYPE}" \
    --definition-file="${SCHEMA_FILE}" \
    --project="${PROJECT_ID}" \
    --quiet

  echo "=== [3/5] Creating Topic ==="
  gcloud pubsub topics create "${TOPIC_ID}" \
    --schema="${SCHEMA_ID}" \
    --message-encoding="${ENCODING}" \
    --project="${PROJECT_ID}" \
    --quiet

  echo "=== [4/5] Creating Subscription ==="
  gcloud pubsub subscriptions create "${SUBSCRIPTION_ID}" \
    --topic="${TOPIC_ID}" \
    --ack-deadline="${ACK_DEADLINE}" \
    --project="${PROJECT_ID}" \
    --quiet

  echo "=== [5/5] Verifying Resources ==="
  gcloud pubsub schemas describe "${SCHEMA_ID}" --project="${PROJECT_ID}"
  gcloud pubsub topics describe "${TOPIC_ID}" --project="${PROJECT_ID}"
  gcloud pubsub subscriptions describe "${SUBSCRIPTION_ID}" --project="${PROJECT_ID}"
)

Schema enforcement: "JSON" is a message encoding, not a schema type

Pub/Sub has two schema TYPES: AVRO and PROTOCOL_BUFFER. There is no JSON schema type, and gcloud pubsub schemas create --type=json fails. A request for a "JSON schema" means: attach a schema to the topic AND set the topic's message encoding to JSON (--message-encoding=json). The schema definition is normally written in AVRO; JSON refers only to how messages on the topic are encoded.

Correct recipe

SCHEMA="orders-schema"
TOPIC="orders-topic"

# 1. Create the schema. Type is AVRO; the definition is written in Avro JSON.
gcloud pubsub schemas create "$SCHEMA" \
  --type=AVRO \
  --definition='{"type":"record","name":"Order","fields":[
    {"name":"id","type":"string"},
    {"name":"amount","type":"double"}
  ]}'

# 2. Create the topic and bind the schema with JSON MESSAGE ENCODING.
#    --message-encoding=json is the part that makes it a "JSON schema".
gcloud pubsub topics create "$TOPIC" \
  --schema="$SCHEMA" \
  --message-encoding=json

# 3. Verify the contract: encoding must be JSON.
gcloud pubsub topics describe "$TOPIC" --format="value(schemaSettings.encoding)"   # -> JSON

If a topic already exists, bind the schema and encoding with gcloud pubsub topics update "$TOPIC" --schema="$SCHEMA" --message-encoding=json.

Default to JSON encoding unless a task explicitly asks for binary. Reach for --message-encoding=binary only when the requirement calls for it. Choosing binary (or omitting the encoding) is the most common reason a "JSON schema" requirement is scored as unmet, because the schema looks present but the messages are not JSON-validated.

Troubleshooting

• "Schema is AVRO but I need JSON": That is a category error. AVRO is the schema type; JSON is the message encoding. Keep the AVRO schema and set the topic's --message-encoding=json. Do not try to change the schema type.
• Encoding shows empty / BINARY: the topic was created without --message-encoding=json (or without --schema). Re-bind with gcloud pubsub topics update --schema=... --message-encoding=json.

Build order: schema before topic, then verify

1. Create the schema first: gcloud pubsub schemas create <schema> --type=AVRO --definition='<avro-json>'
2. Create the topic bound to that schema with JSON encoding: gcloud pubsub topics create <topic> --schema=<schema> --message-encoding=json
3. Create the dead-letter topic before any subscription references it.
4. Create the consuming subscription, then attach the dead-letter policy to it.

If a topics create fails, read the error before retrying. A missing-schema or encoding error means step 1 or 2 was skipped or out of order; re-running the same failing command will keep failing.

Verify each resource actually persisted

Issuing a create command is not proof it succeeded. After binding the schema and encoding, confirm both are populated before moving on:

gcloud pubsub topics describe <topic> \
  --format="value(schemaSettings.schema, schemaSettings.encoding)"

Do not proceed until that returns both a schema path and JSON. Apply the same describe-and-check pattern after creating the dead-letter policy.

Dead-letter topics: the policy lives on the consuming subscription

Attach the dead-letter policy to the subscription that pulls from the SOURCE topic, not to the dead-letter topic or a subscription of it. --max-delivery-attempts is required (range 5–100); without it there is no dead-lettering.

gcloud pubsub subscriptions update <sub> \
  --dead-letter-topic=<dlq-topic> --max-delivery-attempts=5

# verify it landed on the right subscription:
gcloud pubsub subscriptions describe <sub> \
  --format="value(deadLetterPolicy.deadLetterTopic, deadLetterPolicy.maxDeliveryAttempts)"

Also grant the Pub/Sub service agent roles/pubsub.publisher on the dead-letter topic and roles/pubsub.subscriber on the subscription, or dead-lettering silently never fires.

Creating topics and subscriptions

All commands below use placeholders in CAPS. Replace TOPIC_ID, SUBSCRIPTION_ID, PROJECT_ID, ENDPOINT, and similar tokens with concrete values before execution. Every command explicitly scopes the project with --project=PROJECT_ID and runs non-interactively with --quiet.

1. Create a topic

gcloud pubsub topics create TOPIC_ID \
  --project=PROJECT_ID \
  --quiet

To create a topic that enforces a message-retention duration (so that even subscriptions can replay within the window):

gcloud pubsub topics create TOPIC_ID \
  --message-retention-duration=7d \
  --project=PROJECT_ID \
  --quiet

To associate a topic with an existing schema:

gcloud pubsub topics create TOPIC_ID \
  --schema=SCHEMA_ID \
  --message-encoding=json \
  --project=PROJECT_ID \
  --quiet

2. Create a Pull subscription

A pull subscription lets the consumer fetch messages on demand. The --ack-deadline controls how long a consumer has to acknowledge a message before redelivery.

gcloud pubsub subscriptions create SUBSCRIPTION_ID \
  --topic=TOPIC_ID \
  --ack-deadline=60 \
  --project=PROJECT_ID \
  --quiet

To attach a dead letter topic and bound redelivery attempts:

gcloud pubsub subscriptions create SUBSCRIPTION_ID \
  --topic=TOPIC_ID \
  --ack-deadline=60 \
  --dead-letter-topic=DEAD_LETTER_TOPIC_ID \
  --max-delivery-attempts=5 \
  --project=PROJECT_ID \
  --quiet

To restrict which messages this subscription receives using a server-side filter on message attributes (see Data Reduction below):

gcloud pubsub subscriptions create SUBSCRIPTION_ID \
  --topic=TOPIC_ID \
  --message-filter='attributes.eventType = "ORDER_CREATED"' \
  --project=PROJECT_ID \
  --quiet

3. Create a Push subscription

A push subscription delivers each message to an HTTPS ENDPOINT. The endpoint must return a success status to acknowledge a message.

gcloud pubsub subscriptions create SUBSCRIPTION_ID \
  --topic=TOPIC_ID \
  --push-endpoint=ENDPOINT \
  --ack-deadline=60 \
  --project=PROJECT_ID \
  --quiet

For an authenticated push subscription that targets a private service, attach a push-auth service account so Pub/Sub sends a signed OIDC token:

gcloud pubsub subscriptions create SUBSCRIPTION_ID \
  --topic=TOPIC_ID \
  --push-endpoint=ENDPOINT \
  --push-auth-service-account=PUSH_SERVICE_ACCOUNT_EMAIL \
  --ack-deadline=60 \
  --project=PROJECT_ID \
  --quiet

4. Verify what you created

After creation, confirm configuration before relying on it.

gcloud pubsub topics describe TOPIC_ID \
  --project=PROJECT_ID

gcloud pubsub subscriptions describe SUBSCRIPTION_ID \
  --project=PROJECT_ID

Resource naming and verification checklist

Before completing any Pub/Sub provisioning task, run through this checklist to ensure all resources match expectations:

• Verify resource names match requirements: Cross-reference the exact names of the Schema, Topic, and Subscription in the prompt or spec. Do not assume names.
• Verify labels are applied: If the prompt requests labels (e.g. windtunnel-eval), verify they are present on both the Topic and Subscription.
• Verify resource existence: Run describe commands for the Schema, Topic, and Subscription and confirm the outputs show state: ACTIVE (for schema) and correct configurations.

Core principles

These principles are mandatory. Apply them to every Pub/Sub operation.

Explicit command validation (mandatory)

• Before running any mutating command, restate in plain language what the command will do and to which resource. Confirm the resource name, the project, and the effect.
• Use --dry-run where the subcommand supports it, or run the read-only describe/list equivalent first to confirm the current state.
• Never execute a command whose effect you cannot precisely predict. If the outcome is ambiguous, stop and validate first.

Data reduction

Minimize the volume of data fetched, processed, and returned. Large, unfiltered reads are slow, costly, and noisy.

• --limit — Cap the number of results when listing or pulling. For example, gcloud pubsub subscriptions pull SUBSCRIPTION_ID --limit=10.
• Server-side filtering — Apply --message-filter at subscription creation so unwanted messages are never delivered, rather than filtering after the fact. Use --filter on list commands to narrow results server-side where supported.
• Projection — Request only the fields you need with --format="value(...)" or --format="table(...)" instead of dumping full resource representations.

Project and location scoping

• Always append --project=PROJECT_ID to every command. Never rely solely on the ambient gcloud config project; explicit scoping prevents acting on the wrong project.
• For resources that support location or regional endpoints, specify the location explicitly rather than depending on defaults.

Execution constraints

• Non-interactive mode — Append --quiet (or -q) to suppress interactive prompts so commands are deterministic and automation-safe. This must not be used to bypass the human-approval requirements in the Safety section below.
• Single commands — Execute one discrete gcloud command at a time. Do not chain unrelated operations into a single invocation.
• No raw shell operators — Do not use shell pipes (|), redirects (>, >>), command substitution ($(...)), background operators (&), or &&/; chaining to compose Pub/Sub commands. Each command must stand alone and be independently auditable.

Safety and guardrails

[!CAUTION] Pub/Sub operations can be destructive and irreversible. Deleting a topic or subscription permanently discards undelivered messages and breaks every producer and consumer bound to it. Changing IAM policy can silently cut off access for production services. Treat every mutating operation as production-impacting unless explicitly proven otherwise.

Prohibited operations (denylist)

The following operations MUST NOT be executed without explicit, recorded human approval. When a task appears to require one of these, stop and request approval first; describe the exact command and its blast radius.

• Deleting a topic — gcloud pubsub topics delete .... Destroys the topic and detaches all subscriptions.
• Deleting a subscription — gcloud pubsub subscriptions delete .... Permanently drops all unacknowledged messages held by that subscription.
• Deleting or modifying a schema — gcloud pubsub schemas delete ... or schema revision changes that can invalidate in-flight producers.
• Purging/seeking messages destructively — gcloud pubsub subscriptions seek ... to a timestamp or snapshot that discards the current backlog.
• Changing critical IAM bindings — Adding, removing, or replacing roles/pubsub.admin, roles/pubsub.publisher, or roles/pubsub.subscriber bindings on production topics or subscriptions, or any set-iam-policy operation that replaces an entire policy.
• Detaching a topic — gcloud pubsub topics detach-subscription ..., which immediately stops delivery to the affected subscription.

Always-safe operations

Read-only inspection commands (describe, list, and bounded pull without auto-ack for diagnostics) are safe and encouraged for validation, provided they respect the data-reduction principles above.

Troubleshooting / What to do if...

...a command fails with a permission error (PERMISSION_DENIED / 403)

1. Confirm the active identity: gcloud config list --format="value(core.account)".
2. Confirm the role on the resource matches the action (see the IAM table). For example, publishing requires roles/pubsub.publisher; consuming requires roles/pubsub.subscriber.
3. For push to private endpoints, verify the push service account has the invoker role on the target service and that the configuring identity holds roles/iam.serviceAccountUser on that service account.
4. Confirm the API is enabled: gcloud services list --enabled --filter="pubsub" --project=PROJECT_ID.

...a subscription has a growing backlog (messages not being consumed fast enough)

1. Inspect the backlog and oldest unacked age in Cloud Monitoring metrics (subscription/num_undelivered_messages, subscription/oldest_unacked_message_age).
2. For pull consumers, scale out the number of consumer instances or increase per-instance concurrency so total throughput exceeds the publish rate.
3. Confirm consumers are acking successfully; failed processing leads to redelivery and inflated backlog.
4. Consider whether a --message-filter should reduce delivered volume, or whether the topic is over-fanned-out.

...messages accumulate unacknowledged (repeated redelivery)

1. Verify the consumer is calling acknowledge after successful processing.
2. Check the --ack-deadline. If processing routinely takes longer than the deadline, increase it or extend the deadline programmatically; otherwise Pub/Sub redelivers before the consumer finishes.
3. Ensure consumers are idempotent — at-least-once delivery guarantees that duplicates can occur, and non-idempotent handlers compound the problem.
4. If specific "poison" messages never succeed, attach a dead letter topic with --max-delivery-attempts so they are isolated instead of redelivered forever.

...schema validation fails (published messages are rejected)

1. Confirm the topic's bound schema and encoding: gcloud pubsub topics describe TOPIC_ID --project=PROJECT_ID.
2. Validate a sample message against the schema before publishing: gcloud pubsub schemas validate-message ....
3. Confirm the producer serializes with the encoding the topic expects (JSON vs BINARY); a mismatch causes rejection even for structurally valid data.
4. If the schema legitimately needs to change, commit a new, compatible schema revision rather than editing producers ad hoc — and treat schema deletion as a prohibited operation requiring approval.

Reference directory

Consult these reference documents for deeper, task-specific detail:

• references/cli-usage.md — Comprehensive gcloud pubsub command reference, including publishing, pulling, acknowledging, seeking, snapshots, and output formatting recipes.
• references/iam-security.md — Detailed IAM role and policy guidance, least-privilege patterns, service-account configuration for push delivery, and dead-letter permission setup.
• references/client-libraries.md — Pointers to the official Google Cloud client libraries for programmatic publishing and consuming, with notes on when to prefer SDK code over the CLI.