Cloud Run Alert Configuration

Cloud Run Alert Configuration

Production-grade observability for Google Cloud Run using Terraform and PromQL (Cloud Monitoring). Grounded in SRE practices, this skill focuses strictly on actionable user impact and scaling bounds.

CRITICAL RULES

• Prompt-First Fast Path (Skip Discovery When Named):
  • If the user prompt explicitly specifies the target Cloud Run service, job, or worker pool name (e.g., 'video-encoder', 'nightly-reconciliation', 'web-frontend', 'catalog-service', 'api-gateway', 'order-processor'), SKIP all workspace .tf file scanning (find_by_name, code_search, list_dir) and gcloud CLI discovery commands entirely.
  • Do NOT run gcloud, terraform, or file search tools when the target name is already provided in the prompt. Instead, parameterize the project ID (variable "scoping_project_id" { default = "my-gcp-project" }) and target resource name in Terraform variables and proceed immediately to Step 2 (Configure Alerts).
• Autonomous Discovery (Only When Target Name is Omitted):
  • Never Scan Root Monorepo or Unbounded Directories: Never run find_by_name or ls across root workspace directories.
  • Config First: Only if the prompt omits the resource name, check .tf files in the immediate working directory for google_cloud_run_v2_service, google_cloud_run_service, or google_cloud_run_v2_job.
  • CLI Second (Graceful Fallback): Only if unconfigured in prompt or local .tf files, attempt gcloud config get-value project and gcloud run services list. If any gcloud command fails (e.g., auth or metadata errors) or terraform is missing, immediately stop running CLI commands and output parameterized HCL using explicit variable defaults.
• Workload Routing: Always classify the workload target and follow its specific reference guide:
  • For HTTP Services follow services.md
  • For Cloud Run Jobs follow jobs.md
  • For Worker Pools follow worker_pools.md
• Explicit Defaults & User Overrides:
  • Always use explicit defaults for all constants specified in the target workload's reference file (SLO targets, latency thresholds, SLAs, saturation ceilings, rate guards).
  • State the defaults being applied in the final summary output and clearly notify the user that any default constant can be customized or overridden via Terraform variables or prompt input.
• Metric Scope Centralization: Parameterize project = var.scoping_project_id in all Terraform google_monitoring_alert_policy resources so the policy can target either a single project or a centralized Cloud Monitoring Metrics Scope.
• PromQL duration (Retest Window) Rules:
  • Lookbacks $\le$ 25h: Set duration = "300s" (5m buffer) to absorb transient blips and scale-up lag (except immediate job failure alerts which use duration = "0s").
  • Lookbacks $> 25$h (e.g. 3d/7d Slow Burn): Omit duration entirely (or set to 0s). Cloud Monitoring rejects PromQL queries with duration set on lookbacks >25h (INVALID_ARGUMENT).
• Terraform Standards & Mandatory Labels:
  • Output clean, complete .tf configurations using google_monitoring_alert_policy and condition_prometheus_query_language directly in your response.
  • Mandatory User Labels: Every google_monitoring_alert_policy resource MUST include a user_labels block containing:

    user_labels = {
      created-with-google-skill = "cloud-run-alert-configuration"
    }
    

  • Include alert_strategy { auto_close = "604800s" } and parameterize notification_channels = var.notification_channels.

WORKFLOW STEPS

1. Discovery & Target Identification

• Fast Path (Target Named in Prompt): If the user prompt names the target Cloud Run service, job, or worker pool, skip all discovery commands and file searches and proceed directly to Step 2.
• Discovery Fallback (Target Unnamed): Only if no resource name is provided in the prompt, check local .tf files or run gcloud to identify the target workload type and name. If gcloud auth fails, fall back immediately to default Terraform variables (var.scoping_project_id).

2. Configure Alerts

• Route to the corresponding guide to generate the alert policies:
  • HTTP Services: Open services.md. Apply the requested alerting policy or standard suite covering availability SLOs (5xx), request latency (P95/P99), client errors (4xx), container instance saturation, container CPU/memory utilization, traffic anomalies (drop/surge), and billable instance time.
  • Batch Jobs: Open jobs.md. Apply immediate job execution failure alerts (duration = "0s").
  • Worker Pools: Open worker_pools.md. Apply the 4-policy standard suite (Task Success SLO Fast/Slow Burn, Backlog ETD, Message Age SLA).

3. Terraform Generation & Review

• Provide the complete HCL configuration in your response with explicitly parameterized defaults and the mandatory user_labels block (created-with-google-skill = "cloud-run-alert-configuration").
• State the applied defaults and remind the user of their ability to override any constant.
• Provide a clear plain-English breakdown of the PromQL logic and triggering thresholds.

Additional Resources

• Google Cloud Run Documentation
• Google Cloud Monitoring PromQL Documentation
• Google Cloud Alerting Policies in Terraform
• Google SRE Workbook: Alerting on SLOs