Configure Cloud Monitoring Alerts
Configure high-signal alerting policies in Cloud Monitoring using
Terraform (google_monitoring_alert_policy) and PromQL
(condition_prometheus_query_language).
Usage Attribution
For all gcloud CLI, REST API, and MCP attribution requirements when executing
commands with this skill, follow the instructions in
Usage Attribution.
Enforce Core Alerting Rules
- Apply the Four Golden Signals: Cover latency, traffic, errors, and
saturation for the target workload.
- Use Dynamic Grouping: Aggregate metrics by resource labels and
parameterize thresholds with Terraform variables. Never hardcode ephemeral
pod or instance identifiers.
- Default to PromQL: Use Prometheus Query Language (PromQL) for metric
conditions, and use log-based alerts
(
condition_matched_log or google_logging_metric) for lifecycle and
audit events.
- Attach Skill Attribution Labels: Always include
user_labels with
created-with-google-skill = "cloud-monitoring-alert-configuration" on
every generated google_monitoring_alert_policy resource.
- Require User Confirmation Before Mutating Resources: Never execute
commands that modify infrastructure, enable telemetry, create online
monitors, or apply Terraform without waiting for explicit user confirmation.
- Prompt for Notification Channels: Alerting policies without notification
channels incur charges without notifying anyone during an incident. Whenever
notification channels are not specified in the user prompt across any
workflow, explicitly inform the user that no notification channels are
attached to the alerting policies, advise them to configure and attach
notification channels before deploying, and ask at the end of your response
if they want to configure notification channels.
Route by Target Resource Type
Identify the target resource type and read only the matching reference file
directly. Never run directory listings or workspace file searches to look for
queries or .tf files unless the prompt
explicitly provides a file path. If a user asks to fix or validate a PromQL
query without providing the query text or a file path, construct a
representative broken-and-corrected query example directly.
Configure Cloud Run Alerts
- HTTP Services:
cloud_run_services.md
- Batch Jobs:
cloud_run_jobs.md
- Worker Pools:
cloud_run_worker_pools.md
Configure Google Kubernetes Engine Alerts
- PromQL Queries and Golden Signals:
GKE PromQL Queries
- Metrics Catalog and Open-Source Alerts:
GKE Metrics
- Prerequisites and Setup:
GKE Setup
Configure AI Agent Alerts
- Reliability:
Reliability
- Quality:
Quality
- Cost:
Cost
- Safety:
Safety
- Security:
Security
- Telemetry Enablement:
Telemetry
- Traffic Profiles:
Historical
and
No History
Configure Compute Engine and AI Hypercomputer Alerts
- Compute Engine VMs, MIGs, GPU XID Insights, Host or Reservation
Maintenance (
type=scheduled, unscheduled, or multiple), and Machine
or Domain Health:
compute_engine_alerts.md
Configure Generic Resource Alerts
- Generic Workloads, Custom Metrics, MQL Migration, and HCL Deduplication:
generic_alert_policies.md
Delegate Service Level Objective Alerts
Delegate to the google-cloud-slo-alert-configuration skill. That skill
owns the SLO setup wizard, the burn-rate PromQL templates
(references/promql_templates.md), the SLI metric mapping
(references/service_metrics.md), and the App Hub scoping labels
(references/app_hub_labels.md). Do NOT duplicate those templates here or
claim that this skill owns its own SLO PromQL template library.
Delegate whenever the user asks for an availability or latency SLO, an
error budget or burn-rate alert, or a Service Level Objective on any
resource. When SLOs are only one part of a broader request (for example, a full
production alerting suite for a Cloud Run service), delegate the SLO policies
and continue generating the remaining golden-signal policies from the
resource-specific reference in the same response.
Every resource reference carries a Context to hand off list naming the SLI
metric, its good or bad filter, and the grouping labels — under a ## Delegate Service Level Objective Alerts heading for Cloud Run and generic resources, and
inside the SLO section itself for GKE and Agent Platform. Read it and pass that
context on, so the SLO skill does not repeat discovery you have already done.
When delegating, you MUST also:
- State the SLO target and window explicitly if the user gave them, or let the
SLO skill apply its SRE defaults (99.9% over a rolling 28 days) if they did
not.
- Require the returned policies to carry
user_labels with
created-with-google-skill = "cloud-monitoring-alert-configuration" so
attribution stays consistent with the rest of the generated suite. This
literal value does NOT change when you delegate: the label records the
entry-point skill, not whichever skill supplied the template. Do NOT
set created-with-google-skill = "google-cloud-slo-alert-configuration" on
policies generated through this delegation.
- Preserve the burn-rate duration contract, which
google-cloud-slo-alert-configuration does not specify: Fast Burn (1h and
5m windows, factor 14.4) carries duration = "300s", and Slow Burn
([3d] and [6h] lookback, factor 1.0) omits the duration field
entirely — do not emit duration = "0s" in its place. Cloud Monitoring
rejects a non-zero retest buffer when the lookback window exceeds 25 hours.
- When no notification channels are provided, inform the user that no
notification channels are attached to the alerting policies, advise them to
configure and attach notification channels, and ask if they want to
configure notification channels.
[!IMPORTANT]
Always tell the user about the referral, even on the fallback path. Some
reference files in this skill include standalone burn-rate templates to use
when the google-cloud-slo-alert-configuration skill is not installed. If you
emit policies from those fallback templates, your response MUST still tell the
user that google-cloud-slo-alert-configuration owns SLO configuration, state
that the emitted policies use the local fallback templates, and list the
handoff context. Never silently present SLO policies as if this
skill owned them.
[!NOTE]
Exception: Agent Platform (OpenTelemetry-instrumented AI agents on any
runtime, including Vertex AI Agent Engine and Cloud Run) is the one resource
whose Fast and Slow burn policies live in this skill and remain authoritative.
Delegate only for SLO customization — see
Reliability.
Delegate Related Observability Tasks
- Notification Channels: Delegate to
cloud-monitoring-notification-channels-configuration to configure email,
PagerDuty, Pub/Sub, or Slack channels. When an alerting policy has no
notification channels attached, inform the user that no notification
channels are attached and that they should configure notification channels
and attach them to the alerting policy.
- PromQL Authoring: Delegate to
cloud-monitoring-promql-query for
standalone PromQL query authoring or metric exploration.
Follow the Alert Configuration Workflow
- Discover: Identify target resources, metric schemas, and scoping
projects using the lookup commands in the matching reference file (such as
discover_metric_schema.py for Compute
Engine metrics and other Google Cloud resource metrics, or
gather_agent_info.py,
check_telemetry.py,
analyze_traffic.py,
create_online_monitor.py,
list_trace_scope_table_names.py,
and
list_log_scope_table_names.py
for AI agents).
- Route: Load the reference file for the target resource type or delegate
SLOs to
google-cloud-slo-alert-configuration.
- Generate: Produce parameterized Terraform HCL configurations with
required
user_labels.
- Validate: When the loaded reference file requires a validation script
(such as lint_promql.py,
validate_config.py,
lint_syntax.py, or
scan_duplicates.py), run or
print that exact script command. Do not list or search the
scripts/
directory.
- Explain and Prompt: Explain each alert policy in plain English. When no
notification channels are provided, tell the user that no notification
channels are attached to the alerting policy, advise them to configure and
attach notification channels, and ask if they want to configure notification
channels.
1---2name: cloud-monitoring-alert-configuration3metadata:4 source: https://skillmd.com/skills/google/cloud-monitoring-alert-configuration5 version: "1.0.3"6 category: CloudObservabilityAndMonitoring7 support_tier: team8 tags: "cloud-monitoring, alerting, terraform, promql, observability"9 comment: >-10 Consolidated from Cloud Monitoring alerting policy best practices across11 Cloud Run, Google Kubernetes Engine, Compute Engine, and Vertex AI Agent12 Engine.13 canonical_source: https://docs.cloud.google.com/monitoring/alerts.md.txt14description: >-15 Configures Cloud Monitoring alerting policies using Terraform and16 PromQL across Cloud Run, Google Kubernetes Engine (GKE), Compute Engine, AI17 Hypercomputer reservations, Vertex AI Agent Engine, and generic Google Cloud18 resources. Use when creating, reviewing, migrating, or deduplicating Terraform19 alerting policies for latency, traffic, error rates, saturation, host or20 reservation maintenance, GPU XID insights, or AI agent telemetry. Don't use21 for standalone PromQL query authoring without alerting, log querying, or22 billing cost audits.23---2425# Configure Cloud Monitoring Alerts2627Configure high-signal alerting policies in Cloud Monitoring using28Terraform (`google_monitoring_alert_policy`) and PromQL29(`condition_prometheus_query_language`).3031## Usage Attribution3233For all `gcloud` CLI, REST API, and MCP attribution requirements when executing34commands with this skill, follow the instructions in35[Usage Attribution](references/attribution.md).3637## Enforce Core Alerting Rules38391. **Apply the Four Golden Signals**: Cover latency, traffic, errors, and40 saturation for the target workload.412. **Use Dynamic Grouping**: Aggregate metrics by resource labels and42 parameterize thresholds with Terraform variables. Never hardcode ephemeral43 pod or instance identifiers.443. **Default to PromQL**: Use Prometheus Query Language (PromQL) for metric45 conditions, and use log-based alerts46 (`condition_matched_log` or `google_logging_metric`) for lifecycle and47 audit events.484. **Attach Skill Attribution Labels**: Always include `user_labels` with49 `created-with-google-skill = "cloud-monitoring-alert-configuration"` on50 every generated `google_monitoring_alert_policy` resource.515. **Require User Confirmation Before Mutating Resources**: Never execute52 commands that modify infrastructure, enable telemetry, create online53 monitors, or apply Terraform without waiting for explicit user confirmation.546. **Prompt for Notification Channels**: Alerting policies without notification55 channels incur charges without notifying anyone during an incident. Whenever56 notification channels are not specified in the user prompt across any57 workflow, explicitly inform the user that no notification channels are58 attached to the alerting policies, advise them to configure and attach59 notification channels before deploying, and ask at the end of your response60 if they want to configure notification channels.6162## Route by Target Resource Type6364Identify the target resource type and read only the matching reference file65directly. Never run directory listings or workspace file searches to look for66queries or `.tf` files unless the prompt67explicitly provides a file path. If a user asks to fix or validate a PromQL68query without providing the query text or a file path, construct a69representative broken-and-corrected query example directly.7071### Configure Cloud Run Alerts7273* **HTTP Services**:74 [cloud_run_services.md](references/cloud_run_services.md)75* **Batch Jobs**:76 [cloud_run_jobs.md](references/cloud_run_jobs.md)77* **Worker Pools**:78 [cloud_run_worker_pools.md](references/cloud_run_worker_pools.md)7980### Configure Google Kubernetes Engine Alerts8182* **PromQL Queries and Golden Signals**:83 [GKE PromQL Queries](references/kubernetes_promql_queries.md)84* **Metrics Catalog and Open-Source Alerts**:85 [GKE Metrics](references/kubernetes_metrics_and_alerts_catalog.md)86* **Prerequisites and Setup**:87 [GKE Setup](references/kubernetes_gke_configuration_prerequisites.md)8889### Configure AI Agent Alerts9091* **Reliability**:92 [Reliability](references/agent_platform_reliability_alert_policies.md)93* **Quality**:94 [Quality](references/agent_platform_quality_alert_policies.md)95* **Cost**:96 [Cost](references/agent_platform_cost_alert_policies.md)97* **Safety**:98 [Safety](references/agent_platform_safety_alert_policies.md)99* **Security**:100 [Security](references/agent_platform_security_alert_policies.md)101* **Telemetry Enablement**:102 [Telemetry](references/agent_platform_telemetry_enablement.md)103* **Traffic Profiles**:104 [Historical](references/agent_platform_has_historical_traffic_data.md)105 and106 [No History](references/agent_platform_no_historical_traffic_data.md)107108### Configure Compute Engine and AI Hypercomputer Alerts109110* **Compute Engine VMs, MIGs, GPU XID Insights, Host or Reservation111 Maintenance (`type=scheduled`, `unscheduled`, or `multiple`), and Machine112 or Domain Health**:113 [compute_engine_alerts.md](references/compute_engine_alerts.md)114115### Configure Generic Resource Alerts116117* **Generic Workloads, Custom Metrics, MQL Migration, and HCL Deduplication**:118 [generic_alert_policies.md](references/generic_alert_policies.md)119120### Delegate Service Level Objective Alerts121122**Delegate to the `google-cloud-slo-alert-configuration` skill.** That skill123owns the SLO setup wizard, the burn-rate PromQL templates124(`references/promql_templates.md`), the SLI metric mapping125(`references/service_metrics.md`), and the App Hub scoping labels126(`references/app_hub_labels.md`). Do **NOT** duplicate those templates here or127claim that this skill owns its own SLO PromQL template library.128129Delegate whenever the user asks for an **availability or latency SLO**, an130**error budget or burn-rate alert**, or a **Service Level Objective** on any131resource. When SLOs are only one part of a broader request (for example, a full132production alerting suite for a Cloud Run service), delegate the SLO policies133and continue generating the remaining golden-signal policies from the134resource-specific reference in the same response.135136Every resource reference carries a **Context to hand off** list naming the SLI137metric, its good or bad filter, and the grouping labels — under a `## Delegate138Service Level Objective Alerts` heading for Cloud Run and generic resources, and139inside the SLO section itself for GKE and Agent Platform. Read it and pass that140context on, so the SLO skill does not repeat discovery you have already done.141When delegating, you MUST also:1421431. State the SLO target and window explicitly if the user gave them, or let the144 SLO skill apply its SRE defaults (99.9% over a rolling 28 days) if they did145 not.1462. Require the returned policies to carry `user_labels` with147 `created-with-google-skill = "cloud-monitoring-alert-configuration"` so148 attribution stays consistent with the rest of the generated suite. This149 literal value does **NOT** change when you delegate: the label records the150 entry-point skill, not whichever skill supplied the template. Do **NOT**151 set `created-with-google-skill = "google-cloud-slo-alert-configuration"` on152 policies generated through this delegation.1533. Preserve the burn-rate duration contract, which154 `google-cloud-slo-alert-configuration` does not specify: Fast Burn (`1h` and155 `5m` windows, factor `14.4`) carries `duration = "300s"`, and Slow Burn156 (`[3d]` and `[6h]` lookback, factor `1.0`) **omits the `duration` field157 entirely** — do not emit `duration = "0s"` in its place. Cloud Monitoring158 rejects a non-zero retest buffer when the lookback window exceeds 25 hours.1594. When no notification channels are provided, inform the user that no160 notification channels are attached to the alerting policies, advise them to161 configure and attach notification channels, and ask if they want to162 configure notification channels.163164> [!IMPORTANT]165> **Always tell the user about the referral, even on the fallback path.** Some166> reference files in this skill include standalone burn-rate templates to use167> when the `google-cloud-slo-alert-configuration` skill is not installed. If you168> emit policies from those fallback templates, your response MUST still tell the169> user that `google-cloud-slo-alert-configuration` owns SLO configuration, state170> that the emitted policies use the local fallback templates, and list the171> handoff context. Never silently present SLO policies as if this172> skill owned them.173174> [!NOTE]175> **Exception**: Agent Platform (OpenTelemetry-instrumented AI agents on any176> runtime, including Vertex AI Agent Engine and Cloud Run) is the one resource177> whose Fast and Slow burn policies live in this skill and remain authoritative.178> Delegate only for SLO customization — see179> [Reliability](references/agent_platform_reliability_alert_policies.md).180181## Delegate Related Observability Tasks182183* **Notification Channels**: Delegate to184 `cloud-monitoring-notification-channels-configuration` to configure email,185 PagerDuty, Pub/Sub, or Slack channels. When an alerting policy has no186 notification channels attached, inform the user that no notification187 channels are attached and that they should configure notification channels188 and attach them to the alerting policy.189* **PromQL Authoring**: Delegate to `cloud-monitoring-promql-query` for190 standalone PromQL query authoring or metric exploration.191192## Follow the Alert Configuration Workflow1931941. **Discover**: Identify target resources, metric schemas, and scoping195 projects using the lookup commands in the matching reference file (such as196 [discover_metric_schema.py](scripts/discover_metric_schema.py) for Compute197 Engine metrics and other Google Cloud resource metrics, or198 [gather_agent_info.py](scripts/agent_platform/gather_agent_info.py),199 [check_telemetry.py](scripts/agent_platform/check_telemetry.py),200 [analyze_traffic.py](scripts/agent_platform/analyze_traffic.py),201 [create_online_monitor.py](scripts/agent_platform/create_online_monitor.py),202 [list_trace_scope_table_names.py](scripts/agent_platform/list_trace_scope_table_names.py),203 and204 [list_log_scope_table_names.py](scripts/agent_platform/list_log_scope_table_names.py)205 for AI agents).2062. **Route**: Load the reference file for the target resource type or delegate207 SLOs to `google-cloud-slo-alert-configuration`.2083. **Generate**: Produce parameterized Terraform HCL configurations with209 required `user_labels`.2104. **Validate**: When the loaded reference file requires a validation script211 (such as [lint_promql.py](scripts/lint_promql.py),212 [validate_config.py](scripts/kubernetes/validate_config.py),213 [lint_syntax.py](scripts/agent_platform/lint_syntax.py), or214 [scan_duplicates.py](scripts/agent_platform/scan_duplicates.py)), run or215 print that exact script command. Do not list or search the `scripts/`216 directory.2175. **Explain and Prompt**: Explain each alert policy in plain English. When no218 notification channels are provided, tell the user that no notification219 channels are attached to the alerting policy, advise them to configure and220 attach notification channels, and ask if they want to configure notification221 channels.