Agent Platform Alert Configuration

Agent Platform Alert Configuration

Critical Steps

1. Safety & Confirmation Tiers (CRITICAL)

Before executing any commands or writing configurations on behalf of the user, you MUST adhere to the following safety tiers based on the action requested:

1. Tier R: Read-only (check_telemetry.py / gather_agent_info.py)
   • Rule: No confirmation needed. You may execute these scripts immediately to inspect telemetry status or gather agent configuration details.
2. Tier B: Billing & Resource Creation (create_online_monitor.py / provisioning)
   • Rule: Explicit User Confirmation Required. These actions incur additional billing charges and create cloud resources. The agent MUST ALWAYS warn the user explicitly about the potential extra billing costs of BOTH the Online Monitor (specifically mentioning LLM evaluations) and Telemetry (specifically mentioning Cloud Trace/Cloud Logging export). You MUST STOP and ask for explicit approval before proceeding with provisioning or providing setup commands.

2. Prerequisites & Dependencies

Agent Telemetry

• Disclaimer: For Reliability, Cost, Safety, and Security alerts to function, the underlying agent MUST be instrumented to emit OpenTelemetry (OTel) metrics. If the agent does not emit these metrics, the alerting policies will have no data stream to evaluate.

Python Environment

Before executing any python script in this skill you MUST install the required dependencies in your environment. Run this command first:

pip install -r scripts/requirements.txt

3. Input Assumptions

• Explicit Project Adherence: You must ONLY configure alerts, query telemetry, or interact with the Google Cloud Project(s) explicitly provided by the user in the prompt. Do NOT assume or use other projects from your environment or history unless the user explicitly directs you to do so.
• Sequential File Transformations: If the user explicitly asks to copy a file and then modify it, you MUST perform these actions sequentially (copy first, then modify) rather than writing the final content directly.

4. Execution Steps

1. Mandatory Prerequisite Execution Protocol (SEQUENTIAL): Before generating or writing ANY configuration, you MUST execute these steps in order:

   1. Step 1: Streamlined Discovery (Mandatory): Run gather_agent_info.py to automatically identify agent runtime, verify telemetry, metric scopes, linked datasets, and more. This script covers most of the manual verifications listed in subsequent steps.
      • Command: python3 scripts/gather_agent_info.py --project-id {project_id} --agent-name {agent_name}
      • Note: If this script fails, returns partial data, or doesn't produce everything you need, you MUST satisfy requirements by running the manual fallback steps listed in Step 2 and then perform Step 3 below. If Step 1 succeeds and provides all info, SKIP to Step 3 (Pre-existing Policies Verification).
   2. Step 2: Metric Scope Verification (Fallback): Run this ONLY if Step 1 failed to determine the metric scope.
      • Action A (CLI): Run gcloud beta monitoring metrics-scopes list projects/{project_id}. If a scoping project is returned, you MUST deploy policies there.
      • Action B (Code Scan): Search Terraform configurations for google_monitoring_monitored_project resources to extract the scoping project.
      • Action C (Fallback): If ambiguous, ASK the user: "Are you using a multi-project Cloud Monitoring Metric Scope? If so, what is the scoping project ID?"
   3. Step 3: Pre-existing Policies Verification: Avoid duplicates.
      • Action: Scan the target directory to see if aggregated policies already exist targeting the same metrics (grouped by reasoning_engine_id or gen_ai_agent_name). Use scan_duplicates.py to verify.

2. Alert Policy Type Resource Files: You MUST list and read files under references/ with names ending in _alert_policies.md to learn how to configure alert policies based on type. By default you MUST configure all of the following alert types UNLESS the user requests to generate explicit alert policies and/or types. Follow their tables of content to help you find the reference sections you need to read:

   Alert Type	Reference File
   Reliability	reliability_alert_policies.md
   Quality	quality_alert_policies.md
   Cost	cost_alert_policies.md
   Safety	safety_alert_policies.md
   Security	security_alert_policies.md

5. Outputs & Formats

• Always configure the supported alerting policies for the target agent:

  • For Reliability Monitoring: You MUST configure exactly five alerting policies:
    1. Latency (anomaly monitoring)
    2. Error Rate - Fast Burn SLO (1-Hour Window)
    3. Error Rate - Slow Burn SLO (3-Day Window)
    4. Model Call Error Rate (SQL-based Observability Analytics Alerting)
    5. Tool Call Error Rate (SQL-based Observability Analytics Alerting)
  • For Quality Monitoring: You MUST configure exactly three alerting policies (Requires Vertex AI Online Monitors):
    1. Final Response Quality
    2. Tool Use Quality
    3. Hallucination
  • For Cost Monitoring: You MUST configure exactly one cost alerting policy:
    1. Rapid Token Burn Rate (anomaly monitoring)
  • For Safety Monitoring: You MUST configure exactly one safety alerting policy:
    1. High Model Armor Safety Policy Trigger Rate (SQL-based Observability Analytics Alerting)
  • For Security Monitoring: You MUST configure exactly one security alerting policy:
    1. High IAM Permission Denied Trigger Rate (SQL-based Observability Analytics Alerting)

• Terraform Only: Write the generated observability configuration ONLY as Terraform (.tf) files (such as alerts.tf, variables.tf).

  • You ONLY need to install Terraform if you're asked to deploy the alerts AND there is no valid Terraform install. SQL-based alerting using condition_sql requires the provider version >= 6.0.0 (or late 5.x versions supporting the feature).
  • If you are NOT asked to deploy the alerts you do not need to install terraform.

• Dynamic Multi-Resource Alerting (No Single-Resource Pinning): You MUST NOT hardcode specific agent IDs or resource name filters (for example, {gen_ai_agent_name="{agent_name}"} or metric.labels.agent_resource_name="{agent_name}") in alerting conditions unless explicitly requested (for example, "ONLY for this agent"). Merely mentioning a specific agent name or ID in the request does NOT constitute an explicit request to pin/filter; you MUST still default to dynamic grouping to cover all agents. To cover all active agents in the project dynamically:

  Good Example (PromQL Grouping):

  sum(rate(workload_googleapis_com:gen_ai_invoke_agent_duration_count{monitored_resource="generic_node"}[5m])) by (gen_ai_agent_name)
  

  Bad Example (PromQL Hardcoded Filter):

  sum(rate(workload_googleapis_com:gen_ai_invoke_agent_duration_count{monitored_resource="generic_node", gen_ai_agent_name="support-bot"}[5m]))
  

  • For Reliability Metrics using PromQL: ALWAYS use grouping aggregations. Group by gen_ai_agent_name (for example, by (gen_ai_agent_name)). Avoid filtering to a single ID/Name unless requested.
  • For Quality Metrics using Standard Threshold Filters: Omit the agent_resource_name filter entirely. Configure the condition filter to only target the monitored resource type (aiplatform.googleapis.com/OnlineEvaluator) and metric type (aiplatform.googleapis.com/online_evaluator/scores) globally for the project.

  Good Example (SQL Grouping):

  SELECT
    JSON_VALUE(resource.attributes, '$."cloud.resource_id"') as agent_id,
    ...
  FROM ...
  GROUP BY agent_id
  

  Bad Example (SQL Hardcoded Filter):

  SELECT ...
  FROM ...
  WHERE JSON_VALUE(resource.attributes, '$."cloud.resource_id"') = 'support-bot'
  

  • For Downstream Calls using SQL: Omit the ENDS_WITH filter targeting a specific agent name. Instead, extract the agent identifier (for example, JSON_VALUE(resource.attributes, '$."cloud.resource_id"')) and add it to the GROUP BY clause alongside the model or tool name.

• Directory Inference: Prefer the path explicitly provided by the user (if any). Otherwise, deploy configuration files to target Terraform or SRE folders (such as monitoring/, ops/, sre/). Use tools to locate where alert policies or state pointers exist in the project, rather than blindly writing to the root.

• Notification Channels: By default, never configure any notification channels without user input. If the user explicitly provides a notification channel in their prompt, configure the alerts to use it. If no notification channel is provided, you MUST explicitly ask the user in your final response if they would like to configure notification channels. This is a mandatory question and you MUST NOT omit it from your response. IMPORTANT Do NOT make assumptions about notification channels. If you search the codebase for a notification channel you must ALWAYS confirm with the user before using it.

• Plain English Response: You MUST include a plain English explanation for what the alerts do in your response. This must explain in plain English what the alert measures, how the algorithm works, and what a trigger indicates.

6. Output Verification

• Background Task Cleanup: You MUST verify the status of all background tasks that you spawn. Before completing your execution and returning your final response, you MUST terminate or kill any active or hanging background tasks (using the manage_task tool with action kill).
• Validate Configuration: Run the Config Linting tool to make sure all the output files are written with the correct grammar and structure. See details about the tool in the Tooling Scripts section below.

Tooling Scripts

Use the following scripts to discover agents, gather configuration details, resolve duplicates, and validate configs:

1. Agent Information Gathering: Streamlines discovery, environment auditing (Metric Scopes, BQ Datasets, Notification Channels), table derivations (Log & Trace), and Online Evaluator verifications.
   • Command: python3 scripts/gather_agent_info.py --project-id {project_id} --agent-name {agent_name}
2. Duplicate Verification & Merge: Verifies pre-existing alerts in the target folder to ensure changes are merged in-place rather than appended:
   • Command: python3 scripts/scan_duplicates.py {target_tf_dir} --engine-var '${var.gen_ai_agent_name}'
3. Config Linting: Validates PromQL grammar, matching engine labels, and HCL structure:
   • Command: python3 scripts/lint_syntax.py {path_to_tf_file}
   • Self-Correction Loop: If validation fails (exits non-zero or outputs errors), you MUST read the command output, locate the line/file containing the lint error, analyze the PromQL syntax or Terraform HCL issue, apply adjustments in-place, and re-run the lint_syntax.py validation. Repeat this loop until the validation script passes successfully.

Gotchas & Behavioral Corrections

• Raw Error Boundaries: Explain that raw error counts or absolute failed request count boundaries do not scale under changing traffic throughput. Recommend ratio-based error rate alerts instead.
• Safe Threshold Modulation E2E Validation: When verifying a dynamic metric threshold policy end-to-end, do NOT attempt to force real platform errors. Instead, deploy the alert policy with standard safe bounds (Z-score multiplier > 15), then temporarily update standard deviation Z-score limits to a negative value (for example, > -3) to trigger/verify the "Firing" state before reverting. Always get confirmation before taking this action proactively.
• Expected Script Failures:
  • scan_duplicates.py exiting with code 1: Parse the JSON output for duplicate resource targets. Perform in-place upgrade edits, then re-check until it passes with 0.
  • Avoid Redundant Discovery Calls: If gather_agent_info.py successfully returns the Trace or Log table names (or writes them to variables file), do NOT redundantly call list_trace_scope_table_names.py or list_log_scope_table_names.py. These scripts are run internally by gather_agent_info.py and are provided as external Fallbacks only.
  • Script Execution Failures & Self-Correction: If the execution of utility scripts (such as gather_agent_info.py, check_telemetry.py, create_online_monitor.py, analyze_traffic.py, list_log_scope_table_names.py, or list_trace_scope_table_names.py) fails unexpectedly, you MUST read and inspect the stdout/stderr logs or error output. Analyze the error message and attempt to dynamically correct parameters and retry execution before escalating or falling back to manual plans. Consult the relevant domain-specific reference file for detailed troubleshooting steps for specific scripts.
• Distribution Metric Aligner Constraint: Standard ALIGN_MEAN cannot be applied to DELTA distribution metrics like online_evaluator/scores. You MUST use percentile-based aligners (like ALIGN_PERCENTILE_50) to reduce the score distribution into a comparable numeric stream.
• HCL Heredoc Interpolation: When referencing Terraform variables inside PromQL or SQL queries (which are defined as strings), you MUST use the ${var.variable_name} syntax. Bare references like var.variable_name will fail at deployment time.
• Avoid Recursive Directory Operations: You MUST NOT run recursive listing or search commands (such as ls -R, find ., or raw recursive grep) from the repository root if it contains a very large number of files, as this will freeze your session. Always target specific subdirectories.

Supporting Links

• Continuous evaluation with online monitors
• Agent Platform Quality Metrics
• Google Cloud Alerting Policies Guide
• Google Cloud Monitoring PromQL Documentation