Agent Platform Alert Configuration
This skill provides dynamic threshold alerting configurations for Google Cloud /
Vertex AI Reasoning Engines (Agent Platform container deployments) using
extended 1-week lookback retention baselines. Standard static thresholds (e.g.,
"latency > 2s") cause excessive alert noise for AI agents. Dynamic PromQL
baselines solve this.
Safety & Confirmation Tiers (CRITICAL)
Before executing any commands or writing configurations on behalf of the user,
you MUST adhere to the following safety tiers based on the action requested:
- Tier R: Read-only (
check_telemetry.py)
- Rule: No confirmation needed. You may execute these scripts
immediately to inspect the telemetry status of the Reasoning Engine.
- Tier B: Billing & Resource Creation (
create_online_monitor.py /
provisioning)
- Rule: Explicit User Confirmation Required. These actions incur
additional billing charges and create cloud resources. The agent MUST
ask the user directly for approval before proceeding.
CRITICAL RULES
- Always configure both Reliability and Quality alerting policies for the
target agent (6 policies in total):
- For Reliability Monitoring: You MUST configure exactly three
alerting policies:
- Latency (anomaly monitoring)
- Error Rate - Fast Burn SLO (1-Hour Window)
- Error Rate - Slow Burn SLO (3-Day Window)
- For Quality Monitoring: You MUST configure exactly three alerting
policies:
- Final Response Quality
- Tool Use Quality
- Hallucination
- Online Monitor Provisioning & Cost Warning: Quality alerting policies
rely on metrics exported by Online Monitors. You MUST ensure the Online
Monitor is provisioned for the agent and telemetry is enabled:
- Brand New Agents (No Traffic History): When setting up alerts for a
brand new agent, you MUST explicitly ask the user what traffic pattern they
expect (Steady, Seasonal, or Bursty) in your response. If immediate setup is
requested, ask the question but proceed using the default Steady/Consistent
(Short-Window Z-Score) pattern. Follow
no_historical_traffic_data.md.
- PromQL for Reliability (No MQL or Threshold Filters): For the 3
reliability metrics, you MUST use
condition_prometheus_query_language with
PromQL. Do NOT use MQL or standard condition_threshold.
- Standard Threshold Filters for Agent Quality: For the 3 agent quality
metrics, you MUST use standard
condition_threshold filters matching the
monitored resource type aiplatform.googleapis.com/OnlineEvaluator and
metric type aiplatform.googleapis.com/online_evaluator/scores. Do NOT
use PromQL.
- Install Terraform if Necessary: You should use terraform to deploy and
must install terraform if you can't find a valid install.
- Terraform Only: Write the generated observability configuration ONLY as
Terraform (
.tf) files (e.g., alerts.tf, variables.tf).
- Dynamic Multi-Resource Alerting (No Single-Resource Pinning): You MUST
NOT hardcode specific agent IDs or resource name filters (e.g.,
{reasoning_engine_id="[AGENT_ID]"} or
metric.labels.agent_resource_name="[AGENT_NAME]") in alerting conditions
unless explicitly requested. Alerting policies must be written to cover all
active agents in the project dynamically:
- For Reliability Metrics using PromQL: ALWAYS use grouping
aggregations (
by (reasoning_engine_id)) instead of filtering to a
single ID. This allows a single alert policy to dynamically track each
reasoning engine instance separately.
- For Quality Metrics using Standard Threshold Filters: Omit the
agent_resource_name filter entirely. Configure the condition filter to
only target the monitored resource type
(aiplatform.googleapis.com/OnlineEvaluator) and metric type
(aiplatform.googleapis.com/online_evaluator/scores) globally for the
project.
- Check for Pre-existing Policies: Avoid creating duplicate alert policies
for a reasoning engine: scan the target directory or workspace to see if a
policy already exists that targets the same metrics using aggregations
grouped by
reasoning_engine_id.
- Metric Scope Discovery & Project Inference: Centralize alert policies in
a Metric Scope (scoping project) to save costs. Identify if a scope is used
and where policies should live by checking:
- GCP CLI Check: Run
gcloud beta monitoring metrics-scopes list projects/[PROJECT_ID]. If a parent scope
locations/global/metricsScopes/[SCOPING_PROJECT_ID] is returned, a
Metric Scope is active; deploy policies there.
- Infrastructure as Code Scan: Search Terraform configurations for
google_monitoring_monitored_project resources and extract the scoping
project from the metrics_scope attribute.
- Ambiguity Fallback: If unable to determine, ask the user: "Are you
using a multi-project Cloud Monitoring Metric Scope? If so, what is the
scoping project ID?" Deploy policies to the deduced scoping project
(setting the
project attribute in HCL), or default to the local
project.
- Directory Inference: Deploy configuration files to target Terraform or
SRE folders (e.g.
monitoring/, ops/, sre/). Use tools to locate where
alert policies or state pointers exist in the project, rather than blindly
writing to the current working directory.
- Notification Channels: By default, never configure any notification
channels without user input. If the user explicitly provides a notification
channel in their prompt, configure the alerts to use it. If no notification
channel is provided, you MUST explicitly ask the user in your final response
if they would like to configure notification channels. This is a mandatory
question and you MUST NOT omit it from your response. IMPORTANT Do NOT
make assumptions about notification channels. If you search the codebase for
a notification channel you must ALWAYS confirm with the user before using
it.
- Plain English Response: You MUST include a plain English explanation for
what the alerts do in your response. This must explain in plain English what
the alert measures, how the algorithm works, and what a trigger indicates.
- Avoid Recursive Directory Operations: You MUST NOT run recursive listing
or search commands (such as
ls -R, find ., or raw recursive grep) from
the google3 workspace root, as this will hang your session. Always target
specific subdirectories.
- Background Task Cleanup: You MUST check the status of all background
tasks that you spawn. Before completing your execution and returning your
final response, you MUST terminate or kill any active or hanging background
tasks (using the
manage_task tool with action kill).
Algorithm Selection & Policy Mapping Process
Alerting policies for reasoning engine agents MUST map to the correct algorithms
to ensure statistical stability and prevent alert noise or blind spots based on
data classes:
- Latency: Follows workload traffic pattern (Steady -> Z-Score; Seasonal
-> Seasonal Decomposition; Bursty -> Moving Averages).
- Error Rate: ALWAYS use Multi-Window Multi-Burn Rate SLOs (or
ratio-based static thresholds). Error rate is naturally sparse (normally
0). When standard deviation is 0, Z-score computation is mathematically
unstable (division-by-zero or NaN), causing false alert storms.
To resolve the workload traffic pattern (Seasonal, Steady, or Bursty), follow
the instructions corresponding to the availability of historical metrics data:
- Case 1: No historical metrics data available (e.g., brand new agent):
You MUST read and follow:
no_historical_traffic_data.md
- Case 2: Historical metrics data available (e.g., active agent with
traffic): You MUST read and follow:
has_historical_traffic_data.md
Telemetry Metrics and PromQL Examples
All raw telemetry metrics for the Agent Platform are cumulative counters.
Because we monitor their rates or quantiles, we can optimize the PromQL queries
by using longer range windows (e.g., [1w]) for historical averages instead of
expensive avg_over_time subqueries.
| Signal |
Raw Metric |
Type |
Description |
| Latency |
reasoning_engine_request_latencies_bucket |
Counter |
Histogram bucket of request latencies |
| Error Rate |
reasoning_engine_request_count |
Counter |
Cumulative count of requests |
For the specific PromQL queries corresponding to each algorithm, you MUST read
and follow: promql_queries.md
Agent Quality Metrics (Online Monitor)
All agent quality evaluation metrics are exported by Online Monitors to the
monitored resource type aiplatform.googleapis.com/OnlineEvaluator under the
metric type aiplatform.googleapis.com/online_evaluator/scores.
Metric Details & Aligners
Because the scores metric is of value type DISTRIBUTION, standard mean-based
PromQL or arithmetic ALIGN_MEAN aligners are unsupported. You MUST use a
percentile aligner (typically ALIGN_PERCENTILE_50 to evaluate the median
score) within the aggregations block of your condition_threshold.
| Signal |
Metric Name (evaluation_metric_name) |
Target Threshold |
Recommended Aligner |
| Final Response Quality |
final_response_quality_v1 |
< 0.8 (or custom) |
ALIGN_PERCENTILE_50 |
| Tool Use Quality |
tool_use_quality_v1 |
< 0.8 (or custom) |
ALIGN_PERCENTILE_50 |
| Hallucination (Groundedness) |
hallucination_v1 |
< 0.9 (or custom) |
ALIGN_PERCENTILE_50 |
Metric Filter Example
When configuring a quality alert policy in Terraform, use the following filter
expression structure:
resource.type="aiplatform.googleapis.com/OnlineEvaluator"
AND metric.type="aiplatform.googleapis.com/online_evaluator/scores"
AND metric.labels.evaluation_metric_name="[METRIC_NAME]"
Online Monitor & Telemetry Provisioning
Quality metrics are generated by the Online Monitor by evaluating trace data
exported to Cloud Trace. If telemetry is disabled on the reasoning engine, no
traces are sent, and the quality metrics will remain empty.
Prerequisites & Dependencies
Before executing any scripts in this skill (such as check_telemetry.py or
create_online_monitor.py), you MUST install the required dependencies in your
environment. Run this command first:
pip install -r scripts/requirements.txt
Verify Telemetry Status
Before generating any alerting policies, proposing a plan, or provisioning
Online Monitors, you MUST always check if the agent is ready to export traces by
running the telemetry checking script:
- Mandatory Command:
python3 scripts/check_telemetry.py --project-id "[PROJECT_ID]" --agent-resource-name "[AGENT_RESOURCE_NAME]"
- Note on Parameters: The
[AGENT_RESOURCE_NAME] parameter MUST be
the full resource path format
projects/<project_id>/locations/<location>/reasoningEngines/<agent_id>
(e.g. projects/gcp-prod/locations/us-central1/reasoningEngines/556677)
and not just the agent ID itself.
- Dependency Failures: If package imports or dependency installation
fails, try your best to resolve the issues (e.g., by verifying package
installation) and run the script again with the specific project-id and
agent-resource-name parameters. If you cannot run the script
successfully due to missing dependencies, you MUST still attempt to run
it first and then include the complete, pre-populated execution command
in your final response as a verification plan (this applies even if the
user has pre-approved provisioning).
- Detailed Enablement Instructions: For details on the required
environment variables, Terraform setups, and project dependencies, you MUST
read and follow:
telemetry_enablement.md.
Provision the Online Monitor
Because Online Monitors cannot be configured via Terraform, run the provisioning
script to generate the required metrics:
- Command:
python3 scripts/create_online_monitor.py --project-id "[PROJECT_ID]" --agent-resource-name "[AGENT_RESOURCE_NAME]" [--sampling-percentage [PERCENTAGE]]
- Sampling Rate Recommendation: For production agents, configure a
conservative sampling percentage (default: 10%) to control LLM
evaluation costs. For details, refer to
Continuous evaluation with online monitors.
Formatting the Execution Plan
If execution fails (e.g. due to sandbox restrictions or permissions), you MUST
plan it as a concrete next step. To do this, you MUST explicitly include a
section titled "Online Monitor Provisioning Plan" or "Execution Plan" in
your final response, containing the exact concrete python execution command with
all parameter values (such as project ID, region, and agent resource name) fully
populated. Do not merely state that the user should run it.
You MUST format the plan exactly as follows:
Execution Plan: Online Monitor Provisioning
Online Monitor Provisioning Command:
python3 scripts/create_online_monitor.py \
--project-id "[PROJECT_ID]" \
--agent-resource-name "projects/[PROJECT_ID]/locations/[LOCATION]/reasoningEngines/[AGENT_ID]" \
--sampling-percentage [PERCENTAGE]
Verify Telemetry Command (Optional fallback):
python3 scripts/check_telemetry.py \
--project-id "[PROJECT_ID]" \
--agent-resource-name "projects/[PROJECT_ID]/locations/[LOCATION]/reasoningEngines/[AGENT_ID]"
Tooling Scripts
Use the following scripts to resolve duplicates and validate configs before
presenting or applying Terraform changes:
- Duplicate Check & Merge: Checks for pre-existing alerts in the target
folder to ensure changes are merged in-place rather than appended:
- Command:
python3 scripts/validate_config.py --directory [TARGET_TF_DIR] --engine-var "${var.reasoning_engine_id}"
- Config Linting: Validates PromQL grammar, matching engine labels, and
HCL structure:
- Command:
python3 scripts/validate_config.py --file [PATH_TO_TF_FILE]
- Self-Correction Loop: If validation fails (exits non-zero or outputs
errors), you MUST read the command output, locate the line/file
containing the lint error, analyze the PromQL syntax or Terraform HCL
issue, apply adjustments in-place, and re-run the
validate_config.py --file validation. Repeat this loop until the validation script passes
successfully.
Gotchas & Behavioral Corrections
- Duration Buffers (Transient Glitches): To avoid alerts firing on
transient spikes, use duration/retest window buffers appropriately:
- Reliability Metrics (PromQL / Cloud Monitoring):
- For short-lookback alerts querying data under 25 hours (e.g.,
Short-Window Z-Score, Moving Averages, Fast Burn SLO), ALWAYS use a
duration = "300s" (5 minutes) buffer to filter out transient cold
start/deployment spikes.
- For long-lookback alerts querying data longer than 25 hours (e.g.,
Long-Window Z-Score, Seasonal Decomposition, Slow Burn SLO),
duration/retest windows are disabled by the platform. You must not
set a duration (omit it entirely).
- Quality Metrics (Standard Filters / Online Monitor):
- Always use a
duration = "300s" (5 minutes) buffer to filter out
transient scoring dips or evaluation outliers caused by temporary
LLM judge congestion, or edge-case query outliers.
- Dynamic Baseline Adaptation Blind Spot: Explain to users that dynamic
statistical Z-score thresholds compare current rates to a moving statistical
baseline. If a system degrades slowly over days, the standard baseline curve
adapts to this slow drift, making standard Z-score alerts blind to
persistent slow errors. Recommend a hard static threshold alert in parallel
for strict SLA enforcement.
- Seasonal Decomposition Double Alerting: The agent MUST ONLY configure
seasonal decomposition alert policies to track spikes (e.g., latency spikes)
OR drops AND MUST NOT use dual-direction checks (like absolute deviation).
Explain this limitation to the user: comparing to a historical offset (e.g.,
offset 1w) the alert policy triggers twice if tracking both directions
(once for the anomaly, and once 1 week later when the anomaly becomes the
baseline). To prevent this, the generated policy MUST only track either
spikes (using >) or drops (using <), avoiding using abs().
- Raw Error Boundaries: Explain that raw error counts or absolute failed
request count boundaries do not scale under changing traffic throughput.
Recommend ratio-based error rate alerts instead.
- Safe Threshold Modulation E2E Validation: When verifying a dynamic
metric threshold policy end-to-end, do NOT attempt to force real platform
errors. Instead, deploy the alert policy with standard safe bounds (Z-score
multiplier > 15), then temporarily update standard deviation Z-score limits
to a negative value (e.g. > -3) to trigger/verify the "Firing" state before
reverting. Always get confirmation before taking this action proactively.
- Expected Script Failures:
validate_config.py --directory exiting with code 1: Parse the JSON
output for duplicate resource targets. Perform in-place upgrade edits,
then re-check until it passes with 0.
- Script Execution Failures & Self-Correction: If the execution of
utility scripts (such as
check_telemetry.py,
create_online_monitor.py, or analyze_traffic.py) fails unexpectedly,
you MUST read and inspect the stdout/stderr logs or error output.
Analyze the error message (e.g., connection timeouts, invalid
permissions, or missing resources) and attempt to dynamically correct
parameters (such as verifying or correcting the region, project ID, or
resource name format) and retry execution before escalating or falling
back to manual plans.
- Distribution Metric Aligner Constraint: Standard
ALIGN_MEAN cannot be
applied to DELTA distribution metrics like online_evaluator/scores. You
MUST use percentile-based aligners (like ALIGN_PERCENTILE_50) to reduce
the score distribution into a comparable numeric stream.
Supporting Links
1---2name: agent-platform-alert-configuration3metadata:4 source: https://skillmd.com/skills/google/agent-platform-alert-configuration5 category: AiAndMachineLearning6description: >-7 Configures best-practice alerting policies for Google Cloud Vertex AI / Agent8 Platform agents on Agent Runtime. Use when analyzing, writing, or deploying9 alerting policies to monitor agent latency, error rates, and quality metrics10 (response quality, tool use, hallucination). Also use when provisioning online11 monitors for quality evaluation, or analyzing live metrics traffic footprints.12 NOTE: This skill currently only works for the Agent Runtime. Don't use for13 configuring general GCP alert policies or non-agent GCP alerting policies.14allowed-tools: terraform gcloud python15---1617# Agent Platform Alert Configuration1819This skill provides dynamic threshold alerting configurations for Google Cloud /20Vertex AI Reasoning Engines (Agent Platform container deployments) using21extended 1-week lookback retention baselines. Standard static thresholds (e.g.,22"latency > 2s") cause excessive alert noise for AI agents. Dynamic PromQL23baselines solve this.2425## Safety & Confirmation Tiers (CRITICAL)2627Before executing any commands or writing configurations on behalf of the user,28you MUST adhere to the following safety tiers based on the action requested:29301. **Tier R: Read-only (`check_telemetry.py`)**31 * **Rule**: No confirmation needed. You may execute these scripts32 immediately to inspect the telemetry status of the Reasoning Engine.332. **Tier B: Billing & Resource Creation (`create_online_monitor.py` /34 provisioning)**35 * **Rule**: **Explicit User Confirmation Required**. These actions incur36 additional billing charges and create cloud resources. The agent MUST37 ask the user directly for approval before proceeding.3839--------------------------------------------------------------------------------4041## CRITICAL RULES4243* **Always configure both Reliability and Quality alerting policies** for the44 target agent (6 policies in total):45 * **For Reliability Monitoring**: You MUST configure exactly three46 alerting policies:47 1. **Latency** (anomaly monitoring)48 2. **Error Rate - Fast Burn SLO** (1-Hour Window)49 3. **Error Rate - Slow Burn SLO** (3-Day Window)50 * **For Quality Monitoring**: You MUST configure exactly three alerting51 policies:52 1. **Final Response Quality**53 2. **Tool Use Quality**54 3. **Hallucination**55* **Online Monitor Provisioning & Cost Warning**: Quality alerting policies56 rely on metrics exported by Online Monitors. You MUST ensure the Online57 Monitor is provisioned for the agent and telemetry is enabled:58 - [ ] **Ask for Approval**: Both Online Monitors and Telemetry incur59 separate billing charges. Before provisioning them, you MUST warn the60 user about these extra costs. If not pre-approved in the prompt, you61 MUST ask a direct question in your response requesting62 confirmation/approval to proceed (e.g., "Please confirm if you approve63 the extra billing costs for the Online Monitor and Telemetry to64 proceed.").65 - [ ] **Verify Telemetry First**: Before generating any alerting policy66 plan or provisioning Online Monitors, you MUST always verify the67 telemetry status of the Reasoning Engine first using the68 `check_telemetry.py` script as detailed in69 [Verify Telemetry Status](#verify-telemetry-status) below.70 - [ ] **Follow the Guide**: Follow the step-by-step instructions in the71 [Online Monitor & Telemetry Provisioning](#online-monitor--telemetry-provisioning)72 section below.73* **Brand New Agents (No Traffic History)**: When setting up alerts for a74 brand new agent, you MUST explicitly ask the user what traffic pattern they75 expect (Steady, Seasonal, or Bursty) in your response. If immediate setup is76 requested, ask the question but proceed using the default Steady/Consistent77 (Short-Window Z-Score) pattern. Follow78 [no_historical_traffic_data.md](references/no_historical_traffic_data.md).79* **PromQL for Reliability (No MQL or Threshold Filters)**: For the 380 reliability metrics, you MUST use `condition_prometheus_query_language` with81 PromQL. Do **NOT** use MQL or standard `condition_threshold`.82* **Standard Threshold Filters for Agent Quality**: For the 3 agent quality83 metrics, you MUST use standard `condition_threshold` filters matching the84 monitored resource type `aiplatform.googleapis.com/OnlineEvaluator` and85 metric type `aiplatform.googleapis.com/online_evaluator/scores`. Do **NOT**86 use PromQL.87* **Install Terraform if Necessary**: You should use terraform to deploy and88 must install terraform if you can't find a valid install.89* **Terraform Only**: Write the generated observability configuration ONLY as90 Terraform (`.tf`) files (e.g., `alerts.tf`, `variables.tf`).91* **Dynamic Multi-Resource Alerting (No Single-Resource Pinning)**: You MUST92 NOT hardcode specific agent IDs or resource name filters (e.g.,93 `{reasoning_engine_id="[AGENT_ID]"}` or94 `metric.labels.agent_resource_name="[AGENT_NAME]"`) in alerting conditions95 unless explicitly requested. Alerting policies must be written to cover all96 active agents in the project dynamically:97 * **For Reliability Metrics using PromQL**: ALWAYS use grouping98 aggregations (`by (reasoning_engine_id)`) instead of filtering to a99 single ID. This allows a single alert policy to dynamically track each100 reasoning engine instance separately.101 * **For Quality Metrics using Standard Threshold Filters**: Omit the102 `agent_resource_name` filter entirely. Configure the condition filter to103 only target the monitored resource type104 (`aiplatform.googleapis.com/OnlineEvaluator`) and metric type105 (`aiplatform.googleapis.com/online_evaluator/scores`) globally for the106 project.107* **Check for Pre-existing Policies**: Avoid creating duplicate alert policies108 for a reasoning engine: scan the target directory or workspace to see if a109 policy already exists that targets the same metrics using aggregations110 grouped by `reasoning_engine_id`.111* **Metric Scope Discovery & Project Inference**: Centralize alert policies in112 a Metric Scope (scoping project) to save costs. Identify if a scope is used113 and where policies should live by checking:114 1. **GCP CLI Check**: Run `gcloud beta monitoring metrics-scopes list115 projects/[PROJECT_ID]`. If a parent scope116 `locations/global/metricsScopes/[SCOPING_PROJECT_ID]` is returned, a117 Metric Scope is active; deploy policies there.118 2. **Infrastructure as Code Scan**: Search Terraform configurations for119 `google_monitoring_monitored_project` resources and extract the scoping120 project from the `metrics_scope` attribute.121 3. **Ambiguity Fallback**: If unable to determine, ask the user: "Are you122 using a multi-project Cloud Monitoring Metric Scope? If so, what is the123 scoping project ID?" Deploy policies to the deduced scoping project124 (setting the `project` attribute in HCL), or default to the local125 project.126* **Directory Inference**: Deploy configuration files to target Terraform or127 SRE folders (e.g. `monitoring/`, `ops/`, `sre/`). Use tools to locate where128 alert policies or state pointers exist in the project, rather than blindly129 writing to the current working directory.130* **Notification Channels**: By default, never configure any notification131 channels without user input. If the user explicitly provides a notification132 channel in their prompt, configure the alerts to use it. If no notification133 channel is provided, you MUST explicitly ask the user in your final response134 if they would like to configure notification channels. **This is a mandatory135 question and you MUST NOT omit it from your response.** **IMPORTANT** Do NOT136 make assumptions about notification channels. If you search the codebase for137 a notification channel you must ALWAYS confirm with the user before using138 it.139* **Plain English Response**: You MUST include a plain English explanation for140 what the alerts do in your response. This must explain in plain English what141 the alert measures, how the algorithm works, and what a trigger indicates.142* **Avoid Recursive Directory Operations**: You MUST NOT run recursive listing143 or search commands (such as `ls -R`, `find .`, or raw recursive `grep`) from144 the google3 workspace root, as this will hang your session. Always target145 specific subdirectories.146* **Background Task Cleanup**: You MUST check the status of all background147 tasks that you spawn. Before completing your execution and returning your148 final response, you MUST terminate or kill any active or hanging background149 tasks (using the `manage_task` tool with action `kill`).150151--------------------------------------------------------------------------------152153## Algorithm Selection & Policy Mapping Process154155Alerting policies for reasoning engine agents MUST map to the correct algorithms156to ensure statistical stability and prevent alert noise or blind spots based on157data classes:158159* **Latency**: Follows workload traffic pattern (Steady -> Z-Score; Seasonal160 -> Seasonal Decomposition; Bursty -> Moving Averages).161* **Error Rate**: ALWAYS use **Multi-Window Multi-Burn Rate SLOs** (or162 ratio-based static thresholds). Error rate is naturally sparse (normally163 `0`). When standard deviation is `0`, Z-score computation is mathematically164 unstable (division-by-zero or NaN), causing false alert storms.165166To resolve the workload traffic pattern (Seasonal, Steady, or Bursty), follow167the instructions corresponding to the availability of historical metrics data:168169* **Case 1: No historical metrics data available (e.g., brand new agent)**:170 You MUST read and follow:171 [no_historical_traffic_data.md](references/no_historical_traffic_data.md)172* **Case 2: Historical metrics data available (e.g., active agent with173 traffic)**: You MUST read and follow:174 [has_historical_traffic_data.md](references/has_historical_traffic_data.md)175176--------------------------------------------------------------------------------177178## Telemetry Metrics and PromQL Examples179180All raw telemetry metrics for the Agent Platform are cumulative **counters**.181Because we monitor their rates or quantiles, we can optimize the PromQL queries182by using longer range windows (e.g., `[1w]`) for historical averages instead of183expensive `avg_over_time` subqueries.184185Signal | Raw Metric | Type | Description186:------------- | :------------------------------------------ | :------ | :----------187**Latency** | `reasoning_engine_request_latencies_bucket` | Counter | Histogram bucket of request latencies188**Error Rate** | `reasoning_engine_request_count` | Counter | Cumulative count of requests189190--------------------------------------------------------------------------------191192For the specific PromQL queries corresponding to each algorithm, you MUST read193and follow: [promql_queries.md](references/promql_queries.md)194195--------------------------------------------------------------------------------196197## Agent Quality Metrics (Online Monitor)198199All agent quality evaluation metrics are exported by Online Monitors to the200monitored resource type `aiplatform.googleapis.com/OnlineEvaluator` under the201metric type `aiplatform.googleapis.com/online_evaluator/scores`.202203### Metric Details & Aligners204205Because the scores metric is of value type `DISTRIBUTION`, standard mean-based206PromQL or arithmetic `ALIGN_MEAN` aligners are unsupported. You MUST use a207percentile aligner (typically `ALIGN_PERCENTILE_50` to evaluate the median208score) within the `aggregations` block of your `condition_threshold`.209210Signal | Metric Name (`evaluation_metric_name`) | Target Threshold | Recommended Aligner211:------------------------------- | :------------------------------------- | :------------------ | :------------------212**Final Response Quality** | `final_response_quality_v1` | `< 0.8` (or custom) | `ALIGN_PERCENTILE_50`213**Tool Use Quality** | `tool_use_quality_v1` | `< 0.8` (or custom) | `ALIGN_PERCENTILE_50`214**Hallucination (Groundedness)** | `hallucination_v1` | `< 0.9` (or custom) | `ALIGN_PERCENTILE_50`215216### Metric Filter Example217218When configuring a quality alert policy in Terraform, use the following filter219expression structure:220221```filter222resource.type="aiplatform.googleapis.com/OnlineEvaluator"223AND metric.type="aiplatform.googleapis.com/online_evaluator/scores"224AND metric.labels.evaluation_metric_name="[METRIC_NAME]"225```226227### Online Monitor & Telemetry Provisioning228229Quality metrics are generated by the Online Monitor by evaluating trace data230exported to Cloud Trace. If telemetry is disabled on the reasoning engine, no231traces are sent, and the quality metrics will remain empty.232233#### Prerequisites & Dependencies234235Before executing any scripts in this skill (such as `check_telemetry.py` or236`create_online_monitor.py`), you MUST install the required dependencies in your237environment. Run this command first:238239```bash240pip install -r scripts/requirements.txt241```242243#### Verify Telemetry Status244245Before generating any alerting policies, proposing a plan, or provisioning246Online Monitors, you MUST always check if the agent is ready to export traces by247running the telemetry checking script:248249* **Mandatory Command**: `python3 scripts/check_telemetry.py --project-id250 "[PROJECT_ID]" --agent-resource-name "[AGENT_RESOURCE_NAME]"`251 * **Note on Parameters**: The `[AGENT_RESOURCE_NAME]` parameter MUST be252 the full resource path format253 `projects/<project_id>/locations/<location>/reasoningEngines/<agent_id>`254 (e.g. `projects/gcp-prod/locations/us-central1/reasoningEngines/556677`)255 and not just the agent ID itself.256 * **Dependency Failures**: If package imports or dependency installation257 fails, try your best to resolve the issues (e.g., by verifying package258 installation) and run the script again with the specific project-id and259 agent-resource-name parameters. If you cannot run the script260 successfully due to missing dependencies, you MUST still attempt to run261 it first and then include the complete, pre-populated execution command262 in your final response as a verification plan (this applies even if the263 user has pre-approved provisioning).264* **Detailed Enablement Instructions**: For details on the required265 environment variables, Terraform setups, and project dependencies, you MUST266 read and follow:267 [telemetry_enablement.md](references/telemetry_enablement.md).268269#### Provision the Online Monitor270271Because Online Monitors cannot be configured via Terraform, run the provisioning272script to generate the required metrics:273274* **Command**: `python3 scripts/create_online_monitor.py --project-id275 "[PROJECT_ID]" --agent-resource-name "[AGENT_RESOURCE_NAME]"276 [--sampling-percentage [PERCENTAGE]]`277* **Sampling Rate Recommendation**: For production agents, configure a278 conservative sampling percentage (default: **10%**) to control LLM279 evaluation costs. For details, refer to280 [Continuous evaluation with online monitors](https://docs.cloud.google.com/gemini-enterprise-agent-platform/optimize/evaluation/evaluate-online).281282#### Formatting the Execution Plan283284If execution fails (e.g. due to sandbox restrictions or permissions), you MUST285plan it as a concrete next step. To do this, you MUST explicitly include a286section titled **"Online Monitor Provisioning Plan"** or **"Execution Plan"** in287your final response, containing the exact concrete python execution command with288all parameter values (such as project ID, region, and agent resource name) fully289populated. Do not merely state that the user should run it.290291You MUST format the plan exactly as follows:292293### Execution Plan: Online Monitor Provisioning294295**Online Monitor Provisioning Command:**296297```bash298python3 scripts/create_online_monitor.py \299 --project-id "[PROJECT_ID]" \300 --agent-resource-name "projects/[PROJECT_ID]/locations/[LOCATION]/reasoningEngines/[AGENT_ID]" \301 --sampling-percentage [PERCENTAGE]302```303304**Verify Telemetry Command (Optional fallback):**305306```bash307python3 scripts/check_telemetry.py \308 --project-id "[PROJECT_ID]" \309 --agent-resource-name "projects/[PROJECT_ID]/locations/[LOCATION]/reasoningEngines/[AGENT_ID]"310```311312--------------------------------------------------------------------------------313314## Tooling Scripts315316Use the following scripts to resolve duplicates and validate configs before317presenting or applying Terraform changes:3183191. **Duplicate Check & Merge**: Checks for pre-existing alerts in the target320 folder to ensure changes are merged in-place rather than appended:321 * Command: `python3 scripts/validate_config.py --directory [TARGET_TF_DIR]322 --engine-var "${var.reasoning_engine_id}"`3232. **Config Linting**: Validates PromQL grammar, matching engine labels, and324 HCL structure:325 * Command: `python3 scripts/validate_config.py --file [PATH_TO_TF_FILE]`326 * **Self-Correction Loop**: If validation fails (exits non-zero or outputs327 errors), you MUST read the command output, locate the line/file328 containing the lint error, analyze the PromQL syntax or Terraform HCL329 issue, apply adjustments in-place, and re-run the `validate_config.py330 --file` validation. Repeat this loop until the validation script passes331 successfully.332333--------------------------------------------------------------------------------334335## Gotchas & Behavioral Corrections336337* **Duration Buffers (Transient Glitches)**: To avoid alerts firing on338 transient spikes, use duration/retest window buffers appropriately:339 * **Reliability Metrics (PromQL / Cloud Monitoring)**:340 * For short-lookback alerts querying data under 25 hours (e.g.,341 Short-Window Z-Score, Moving Averages, Fast Burn SLO), ALWAYS use a342 `duration = "300s"` (5 minutes) buffer to filter out transient cold343 start/deployment spikes.344 * For long-lookback alerts querying data longer than 25 hours (e.g.,345 Long-Window Z-Score, Seasonal Decomposition, Slow Burn SLO),346 duration/retest windows are disabled by the platform. You must **not347 set a duration** (omit it entirely).348 * **Quality Metrics (Standard Filters / Online Monitor)**:349 * Always use a `duration = "300s"` (5 minutes) buffer to filter out350 transient scoring dips or evaluation outliers caused by temporary351 LLM judge congestion, or edge-case query outliers.352* **Dynamic Baseline Adaptation Blind Spot**: Explain to users that dynamic353 statistical Z-score thresholds compare current rates to a moving statistical354 baseline. If a system degrades slowly over days, the standard baseline curve355 adapts to this slow drift, making standard Z-score alerts blind to356 persistent slow errors. Recommend a hard static threshold alert in parallel357 for strict SLA enforcement.358* **Seasonal Decomposition Double Alerting**: The agent MUST ONLY configure359 seasonal decomposition alert policies to track spikes (e.g., latency spikes)360 OR drops AND MUST NOT use dual-direction checks (like absolute deviation).361 Explain this limitation to the user: comparing to a historical offset (e.g.,362 `offset 1w`) the alert policy triggers twice if tracking both directions363 (once for the anomaly, and once 1 week later when the anomaly becomes the364 baseline). To prevent this, the generated policy MUST only track either365 spikes (using `>`) or drops (using `<`), avoiding using `abs()`.366* **Raw Error Boundaries**: Explain that raw error counts or absolute failed367 request count boundaries do not scale under changing traffic throughput.368 Recommend ratio-based error rate alerts instead.369* **Safe Threshold Modulation E2E Validation**: When verifying a dynamic370 metric threshold policy end-to-end, do NOT attempt to force real platform371 errors. Instead, deploy the alert policy with standard safe bounds (Z-score372 multiplier > 15), then temporarily update standard deviation Z-score limits373 to a negative value (e.g. > -3) to trigger/verify the "Firing" state before374 reverting. Always get confirmation before taking this action proactively.375* **Expected Script Failures**:376 * `validate_config.py --directory` exiting with code 1: Parse the JSON377 output for duplicate resource targets. Perform in-place upgrade edits,378 then re-check until it passes with 0.379 * **Script Execution Failures & Self-Correction**: If the execution of380 utility scripts (such as `check_telemetry.py`,381 `create_online_monitor.py`, or `analyze_traffic.py`) fails unexpectedly,382 you MUST read and inspect the stdout/stderr logs or error output.383 Analyze the error message (e.g., connection timeouts, invalid384 permissions, or missing resources) and attempt to dynamically correct385 parameters (such as verifying or correcting the region, project ID, or386 resource name format) and retry execution before escalating or falling387 back to manual plans.388* **Distribution Metric Aligner Constraint**: Standard `ALIGN_MEAN` cannot be389 applied to `DELTA` distribution metrics like `online_evaluator/scores`. You390 MUST use percentile-based aligners (like `ALIGN_PERCENTILE_50`) to reduce391 the score distribution into a comparable numeric stream.392393--------------------------------------------------------------------------------394395## Supporting Links396397* [Continuous evaluation with online monitors](https://docs.cloud.google.com/gemini-enterprise-agent-platform/optimize/evaluation/evaluate-online)398* [Agent Platform Quality Metrics](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/rubric-metric-details)399* [Google Cloud Alerting Policies Guide](https://cloud.google.com/monitoring/alerts)400* [Google Cloud Monitoring PromQL Documentation](https://cloud.google.com/monitoring/promql)