Automate Test Triage

automate-test-triage

Note to AI Agents: Use this skill to automate triaging failing tests from GCS test-metadata JSON files over a multi-day window for monitoring purposes. Do NOT modify source files or create code fixes.

Prerequisites

• Access to gs://nightly-test-data/test-metadata/ via gcloud storage cat (requires roles/storage.objectViewer permission; verify with gcloud auth login if permission denied).
• gh CLI installed for querying GitHub issues.

Execution Steps

1. Execute Triage Script

Run the automated triage script from the workspace root:

• Script: .agents/skills/automate-test-triage/scripts/triage.py
• Command: python3 .agents/skills/automate-test-triage/scripts/triage.py (or specify a target date with --date YYYY-MM-DD, e.g., python3 .agents/skills/automate-test-triage/scripts/triage.py --date 2026-07-23)
• Output File: tmp/test-status/test-report-<date>.md (e.g., tmp/test-status/test-report-2026-07-28.md)

2. Triage & Aggregation Rules

The script performs the following steps:

1. Fetch Metadata: Reads gs://nightly-test-data/test-metadata/{version}/{date}-{version}.json for GA and Beta providers across the 7-day window ending on the target date (defaults to today) using gcloud storage cat. If a GCS permission or authentication failure occurs, the script immediately reports an error with remediation instructions (gcloud auth login, roles/storage.objectViewer) and aborts execution.
2. Extract Error Output: Reads error_message and log_link directly from each test entry in test-metadata.
3. High-Impact Classification (Section 1): Identifies actionable failures in today's latest run flagged by:
   • 🚨 Critical Severity: Provider panic/crash (panic:, runtime error:, SIGSEGV) or API enablement errors in CI test environment projects, regardless of test count.
   • ⚠️ High Volume: Error signatures affecting $\ge 3$ tests in the latest run.
4. Human-Action Separation (Section 2): Identifies non-actionable failures requiring human intervention (e.g., Quota / Rate Limit / Stockout, Internal Error / Error Code 13, Tenant Project Creation failure).
5. Multi-Day Persistence Filtering (Section 3): Identifies actionable tests that failed in the latest run AND failed at least 4 out of the past 7 days (filtering out one-off flaky runs and non-actionable errors).
6. Error Sanitization & Dedup: Replaces dynamic tokens (project IDs, project/org numbers, timestamps, UUIDs, SA emails) to compare error signatures across Beta and GA runs.
7. GitHub Issue Matching: Queries open test-failure issues from hashicorp/terraform-provider-google via gh issue list and matches test names against issue titles.
8. Report Generation: Outputs a 4-section Markdown report with a Table of Contents where every table includes clickable GCS debug log links ([Log](url)), error summaries, and cross-links to Section 4. Sections 1, 2, 3, and 4 are open by default (<details open>):
   • Section 1 (<details open>): High-Impact Actionable Errors in Latest Run (Critical Severity & High Volume; includes note indicating 500-char truncation and linking to Section 4).
   • Section 2 (<details open>): Test Failures Requiring Human Action (Non-Actionable by Agents; includes note indicating 500-char truncation and linking to Section 4).
   • Section 3 (<details open>): Persistent Actionable Failures Grouped by Error Signature (Past 7 Days; includes note indicating 500-char truncation and linking to Section 4).
   • Section 4 (<details open>): Detailed Test Failures Grouped by Service Package (sorted alphabetically A–Z; the first service package is open by default (<details open>) and remaining service packages are collapsed by default (<details>); displays full untruncated error messages, all latest-run test failures per package, and a Human Action Required? column).