Diagnose Job Run Symptoms

Diagnose Job Run Symptoms

Sippy Symptoms are known-failure signatures for OpenShift CI. A symptom is a rule made of a file pattern (a glob over a CI job run's artifact files, e.g. **/build-log.txt) and a matcher (string = substring, regex = regular expression, none = file merely exists, cel = a compound CEL expression over other label names). When a symptom matches a job run's artifacts, Sippy applies one or more Labels — human-readable tags like InfraFailure — to that run. Labels appear in the Sippy UI and Spyglass and help everyone quickly recognize known failure modes without re-debugging them. You do not need any prior Sippy knowledge to use this skill.

This skill takes a Prow job run URL and explains which symptoms matched the run and what each applied label means — including the matched file and text.

When to Use This Skill

Use this skill when:

• A CI job failed and you want to know if it is a known failure mode before debugging it from scratch
• You are about to create a new symptom and want to check what already matched the run (see manage-symptoms)
• You want a plain-language explanation of the labels shown on a run in the Sippy UI or Spyglass

Prerequisites

1. Default mode (already-applied labels): only network access to public GCS (https://storage.googleapis.com) and the public Sippy API (https://sippy.dptools.openshift.org) — no authentication required.

2. Deep mode (--deep, server-side dry-run rescan): a Bearer token from the DPCR cluster.

   • Must be logged into the DPCR cluster via oc login
   • Cluster API: https://api.cr.j7t7.p1.openshiftapps.com:6443
   • Use the oc-auth skill to obtain the token (see the token-acquisition snippet in reevaluate-job-runs/SKILL.md); prefer export SIPPY_TOKEN=... over --token — argv is visible in process listings

3. Python 3: Python 3.6 or later, standard library only.

Implementation Steps

Step 1: Default mode — explain already-applied labels

python3 plugins/ci/skills/diagnose-job-run-symptoms/diagnose_job_run.py \
  "https://prow.ci.openshift.org/view/gs/test-platform-results/logs/<job>/<build_id>"

For each applied label the summary output shows: the label title and explanation, the symptom that applied it (summary, matcher type, file pattern, match string), and the actual matched file and matched text from the run's artifacts.

Step 2: Deep mode — server-side rescan

Use when the run predates the current symptom set or shows no labels. This asks Sippy to re-scan the run with dry_run: true — it writes nothing and reports what would match now:

python3 plugins/ci/skills/diagnose-job-run-symptoms/diagnose_job_run.py \
  "<prow_url>" --deep

Note: deep mode reports label IDs only — the matched file/text detail is only available in default mode (it comes from the GCS artifacts).

Step 3: Nothing matched?

If default mode finds no labels, first suggest --deep: the run may simply never have been scanned, and default mode cannot distinguish that from "scanned, nothing matched". If a deep rescan also finds nothing and the user has identified the failure cause, guide them to the manage-symptoms skill to create a new symptom (and reevaluate-job-runs to apply it retroactively to past runs).

Arguments:

• prow_url: Prow job run URL (https://prow.ci.openshift.org/view/gs/..., positional, required)

Options:

• --deep: Server-side dry-run rescan via the reevaluate API (requires a token)
• --token <token>: Bearer token from the oc-auth skill (only needed with --deep; optional if the SIPPY_TOKEN environment variable is set, which is preferred — argv is visible in process listings; --token takes precedence)
• --format json|summary: Output format (default: summary)

API Details

Default mode uses the public GCS JSON API (no auth):

• List label artifacts: GET https://storage.googleapis.com/storage/v1/b/{bucket}/o?prefix={path}/artifacts/job_labels/
• Download each object: GET https://storage.googleapis.com/storage/v1/b/{bucket}/o/{object}?alt=media

GCS job_labels artifact schema (verified live 2026-07): each *.json object under artifacts/job_labels/ contains one wrapped entry (a label-summary.html file is also present and skipped):

{
  "symptom_label_v1": {
    "symptom": {"id": "...", "summary": "...", "matcher_type": "string",
                "file_pattern": "...", "match_string": "...", "label_ids": ["..."]},
    "label": {"id": "...", "label_title": "...", "explanation": "..."},
    "file_match": "artifacts/.../nodes.json",
    "text_match": "the exact text that matched"
  }
}

The script also cross-references GET /api/jobs/labels and GET /api/jobs/symptoms on the public Sippy API to enrich entries with current explanations (embedded copies are snapshots from labeling time).

Deep mode calls POST https://sippy-auth.dptools.openshift.org/api/jobs/runs/reevaluate with {"prow_job_build_ids": ["<build_id>"], "dry_run": true} — same request/response as the reevaluate-job-runs skill.

Error Handling

• Bad / non-Prow URL: Must contain /view/gs/ and end in a numeric build ID (exit 1 client-side).
• GCS 404 or no job_labels artifacts: The run may be too old (artifacts pruned) or was never scanned — suggest --deep to rescan server-side.
• No labels found: Not an error — the script prints guidance (try --deep first, then create a new symptom via manage-symptoms).
• 401/403 or HTML login page in deep mode: Token missing/expired (the SSO proxy may return a login page instead of 401) — refresh via the oc-auth skill.
• HTML gateway error page or non-JSON body in deep mode: Transient gateway error (likely 504) — retry later.
• Sippy API unreachable: exit 1 with a clear message.

Exit Codes:

• 0: Diagnosis completed (including "no labels found")
• 1: Validation, network, or API error

See Also

• Related Skill: list-symptoms (browse/search the symptom and label catalogs)
• Related Skill: manage-symptoms (create a new symptom when nothing matched)
• Related Skill: reevaluate-job-runs (apply symptoms retroactively to past runs)
• Related Skill: oc-auth (token for deep mode)
• Related Skill: prow-job-analyze-test-failure (deeper manual failure analysis)