Manage Symptoms

Manage Symptoms

Sippy Symptoms are known-failure signatures for OpenShift CI. A symptom is a rule made of a file pattern (a glob over a CI job run's artifact files, e.g. **/build-log.txt) and a matcher (string = substring, regex = regular expression, none = file merely exists, cel = a compound CEL expression over other label names). When a symptom matches a job run's artifacts, Sippy applies one or more Labels — human-readable tags like InfraFailure — to that run. Labels appear in the Sippy UI and Spyglass and help everyone quickly recognize known failure modes without re-debugging them. You do not need any prior Sippy knowledge to use this skill.

When to Use This Skill

Use this skill when you need to:

• Auto-label a recurring failure pattern you found in CI logs, so future runs are recognized without re-debugging
• Correct an existing symptom's matcher, file pattern, or match string
• Retire an obsolete symptom that no longer applies

Prerequisites

1. OpenShift CLI Authentication: Required for authenticating to the sippy-auth API

   • Must be logged into the DPCR cluster via oc login
   • Cluster API: https://api.cr.j7t7.p1.openshiftapps.com:6443
   • Use the oc-auth skill to obtain the Bearer token

2. Python 3: Python 3.6 or later

   • Check: python3 --version
   • Uses only standard library (no external dependencies)

Implementation Steps

Step 1: Check for Duplicates

Before creating a symptom, search the existing catalog to avoid duplicates:

python3 plugins/ci/skills/list-symptoms/list_symptoms.py --search "AuthFailure" --format summary

If an equivalent symptom already exists, prefer updating it instead of creating a new one.

Step 2: Verify the Target Labels Exist

Symptoms can only reference labels that already exist:

python3 plugins/ci/skills/list-symptoms/list_symptoms.py --labels --format summary

If a label is missing, create it first with the manage-labels skill. (The script also verifies label IDs against the labels API before submitting; use --skip-label-check only if the labels API is unreachable.)

Step 3: Obtain Authentication Token

Use the oc-auth skill to obtain a Bearer token from the DPCR cluster:

# Get token from the DPCR cluster context
# The oc-auth skill's curl_with_token.sh uses this cluster for sippy-auth
DPCR_CLUSTER="https://api.cr.j7t7.p1.openshiftapps.com:6443"

# Find the oc context for the DPCR cluster and get the token
CONTEXT=$(oc config get-contexts -o name 2>/dev/null | while read -r ctx; do
  server=$(oc config view -o jsonpath="{.clusters[?(@.name=='$(oc config view -o jsonpath="{.contexts[?(@.name=='$ctx')].context.cluster}" 2>/dev/null)')].cluster.server}" 2>/dev/null || echo "")
  server_clean=$(echo "$server" | sed -E 's|^https?://||')
  if [ "$server_clean" = "api.cr.j7t7.p1.openshiftapps.com:6443" ]; then
    echo "$ctx"
    break
  fi
done)

if [ -z "$CONTEXT" ]; then
  echo "Error: Not logged into DPCR cluster. Please run: oc login $DPCR_CLUSTER"
  exit 1
fi

export SIPPY_TOKEN=$(oc whoami -t --context="$CONTEXT" 2>/dev/null)
if [ -z "$SIPPY_TOKEN" ]; then
  echo "Error: Failed to get token. Please re-authenticate to DPCR cluster."
  exit 1
fi

Prefer exporting SIPPY_TOKEN as above rather than passing --token on the command line — command-line arguments are visible in process listings. --token still works and takes precedence over the environment variable.

Step 4: Confirm the Payload with the User

Before any create or update, show the user the full payload that will be sent (summary, matcher type, file pattern, match string, label IDs) and get their confirmation. Before delete, you MUST show the symptom (list-symptoms --id <id> --format summary) and get explicit confirmation — never run delete without the user confirming the specific symptom.

Step 5: Create a Symptom

python3 plugins/ci/skills/manage-symptoms/manage_symptoms.py create \
  --summary "AWS could not validate access credentials" \
  --matcher-type string \
  --file-pattern "build-log.txt" \
  --match-string "api error AuthFailure: AWS was not able to validate the provided access credentials" \
  --label-ids InfraFailure

The symptom id is generated by the server from the summary — do not pass --id on create.

Step 6: Update a Symptom

Only pass the flags you want to change — the script fetches the existing symptom and merges, because the API's PUT is a full replacement:

python3 plugins/ci/skills/manage-symptoms/manage_symptoms.py update \
  --id AWSCouldNotValidateAccessCredentials \
  --match-string "api error AuthFailure"

To remove all labels from a symptom, pass an explicit empty value: --label-ids "" clears the label list on update (omitting the flag preserves the existing labels).

Step 7: Delete a Symptom

Delete is a soft delete on the server side. Requires explicit user confirmation first (see Step 4):

python3 plugins/ci/skills/manage-symptoms/manage_symptoms.py delete \
  --id AWSCouldNotValidateAccessCredentials

Step 8: Verify

After create/update, verify the result:

python3 plugins/ci/skills/list-symptoms/list_symptoms.py --id <new-id> --format summary

Optionally test the symptom against a known-affected run with the reevaluate-job-runs skill using --dry-run to preview matches without writing anything.

Arguments:

• action: create, update, or delete (positional, required)

Options:

• --token <token>: Bearer token from the oc-auth skill (optional if the SIPPY_TOKEN environment variable is set, which is preferred — argv is visible in process listings; --token takes precedence)
• --id <id>: Symptom ID (required for update/delete; server-generated on create)
• --summary <text>: Short unique description (required for create, max 200 characters)
• --matcher-type string|regex|none|cel: How the match string is interpreted
• --file-pattern <glob>: Artifact glob, e.g. **/build-log.txt (required for non-CEL matchers)
• --match-string <text>: Substring, regex, or CEL expression
• --label-ids <list>: Comma-separated label IDs to apply on match; on update, --label-ids "" clears all labels
• --skip-label-check: Skip verifying label IDs against the labels API
• --format json|summary: Output format (default: json)

API Details

Base URL (writes): https://sippy-auth.dptools.openshift.org/api/jobs/symptoms

• Create: POST /api/jobs/symptoms
• Update: PUT /api/jobs/symptoms/{id} (full replacement — the script fetches the existing symptom and merges your changes, so only pass flags you want to change)
• Delete: DELETE /api/jobs/symptoms/{id} (soft delete)

Authentication: Authorization: Bearer <token> from the DPCR cluster.

Symptom fields:

Matcher-type rules:

• string / regex: require both file_pattern and match_string
• none: requires only file_pattern (matches when the file exists)
• cel: requires only match_string (a CEL expression over other label names)

Error Handling

• Client-side validation: Missing summary, over-long summary, invalid matcher type, or missing file_pattern/match_string for the chosen matcher are caught locally before any request (exit 1).
• Label not found: If a --label-ids value does not exist, validation fails and points you to the manage-labels skill to create it first.
• 401/403: Token missing or expired — refresh it via the oc-auth skill.
• 501: You hit the read-only Sippy instance with a write; make sure the sippy-auth base URL is used (the script already does).
• 400: Server-side validation failure — the server's message is shown in the detail field of the output.
• Concurrent edits: The update flow is read-merge-replace with no server-side concurrency control, so near-simultaneous edits can overwrite each other — re-check the symptom after updating if others may be editing.

Exit Codes:

• 0: Success
• 1: Validation error, API error, or network error

See Also

• Related Skill: oc-auth (provides authentication tokens for sippy-auth)
• Related Skill: list-symptoms (search/inspect symptoms and labels, no auth needed)
• Related Skill: manage-labels (create labels before symptoms reference them)
• Related Skill: reevaluate-job-runs (apply or preview symptoms on past runs)
• Related Skill: diagnose-job-run-symptoms (explain which symptoms matched a run)