# Payload Experimental Reverts > Experimentally test medium-confidence payload candidates by opening draft revert PRs and triggering payload jobs - Skill: `openshift-eng/payload-experimental-reverts` (Agent Skill) - Install (CLI): `npx skillmds@latest add openshift-eng/payload-experimental-reverts` - Raw SKILL.md: https://api.skillmd.com/api/skills/openshift-eng/payload-experimental-reverts/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: openshift-eng (https://skillmd.com/u/openshift-eng) - Updated: 2026-09-10 - Page: https://skillmd.com/skills/openshift-eng/payload-experimental-reverts --- # Payload Experimental Reverts This skill experimentally tests medium-confidence candidate PRs by opening draft revert PRs, triggering payload jobs, and evaluating results. It operates in two phases separated by a CI wait period. All state is tracked in the payload results YAML file via the `payload-results-yaml` skill — no separate tracking file is created. ## When to Use This Skill Use this skill when the `/ci:payload-experiment` command identifies candidate PRs with medium confidence (score 60-84) that cannot be conclusively attributed to a failure through static analysis alone. The experiment creates real tests to determine causality. **Inputs** (passed in-context by the caller): - `results_yaml_path`: Path to the payload results YAML file (e.g., `./payload-results-{tag}.yaml`) - `candidates`: List of medium-confidence PRs to test experimentally, each with: - `pr_url`, `pr_number`, `component`, `title`, `confidence_score` - `failing_jobs`: List of `{job_name, prow_url, is_aggregated, underlying_job_name}` ## Required Skills Before starting, you **MUST** load the following skills (they define output schemas used when updating results): 1. **`payload-results-yaml`** — schema for the payload results YAML file 2. **`payload-autodl-json`** — schema for the autodl JSON data file ## Prerequisites 1. **GitHub CLI (`gh`)**: Installed and authenticated 2. **JIRA MCP**: Configured for creating TRT issues (needed in Phase 2 for confirmed causes) 3. **Repository Access**: User must have push access to their fork of each target repository ## Implementation Steps ### Phase 1: Set Up Experiments For each medium-confidence candidate, launch a **parallel subagent** (do NOT set the `model` parameter): #### 1.1: Check for Merge Conflicts Before opening a revert PR, preemptively check whether the revert will have merge conflicts: ```bash # Clone the repo (shallow for speed) git clone -b --depth 50 "https://github.com//.git" /tmp/experiment-check- cd /tmp/experiment-check- # Attempt the revert without committing git revert -m1 --no-commit # Check for conflicts git status --porcelain ``` If conflicts exist: - Append an action entry with `type: "experiment"`, `status: "skipped_conflict"` to this candidate's `actions` array - Skip to the next candidate - Do NOT attempt to resolve conflicts for experimental reverts If no conflicts, abort the dry-run revert and proceed: ```bash git revert --abort 2>/dev/null || git checkout -- . ``` #### 1.2: Open Draft Revert PR Load the `revert-pr` skill and follow its workflow with `--draft`: - PR URL: the candidate PR - JIRA ticket: use a placeholder like `NO-JIRA` (real ticket is created in Phase 2 only for confirmed causes) - `--draft`: Create as a draft PR - `--context`: "Experimental revert for {stream} {architecture} payload {payload_tag}. Testing whether reverting this PR resolves blocking job failures." - Do NOT prompt the user for any input Record the draft revert PR URL. #### 1.3: Trigger Payload Jobs and Collect Run URLs Use the `trigger-payload-job` skill (`plugins/ci/skills/trigger-payload-job/SKILL.md`) to trigger payload validation jobs on the draft revert PR and collect the resulting URLs. Pass: - `pr_url`: The draft revert PR URL - `jobs`: The `failing_jobs` list for this candidate (includes `job_name`, `is_aggregated`, `underlying_job_name` for each job) #### 1.4: Record Experiment Use the `payload-results-yaml` skill to append an action entry to the candidate's `actions` array: - `type`: `"experiment"` - `status`: `"pending"` - `revert_pr_url`, `revert_pr_state: "draft"`, `payload_jobs`, `result_summary: ""`, `jira_key: ""`, `jira_url: ""` See the `payload-results-yaml` skill for the full schema. **Throttling**: Never test more than 5 candidates. If there are more than 5, test only the top 5 by confidence score. **Job triggering limits**: Across all experiments combined: trigger at most 5 non-aggregated jobs and at most 1 aggregated job. Prioritize jobs from higher-confidence candidates. When a candidate is processed but **all** of its jobs were skipped due to these limits (i.e., none were actually triggered), do NOT leave it with `status: "pending"`. Instead set: - `status`: `"deferred"` - `payload_jobs`: one entry per skipped job with `command` set and `test_url`, `test_prow_url` all set to `"skipped_due_to_limits"` - `result_summary`: `"All jobs skipped due to cross-experiment triggering limits"` When a candidate has **some** jobs triggered and some skipped, mark the triggered jobs normally and add entries for skipped jobs with the `"skipped_due_to_limits"` marker so the record is complete. The action's `status` should be `"pending"` in this case (it has real jobs to check). Candidates beyond the top 5 that were never processed at all should get an action entry with: - `type`: `"experiment"` - `status`: `"deferred"` - `result_summary`: `"Deferred — exceeded maximum of 5 experimental candidates"` ### Update Payload Results YAML After all Phase 1 subagents complete, use the `payload-results-yaml` skill to update the results file at `results_yaml_path` with the action entries for each candidate that was processed or deferred. ### Update autodl JSON Use the `payload-autodl-json` skill's "Update Experiment Status" Phase 1 operation to update the autodl JSON file for each candidate that had a draft revert PR created. --- ### Phase 2: Collect Results and Act Phase 2 is invoked after a CI wait period (typically 1-4 hours). If the results YAML contains any action entry with `type: "experiment"` and `status: "pending"`, enter Phase 2. Phase 2 processes only pending experiments — candidates with other statuses are left unchanged. #### 2.1: Read Payload Results YAML Read the results YAML at `results_yaml_path` using the `payload-results-yaml` skill. Find all candidates that have an action entry with `type: "experiment"` and `status: "pending"`. Skip actions with `status: "deferred"` — these had no jobs triggered and cannot be evaluated. #### 2.2: Check Job Results For each pending experiment action: 1. Fetch the `test_url` from the action's `payload_jobs` 2. Check for "AllJobsFinished" status on the page 3. If not finished, leave the action's `status` as `"pending"` — do NOT change it. The caller can invoke Phase 2 again later to re-check. 4. If finished, check individual prow job results (pass/fail) by fetching each `test_prow_url` #### 2.3: Act on Results For each completed experiment: **PASS** (payload jobs pass with the revert applied — the revert fixed the problem): The candidate PR is confirmed as the cause. Execute: 1. **Create TRT JIRA bug**: Same format as `stage-payload-reverts` Substep 1 2. **Promote draft to real PR**: ```bash gh pr ready ``` Update the PR title to include the JIRA key and remove any "NO-JIRA" placeholder: ```bash gh pr edit --title ": Revert # \"\"" ``` Update the PR body to include the JIRA reference and full Revertomatic template. 3. Update the action entry: `status: "passed"`, `revert_pr_state: "open"`, `jira_key`, `jira_url` **FAIL** (payload jobs still fail with the revert applied — the PR is innocent): 1. Post a comment on the draft PR explaining the result: ``` Experiment result: payload jobs still fail with this PR reverted. This PR is not the cause of the blocking job failures in {payload_tag}. Closing this draft. ``` 2. Close the draft PR: ```bash gh pr close ``` 3. Update the action entry: `status: "failed"`, `revert_pr_state: "closed"` **ALL FAIL** (no single revert fixes the problem): If all experiments fail, close all remaining draft PRs and note in the result summaries that the failures may be caused by an interaction between multiple PRs or by infrastructure issues. #### 2.4: Update Payload Results YAML Use the `payload-results-yaml` skill to update the results file at `results_yaml_path`: - For each completed candidate, update the relevant action entry's `status`, `result_summary`, `revert_pr_state`, `jira_key`, `jira_url` - Candidates whose jobs are still running keep their action entry's `status: "pending"` (unchanged) #### 2.5: Update autodl JSON Use the `payload-autodl-json` skill's "Update Experiment Status" Phase 2 operation to update the autodl JSON file for each completed experiment. Return results to the caller. If any candidates remain `pending`, inform the caller that Phase 2 should be re-invoked later to collect remaining results. ## Error Handling - If a revert PR cannot be created (e.g., fork issues), skip that candidate and record the error. - If payload job triggering fails, record the error but keep the draft PR open for manual testing. - If the pr-payload-tests URL cannot be extracted, record the draft PR URL and note manual checking is required. - Do not let one failed experiment block processing of others. ## See Also - Related Skill: `payload-results-yaml` - Schema and operations for the payload results YAML - Related Skill: `revert-pr` - The git revert workflow (`plugins/ci/skills/revert-pr/SKILL.md`) - Related Skill: `trigger-payload-job` - Triggers payload jobs and collects URLs (`plugins/ci/skills/trigger-payload-job/SKILL.md`) - Related Skill: `stage-payload-reverts` - Stages high-confidence reverts (`plugins/ci/skills/stage-payload-reverts/SKILL.md`) - Related Command: `/ci:payload-experiment` - Command for experimental reverts (`plugins/ci/commands/payload-experiment.md`)