Debug updatecli workflow failures
Investigate the failed runs of the workflows below in the lookback window given by the user (default 7 days). If the user names a run, start from that run.
Workflows in scope
| Workflow name | File | updatecli config |
|---|---|---|
bump-elastic-stack-snapshot |
.github/workflows/bump-elastic-stack.yml |
.ci/updatecli/bump-elastic-stack-snapshot.yml |
bump-golang |
.github/workflows/bump-golang.yml |
.ci/updatecli/bump-golang.yml |
update-beats |
.github/workflows/update-beats.yml |
.ci/updatecli/update-beats.yml |
update-compose |
.github/workflows/update-compose.yml |
updatecli-compose.yaml (policies, values in .ci/updatecli/values.d/updatecli-compose.yml, .ci/updatecli/values.d/ironbank.yml and .ci/updatecli/values.d/scm.yml) |
All of them run elastic/oblt-actions/updatecli/run@v1, some of them in a matrix over the
active branches returned by elastic/oblt-actions/elastic/active-branches (the filter job).
Shared SCM values live in .ci/updatecli/values.d/scm.yml.
Step 1 — List and inspect the failed runs
List the runs of each workflow in the window first (see Step 3), then inspect each failed run:
gh run view <run_id> --repo elastic/apm-server --json name,conclusion,createdAt,event,jobs,url
gh run view <run_id> --repo elastic/apm-server --log-failed
Note which matrix jobs (branches) failed and which step failed. Logs can be long; focus on
lines containing ERROR, ##[error], ✗ and the SUMMARY block updatecli prints at the end.
Step 2 — Classify the failure
Pattern A — pull request already exists
ERROR: action stage: "creating pull request: A pull request already exists for elastic:updatecli_<branch>_<pipelineid>."
updatecli pushed the branch but failed to find or reuse the open PR. Check the PR mentioned in
Existing GitHub pull request found::
gh pr view <number> --repo elastic/apm-server --json state,createdAt,headRefName,mergeable
- PR
createdAtwithin seconds of the failing step → race / retry in the GitHub API during PR creation. The PR is fine; the failure is cosmetic. - PR is old and still open → the automation PR is stale (not merged, failing CI, or conflicts). Report why it is stuck.
- PR has since been merged or closed (e.g. superseded by a newer bump) → no action needed. Mention it for completeness.
Pattern B — GitHub API transient error
Log contains 429 (Too Many Requests), 503 Service Unavailable, unable to query GitHub API rate limit,
secondary rate limit or Failed to download archive ... after 3 attempts.
Transient. Check whether the next scheduled run succeeded.
Pattern C — source not available
Failure in a source stage, e.g. the snapshot JSON at
https://storage.googleapis.com/artifacts-api/snapshots/<branch>.json returns 404 or an empty
build_id, or a Go / Beats version cannot be resolved. Usually the branch was just created or
retired: compare against the active branches matrix.
Pattern D — updatecli configuration / version error
Template errors, requiredEnv missing or unknown resource kind. Read the updatecli config file
for the workflow and point to the exact line to change.
WARNING: lines such as Engine "dasel/v1" is deprecated show up in successful runs too.
They are not a root cause unless an ERROR line refers to them; mention them only as a
follow-up clean-up.
Pattern E — target / condition failure
A target or condition stage fails (file not found, yaml key missing, regex does not match).
The target files likely moved or changed format on that branch. Read the file on that branch:
gh api -H "Accept: application/vnd.github.raw" "repos/elastic/apm-server/contents/<path>?ref=<branch>"
Pattern F — authentication
Bad credentials, Resource not accessible by integration, or the Get token step fails.
The GitHub App token (OBS_AUTOMATION_APP_*) is missing permissions or expired.
Anything else: quote the relevant log lines and describe it as unclassified.
Step 3 — Check recurrence
List recent runs of each workflow (default lookback: 7 days):
gh run list --repo elastic/apm-server --workflow <file> --limit 100 \
--created ">=<YYYY-MM-DD>" --json databaseId,conclusion,createdAt,url
For every failed run in the window, fetch --log-failed and classify it with Step 2. Build a
timeline: first occurrence, how many runs failed with the same pattern, which branches, and
whether the next run recovered on its own. Logs older than the retention period may be gone —
say so rather than guessing.
Step 4 — Summarise and propose a fix
Report:
- Which workflows, runs and branches failed, and since when
- The exact error lines
- The pattern (A–F or unclassified) and whether it is transient or needs action
- Whether it is recurring, with the frequency over the lookback window
- The specific file(s) and lines to change and what to change, or why no change is needed
- Any risk or trade-off of the fix
When filing an issue, follow the issue-reporting instruction file at
.apm/instructions/updatecli-failure-issue.instructions.md, which defines the required
title, body sections and format.
Do not re-run, cancel or modify workflow runs, and do not close or merge pull requests.