Perf Weekly Status Report

Performance Weekly Status Report

Generate Gmail-compatible HTML reports summarizing ScyllaDB Enterprise performance test results from the past week using the Argus CLI.

First Step: Ask the User Which Build Type to Report

Before collecting any data, ask the user which build type to report on.

Use the interactive question tool to present two options:

1. Master (~dev) -- Filter to dev versions matching ^\d{4}\.\d+\.\d+.+dev$. These are builds from the master/development branch.
2. Release -- Filter to release versions matching ^\d{4}\.\d+\.\d+$ (no dev suffix). These are builds from stable release branches.

The chosen build type determines:

• The version filter regex applied in Phase 2
• The header subtitle text ("Master (~dev) builds only" vs "Release builds only")
• The output filename (perf-weekly-status-report.html vs perf-weekly-status-report-release.html)

If the user has already specified the build type in their prompt (e.g., "generate report for release versions"), skip the question and proceed with that choice.

Essential Principles

Version Filtering by Build Type

Master mode: Filter to dev versions matching ^\d{4}\.\d+\.\d+.+dev$ -- exclude release builds.

Release builds (2026.1.5) run on stable branches and represent prior releases. The master report only includes master/dev versions (e.g., 2026.3.0~dev) because these represent the latest development state being validated.

Release mode: Filter to release versions matching ^\d{4}\.\d+\.\d+$ -- exclude dev builds.

Dev builds run on the master branch and represent unreleased code. The release report only includes release versions (e.g., 2026.2.3, 2026.1.10) because these represent validated stable releases. Note that release reports often span multiple versions (e.g., 2026.2.3 and 2026.1.10 in the same week).

Use Argus CLI Directly

The argus binary (Go CLI) is the data source -- not the Python argus client library.

The Go-based argus CLI provides run list and run results subcommands that return JSON. This is faster and more reliable than the Python library for batch data collection. Always use --url https://argus.scylladb.com to target the production Argus instance.

Resolve the binary from PATH (which argus) -- do not hardcode a path under any particular user's home directory. A typical install lives in ~/.local/bin/argus.

Name collision: argus --version is not a valid flag. Verify with argus --help and confirm run and issue subcommands are listed.

Authenticate Argus Before Collecting

The first argus call opens a browser and blocks on Waiting for login....

This violates the repo-wide rule that commands must be non-interactive, and in a headless or automated run it hangs until timeout. Do a cheap pre-flight call before the batch collection so the login happens once, visibly, at a predictable moment:

argus run list --test-id <ANY_TEST_ID> --limit 1 --url https://argus.scylladb.com

If the output contains Waiting for login..., tell the user a browser login is required and wait for it to complete before proceeding. Never launch the 20-test collection loop as the first Argus call -- a login prompt mid-fan-out produces confusing partial failures.

Gmail-Compatible HTML

Output must render correctly in Gmail, which strips <style> blocks and external CSS.

Gmail only supports inline CSS styles on elements. Never use <style> tags, <link> stylesheets, or CSS classes. All styling must be inline via style="..." attributes. Keep the HTML structure simple: use tables for layout, not divs with flexbox or grid.

Overview Table: Group by Workload

The overview table shows each workload as its own row. Columns: Category | Test | Workload | Status | Cause | Issues | Link.

• Category column shows the category name only on the first row of that category group (empty on subsequent rows)
• Test column shows the test name only on the first workload row for that test (empty on subsequent rows)
• Workload column shows the actual workload name (mixed, read, write, read_disk_only). For microbenchmarks use "-" since they don't have separate workloads
• Status column shows just the status badge (PASSED/FAILED/ERROR/RUNNING) of the latest run -- no counts
• Cause column shows why the run failed, and is empty for PASSED rows. See Cause Column
• Issues column shows linked Jira issue keys as clickable links (comma-separated if multiple). When a FAILED row has no linked issue, it shows Investigation in progress instead -- italic amber (#fd7e14), not a link. PASSED rows with no issue stay empty. This is the only place unlinked failures are reported; there is no separate table for them.
• Link column shows an Argus link to the specific run for that workload
• Each run covers a single workload, so each workload has its own run(s)

One row per workload per version: the LATEST run wins. When a build re-ran a workload (or a sub-test was re-run in a later build), sort that workload's runs by start_time and keep the last one. Earlier attempts are neither shown nor counted; the Summary's note line is the only place they are accounted for.

Per-version tables: When the report spans multiple versions, create a separate Overview table for each version with a version label above it (e.g., "Version 2026.2.3"). When only a single version is present, show one table without a version label.

Microbenchmark handling: Microbenchmarks don't report results in the same table structure as performance tests. When a microbenchmark run has no workload-specific tables, use "-" as the workload value. Each microbenchmark test appears as a single row in the overview with workload="-".

Runs column format: Removed. Status column shows just the badge.

Status column format: Show just the status badge (PASSED/FAILED/ERROR/RUNNING) of the latest run for that workload. No counts.

Cause Column: Failed Value Plus Error Threshold

Every FAILED row states the metric that failed, the step, the measured value AND the configured error threshold. A bare "FAILED" forces the reader into Argus; the threshold is what makes the number meaningful.

Format one entry per failed metric:

<metric> <STATUS> at <step> step (<value> <unit>, threshold <limit> <unit>)

Real examples:

Throughput read ERROR at unthrottled step (571,128 op/s, threshold 580,000 op/s)
P99 read ERROR at 1,500,000 op/s step (916.98 ms, threshold 10 ms)
duration ERROR at decommission_nodes step (1,608 s, threshold 1,600 s)

• Units: P90/P99 -> ms, Throughput -> op/s, duration -> s, microbenchmark counters -> none.
• Numeric steps are rendered with thousands separators plus op/s (1500000 -> 1,500,000 op/s). Named steps (unthrottled, decommission_nodes) are printed as-is. Microbenchmark tables have no meaningful step -- omit the at <step> step clause entirely.
• When a run has several failed metrics, put each one on its own line (<br>-separated inside the cell). Do not join them with ; -- a multi-failure cell is unreadable as one run-on line.
• When a run failed without any failed result table, state the real cause from argus run events instead, e.g. OVERSIZED_ALLOCATION error in DB log (all result tables PASS) or cassandra-stress write pre-population failed (CL=ALL, OperationOnKey timeouts) -- no result tables produced.

Result Status: the Argus Run Status Is Authoritative

Report the status Argus reports, and let the run status -- not the result tables -- decide it.

Show the run's real Argus status -- never fold one status into another. test_error is an ERROR badge, not FAILED: the two mean different things (a test that ran and failed its checks vs a test that never produced a result), and collapsing them hides which is which.

A run's own status decides its badge, and table statuses never override it: a run Argus marks passed stays PASSED even if one of its result tables reports FAIL or ERROR. Those table-level errors have already been reviewed and dismissed by whoever set the run status; re-flagging them produces phantom regressions.

Cause text still comes from the failed result tables -- but only for runs that are not passed.

Error Thresholds: Resolve default | step Across TWO Files

The threshold for a step is NOT always in the test's threshold config. The per-workload default block lives in defaults/test_default.yaml, and every step without an explicit override inherits it. Reading only the test-specific file makes steps look like they have no threshold at all.

sct_config deep-merges every config file (anyconfig with ac_merge=MS_DICTS), then send_result_to_argus resolves one step as:

error_thresholds = error_thresholds[workload]["default"] | error_thresholds[workload].get(step_name, {})

So resolve a limit in exactly this order:

1. Read the run's config_files and scm_revision_id from argus run details -- the perf jobs run branch-perf-v17, so read the YAMLs at that revision (git show <sha>:<path>), never from the checked-out branch.
2. Load latency_decorator_error_thresholds from defaults/test_default.yaml at that revision. It defines <workload>.default.P99 <op>.fixed_limit: 10 (10 ms) for every workload.
3. Deep-merge each configurations/performance/latency-decorator-error-thresholds-*.yaml listed in config_files on top of it.
4. For the failing step: merged[workload]["default"] | merged[workload].get(step, {}), then read [metric]["fixed_limit"].

A step-specific entry always wins over the default (e.g. i4i tablets read has P99 read: 1 at the 150,000 step), but a step the file never mentions -- such as the i8g 1500000 read step, or any throttled step in a write section that only lists unthrottled -- correctly falls back to 10 ms.

Microbenchmarks use a different code path. set_validation_rules in sdcm/argus_results.py puts the metric directly under the workload (write.instructions_per_op.fixed_limit) with no step level and no default merge, defaulting to ValidationRule(best_pct=5) when unset.

Running Tests

Include runs with status running in the report if they match the version filter.

Running tests should appear in the Overview table with a RUNNING status badge (blue, #17a2b8). They are counted separately in the Summary (not as passed or failed). Determine the workload from partial results (argus run results) if available, or from Jenkins sub_tests positional mapping. Do not mark running tests Investigation in progress -- that marker is for FAILED rows only.

Runs With No Results: Recover the Workload From Jenkins

A test_error run often has ZERO result tables, so its workload cannot be read from table names. Do not drop these runs.

This is common and can dominate a week: a run that dies during provisioning has end_time of 1970-01-01, an empty results array, and no field anywhere in the run object naming its workload. Since the skill's normal path derives workload from table names ("<workload> - <step> - latencies"), these runs would silently vanish from the overview -- making a week of failures look like a week with few runs.

Recover the workload from the Jenkins build that produced the run:

1. Each run object has build_id and build_number. Fetch that build's parameters (Jenkins MCP get_build_parameters, or the Jenkins REST API).
2. The sub_tests parameter is an ordered JSON list of the workloads that build ran, e.g. ["test_read_gradual_increase_load", "test_write_gradual_increase_load"].
3. Sub-tests execute sequentially, each creating one Argus run. Sort that build's runs by start_time and zip them positionally against sub_tests.
4. Validate the mapping against the runs in the same build that do have tables -- their table-name workload must match the position they were assigned. If it doesn't, stop and report the ambiguity rather than guessing.

Map sub-test names to workload labels: test_read_gradual_increase_load -> read, test_write_gradual_increase_load -> write, test_mixed_gradual_increase_load -> mixed, test_read_disk_only_gradual_increase_load -> read_disk_only, test_latency_mixed_with_nemesis -> mixed.

Also determine why the run produced nothing using argus run events:

argus run events --run-id <RUN_UUID> --url https://argus.scylladb.com

This returns CRITICAL and ERROR events for the run. The common cause is AWS capacity:

(TestFrameworkEvent Severity.CRITICAL) Failed to provision aws resources: CapacityReservationError: Failed to create capacity reservation in any availability zone.

State the real reason (e.g. CapacityReservationError) rather than reporting a bare ERROR.

Manually Triggered Re-runs Are Not Part of the Weekly Picture

Exclude every run whose started_by is avi. These are ad-hoc re-runs kicked off by hand during an investigation -- often a burst of partial runs on the same job across several builds -- and they are not the scheduled weekly result. Counting them distorts both the totals and the failure picture.

started_by comes from argus run details, not from argus run list. It has been wrong before, so confirm the classification against the Jenkins build causes before relying on it:

curl -s -u "$JENKINS_USER:$JENKINS_TOKEN" \
  "https://jenkins.scylladb.com/<job path>/<build>/api/json?tree=actions[causes[shortDescription,userId,userName]]"

A manual run reads Started by user Avi Kivity; a scheduled one reads Started by upstream project "scylla-master/sct_triggers/perf-regression-trigger" build number N (Argus attributes those to the job owner, which is not avi). Credentials come from the SCT KeyStore: KeyStore().get_json("jenkins.json") -> username / password.

Other trigger values (timer, a job owner's username) are all kept.

Report how many runs this dropped to the user in the review step.

CapacityReservationError Runs: Never Reported

Exclude every run whose failure is CapacityReservationError from the report entirely -- whether or not it was re-run, and whether or not a later attempt also failed.

These runs died during AWS provisioning. They carry no result tables and no signal about ScyllaDB; listing them makes an infrastructure outage look like a product regression and inflates the failure count. Drop them from the Overview and from the Summary totals alike.

Detect them via argus run events -- the message contains:

(TestFrameworkEvent Severity.CRITICAL) Failed to provision aws resources: CapacityReservationError: Failed to create capacity reservation in any availability zone.

Apply the same treatment to aborted runs whose workload was re-run in a later build (a sub-test that was cancelled and retried); count only the run that actually produced results.

Report the number of excluded runs to the user in the review step -- the report itself stays silent about them.

Unlinked Failures: Mark Them in the Issues Column

A FAILED run with no linked Argus issue is reported in the Test Overview's Issues column as Investigation in progress -- not in a section of its own.

<td style="...">P99 read ERROR at 700,000 op/s step (18,236.83 ms, threshold 50 ms)</td>
<td style="..."><i style="color:#fd7e14;">Investigation in progress</i></td>

Keeping it in the row means the reader sees, in one pass, which failures are accounted for by a ticket and which are not -- no cross-referencing a second table. Earlier versions of this skill rendered a separate "Failed, investigation in progress" table; that section no longer exists.

The marker applies to the run shown in the row -- the latest run for that version/test/workload. A failure already superseded by a later passing run never reaches the table, so it is never marked.

Still print the list of unlinked failures to the user during the review step, so they can investigate before the report is finalised.

Version Display

Show full version with build date AND revision hash: 2026.3.0.dev-20260612.91ada5517d59.

The full version is constructed from the packages[] array: take scylla-server-target package's version field (normalize ~ to .), append . + the date field, append . + the revision_id field. Example: version=2026.3.0~dev, date=20260612, revision_id=91ada5517d59 becomes 2026.3.0.dev.20260612.91ada5517d59.

The "short version" (e.g., 2026.3.0.dev) is derived by normalizing ~ to . in the scylla_version field.

When the period spans more than one version, the Overview gets one table per version -- on master as well as on release. Master reports one product version (e.g. 2026.4.0~dev) but roll through several builds a week; group by the full version (2026.4.0.dev.20260830.dc4ad9bfdb94), exactly as release groups by 2026.2.6. The Summary lists only the versions whose results are reported, each with the number of tests it contributes, so the per-version counts add up to Total Tests:

Summary for Scylla version 2026.4.0.dev, builds:

  2026.4.0.dev.20260902.5f352afcf41a (4 tests)
  2026.4.0.dev.20260901.d0f9c3a48a11 (4 tests)
  2026.4.0.dev.20260830.dc4ad9bfdb94 (13 tests)

A build that ran in the period but whose every run was superseded contributes no rows, so it is not listed. Do not add a Version column to the table -- the per-version heading already says it. Group a run by its scylla_version; for rolling-upgrade tests that is the upgrade target, so read the full version from the scylla-server-upgrade-target package rather than scylla-server-target (which holds the base version there). Microbenchmark runs often have no date/revision on scylla-server-target -- fall back to the scylla-server package.

Runs Whose Results Endpoint Errors Out

argus run results can fail server-side for an entire test -- it returns Error: server returned status "error" for every run of that test, with or without --test-id, and retrying does not help. Do not silently drop those runs.

Recover what you can and say so plainly in the summary you give the user:

1. argus run events still works. A FailedResultEvent names the exact table that failed -- Argus validation failed for the result in read_disk_only - 165000 - latencies -- which gives both the workload and the failing step.
2. A TestFrameworkEvent carries source=PerformanceRegressionTest.test_latency_read_with_nemesis(), which also identifies the workload.
3. The measured value is not recoverable from events. Take it from the previous week's report if the same run appears there, and tell the user which rows came from that fallback rather than from a live Argus fetch.

Report the affected test names to the user -- a persistent failure of this endpoint is an Argus bug worth raising, not a quirk to work around silently.

Table Width

Use width="950" for the main content table.

The Cause column carries long text -- metric, step, value and threshold, sometimes several lines -- and 700px squeezes it into unreadable wrapping. 950px still fits inside a typical desktop Gmail reading pane. Inner tables stay at width="100%" so they follow the outer table.

Output Location

The report file (perf-weekly-status-report.html) must NOT be saved into the SCT repository.

Write it anywhere outside the working tree -- the agent's own scratchpad/temp directory is the natural choice, or the user's home directory. Never commit it to the repo. (/tmp/opencode/ also works but is specific to one agent harness; don't treat it as required.)

When to Use

• Generating a weekly performance status email for stakeholders
• Creating an HTML summary of enterprise perf test results from the past week
• Aggregating latency and throughput data across multiple Argus test IDs
• Producing a Gmail-friendly report of test pass/fail status with detailed metrics
• When asked for "perf weekly report", "perf status", or "weekly performance summary"

When NOT to Use

• Comparing two specific versions (use perf-comparison-report skill instead)
• Investigating root cause of a specific regression
• Running or configuring performance tests
• Generating reports for non-enterprise (OSS) tests

Test Registry

These are the enterprise performance tests tracked in the weekly report:

Test names in this table are the Jenkins job names with the scylla-enterprise-perf- prefix stripped; the jobs live under scylla-enterprise/perf-regression/ and their pipeline definitions under jenkins-pipelines/performance/branch-perf-v17/scylla-enterprise/perf-regression/.

cql-raw vs simple-query: these are different tests, not architecture variants of one. cql-raw runs microbenchmarking_test.PerfCqlRawTest, simple-query runs PerfSimpleQueryTest. Beware the scylla-staging/yulia/... copies of the cql-raw jobs: they default test_name to PerfSimpleQueryTest, so a staging run can silently be a different test. Confirm via the run's test_method field before treating a cql-raw run as a cql-raw result.

Most of this registry is usually empty for a given week, and that is expected. These tests are not all scheduled weekly, and several run predominantly on release branches -- their runs get filtered out by the master-only rule. It is normal for whole categories (i8g Vnodes, i4i Tablets, i4i Vnodes) to contribute zero rows because they only ran release builds such as 2026.2.2 / 2026.1.9.

Do not treat a mostly-empty result as a collection failure, and do not list tests with no runs in the report. Do state in the "Issues Found in the Runs" section which categories had no runs, so readers can tell "passed" apart from "never ran".

Input Parameters

Example user prompts with time window:

• "Generate perf weekly report" -- uses default 7 days
• "Generate perf report for last 14 days" -- uses 14 days
• "Performance status for the past month" -- uses 30 days

The time window is computed as: --after $(date -d '<N> days ago' +%s)

Argus CLI Quick Reference

List runs for a test

argus run list \
  --test-id <TEST_UUID> \
  --after $(date -d '<N> days ago' +%s) \
  --limit 200 \
  --full \
  --url https://argus.scylladb.com

Returns JSON array of run objects. Key fields:

• id -- Run UUID (needed for run results)
• scylla_version -- Version string to filter by
• status -- "passed", "failed", "test_error", "running"
• packages[].version -- Package versions (alternative version source)
• build_number -- Jenkins build number

Fetch results for a run

argus run results \
  --run-id <RUN_UUID> \
  --url https://argus.scylladb.com

Returns JSON array of result tables. Each table has:

• name -- Format: "<workload> - <step> - latencies" or "<workload> - <step> - stalls - REACTOR_STALLED"
• status -- "PASS", "FAIL", "ERROR"
• rows[].cells -- Key-value pairs with value and status per metric:
  • "P90 <op>" -- 90th percentile latency in ms
  • "P99 <op>" -- 99th percentile latency in ms
  • "Throughput <op>" -- Actual throughput in op/s

Fetch events (errors) for a run

argus run events \
  --run-id <RUN_UUID> \
  --url https://argus.scylladb.com

Returns JSON array of CRITICAL and ERROR event objects. Key fields:

• severity -- "CRITICAL" or "ERROR"
• message -- Event message text (contains the error description)

Use this to determine why a test_error run failed (e.g., CapacityReservationError). This is more reliable than fetching Jenkins console output, which often requires authentication.

Fetch issues for a run

argus issue list \
  --run-id <RUN_UUID> \
  --url https://argus.scylladb.com

Returns JSON array of issue objects. Key fields:

• key -- Jira issue key (e.g., "SCYLLADB-2794")
• title -- Issue title/summary
• state -- Jira state ("new", "todo", "done", "duplicate")
• url -- Direct Jira link

Note: the Argus payload does NOT include Jira creation dates -- but you can look them up directly instead of asking the user (see below).

argus issue list frequently returns [] even for failed runs. Linking a run to a Jira ticket is a manual step in Argus, so an unlinked failure is normal, not a collection bug. When every failed run returns no issues, do not conclude there are no relevant issues -- ask the user whether a known ticket covers the failure, and say plainly in your summary that the attribution came from them rather than from Argus.

Issue Collection Spans Every Counted Run, Not Just the Reported Rows

Collect issues from every run that survived the exclusions -- including runs superseded by a later attempt -- even though only the latest run per version/test/workload reaches the Overview.

An issue linked to a failure that a later run cleared is still an issue the week hit, and the New / Reproduced tables are the week's issue ledger, not a per-row index. So the tables can legitimately name a ticket with no matching FAILED row in the Overview: for example a setUp failure that was re-run successfully an hour later. Note this in your summary to the user rather than dropping it.

The Investigation in progress marker works the other way round -- it is a property of a rendered row, so only runs that reach the Overview can carry it.

Resolve new vs reproduced from Jira, not from the user

Fetch each issue's created field via the Atlassian MCP getJiraIssue tool (fields: ["summary","status","created","resolution"]) and compare it to the report window:

• created inside the window -> New Issues - Regression
• created before the window -> Reproduced Issues

Use status / resolution for the State column, and to support any "no progress since last week" statement -- an issue whose updated timestamp predates the previous report genuinely has had no activity. Only fall back to asking the user if Jira is unreachable.

Report Structure

The output HTML file must contain:

1. Header -- Report title, date range, "Master (~dev) builds only" indicator
2. Summary -- Title format: "Summary for Scylla version(s) {full_version}", listing only the versions whose results are reported, each with its test count. Body: Total tests, passed, failed/error, running. Counts follow the Overview: one per reported row -- the latest run per version, test and workload -- so the per-version counts add up to Total Tests and the reader can reconcile the box against the table. Add a note line underneath giving the number of scheduled runs actually executed in the period and how many of those failed, so superseded re-runs are still visible: "Latest run per version, test and workload; 52 scheduled runs in total were executed in the period (13 of them failed, the rest of the failures were superseded by a later run). Manually triggered ad-hoc re-runs are not included."
3. Issues Found in the Runs -- (formerly "Conclusion") The warning banner is the required content; the hierarchical bullet-point summary underneath it is optional and is usually omitted -- both reference renderings are banner-only, because the Cause and Issues columns already carry the per-test detail. Include bullets only when the week needs narrative the table cannot express. When included: top-level items are test names in bold (prefixed with - ), sub-items are specific observations (prefixed with •), and version numbers are bold. Do NOT mention CapacityReservationError runs (they are excluded from the report entirely). Do NOT mention issue numbers per test in the conclusion body -- issue references belong only in the warning banner; the Issues column in Overview and the Known Issues table provide the per-run linkage. Per-version grouping: When the report spans multiple versions, create a separate list per version with a bold version header (e.g., "Version 2026.2.3:"). When only a single version is present, omit the version header and list tests directly. Warning banner (optional): After collecting all issues, present the de-duplicated list to the user and ask which issues (if any) should be highlighted in a warning banner at the top of the section. If the user selects issues, render a yellow/red banner with ⚠ icon stating "No updates on [issues] during last week." A second ⚠ line lists the issues opened during the period ("New issues opened during last week: ..."). Issue numbers in the banner must be clickable links (e.g., <a href="...">SCYLLADB-3459</a>), not plain text. If the user selects none, omit the banner entirely. The agent MUST print the generated conclusion text to the user and ask for confirmation or edits BEFORE saving it into the final HTML report file. This ensures the user can review and adjust the conclusion wording.
4. New Issues - Regression -- Shown after "Issues Found in the Runs" when new issues exist. Lists Jira issues whose tickets were created during the report period (i.e., newly filed regressions). Classify by fetching each issue's created date from Jira (Atlassian MCP getJiraIssue) and comparing it to the report window; only ask the user if Jira is unreachable. If no new issues, this section is omitted.
5. Reproduced Issues -- Always shown after New Issues (or after "Issues Found in the Runs" if no new issues). Lists issues linked to runs whose Jira tickets were created before the report period. If no reproduced issues, displays "No reproduced issues in this period."
6. Test Overview -- Grouped by category, then test, then workload. Columns: Category | Test | Workload | Status | Cause | Issues | Link. Cause states the failed value and its error threshold, one line per failed metric; it is empty for PASSED rows. Issues shows linked Jira keys, or Investigation in progress for a FAILED row with no ticket. Microbenchmarks use "-" as workload. When multiple versions exist -- including several master builds -- create a separate table per version with the full version as a label above it; with a single version, show one table without a label.

There is no "Detailed Results" section -- the Cause column carries the failure detail, and the Argus link in every row carries the rest.

Reference Index

Success Criteria

A valid weekly status report:

• User asked to choose build type (master or release) before data collection, unless already specified in prompt
• Filters exclusively to the chosen build type (master ~dev or release -- no mixing)
• Shows only tests that were actually run (no "NO_RUNS" entries)
• Uses table-based layout with inline CSS only (no style blocks, no div layout, no border-radius)
• Uses bgcolor attribute alongside background-color for Gmail compatibility
• Renders correctly when opened in a browser and in Gmail
• Summary title format: "Summary for Scylla version {full_version}" with build date and revision hash
• Overview table: grouped by category/test/workload with Argus links in Link column
• Overview table: microbenchmarks appear with "-" as workload
• Overview Status column: just the status badge (PASSED/FAILED/ERROR/RUNNING) -- no counts
• Every badge shows the run's real Argus status -- test_error renders as ERROR (orange #fd7e14), never folded into FAILED
• Running tests included in Overview with RUNNING badge (blue #17a2b8) when they match version filter, counted on their own Summary line
• Aborted runs: excluded when the workload was re-run later, otherwise shown with an ABORTED badge (gray #6c757d) rather than silently passing
• Overview columns: Category | Test | Workload | Status | Issues | Link (no Runs column, no Version column)
• Overview Issues column: linked Jira keys as clickable links, comma-separated; empty for runs with no issues
• Overview: separate table per FULL version (master builds included) with a version header when several exist; no header with a single version
• Overview has no Version column -- the per-version heading carries it
• Overview: one row per workload per version -- the LATEST run, earlier attempts not duplicated
• Overview columns include Cause between Status and Issues; Cause empty on PASSED rows
• Cause states metric, step, measured value AND configured threshold with units
• Cause with several failed metrics puts each metric on its own line (<br>), never joined with ;
• Thresholds resolved as default | step across defaults/test_default.yaml + the test's threshold YAML, read at the run's scm_revision_id
• A step with no explicit override falls back to the default P99 limit (10 ms) -- never reported as "no threshold"
• The run status alone decides the badge; a passed run with an errored result table stays PASSED
• Report contains NO "Detailed Results" section
• "Issues Found in the Runs" section grouped by version when multiple versions present; single list when only one version
• Argus link format uses /test/ (singular), not /tests/ (plural)
• States the reporting period in the header
• Main content table width is 950px; inner tables at 100%
• Output file is NOT saved into the SCT repository (scratchpad/temp dir or home dir)
• "Issues Found in the Runs" text is printed to the user for review/editing BEFORE being saved into the HTML report
• "Issues Found in the Runs" carries the warning banner; hierarchical bullets are optional and, when present, use bold test names as top-level items with indented sub-bullets and bold version numbers
• That section does not mention CapacityReservationError runs (they are excluded from the report entirely)
• Warning banner: user asked to select which issues to highlight from the found issues list; omitted if user selects none or no issues found
• Warning banner: issue numbers rendered as clickable links, not plain text
• Issues are split into "New Issues - Regression" (created during period) and "Reproduced Issues" (pre-existing)
• Issue classification comes from Jira created dates; the user is asked only for unlinked failures or if Jira is unreachable
• argus resolved from PATH, not a hardcoded home directory; login handled by a pre-flight call
• Every run in the window is accounted for, including test_error runs with zero result tables
• Workload for runs with no result tables recovered from argus run events or the Jenkins sub_tests order, and validated against runs that do have tables
• Test error causes determined via argus run events, not Jenkins console
• CapacityReservationError runs excluded from every section AND from the Summary counts
• Aborted runs whose workload was re-run in a later build are excluded too
• Count of excluded runs reported to the user, though the report itself stays silent about them
• FAILED rows with no linked Argus issue show Investigation in progress in the Issues column, in italic amber
• Report contains NO separate "Failed, investigation in progress" section
• The list of unlinked failures is still printed to the user during the review step
• Categories with no runs are named in "Issues Found in the Runs", so "passed" is distinguishable from "never ran"
• Summary lists only the versions whose results are reported, each with its test count, and those add up to Total Tests
• Summary counts follow the Overview rows (latest run per version/test/workload), with a note giving the total scheduled runs executed
• Runs whose started_by is avi are excluded, and the classification was confirmed against Jenkins build causes
• Tests whose argus run results endpoint errors out are still reported, with workload recovered from events and the fallback disclosed to the user
• Data tables use border="1" plus per-cell borders (CSS-only cell borders get stripped)
• Status badges use bgcolor on a <td>, not a bare <span>
• Rendering verified visually i

…(truncated)