Run the validators CI runs
Purpose
Local builds are tuned for iteration speed, so projects disable the slow static
analysers and enable them only in CI. The failure mode is specific and expensive:
you run the project's own documented gate, it passes, you report "done" — and CI
fails on a warning your build never computed. The reviewer then sees a red PR they
already reviewed, which spends their attention on your build hygiene instead of your
change.
The second half matters as much as the first. When a linter does fire, the reviewer
usually quotes one instance. That instance is an example, not the requirement: the
rule is the requirement. Fixing only what was quoted leaves the same violation
elsewhere in your diff, and the next CI run finds it — costing a second round for
the same lesson.
When to use
- Before saying "done", "green", "tests pass", or "ready for review".
- Immediately after a lint/static-analysis failure appears in CI.
- When CI is red and the local build is green — assume the validator gap first.
When NOT to use
Not for the slow test suites — running every module's tests locally is often
impractical and CI exists for that. This is about validators: they are fast,
deterministic, and their absence locally is silent.
The practice (checklist)
Treat the flagged run as mandatory before any "done" — the whole point is that
its absence produces no error to notice.
Rationalizations
| Shortcut |
Why it fails |
"gradlew check is the documented gate, and it passed." |
It is documented and partial. The project disables validators locally by design; the documented command is not the CI command. |
| "CI will tell me." |
It will — after the reviewer sees red on a PR they already reviewed. You spend their attention instead of 90 seconds of yours. |
| "I fixed the line they pointed at." |
The linter quoted one instance of a rule. Other instances of the same rule are still in your diff; the next run finds them. |
| "It's only a warning." |
CI commonly runs -Werror. There, a warning is a compile failure. |
| "The failure is unrelated / flaky." |
Maybe — but check before saying so. "Known flaky" is the label most often applied to a real regression. |
RECEIPT
apache/solr PR #4640, 2026-08 — ./gradlew check -x test passed locally while the PR
sat red. The build states the gap itself, verbatim from its own output:
WARNING: errorprone disabled (skipped on builds not running inside CI environments, pass -Pvalidation.errorprone=true to enable)
Under CI's -Werror the skipped rule became a hard failure —
error: warnings found and -Werror specified → compileTestJava FAILED — taking
gradle check, Run SolrJ Tests and the Crave run red with it, on a
[UnnecessarilyFullyQualified] warning the local build never computed.
The rule-vs-instance half, same PR. The reviewer quoted one fully-qualified name. It
was fixed, the build recompiled clean without errorprone, and "clean" was reported —
while four more instances of the same rule in a second file were still failing. Only
-Pvalidation.errorprone=true surfaced them. A diff-wide grep for the pattern would have
found all five the first time.
Lifecycle
- Signals it worked: CI's first run on a push is green; no reviewer ever sees a red
check on a PR they have already reviewed.
- What to log on a misfire: the project, the validator, and the exact opt-in flag —
each house names these differently, and the list is the asset. Record it in
LEDGER.md.
- Death criterion: obsolete for any project whose local default gate equals its CI
gate; check the build output for a skip notice before assuming that.
- Relates to: sibling to obey-the-houses-own-tooling — that one is about generating
artifacts with the house's tools, this one about validating with them. Both fail the
same way: silently, with a green result.
1---2name: run-the-validators-ci-runs3description: A green local build does not mean a green CI. Projects skip expensive validators (errorprone, RAT, forbidden-apis, license checks) on developer machines and enable them only in CI, where the same warning becomes a hard failure under -Werror. So "check passed locally" can be true while CI is red on the very file you just changed. Before claiming a change is done, find the project's opt-in flags for the validators its CI enables, and run with them. And when a validator does fire, fix the RULE across the whole diff — not the one line the reviewer pointed at. Use before saying "done", "green", or "ready for review". Trigger terms: check passed, build is green, CI is red but it works locally, errorprone, -Werror, lint failure, skipped on builds not running inside CI, warnings found.4---56# Run the validators CI runs78## Purpose910Local builds are tuned for iteration speed, so projects disable the slow static11analysers and enable them only in CI. The failure mode is specific and expensive:12you run the project's own documented gate, it passes, you report "done" — and CI13fails on a warning your build never computed. The reviewer then sees a red PR they14already reviewed, which spends their attention on your build hygiene instead of your15change.1617The second half matters as much as the first. When a linter does fire, the reviewer18usually quotes *one* instance. That instance is an example, not the requirement: the19**rule** is the requirement. Fixing only what was quoted leaves the same violation20elsewhere in your diff, and the next CI run finds it — costing a second round for21the same lesson.2223## When to use2425- Before saying "done", "green", "tests pass", or "ready for review".26- Immediately after a lint/static-analysis failure appears in CI.27- When CI is red and the local build is green — assume the validator gap first.2829## When NOT to use3031Not for the slow *test* suites — running every module's tests locally is often32impractical and CI exists for that. This is about **validators**: they are fast,33deterministic, and their absence locally is silent.3435## The practice (checklist)3637- [ ] Find the flags. Grep the build for validator toggles and read the CI workflow38 for what it actually enables:39 `grep -rn "validation\.\|skipIf\|onlyIf.*CI\|System.getenv(\"CI\")" build*.gradle* gradle/`40 and `cat .github/workflows/*.yml | grep -iE "gradlew|mvn|task"`.41- [ ] Run the gate **with** them, not just the bare gate. *Done when* the command42 you quote in your status includes the flags.43- [ ] Read the build's own output for skip notices. A line like44 `WARNING: errorprone disabled (skipped on builds not running inside CI45 environments, pass -Pvalidation.errorprone=true to enable)` is the project46 telling you your green is partial.47- [ ] When a validator fires, fix the **rule across the diff**, not the quoted line:48 `for f in $(git diff --name-only main...HEAD); do grep -n '<pattern>' "$f"; done`49 *Done when* the pattern returns nothing across every changed file.50- [ ] Re-run the flagged gate after the fix, before reporting.5152Treat the flagged run as **mandatory before any "done"** — the whole point is that53its absence produces no error to notice.5455## Rationalizations5657| Shortcut | Why it fails |58|---|---|59| "`gradlew check` is the documented gate, and it passed." | It is documented *and* partial. The project disables validators locally by design; the documented command is not the CI command. |60| "CI will tell me." | It will — after the reviewer sees red on a PR they already reviewed. You spend their attention instead of 90 seconds of yours. |61| "I fixed the line they pointed at." | The linter quoted one instance of a rule. Other instances of the same rule are still in your diff; the next run finds them. |62| "It's only a warning." | CI commonly runs `-Werror`. There, a warning *is* a compile failure. |63| "The failure is unrelated / flaky." | Maybe — but check before saying so. "Known flaky" is the label most often applied to a real regression. |6465## RECEIPT6667**apache/solr PR #4640, 2026-08** — `./gradlew check -x test` passed locally while the PR68sat red. The build states the gap itself, verbatim from its own output:6970> `WARNING: errorprone disabled (skipped on builds not running inside CI environments,71> pass -Pvalidation.errorprone=true to enable)`7273Under CI's `-Werror` the skipped rule became a hard failure —74`error: warnings found and -Werror specified` → `compileTestJava FAILED` — taking75`gradle check`, `Run SolrJ Tests` and the Crave run red with it, on a76`[UnnecessarilyFullyQualified]` warning the local build never computed.7778**The rule-vs-instance half, same PR.** The reviewer quoted one fully-qualified name. It79was fixed, the build recompiled clean *without* errorprone, and "clean" was reported —80while **four more instances of the same rule** in a second file were still failing. Only81`-Pvalidation.errorprone=true` surfaced them. A diff-wide grep for the pattern would have82found all five the first time.8384## Lifecycle8586- **Signals it worked:** CI's first run on a push is green; no reviewer ever sees a red87 check on a PR they have already reviewed.88- **What to log on a misfire:** the project, the validator, and the exact opt-in flag —89 each house names these differently, and the list is the asset. Record it in90 [`LEDGER.md`](../../LEDGER.md).91- **Death criterion:** obsolete for any project whose local default gate equals its CI92 gate; check the build output for a skip notice before assuming that.93- **Relates to:** sibling to obey-the-houses-own-tooling — that one is about *generating*94 artifacts with the house's tools, this one about *validating* with them. Both fail the95 same way: silently, with a green result.
serhiy-bzhezytskyy