# Security Regression Tests > Turning a confirmed and fixed finding into a permanent guard: proving the test fails without the fix, asserting the effect and not only the status code, seeding the two principals an authorization test needs, making a timing or out-of-band proof deterministic enough for CI, and keeping the test in a job that can actually block. Use once a security finding has been confirmed and fixed, or when a security test is failing, skipped, or about to be deleted. - Skill: `shieldnet-360/security-regression-tests` (Agent Skill, multi-file: 3 files) - Install (CLI): `npx skillmds@latest add shieldnet-360/security-regression-tests` - Raw SKILL.md: https://api.skillmd.com/api/skills/shieldnet-360/security-regression-tests/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Security - Author: ShieldNet-360 (https://skillmd.com/u/shieldnet-360) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/shieldnet-360/security-regression-tests --- # Security Regression Tests ## Rules (for AI agents) ### ALWAYS - **Prove the test fails without the fix.** Revert the patch, or mutate the guard, and confirm the new test goes red — then restore. A test written after the fix can be vacuously green: wrong route, wrong fixture user, wrong assertion target, a typo in the payload. It will stay green forever and nobody will learn anything from it. This is the one step that distinguishes a guard from a decoration, and it is the one most often skipped. - Assert **both directions**: the attack input now yields the secure outcome, *and* a legitimate input still succeeds. A deny-only test passes equally well against an endpoint that is broken for everyone, so it will hold the line and tell you nothing when someone disables the feature entirely. - Assert the **effect**, not only the status code. A `403` proves the request was refused; it does not prove the write did not land somewhere else, that the record was not returned in a different shape, or that a queue message was not emitted. Check the state the attack was trying to reach. - Seed the **fixtures the assertion depends on** before writing it. A cross-tenant test needs two principals and two owned resources, deterministically created — if both test users end up in the same tenant, the deny assertion is untestable and passes for the wrong reason. - Make it **deterministic and offline**. Freeze the clock for expiry and TTL assertions, stub the HTTP client, dialer or resolver rather than reaching the network, fix random seeds, and avoid sleep-based timing assertions and state shared between tests. Note that a canonical SSRF payload such as `http://169.254.169.254/latest/meta-data/` is a *live, reachable* address on a cloud CI runner, so the naive test both leaves the sandbox and returns a different verdict on a laptop. - Graduate a live proof-of-concept by keeping its **shape and dropping its mechanism**. `dynamic-verification` confirms some classes with a timing delta or an out-of-band callback, and neither survives in CI: replace the time-based payload with an assertion that the query was parameterized, and the callback with a stubbed dialer asserting no outbound request was attempted. - Put the test in a **job that can block the merge**. A committed test in a job that is `continue-on-error`, not a required check, or nightly-only is documentation. `cicd-security` owns the gate configuration itself. - Name it so a reviewer sees what it guards — `test_idor_orders_cross_tenant_403` — and reference the finding in the test body or the commit, so the next person to see it fail knows what it is protecting rather than what it is blocking. ### NEVER - Close a confirmed finding without a regression test. "Fixed by inspection" rots: the next refactor reintroduces it, and there is nothing to notice. - **Skip, quarantine, `xfail`, or delete** a security test to get a build green. A skipped security test is a removed control that still looks like a present one, and it is indistinguishable in a passing build from one that runs. A failing security test is either a real regression or a wrong test — both need a person, neither needs a decorator. - Relax an assertion until it passes. Widening an expected status set, dropping the effect check, or accepting either outcome converts the guard into a formality. - Put a **live-target probe in CI** as the verification. Attack traffic from a build agent is non-deterministic, sometimes illegal against infrastructure you do not own, and the reason the committed test exists. - Put a **real** secret in a fixture to exercise a secret check. Use the documented sentinels that scanners already exclude by name — `secret-detection` owns them. ### KNOWN FALSE POSITIVES - A regression test necessarily **commits attack payloads**: `' OR 1=1--`, ``, `{{7*7}}`, `../../etc/passwd`, a fake token. The project's own SAST and secret scanners will re-flag them on the next run. That is the test doing its job — exclude the fixture path, do not weaken the payload. - A **refuted** candidate never triggered the rule: there is no fix, so there is nothing to pin. This is not an exemption, it is the precondition not being met. - A finding fixed in configuration rather than code — a base image, a workflow permission, a cloud setting, a dependency bump — is pinned by the **scanner gate** rather than a unit test, because there is no application behaviour to assert. The gate then needs the same both-directions proof: a fixture that fails it and one that passes, or nobody can tell a working gate from a misconfigured one. - A test asserting a *narrower* outcome than the original report — a `404` where the report said `403`, because the application hides existence — is correct if that is the intended behaviour. Match the application's contract, not the report's wording. ## Context (for humans) The failure this skill exists to prevent is not the missing test; it is the test that exists and proves nothing. There are three common ways to get one, and the rules above are ordered by how often each occurs: the test never fails because it does not actually reach the vulnerable path, the test only checks the deny direction so it stays green against a completely broken endpoint, and the test checks a status code while the dangerous effect still happens. The second theme is that a security test's value decays through ordinary maintenance. Nobody deletes a control deliberately; someone adds `@pytest.mark.skip` to unblock a release, and the skip outlives the release. A build with a skipped security test and a build with a passing one look identical from the outside, which is why the rule against skipping is stated as flatly as it is. Where a finding is confirmed live rather than by reading code, the proof usually cannot be committed as-is — a timing delta and an out-of-band callback are both properties of a running system. What transfers is the claim being made, not the way it was demonstrated. ## References - `references/where-verification-runs.md` — live probe versus CI regression test, and why CI must never send attack traffic - `references/test-by-class.md` — attack input, secure outcome, the effect to assert and the paired control case, per finding class - [OWASP Web Security Testing Guide](https://owasp.org/www-project-web-security-testing-guide/).