Identifying Slop Tests
Overview
Slop testing is testing that proves code satisfied a prompt, plan, rubric, or static design shape instead of proving behavior through a real consumer surface.
Core gate: Would an independent engineer building this for themselves write this test because it protects real behavior? If not, treat it as validation theater until proven otherwise.
When to Use
Use this skill when a test asserts:
- Constants, enum members, descriptor counts, registry keys, headings, labels, descriptions, or prose.
- Prompt/golden-plan/template sections like
## QA Scenarios, #### Evidence, or self-check headings.
- Example objects that parse back into themselves.
- Advisory metadata such as capabilities, recommendations, notes, reasons, or non-enforcing configuration.
- Schema fields that exist only because a plan said the design should contain them.
Do not use this skill to delete every static assertion. Exact text or exact shape is valid when it is a real product surface, public compatibility contract, protocol, migration boundary, or file format consumed by another tool.
When you do not have enough repository or filesystem context to prove whether a static shape is externally consumed, do not delete the test. Mark it for developer verification instead.
Core Pattern
Validation theater asks, "Did the implementation remember the planned structure?"
Behavior testing asks, "Does the system still work through the surface that depends on this?"
// Slop: proves the prompt template survived.
expect(prompt).toContain("## QA Scenarios")
// Better: proves the consumer can use the generated artifact.
const agentFile = await syncGeneratedAgent(profile)
expect(await canOpenCodeLoadAgent(agentFile)).toBe(true)
Identification Gates
Ask these before keeping or writing the test:
| Gate |
Slop signal |
Keep only if |
| Behavior |
The test can pass while the feature is unusable. |
It exercises a real outcome or failure path. |
| Surface |
The assertion never leaves internal data shape. |
The shape is consumed by a user, tool, API, file format, or runtime. |
| Change detection |
Failing test would only mean wording/count changed. |
Failing test signals a broken contract or behavior regression. |
| Independence |
The test exists because a prompt demanded rigor. |
An engineer would write it without seeing the prompt. |
| Replacement |
No one can name the behavior it protects. |
It can be replaced by a behavior-through-surface check. |
If code could be useless while the test still passes, the test is slop unless the assertion protects a known external contract or consumer surface.
Common Slop Test Types
| Type |
Symptom |
Replace with |
| Fact-checking constants |
expect(count).toBe(18) |
Exercise the consumer that depends on all generated items. |
| Template compliance |
expect(markdown).toContain("## Evidence") |
Parse/load/use the generated document through its consumer. |
| Schema ceremony |
Dummy object proves a field is required. |
Boundary test for invalid real input or remove the unneeded field. |
| Example tautology |
parse(example) === example |
Behavior that registered/loaded example enables. |
| Advisory metadata |
recommendedCapabilities.reasoningLevel === "deep" |
Runtime routing/enforcement, or delete the metadata. |
| Registry bookkeeping |
Object keys equal a hardcoded list. |
Tool lookup/dispatch behavior for registered tools. |
Default Policy
- Do not add tests to make a refactor look rigorous.
- Delete slop tests when their asserted structure is removed.
- Rewrite slop tests only when a real behavior would otherwise lose coverage.
- Prefer existing behavioral tests over new unit tests for deletion/consolidation refactors.
- Add a new test only for an actual runtime boundary, compatibility contract, safety invariant, or user-visible behavior.
Better Test Targets
Good tests usually protect one of these:
- Generated files can be loaded by the tool that consumes them.
- Invalid input is rejected at a real boundary with an actionable error.
- State transitions preserve invariants and do not corrupt persisted data.
- Destructive actions are prevented or require the intended confirmation.
- Refactors preserve behavior through CLI, API, UI, file sync, or library import surfaces.
- Failure modes leave recoverable state.
Rationalizations
| Excuse |
Reality |
| "This proves the plan was followed." |
Plans are not runtime behavior. |
| "It documents the expected structure." |
Documentation belongs in docs unless a consumer depends on exact structure. |
| "It is cheap coverage." |
Coverage that cannot fail for real bugs is noise. |
| "The field may matter later." |
Future consumers do not justify current tests. |
| "The schema is important." |
Test the boundary behavior, not the schema's self-description. |
| "Without this, refactor could change wording." |
Wording matters only if it is a contract or user surface. |
Red Flags
Stop and re-evaluate when you see:
toContain against prompt headings, plan sections, or generated prose.toEqual(exampleDescriptor) after parsing or registering the same example.- Hardcoded descriptor counts, role counts, tool counts, or section counts.
- Assertions over
recommended*, metadata, notes, capabilities, reason, or mode fields with no enforcement path.
- Tests named after design artifacts: "golden plan contract", "self-check criteria", "scaffold integrity".
- A test whose failure message would only mean "the code no longer matches the prompt wording."
All of these mean: identify the real consumer surface, replace with a behavior test if needed, otherwise delete the test with the dead structure.
1---2name: slop-tests3description: Use when reviewing, writing, or refactoring tests that assert constants, headings, descriptor counts, schema shape, generated templates, examples, advisory metadata, or plan/rubric compliance instead of runtime behavior.4license: CC-BY-NC-4.05---67# Identifying Slop Tests89## Overview1011Slop testing is testing that proves code satisfied a prompt, plan, rubric, or static design shape instead of proving behavior through a real consumer surface.1213Core gate: **Would an independent engineer building this for themselves write this test because it protects real behavior?** If not, treat it as validation theater until proven otherwise.1415## When to Use1617Use this skill when a test asserts:1819- Constants, enum members, descriptor counts, registry keys, headings, labels, descriptions, or prose.20- Prompt/golden-plan/template sections like `## QA Scenarios`, `#### Evidence`, or self-check headings.21- Example objects that parse back into themselves.22- Advisory metadata such as capabilities, recommendations, notes, reasons, or non-enforcing configuration.23- Schema fields that exist only because a plan said the design should contain them.2425Do not use this skill to delete every static assertion. Exact text or exact shape is valid when it is a real product surface, public compatibility contract, protocol, migration boundary, or file format consumed by another tool.2627When you do not have enough repository or filesystem context to prove whether a static shape is externally consumed, do not delete the test. Mark it for developer verification instead.2829## Core Pattern3031Validation theater asks, "Did the implementation remember the planned structure?"3233Behavior testing asks, "Does the system still work through the surface that depends on this?"3435```ts36// Slop: proves the prompt template survived.37expect(prompt).toContain("## QA Scenarios")3839// Better: proves the consumer can use the generated artifact.40const agentFile = await syncGeneratedAgent(profile)41expect(await canOpenCodeLoadAgent(agentFile)).toBe(true)42```4344## Identification Gates4546Ask these before keeping or writing the test:4748| Gate | Slop signal | Keep only if |49| --- | --- | --- |50| Behavior | The test can pass while the feature is unusable. | It exercises a real outcome or failure path. |51| Surface | The assertion never leaves internal data shape. | The shape is consumed by a user, tool, API, file format, or runtime. |52| Change detection | Failing test would only mean wording/count changed. | Failing test signals a broken contract or behavior regression. |53| Independence | The test exists because a prompt demanded rigor. | An engineer would write it without seeing the prompt. |54| Replacement | No one can name the behavior it protects. | It can be replaced by a behavior-through-surface check. |5556If code could be useless while the test still passes, the test is slop unless the assertion protects a known external contract or consumer surface.5758## Common Slop Test Types5960| Type | Symptom | Replace with |61| --- | --- | --- |62| Fact-checking constants | `expect(count).toBe(18)` | Exercise the consumer that depends on all generated items. |63| Template compliance | `expect(markdown).toContain("## Evidence")` | Parse/load/use the generated document through its consumer. |64| Schema ceremony | Dummy object proves a field is required. | Boundary test for invalid real input or remove the unneeded field. |65| Example tautology | `parse(example) === example` | Behavior that registered/loaded example enables. |66| Advisory metadata | `recommendedCapabilities.reasoningLevel === "deep"` | Runtime routing/enforcement, or delete the metadata. |67| Registry bookkeeping | Object keys equal a hardcoded list. | Tool lookup/dispatch behavior for registered tools. |6869## Default Policy7071- Do not add tests to make a refactor look rigorous.72- Delete slop tests when their asserted structure is removed.73- Rewrite slop tests only when a real behavior would otherwise lose coverage.74- Prefer existing behavioral tests over new unit tests for deletion/consolidation refactors.75- Add a new test only for an actual runtime boundary, compatibility contract, safety invariant, or user-visible behavior.7677## Better Test Targets7879Good tests usually protect one of these:8081- Generated files can be loaded by the tool that consumes them.82- Invalid input is rejected at a real boundary with an actionable error.83- State transitions preserve invariants and do not corrupt persisted data.84- Destructive actions are prevented or require the intended confirmation.85- Refactors preserve behavior through CLI, API, UI, file sync, or library import surfaces.86- Failure modes leave recoverable state.8788## Rationalizations8990| Excuse | Reality |91| --- | --- |92| "This proves the plan was followed." | Plans are not runtime behavior. |93| "It documents the expected structure." | Documentation belongs in docs unless a consumer depends on exact structure. |94| "It is cheap coverage." | Coverage that cannot fail for real bugs is noise. |95| "The field may matter later." | Future consumers do not justify current tests. |96| "The schema is important." | Test the boundary behavior, not the schema's self-description. |97| "Without this, refactor could change wording." | Wording matters only if it is a contract or user surface. |9899## Red Flags100101Stop and re-evaluate when you see:102103- `toContain` against prompt headings, plan sections, or generated prose.104- `toEqual(exampleDescriptor)` after parsing or registering the same example.105- Hardcoded descriptor counts, role counts, tool counts, or section counts.106- Assertions over `recommended*`, `metadata`, `notes`, `capabilities`, `reason`, or `mode` fields with no enforcement path.107- Tests named after design artifacts: "golden plan contract", "self-check criteria", "scaffold integrity".108- A test whose failure message would only mean "the code no longer matches the prompt wording."109110All of these mean: identify the real consumer surface, replace with a behavior test if needed, otherwise delete the test with the dead structure.
Arcadi4