Scan Generated Files For Artifacts

Scan Generated Files for Artifacts

Type: Open-source — client-agnostic methodology, no project-specific detail.

Created by akbarsha — https://github.com/iamakbarsha1

Distilled from a review that approved a generated markdown file which compiled and tested clean — its tail carried a leaked tool-call closing tag and shipped as visible garbage.

Licence: Released under CC BY 4.0 — share and adapt for any purpose with credit. Full text: LICENSE at the repository root.

Feedback & Support: If a rule here proves wrong or needs sharpening, open an issue on the repository or contact the author at the profile link above. If the problem is the agent not following a rule below rather than the rule itself, that's an execution failure — acknowledge and correct it.

The core rule

Tests exercise code behaviour; they say nothing about the bytes of a non-code file. When an agent writes a file through a tool, the harness framing around that write — a closing </content> or invoke/function tag, a stray markdown fence, content past the point the spec said the file should end — can leak into the file itself. It parses fine, compiles fine, tests green, and ships as visible garbage a reader sees on the first or last screen. A review that runs only the test suite misses this whole class. So for every file an agent produced with a write tool, spend one cheap grep on its head and tail before sign-off, and confirm it ends exactly where it should.

Checks

• Grep the head and tail of every agent-written file for harness residue. Search the changed non-code files — especially the first and last few lines — for tool/harness tag fragments: </content>, closing invoke/function tags, orphaned markdown fences, or any framing that belongs to the write mechanism, not the content. A trivial grep catches what the test suite structurally cannot. (A review approved a generated markdown doc that compiled and tested clean; its tail carried a leaked tool-call closing tag and shipped as visible garbage.)
• Confirm the file ends where the spec says it ends. Content past the intended EOF — a duplicated section, a trailing fragment, a second copy of the closing block — reads as garbage but breaks no test. Check the last lines against what the file was supposed to contain, not just that it is non-empty. (Illustrative: an agent-generated config had the intended body followed by a repeated tail block; every parser accepted it and the duplicate silently overrode the real values.)
• Never let a green test suite stand in for a byte-level check on non-code output. Docs, configs, fixtures, and data files pass tests they were never the subject of. Treat "tests pass" and "the file is clean" as two separate claims, and prove the second one directly for anything an agent wrote. (Illustrative: a README shipped with an XML fence fragment in its last paragraph; CI was green because no test reads the README.)

Pre-flight check — before you approve a review that includes generated files

• Every file an agent produced via a write tool was grepped at head and tail for harness/tool-framing residue (</content>, closing invoke/function tags, stray fences).
• Each such file ends exactly where its spec says it should — no content past the intended EOF, no duplicated tail block.
• "Tests pass" was not treated as proof the file's bytes are clean; the byte-level check was run separately for non-code output.

If any box is unchecked, a generated file may carry invisible artifacts that test clean and ship as garbage — run the scan before approving.