Audit Py Codebase

audit-py-codebase

Perform a single critical audit pass on a Python codebase folder. Designed to be re-invoked until convergence (zero remaining issues).

Instructions

You are performing a single audit pass on the Python codebase at $ARGUMENTS. This skill is idempotent and convergent — each invocation examines the current state, fixes what it can, and reports what remains.

Phase 0: Baseline Snapshot

1. Run ruff check <folder> --statistics and capture violation count.
2. Run pyright <folder> --level strict and capture error count.
3. Run pytest --tb=line -q (if tests exist) and capture pass/fail count.
4. Record these numbers — you will compare against them at the end.

Phase 1: Type Annotations

Ensure every variable, parameter, and return value is properly annotated.

Rules:

• Every function/method has fully annotated parameters and -> ReturnType.
• Every assignment where the type is non-obvious has an annotation (e.g., results: list[dict[str, Any]] = []).
• Class attributes are annotated at class body level or in __init__.
• Use modern syntax: list[X] not List[X], X | None not Optional[X] (Python 3.11+).
• No untyped Any without a # type: ignore[<rule>] comment explaining why.
• Run pyright --level strict after changes — target zero errors.

Phase 2: Docstrings (PEP-257 + NumPy Style)

Module-level: Every .py file starts with a docstring describing its purpose, responsibilities, and key exports.

Functions/Methods (NumPy style):

def function_name(param: Type) -> ReturnType:
    """
    Short summary (imperative mood, one line).

    Extended description if the function is non-trivial.

    Parameters
    ----------
    param : Type
        Description of the parameter.

    Returns
    -------
    ReturnType
        Description of what is returned.

    Raises
    ------
    SpecificError
        When this error is raised.
    """

Classes (NumPy style):

class ClassName:
    """
    Short summary.

    Extended description.

    Attributes
    ----------
    attr_name : Type
        Description.

    Notes
    -----
    Any important implementation notes.
    """

Rules:

• Every public module, class, function, method gets a full docstring.
• Private helpers (_func) get at minimum a one-line docstring.
• __init__ documents parameters under the class docstring's Parameters section.
• Static methods and class methods follow the same NumPy format.
• Run ruff check --select D to verify formatting compliance.

Phase 3: Code Quality (SOLID, DRY, Pure Functions)

Scan for and refactor:

Rules:

• Do NOT over-abstract. Only extract when duplication represents the same concept.
• Prefer composition over inheritance.
• Ensure refactored code passes existing tests before proceeding.

Phase 4: Error Handling

Rules:

• No bare except: or broad except Exception: without re-raise.
• No swallowed exceptions (except E: pass) — at minimum, log the error.
• External boundaries (APIs, file I/O, user input) MUST have try/except with specific types.
• Use raise ... from e to preserve exception chains.
• Create domain-specific exceptions where generic ones are used repeatedly.
• Functions should raise on failure, not return sentinel values (None, -1, False).

Phase 5: Logging

Rules:

• Every module: logger = logging.getLogger(__name__) — no print() for operational output.
• Appropriate levels: DEBUG (internals), INFO (flow), WARNING (recoverable), ERROR (failures).
• Use lazy formatting: logger.info("Processing %s", case_id) — not f-strings.
• Never log sensitive data (credentials, PHI, PII, tokens).
• Verify a central logging configuration exists (file, function, or config dict).
• Add at least one test using caplog fixture to verify critical log messages fire.

Phase 6: Testing (pytest + Fixtures)

Rules:

• Run pytest --cov=<folder> --cov-report=term-missing to identify untested code.
• Shared test state goes in conftest.py as fixtures — no test-level setup boilerplate.
• Use @pytest.fixture for reusable objects, @pytest.mark.parametrize for edge cases.
• Test pure logic thoroughly (unit), test boundaries with mocks (integration).
• Every custom exception should have a test that triggers it.
• Every error handling path (Phase 4) should have a test that exercises it.
• Async tests: use pytest-asyncio with asyncio_mode = "auto".

Phase 7: Final Clean Code Gate

Run in sequence — all must pass with zero violations:

1. ruff format <folder> — apply formatting.
2. ruff check <folder> --fix — auto-fix what's possible.
3. ruff check <folder> — confirm zero remaining.
4. pyright <folder> --level strict — confirm zero errors.
5. Review any # type: ignore — each must have a [rule] code and justification.

Phase 8: Smoke Tests

1. pytest tests/ -v --tb=short — full suite must pass.
2. python -c "import <package>" — confirm no import errors.
3. If an entry point exists (server, CLI), start it and confirm it boots without errors.
4. Compare final ruff/pyright counts against Phase 0 baseline.

Output: Audit Scorecard

End every invocation with this exact format:

── Audit Scorecard ─────────────────────────────────────────
  Pass:              N (where N is which re-invocation this is, default 1)
  Folder:            <folder_path>
  
  pyright errors:    [before] → [after]
  ruff violations:   [before] → [after]
  missing docstrings:[before] → [after]
  test coverage:     [before]% → [after]%
  tests passing:     [before] → [after]
  
  Issues fixed this pass:    X
  Issues remaining:          Y
  Items needing human decision:
    - [list each item with file:line and reason]

  Recommendation:    RE-RUN | CONVERGED ✓
────────────────────────────────────────────────────────────

Recommendation logic:

• RE-RUN if any issues remain that another pass could fix.
• CONVERGED ✓ if pyright=0, ruff=0, all tests pass, and no new issues found.

Important Constraints

• Commit changes at the end of each phase (not at the end of the full pass) with a message like audit: phase N — <description>.
• If a refactoring might break things, run pytest immediately after — do not accumulate risk.
• Do NOT add features, new functionality, or speculative abstractions.
• Do NOT delete tests or weaken assertions to make them pass.
• If unsure whether a change is safe, flag it in "Items needing human decision" and skip it.