Python Programmer

Python Programmer

Related skills:

• software-engineer — Core engineering philosophy, system design principles
• functional-programmer — When functional approaches are clearer
• test-driven-development — Testing philosophy and TDD principles

When to Use This Skill

Core Philosophy

The Zen of Python (PEP 20)

Python's design philosophy is captured in "The Zen of Python" (import this). Key principles:

Quote to remember: "Explicit is better than implicit. Simple is better than complex. Readability counts." — Tim Peters, PEP 20

In practice:

• Favor clarity over cleverness
• One obvious way beats multiple equivalent ways
• Code is read more than written (optimize for readers)
• Practicality beats purity (Python isn't a pure functional or OO language)

Staff insight: The Zen is philosophy, not law. Sometimes implicit is fine (e.g., context managers hide __enter__ and __exit__). Sometimes there are two ways (e.g., list comprehension vs map). The Zen guides judgment; it doesn't eliminate it.

The Zen says both "Explicit is better than implicit" and to use Python idioms. When they conflict, optimize for readers.

Heuristics:

• If you need a comment explaining the trick, the trick is too clever
• Nested comprehensions beyond 2 levels need explicit loops
• Walrus operator (:=) in comprehension conditions is usually too clever
• Code golf is not a virtue — readable beats concise

Python culture prefers EAFP (i.e., try/except) over LBYL (i.e., check-then-act). Use EAFP for dict lookups, file access, network calls, and anywhere race conditions matter. Use LBYL for pre-flight validation before expensive operations and performance-critical tight loops where the exceptional case is common.

Staff insight: EAFP is about correctness and clarity, not performance. The file existence check has a race condition; the exception handling doesn't. But exceptions have real cost (e.g., stack unwinding, traceback construction) — they're fine when the exceptional case is rare, not for expected control flow in loops.

Prefer protocols (i.e., behavior) over explicit type checking. Don't use isinstance except with abstract base classes at system boundaries. Use typing.Protocol for duck-typed interfaces — it gives you structural subtyping with type checker support. Use @runtime_checkable only when you genuinely need runtime protocol checking.

Safety Constraints

Fundamental Principles

List/dict/set comprehensions are Pythonic for simple transformations.

Complexity limits:

• One for clause: usually fine
• Two for clauses: acceptable for obvious Cartesian products
• Three+ for clauses: use explicit loops
• Walrus operator (:=) in conditions: almost always too clever
• More than one filter condition: use explicit loops
• Any logic requiring explanation: use explicit loops
• Side effects: never use comprehensions

Staff insight: Comprehensions are readable when they fit on one line and scan left-to-right. The moment you nest, chain conditions, or use walrus operators, you're optimizing for concision over clarity.

Always use with for files, locks, and database connections. Use contextlib.contextmanager for simple cases, ExitStack for dynamic resource management. Don't write try/finally when a context manager expresses intent better.

Use generators for large/infinite sequences and one-pass iteration. Use lists when you need multiple passes, random access, or length upfront. Generator expressions in function calls avoid intermediate lists: sum(x**2 for x in numbers).

Default arguments are evaluated once at function definition. Use None as a sentinel for mutable defaults:

def append_to(element, to=None):
    if to is None:
        to = []
    to.append(element)
    return to

Type System

Type hints are required for all production code. They provide explicit, machine-readable contracts; enable static analysis; and are critical for LLM-assisted development.

Where to use type hints:

• All public APIs, function signatures, and class attributes: required
• Internal functions in non-trivial modules: required
• Local variables: only when type isn't obvious from context
• Throwaway scripts (i.e., < 50 lines, one-time use): optional

Union types (3.10+): Use X | None instead of Optional[X]. Use int | str instead of Union[int, str].

Type parameter syntax (3.12+, PEP 695): Use the bracket syntax for generics:

# Modern (3.12+)
def first[T](items: list[T]) -> T: ...
type Vector[T] = list[T]

# Legacy (pre-3.12) — don't use in new code targeting 3.12+
from typing import TypeVar
T = TypeVar('T')
def first(items: list[T]) -> T: ...

Deferred annotations (3.14+, PEP 649/749): Annotations are now evaluated lazily by default. from __future__ import annotations (PEP 563) is superseded — don't use it in new code. It still works but has different semantics: PEP 563 stringifies annotations, while PEP 649 stores an evaluator function. For code targeting 3.10-3.13, PEP 563 remains useful for forward references.

Other useful features:

• TypedDict for structured dictionaries
• Protocol for structural subtyping (i.e., duck typing with types)
• ParamSpec and Concatenate for higher-order functions
• typing.assert_never for exhaustiveness checking in match statements
• @override decorator (3.12+) for explicit method overriding

ty (Astral): Preferred for most projects. Written in Rust; dramatically faster than alternatives. Currently in beta (v0.0.x) but effective at catching real bugs in typical codebases. Use ty unless you need Pydantic or Django ORM integration, which it doesn't yet support.

mypy: The reference implementation with the broadest ecosystem support. Use mypy when you need its plugin API for dynamic frameworks (e.g., Pydantic, SQLAlchemy, Django ORM) or when your project already uses it. Configure with strict = true in pyproject.toml.

pyright/basedpyright: The dominant IDE type checker (powers Pylance in VS Code). Richer type narrowing than mypy; checks unannotated code by default. Consider basedpyright for LSP integration in non-VS Code editors.

Documentation

Python documentation uses Sphinx with reStructuredText. Comprehensive documentation is mandatory for all production code.

Every module: Module-level docstring explaining purpose and main components.

Every public class: Class docstring with purpose, responsibilities, and invariants.

Every public function/method:

• One-sentence summary (first line)
• Parameters with :param: and :type: (when :type: adds constraints beyond the signature)
• Return value with :returns: and :rtype:
• Exceptions with :raises:
• Usage examples for non-trivial functions

Every test: Docstring explaining what behavior is being tested and why.

See references/docstring-example.py for a complete example.

Include :type: alongside type hints only when it adds information beyond the signature:

Omit :type: when it would mechanically duplicate the signature. The goal is useful documentation, not ceremony.

For new projects, use Sphinx/reST style (:param:, :type:, :returns:, :rtype:, :raises:). For existing projects, follow the established convention — don't convert a Google-style codebase to Sphinx mid-project. pydoclint supports all three styles (sphinx, google, numpy); configure it to match your project.

Python Tooling

Ruff (linting and formatting):

• Replaces Black, Flake8, isort, pydocstyle, and pyupgrade in a single Rust-based tool
• ruff check for linting, ruff format for Black-compatible formatting
• 10-100x faster than the tools it replaces
• Includes partial Bandit rules (S rule set) for common security checks
• Configure in pyproject.toml under [tool.ruff]

pydoclint (docstring linting):

• Validates docstring sections (params, returns, raises) match function signatures
• Replaces darglint, which was archived December 2022 — do NOT use darglint
• Supports Sphinx, Google, and NumPy docstring styles
• Runs standalone or as a flake8 plugin (install with pydoclint[flake8])
• Required because Ruff's DOC rules are still in preview and don't yet support Sphinx style or DOC101 (i.e., missing parameter detection)
• Configure in pyproject.toml under [tool.pydoclint]

Bandit (security linting — when needed):

• Security vulnerability scanner for Python code
• Ruff's S rules cover most common security checks; use standalone Bandit only for security-critical projects needing full coverage (e.g., cryptographic vulnerability detection, severity classifications)
• Configure in pyproject.toml

Type checker: See <type_checkers> — use ty, mypy, or pyright depending on your project's needs.

uv (primary project tool):

• Ultra-fast Python package installer, project manager, and Python version manager
• Handles project initialization (uv init), dependency management (uv add/uv remove), lockfiles (uv.lock), building (uv build), publishing (uv publish), and script execution (uv run)
• Use for all new projects

Hatch (matrix testing):

• Use alongside uv specifically when you need declarative local multi-Python-version testing:

  [[tool.hatch.envs.hatch-test.matrix]]
  python = ["3.12", "3.13", "3.14"]
  

  Then hatch test --all runs tests across all versions locally.
• If you only test against one Python version locally and rely on CI for matrix testing, uv alone suffices
• Hatch also provides VCS-driven versioning (hatch-vcs) and custom build hooks if needed

Dependency Version Specification

NEVER guess or assume dependency versions. Verify current versions on PyPI before adding any dependency.

Staff insight: LLMs confidently hallucinate version numbers. The cost of a PyPI search is trivial compared to debugging phantom compatibility issues. This is especially critical for fast-moving ecosystems where major versions ship monthly.

See references/pyproject-example.toml for a starter project configuration.

Testing

Use pytest for all new projects. It's the community standard: plain functions, simple assert with introspection, powerful fixtures, rich plugin ecosystem. Use unittest only in existing codebases that already use it.

Core testing principle (restated from test-driven-development skill): Mock at architectural boundaries (e.g., external systems, injected dependencies), not internal implementation details. The unittest.mock module is still useful with pytest.

Async Patterns

Common mistakes:

• Mixing sync and async code (e.g., blocking the event loop with synchronous requests or psycopg2)
• Using sync libraries in async contexts — use aiohttp, asyncpg, motor, redis.asyncio instead
• Premature async adoption when threads suffice

Bridge sync-to-async with asyncio.to_thread() (3.9+) when needed for CPU-bound work in async contexts.

Staff insight: Start with synchronous code. Add threads for I/O concurrency. Move to async only when you have many concurrent connections and measurable evidence it helps.

Modern Python (3.12+)

• PEP 695: Type parameter syntax (def func[T](x: T) -> T) — see <modern_type_syntax>
• @override decorator: Explicit intent for method overriding, caught by type checkers
• F-string improvements: Multi-line expressions, nesting, reused quote types all allowed
• distutils removed from stdlib; use uv build

• Free-threaded build (experimental): GIL-optional via --disable-gil build flag; ~40% single-threaded overhead in this version
• Improved REPL: Color output, multi-line editing, better history
• locals() semantics change: Now returns a copy, not a proxy to the frame

• PEP 649/749: Deferred annotation evaluation is now the default — see <modern_type_syntax>
• PEP 750: Template strings (t"Hello {name}") — evaluate to Template objects instead of str, enabling safe SQL interpolation, HTML templating, and structured logging
• PEP 779: Free-threaded build is now officially supported (Phase II), no longer experimental; single-threaded overhead reduced to ~5-10%
• PEP 758: except clauses no longer require parentheses for multiple exceptions: except TimeoutError, ConnectionError:
• PEP 734: concurrent.interpreters — multiple interpreters per process with InterpreterPoolExecutor
• PEP 768: Zero-overhead external debugger interface; pdb can attach to running processes

Python's Global Interpreter Lock historically prevented true parallelism for CPU-bound threads. This is changing.

Current state (3.14):

• The standard build still has the GIL (default behavior unchanged)
• The free-threaded build (PEP 779) removes the GIL and is officially supported, with ~5-10% single-threaded overhead
• Free-threading enables true parallel threads for CPU-bound work without multiprocessing

Practical guidance today:

• For I/O-bound concurrency: asyncio or threads (both work with or without GIL)
• For CPU-bound parallelism: multiprocessing remains the safe default; free-threaded builds are viable for early adopters
• asyncio.to_thread() (3.9+) bridges sync code into async contexts

Staff insight: Free-threading is the future but ecosystem support (e.g., C extensions, third-party libraries) is still maturing. For most projects, multiprocessing for CPU-bound work and threads/async for I/O-bound work remain the pragmatic choices.

When Python Works and Struggles

Python struggles with: Performance-critical tight loops (use NumPy or drop to C/Rust), real-time systems, mobile development, systems programming, and CPU-bound parallelism (though free-threading is changing this).

Staff insight: Use Python where its strengths (e.g., development speed, ecosystem, readability) outweigh its weaknesses. Don't force it into low-level, high-performance, or mobile domains.

Common Pitfalls

• Late binding in closures: Closures capture variables by reference. Use default arguments (lambda i=i: i) or functools.partial to capture values in loop-generated functions
• Truthiness confusion: Empty containers, zero, None, and False are all falsy. Use is None when checking for None specifically, not not x
• global abuse: Pass parameters explicitly or use classes. functools.lru_cache replaces most caching-via-global patterns
• Import cycles: Use TYPE_CHECKING guard for type-only imports; restructure modules to break real circular dependencies
• pathlib neglect: Prefer pathlib.Path over os.path for path manipulation — it's more readable and less error-prone
• Lambda assignment: Don't assign lambdas to variables (PEP 8 E731) — use def for named functions

Dataclasses and attrs

Dataclasses aren't a replacement for all classes — they're for data-focused classes. Use them for configuration, API responses, and value objects. Don't shoehorn behavior-heavy classes into dataclasses.

Resources

Tooling:

• Ruff Documentation: https://docs.astral.sh/ruff/
• uv Documentation: https://docs.astral.sh/uv/
• ty Documentation: https://docs.astral.sh/ty/
• pydoclint Documentation: https://jsh9.github.io/pydoclint/
• MyPy Documentation: https://mypy.readthedocs.io/
• pytest Documentation: https://docs.pytest.org/
• Hatch Documentation: https://hatch.pypa.io/

Style Guides:

• Google Python Style Guide: https://google.github.io/styleguide/pyguide.html

Python programming emphasizes:

• Readability over cleverness — Code is read more than written; don't show off
• Type hints everywhere — Essential for code quality and LLM-assisted development; use PEP 695 syntax on 3.12+
• Comprehensive documentation — Sphinx docstrings with meaningful content, not mechanical :type: duplication
• Modern tooling — Ruff for linting/formatting, pydoclint for docstring validation, uv for project management, ty or mypy for type checking
• EAFP over LBYL — Try and catch exceptions rather than checking first
• Duck typing with Protocols — Accept behavior, not types
• Simple Pythonic idioms — Comprehensions for simple cases, explicit loops for complex ones

All new projects must use uv for project management, with Ruff, pydoclint, and a type checker (ty or mypy) enabled in CI/CD. Do NOT use darglint (archived 2022). Type hints and documentation are mandatory for all production code (exception: throwaway scripts). Follow existing project conventions rather than fighting established codebases.