# Python Engineering
> Modern Python engineering standards and best practices. Use this skill whenever a user wants to: scaffold a Python project, configure tooling (uv, ruff, ty, mypy, structlog, pytest, hypothesis, pydantic-settings, opentelemetry, pip-audit), set up pyproject.toml, src-layout, pre-commit, CI/CD, Docker — for an existing, inherited, or legacy project as much as a greenfield one (assessing and modernizing current setup, not just scaffolding new) — or asks about Python architecture, packaging, testing, type checking, observability, security, async patterns, typing.Protocol, dependency injection, CLAUDE.md, or Cursor rules. Covers hexagonal architecture, functional core/imperative shell, property-based testing, snapshot testing, testcontainers, Trusted Publishers, and Sigstore.
- Skill: `grimaldost/python-engineering` (Agent Skill, multi-file: 16 files)
- Install (CLI): `npx skillmds@latest add grimaldost/python-engineering`
- Raw SKILL.md: https://api.skillmd.com/api/skills/grimaldost/python-engineering/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: grimaldost (https://skillmd.com/u/grimaldost)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/grimaldost/python-engineering
---
# Modern Python Engineering Standards
Apply these standards whenever scaffolding, advising on, or generating Python code
and configuration.
---
## Core Philosophy
1. **Ecosystem-First**: Before writing custom logic, evaluate standardized
solutions. Custom scripts are liabilities; community tools are assets.
2. **Fail Fast**: Catch errors at startup (typed config) and at commit time
(pre-commit hooks), not in production.
3. **src layout is mandatory** — never flat layout for production projects.
4. **Single quotes** for code literals; **double quotes** for docstrings.
5. **Never suggest `pip`, `poetry`, or `virtualenv`** — always `uv`.
6. **Astral-first toolchain**: prefer `uv` + `ruff` + `ty` for the unified,
high-performance developer experience.
7. **Protocol-first typing**: prefer `typing.Protocol` (structural subtyping)
over ABCs for interfaces. Reserve ABCs for shared implementation only.
8. **Async for I/O-bound services**: reach for `async`/`await` on web services and
I/O-bound code — not CPU-bound work or one-shot scripts. Use `asyncio.TaskGroup`
over `asyncio.gather()`.
9. **Observable when it runs as a service**: for anything long-running, logging
alone is insufficient — instrument with OpenTelemetry traces, metrics, and
correlated structured logs. Right-size it; a CLI or one-off script doesn't need
distributed tracing.
10. **Secure the supply chain**: pin dependencies with hashes, audit for CVEs
in CI, use Trusted Publishers for PyPI releases.
---
## Modifying existing code (the edit lane)
Most work in an existing project is an *edit*, not a scaffold. When the task
is to change a few files in a project that already has its layout, tooling,
and CI, the scaffolding, Docker, observability, and CI sections below are
not the relevant rules — skip them. Match what the surrounding code already
does, and apply only the rules that govern the lines you touch (for a
**micro-edit** — a one-line docstring or string change — even these are mostly
moot: match the touched line's local form, run the project's gate, and stop):
- **Match the local convention first — project config governs.** Read 2–3
nearby files before editing. The project's existing patterns — even where
they differ from this skill's greenfield defaults — are the contract for an
edit; a one-file modernization that diverges from the rest of the module is
noise, not improvement. Where the project states its own conventions
(`AGENTS.md`, `CLAUDE.md`, `ruff.toml`, a style guide), **those govern and
this skill's defaults are the fallback** — defer, don't override.
- **Protocol-first typing for new interfaces.** A new seam introduced inside
existing code prefers `typing.Protocol` (structural) over an ABC, unless the
surrounding code already commits to ABCs. See the Typing Philosophy section.
- **`@override` semantics.** When overriding a real base-class method, annotate
it `@override` (PEP 698); do not add `@override` to a class that only
structurally satisfies a `Protocol` — see the caveat in Typing Philosophy.
- **Import hygiene.** Keep imports at module top, grouped stdlib / third-party /
first-party (ruff's isort handles ordering). When a format-on-save or
autofix hook strips unused imports, add an import in the *same* edit that
first references it — an "import now, use later" split loses the import to
the hook between edits.
- **Quoting and docstrings.** Single quotes for code literals, double quotes
for docstrings (the project's ruff config enforces this; an edit that fights
it just gets reformatted).
- **Don't widen the scope.** An edit's diff is its scope. Adjacent cleanup,
renames, and "while I'm here" refactors belong in a separate change — the
same scope discipline `data-engineering-discipline` applies to a migration.
If the task is actually to *modernize* an inherited project's tooling (assess
the current setup and bring it to standard), that is the broader scope the
rest of this skill and `scripts/doctor.py` cover — run the doctor, then work
through its findings.
---
## The Canonical Stack
| Layer | Tool | Notes |
|------------------|-----------------------|--------------------------------------------|
| Package manager | `uv` | Replaces pip + virtualenv + poetry + pyenv |
| Build backend | `uv_build` | Default from `uv init`, PEP 621-compliant |
| Lint + Format | `ruff` | Single-quote enforced in both linter+fmt |
| Type checking | `ty` (primary) | Astral's Rust-based checker (beta, fast) |
| | `mypy` (stable alt) | Use if `ty` coverage is insufficient |
| Config mgmt | `pydantic-settings` | Typed env vars, SecretStr for secrets |
| Logging | `structlog` | JSON in prod, pretty in dev |
| Observability | `opentelemetry` | Traces + metrics; correlate with structlog |
| Testing | `pytest` | Unit (pure) + integration (I/O) |
| Property testing | `hypothesis` | For data processing, serialization, algos |
| Security audit | `pip-audit` | CVE scanning against OSV database in CI |
| CLI (if needed) | `typer` | Type-hint-driven argument parsing |
| Web (if needed) | `fastapi` | Interface layer only — keep logic in core/ |
| HTTP client | `httpx` | Sync + async, HTTP/2; default for new code |
### A note on type checkers
**`ty`** (by Astral, the ruff/uv team) is the forward-looking choice: much faster
than mypy (Astral reports 10-100× on large codebases), built-in language server,
first-class intersection types,
and advanced reachability analysis. It is currently in **beta** (0.0.x) and
evolving rapidly toward a 1.0 release.
**`mypy`** remains the battle-tested, stable option for projects that need
full typing ecosystem coverage today.
**Recommendation**: Use `ty` for new projects and active development (fast
feedback loop). Keep `mypy` in CI as a secondary check if your project relies
on mypy plugins (e.g., `django-stubs`, `sqlalchemy-stubs`). For VS Code /
Cursor users, install the `ty` extension for the language server.
---
## Typing Philosophy
**Prefer `Protocol` over ABCs** for interface definitions. Protocols enable
structural subtyping (static duck typing) — any class that implements the
required methods satisfies the Protocol without inheriting from it. This
aligns with Python's duck-typing nature and keeps classes decoupled.
```python
from typing import Protocol
class Repository(Protocol):
"""Any class with save() and get() satisfies this contract."""
def save(self, entity: dict) -> str: ...
def get(self, entity_id: str) -> dict | None: ...
# No inheritance needed — just implement the methods
class PostgresRepository:
def save(self, entity: dict) -> str: ...
def get(self, entity_id: str) -> dict | None: ...
```
Reserve ABCs only when you need: runtime `isinstance()` enforcement, shared
method implementations via inheritance, or `@abstractmethod` guarantees.
### Python 3.14 typing improvements
**PEP 649 (deferred annotations)**: On Python 3.14+, annotations are lazily
evaluated. Forward references work natively — no need for
`from __future__ import annotations`. Drop that import on new 3.14+ projects.
**Other modern typing patterns**: use `Self` for fluent APIs, `@override` for
explicit method overriding, `ParamSpec` for typed decorators, and
`TypeIs` / `TypeGuard` for type narrowing in guards.
> **`@override` caveat (PEP 698).** `@override` requires an actual base-class
> method to override. On a plain structural class that satisfies a `Protocol`
> *without* subclassing it — the Protocol-first default above — do not add
> `@override`: there is no base method, so the type checker flags it as an
> error. Structural conformance and `@override` are mutually exclusive; reach
> for `@override` only inside a real inheritance chain (an ABC subclass, or a
> class that explicitly subclasses its base).
→ Full architecture rationale: **Read `references/ecosystem_rationale.md`**
---
## Project Layout Decision
Ask (or infer) whether the project is a **Library/Data tool** or an
**Application**.
- **Library layout** → domain-centric modules exposed directly
(`src/mylib/datatools/`, `src/mylib/core/`)
- **Application layout** → decouple logic from entry points
(`src/myapp/core/` + `src/myapp/interface/`)
- **Tests** → a top-level `tests/` tree, **never inside `src/`**. Colocated
`src/.../tests/` modules ship inside the built wheel and pollute the installed
package; for per-unit suites (datajobs, plugins) mirror the package under
`tests///`. (`doctor.py` flags any `test_*.py` found under `src/`.)
The application layout follows the **functional core / imperative shell**
pattern: `core/` contains pure business logic (no I/O, fully testable),
while `interface/` contains thin adapters (FastAPI routes, CLI commands,
database clients). Connect them via `Protocol` interfaces and constructor
injection.
→ Full directory trees and the authoritative `pyproject.toml` template:
**Read `references/project_templates.md`**
---
## Scaffolding Protocol
When generating a project for a user:
1. **Resolve names** from the project name they provide:
- PyPI: `kebab-case` (e.g., `my-cool-tool`)
- Package: `snake_case` (e.g., `src/my_cool_tool/`)
- Classes: `PascalCase`
2. **Mandatory substitutions in `pyproject.toml`**:
- `[project] name`
- `[tool.ruff.lint.isort] known-first-party = ["actual_name"]`
3. **Strip irrelevant optional deps**: if user doesn't mention data processing,
remove `datatools` extras; if no CLI, remove `typer`; if no web, remove
`fastapi`.
4. **Initialization command**: always `uv init --lib ` to get the
src layout with `uv_build` from the start.
---
## Dependency Management (PEP 735)
Use `[dependency-groups]` (PEP 735) for development dependencies — **not**
`[project.optional-dependencies]`, which is for end-user feature extras
(`pip install mylib[cli]`). Dev deps are local-only and must not ship with the
package. Group into `lint` / `test` / `security`, plus a `dev` group that
includes them.
→ The authoritative `[dependency-groups]` block:
**Read `references/project_templates.md`**
---
## Ruff Single-Quote Enforcement
Critical: both the **formatter** and **linter** must be aligned or they fight.
```toml
[tool.ruff.format]
quote-style = "single"
[tool.ruff.lint.flake8-quotes]
inline-quotes = "single"
docstring-quotes = "double"
multiline-quotes = "double"
```
Ruff 0.8+ deltas: the `TCH` category is now `TC` (use `"TC"` in `select`);
`ANN101` / `ANN102` were removed (don't list them in `ignore`); block
suppression via `# ruff: disable[RULE]` / `# ruff: enable[RULE]` avoids per-line
`# noqa`.
---
## Configuration Management Pattern
Never read `os.getenv()` directly. Use a typed `Settings(BaseSettings)` from
`pydantic-settings`, with `SecretStr` for secrets (masked in logs). Keep `.env`
gitignored and commit `.env.template` to document required keys.
```python
from pydantic import SecretStr
from pydantic_settings import BaseSettings, SettingsConfigDict
class Settings(BaseSettings):
API_KEY: SecretStr
model_config = SettingsConfigDict(env_file='.env', extra='ignore')
settings = Settings()
```
→ Why typed config (fail-fast at startup, validation): **Read
`references/ecosystem_rationale.md`**
---
## Observability
Observability has three pillars: **structured logs**, **distributed traces**,
and **metrics**. Use `structlog` for logging and `opentelemetry` for
traces/metrics. Correlate them by injecting trace IDs into log entries.
→ Full structlog + OpenTelemetry setup, stdlib bridge, FastAPI middleware,
and pytest integration: **Read `references/observability.md`**
The minimum-viable `configure_logging()` (JSON in prod, pretty in dev) and the
full trace-correlated setup both live in **`references/observability.md`** —
call it once at application startup (CLI entry point or FastAPI `lifespan`).
---
## Pre-commit Hooks
`scripts/scaffold.py` writes the canonical `.pre-commit-config.yaml`
(trailing-whitespace, end-of-file-fixer, check-yaml, check-added-large-files,
plus `ruff` + `ruff-format` at the revs pinned in `stack.toml`). Install with
`uv run pre-commit install`. Enforcement is a ladder, not an assumption: on a
harness with act-time hooks (Claude Code) this plugin formats each edit and
blocks pip/poetry in uv projects as it happens; elsewhere the same rules hold
at commit time via this pre-commit config (plus the exported
`check-uv-hygiene` hook), and as advisory text where neither exists.
> **Note on ty in pre-commit**: `ty` has no official pre-commit hook yet. Run it
> via `uv run ty check src` in CI or as a local script.
---
## Key uv Commands
```bash
uv init --lib my-project # Scaffold with src layout + uv_build
uv add # Add runtime dependency
uv add --dev # Add dev dependency (PEP 735)
uv add --group lint # Add to specific dependency group
uv sync # Install all deps from lockfile
uv sync --frozen # CI: install without updating lockfile
uv run pytest # Run tests in project venv
uv run ruff check src # Lint
uv run ruff format src # Format
uv run ty check src # Type check (ty)
uv run mypy src # Type check (mypy, if used)
uv python install 3.14 # Install a specific Python version
uv python pin 3.14 # Pin project to Python 3.14
```
---
## AI-Assistant Configuration
Modern Python projects include configuration files for AI coding assistants.
These are committed to the repo and provide persistent project context.
- **`CLAUDE.md`** — Read by Claude Code at conversation start. Include
build/test commands, architecture decisions, code style preferences.
Keep under ~50 concise instructions. Use `/init` to auto-generate.
- **`.cursor/rules/*.mdc`** — Cursor rules with frontmatter-based glob
matching (e.g., scope rules to `app/routers/*.py`).
- **`.github/copilot-instructions.md`** — Repo-level instructions for
GitHub Copilot Chat and Code Review.
→ Full templates and patterns: **Read `references/ai_config.md`**
---
## Supply-Chain Security
Add these to every CI pipeline:
```bash
uv run pip-audit # Scan for known CVEs (OSV database)
uv run ruff check src --select S # Bandit-equivalent security linting
```
For package publishers: use **Trusted Publishers** on PyPI (OIDC-based, no
long-lived API tokens) and **Sigstore attestations** (automatic with the
canonical GitHub Action). Pin all dependencies with cryptographic hashes
via `uv lock`.
→ Full security setup and CI integration: **Read `references/security.md`**
---
## Reference Files
Exhaustive templates and patterns — read on demand:
- **`project_templates.md`** — `pyproject.toml` master config, both directory layouts, GitHub Actions CI.
- **`ecosystem_rationale.md`** — why each tool, Protocol vs ABC, functional core / imperative shell, DI.
- **`observability.md`** — structlog + OpenTelemetry setup, trace correlation, FastAPI middleware.
- **`testing_and_qa.md`** — pytest, Hypothesis, mutation/snapshot testing, testcontainers, unit/integration split.
- **`docker_patterns.md`** — multi-stage Dockerfile with uv, CI/CD Docker caching.
- **`security.md`** — Trusted Publishers, Sigstore, pip-audit, dependency pinning.
- **`ai_config.md`** — CLAUDE.md, Cursor rules, Copilot instructions.
- **`currency_review.md`** — quarterly review protocol for keeping the skill current.
---
## Scripts, freshness, and keeping current
> **last-reviewed: 2026-06-04.** Pinned versions live in `stack.toml` — the
> single source of truth this skill cites instead of repeating numbers in prose.
> When exact current versions matter, verify against PyPI / context7 at the
> point of use; a pinned floor can lag the latest release.
Bundled scripts (`scripts/`):
```bash
python scripts/scaffold.py my-cool-tool # new project to standard (uv init + canonical pyproject + pre-commit)
python scripts/doctor.py [path] # audit an existing project against the standard
python scripts/check_versions.py # compare stack.toml pins to latest on PyPI (--json for CI)
```
**Freshness loop (Tier 3).** `check_versions.py` detects drift (exits non-zero
when a pin is behind a newer minor/major); a monthly CI cron opens a drift
issue; the **`/refresh-stack`** command reviews each changelog and proposes the
`stack.toml` + guidance updates for you to approve — mechanical bumps applied on
approval, guidance edits never auto-applied. See `references/currency_review.md`
for the full review checklist.