Python Packaging

Python Packaging

Comprehensive guide to creating, structuring, and distributing Python packages using modern packaging tools, pyproject.toml, and publishing to PyPI.

Core Concepts

1. Package Structure

• Source layout: src/package_name/ (recommended)
• Flat layout: package_name/ (simpler but less flexible)
• Package metadata: pyproject.toml, setup.py, or setup.cfg
• Distribution formats: wheel (.whl) and source distribution (.tar.gz)

2. Modern Packaging Standards

• PEP 517/518: Build system requirements
• PEP 621: Metadata in pyproject.toml
• PEP 660: Editable installs
• pyproject.toml: Single source of configuration

3. Build Backends

• setuptools: Traditional, widely used
• hatchling: Modern, opinionated
• flit: Lightweight, for pure Python
• poetry: Dependency management + packaging

4. Distribution

• PyPI: Python Package Index (public)
• TestPyPI: Testing before production
• Private repositories: JFrog, AWS CodeArtifact, etc.

Quick Start

Minimal Package Structure

my-package/
|-- pyproject.toml
|-- README.md
|-- LICENSE
|-- src/
|   `-- my_package/
|       |-- __init__.py
|       `-- module.py
`-- tests/
    `-- test_module.py

Minimal pyproject.toml

[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"

[project]
name = "my-package"
version = "0.1.0"
description = "A short description"
authors = [{name = "Your Name", email = "you@example.com"}]
readme = "README.md"
requires-python = ">=3.8"
dependencies = [
    "requests>=2.28.0",
]

[project.optional-dependencies]
dev = [
    "pytest>=7.0",
    "black>=22.0",
]

Package Structure Patterns

Pattern 1: Source Layout (Recommended)

my-package/
|-- pyproject.toml
|-- README.md
|-- LICENSE
|-- .gitignore
|-- src/
|   `-- my_package/
|       |-- __init__.py
|       |-- core.py
|       |-- utils.py
|       `-- py.typed          # For type hints
|-- tests/
|   |-- __init__.py
|   |-- test_core.py
|   `-- test_utils.py
`-- docs/
    `-- index.md

Advantages:

• Prevents accidentally importing from source
• Cleaner test imports
• Better isolation

pyproject.toml for source layout:

[tool.setuptools.packages.find]
where = ["src"]

Pattern 2: Flat Layout

my-package/
|-- pyproject.toml
|-- README.md
|-- my_package/
|   |-- __init__.py
|   `-- module.py
`-- tests/
    `-- test_module.py

Simpler but:

• Can import package without installing
• Less professional for libraries

Pattern 3: Multi-Package Project

project/
|-- pyproject.toml
|-- packages/
|   |-- package-a/
|   |   `-- src/
|   |       `-- package_a/
|   `-- package-b/
|       `-- src/
|           `-- package_b/
`-- tests/

Detailed patterns and worked examples

Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient. For advanced patterns, see references/advanced-patterns.md.