Python

Python Backend Development Skill

File Organization

This skill uses a split structure for HIGH-RISK requirements:

• SKILL.md: Core principles, patterns, and essential security (this file)
• references/security-examples.md: Complete CVE details and OWASP implementations
• references/advanced-patterns.md: Advanced Python patterns and optimization
• references/threat-model.md: Attack scenarios and STRIDE analysis

Validation Gates

1. Overview

Risk Level: HIGH

Justification: Python backend services handle authentication, database access, file operations, and external API communication. Vulnerabilities in input validation, deserialization, command execution, and cryptography can lead to data breaches and system compromise.

You are an expert Python backend developer specializing in secure, maintainable, and performant services.

Core Expertise Areas

• Type annotations and runtime validation
• Async programming with asyncio
• Security: input validation, cryptography, secrets management
• Testing: pytest, property-based testing, security testing
• Database access with SQLAlchemy/asyncpg
• API development with FastAPI/Starlette

2. Core Responsibilities

Fundamental Principles

1. TDD First: Write tests before implementation, design API through test cases
2. Performance Aware: Use async, generators, efficient data structures by default
3. Type Safety: Use type hints everywhere, validate at runtime boundaries
4. Defense in Depth: Multiple validation layers, fail securely
5. Secure Defaults: Use safe libraries, reject unsafe operations
6. Explicit over Implicit: Clear error handling, explicit dependencies
7. Testability: Design for testing, write security tests

Decision Framework

2.1 Implementation Workflow (TDD)

Step 1: Write Failing Test First

import pytest
from my_service import UserService, UserNotFoundError

class TestUserService:
    @pytest.mark.asyncio
    async def test_get_user_returns_user_when_exists(self, db_session):
        service = UserService(db_session)
        user_id = await service.create_user("alice", "alice@example.com")
        user = await service.get_user(user_id)
        assert user.username == "alice"

    @pytest.mark.asyncio
    async def test_get_user_raises_when_not_found(self, db_session):
        service = UserService(db_session)
        with pytest.raises(UserNotFoundError):
            await service.get_user(99999)

    @pytest.mark.asyncio
    async def test_create_user_validates_email(self, db_session):
        service = UserService(db_session)
        with pytest.raises(ValueError, match="Invalid email"):
            await service.create_user("bob", "not-an-email")

Step 2: Implement Minimum to Pass

class UserNotFoundError(Exception): pass

class UserService:
    def __init__(self, db: AsyncSession):
        self.db = db

    async def get_user(self, user_id: int) -> User:
        user = await self.db.get(User, user_id)
        if not user:
            raise UserNotFoundError(f"User {user_id} not found")
        return user

    async def create_user(self, username: str, email: str) -> int:
        if "@" not in email:
            raise ValueError("Invalid email format")
        # ... minimal implementation to pass tests

Step 3: Refactor if Needed

• Extract common patterns, add type hints, ensure errors don't leak internals

Step 4: Run Full Verification

pytest --cov=src           # All tests pass
mypy src/ --strict         # Type check passes
bandit -r src/ -ll         # Security scan passes
pip-audit && safety check  # Dependencies clean

2.2 Performance Patterns

Pattern 1: Async I/O with asyncio.gather

# BAD: Sequential requests (slow)
for url in urls:
    response = await client.get(url)  # Waits for each one

# GOOD: Concurrent requests with gather
tasks = [client.get(url) for url in urls]
responses = await asyncio.gather(*tasks)  # All at once

Pattern 2: Generators for Large Data Processing

# BAD: Load all into memory
return [process(line) for line in f.readlines()]  # OOM risk

# GOOD: Generator yields one at a time
def process_large_file(filepath: str) -> Iterator[dict]:
    with open(filepath) as f:
        for line in f:
            yield process(line)  # Memory efficient

Pattern 3: Efficient Data Structures

# BAD: List for membership testing - O(n)
required in user_perms_list  # Slow for large lists

# GOOD: Set for membership testing - O(1)
required in user_perms_set  # Fast lookup

# BAD: Repeated string concatenation
result = ""; for f in fields: result += f + ", "  # Creates new string each time

# GOOD: Join for string building
", ".join(fields)  # Single allocation

Pattern 4: Connection Pooling

# BAD: New connection per request
engine = create_async_engine(DATABASE_URL)  # Connection overhead each time

# GOOD: Reuse pooled connections
engine = create_async_engine(DATABASE_URL, pool_size=20, max_overflow=10)
async_session = sessionmaker(engine, class_=AsyncSession)

async def get_user(user_id: int):
    async with async_session() as session:  # Reuses pooled connection
        return await session.get(User, user_id)

Pattern 5: Batch Database Operations

# BAD: Individual inserts (N round trips)
for user in users:
    db.add(User(**user)); await db.commit()  # N commits = slow

# GOOD: Batch insert (1 round trip)
stmt = insert(User).values(users)
await db.execute(stmt); await db.commit()  # Single commit

# GOOD: Chunked for very large datasets
for i in range(0, len(users), 1000):
    await db.execute(insert(User).values(users[i:i+1000]))
await db.commit()

3. Technical Foundation

Version Recommendations

Security Dependencies

# pyproject.toml
[project]
dependencies = [
    "pydantic>=2.0", "email-validator>=2.0",      # Validation
    "cryptography>=41.0", "argon2-cffi>=21.0",    # Cryptography
    "PyJWT>=2.8", "sqlalchemy>=2.0", "asyncpg>=0.28",
    "httpx>=0.25", "bandit>=1.7",
]

[project.optional-dependencies]
dev = ["pytest>=7.0", "pytest-asyncio>=0.21", "hypothesis>=6.0", "safety>=2.0", "pip-audit>=2.0"]

4. Implementation Patterns

Pattern 1: Type-Safe Input Validation

from pydantic import BaseModel, Field, field_validator, EmailStr
from typing import Annotated
import re

class UserCreate(BaseModel):
    """Validated user creation request."""
    username: Annotated[str, Field(min_length=3, max_length=50)]
    email: EmailStr
    password: Annotated[str, Field(min_length=12)]

    @field_validator('username')
    @classmethod
    def validate_username(cls, v: str) -> str:
        if not re.match(r'^[a-zA-Z0-9_-]+$', v):
            raise ValueError('Username must be alphanumeric')
        return v

    @field_validator('password')
    @classmethod
    def validate_password_strength(cls, v: str) -> str:
        if not all([re.search(r'[A-Z]', v), re.search(r'[a-z]', v), re.search(r'\d', v)]):
            raise ValueError('Password needs uppercase, lowercase, and digit')
        return v

Pattern 2: Secure Password Hashing

from argon2 import PasswordHasher
from argon2.exceptions import VerifyMismatchError

ph = PasswordHasher(time_cost=3, memory_cost=65536, parallelism=4)

def hash_password(password: str) -> str:
    return ph.hash(password)

def verify_password(password: str, hash: str) -> bool:
    try:
        ph.verify(hash, password)
        return True
    except VerifyMismatchError:
        return False

Pattern 3: Safe Database Queries

from sqlalchemy import select, text
from sqlalchemy.ext.asyncio import AsyncSession

# NEVER: f"SELECT * FROM users WHERE username = '{username}'"

async def get_user_safe(db: AsyncSession, username: str) -> User | None:
    stmt = select(User).where(User.username == username)
    result = await db.execute(stmt)
    return result.scalar_one_or_none()

async def search_users(db: AsyncSession, pattern: str) -> list:
    stmt = text("SELECT * FROM users WHERE username LIKE :pattern")
    result = await db.execute(stmt, {"pattern": f"%{pattern}%"})
    return result.fetchall()

Pattern 4: Safe File Operations

from pathlib import Path

def safe_read_file(base_dir: Path, user_filename: str) -> str:
    if '..' in user_filename or user_filename.startswith('/'):
        raise ValueError("Invalid filename")

    file_path = (base_dir / user_filename).resolve()
    if not file_path.is_relative_to(base_dir.resolve()):
        raise ValueError("Path traversal detected")

    return file_path.read_text()

Pattern 5: Safe Subprocess Execution

import subprocess

ALLOWED_PROGRAMS = {'git', 'python', 'pip'}

def run_command_safe(program: str, args: list[str]) -> str:
    if program not in ALLOWED_PROGRAMS:
        raise ValueError(f"Program not allowed: {program}")

    result = subprocess.run(
        [program, *args],
        capture_output=True, text=True, timeout=30, check=True,
    )
    return result.stdout

5. Security Standards

5.1 Domain Vulnerability Landscape

See references/security-examples.md for complete CVE details and mitigation code

5.2 OWASP Top 10 Mapping

5.3 Essential Security Patterns

from pydantic import BaseModel, field_validator
import os, logging

# Secure base model - reject unknown fields, strip whitespace
class SecureInput(BaseModel):
    model_config = {'extra': 'forbid', 'str_strip_whitespace': True}

    @field_validator('*', mode='before')
    @classmethod
    def reject_null_bytes(cls, v):
        if isinstance(v, str) and '\x00' in v:
            raise ValueError('Null bytes not allowed')
        return v

# Secrets from environment (NEVER hardcode)
API_KEY = os.environ["API_KEY"]
DB_URL = os.environ["DATABASE_URL"]

# Safe error handling - log details, return safe message
class AppError(Exception):
    def __init__(self, message: str, internal: str = None):
        self.message = message
        if internal:
            logging.error(f"{message}: {internal}")

    def to_response(self) -> dict:
        return {"error": self.message}

See references/advanced-patterns.md for secrets manager integration

6. Testing & Validation

Security Testing Commands

bandit -r src/ -ll          # Static analysis
pip-audit && safety check   # Dependency vulnerabilities
mypy src/ --strict          # Type checking

Security Test Examples

import pytest
from pathlib import Path

def test_sql_injection_prevented(db):
    for payload in ["'; DROP TABLE users; --", "' OR '1'='1", "admin'--"]:
        assert get_user_safe(db, payload) is None

def test_path_traversal_blocked():
    base = Path("/app/data")
    for attack in ["../etc/passwd", "..\\windows\\system32", "foo/../../etc/passwd"]:
        with pytest.raises(ValueError, match="traversal|Invalid"):
            safe_read_file(base, attack)

def test_command_injection_blocked():
    with pytest.raises(ValueError, match="not allowed"):
        run_command_safe("rm", ["-rf", "/"])

See references/security-examples.md for comprehensive test patterns

7. Common Mistakes & Anti-Patterns

8. Pre-Deployment Checklist

Phase 1: Before Writing Code

• Requirements understood and documented
• API design reviewed (inputs, outputs, errors)
• Security threat model considered
• Test cases written first (TDD)
• Edge cases and error scenarios identified

Phase 2: During Implementation

• Following TDD workflow (test -> implement -> refactor)
• Using performance patterns (async, generators, pooling)
• All inputs validated with Pydantic
• DB queries parameterized/ORM
• File ops check path containment
• Subprocess uses list args
• Passwords hashed with Argon2id
• Secrets from environment only

Phase 3: Before Committing

• All tests pass: pytest --cov=src
• Type check passes: mypy src/ --strict
• Security scan passes: bandit -r src/ -ll
• Dependency audit passes: pip-audit && safety check
• No hardcoded secrets in code
• Errors don't leak internal details
• Debug mode disabled
• Logging configured (no PII/secrets)

9. Summary

Create Python code that is type safe, secure, testable, and maintainable.

Security Essentials:

1. Validate and sanitize all user input
2. Use parameterized queries for database ops
3. Never use shell=True with user input
4. Hash passwords with Argon2id
5. Load secrets from environment
6. Keep dependencies updated and audited

For attack scenarios and threat modeling, see references/threat-model.md