Python Infrastructure
System-reliability concerns for Python services in this project, grouped because real code uses them together: a task you queue (background-jobs) needs retries (resilience), instrumentation (observability), and often touches the cache (Redis) on the same call path.
Preferred stack
| Concern | Tool | Notes |
|---|---|---|
| Message bus / task queue | NATS JetStream (via nats-py) |
Durable streams, consumer groups, replay. Replaces Celery/RabbitMQ here. |
| Durable multi-step workflows | Dapr Workflow (via dapr-ext-workflow) |
Activity-level checkpointing via the Dapr sidecar (one per pod). When vs JetStream: references/dapr-workflows.md § When to use. |
| HTTP | FastAPI | See sibling fastapi skill. |
| Cache | Redis | redis.asyncio for async workers. |
| Retries / backoff | tenacity | Exponential + jitter, by default. |
| Observability | OpenTelemetry (OTLP) | Traces + metrics + logs. See sibling otel. |
| Logging | stdlib logging → OTel handler |
Don't pull in structlog; OTel forwards stdlib records. |
| HTTP client | httpx (async) | Replaces requests. |
Scope routing
| If you need to… | Read |
|---|---|
| Handle an operation that can fail transiently (network/IO/3rd-party API) — what to retry, with what backoff, when to stop, circuit-breakers | references/resilience.md |
Run work out-of-request as a single idempotent action or fanout/event distribution — queue a task, design a worker, persist job state, retry/DLQ patterns (NATS JetStream + nats-py) |
references/background-jobs.md |
| Run a multi-step workflow where re-running step 1 on crash is bad — activity-level recovery, retry policies, scheduling (Dapr Workflow, sidecar required) | references/dapr-workflows.md |
| Know what's happening in production — instrument a service with OTel traces/metrics/logs, four golden signals | references/observability.md |
| Avoid repeated expensive lookups — Redis as a cache (TTL, invalidation) or short-lived coordination (rate limits, dedup windows, single-flight locks) | references/caching.md |
JetStream vs Dapr Workflow in depth: references/dapr-workflows.md § When to use. All five at once for one feature? Instrument first, then queue / workflow + retry + cache.
Cross-skill boundaries
writing-python— how to write the function. This skill — how it survives in production.writing-python→references/error-handling.md— what exception to raise. This skill — what to do when it's raised across a network boundary.writing-python→references/resource-management.md— how to clean up resources (context managers). This skill — how to keep retrying when resources fail to acquire.fastapi— request handlers and DI. This skill — what runs around them.otel— full OTel reference (Python SDK, signals, attributes, Collector). This skill'sobservability.mdpins project conventions on top.
Top gotchas
- Retry without backoff is a DoS amplifier — default to exponential backoff + jitter from day one (
references/resilience.md). - Retrying non-idempotent operations duplicates side effects — pair retry with an idempotency key OR mark the operation non-retryable (
references/background-jobs.md§ Make tasks idempotent). - Synchronous code inside an async worker blocks the event loop — use
httpx.AsyncClient, or run sync code in an executor. - Logs and metrics serve different audiences — don't try to derive one from the other; instrument both.
- Trace context is lost across queue boundaries unless you
inject/extractit via message headers (references/observability.md§ Trace context across queue boundaries). - Redis cache stampede — N clients miss a hot key simultaneously and all recompute; use single-flight locks (
references/caching.md§ Cache stampede protection).