Python Automation Skill
This skill provides concrete Python patterns to implement robust REST API automations.
When to use this skill
- Python scripts that call one or more REST APIs
- ETL jobs, sync tools, webhook handlers
- CLI tools or small services that integrate external systems
Preferred stack (defaults)
- HTTP client: httpx (preferred) or requests
- Config: env vars +
.env (optional) with pydantic-settings if appropriate
- Logging: stdlib
logging with structured-ish fields
- Testing: pytest (+ respx for httpx mocking when useful)
If the project already uses different tools, follow project conventions.
Reference architecture (small but scalable)
client.py: API client wrapper (auth headers, retries, pagination helpers)
models.py: typed payload models (dataclasses or pydantic)
sync.py: orchestration logic (fetch -> transform -> upsert)
main.py: CLI entrypoint
tests/: unit tests for transform + client behavior
HTTP rules (mandatory)
- Always set timeouts (connect + read)
- Centralize request sending in one function so retries/logging are consistent
- Never log secrets (Authorization headers, tokens)
Retry policy guidance
Retry on:
- network errors/timeouts
- 429 (respect Retry-After when present)
- 500–599
Optional: 408, and 409 only if operation is safe and semantics known
Do NOT retry on:
- most 400–499 (unless explicitly safe)
Timeouts
- Set explicit timeouts; do not rely on defaults.
- Use smaller connect timeout; moderate read timeout.
Pagination patterns
Support at least one helper that can handle:
next URL in response
- cursor token in response
- page/limit parameters
Add a hard stop:
- max pages OR max items OR max elapsed time
Idempotency patterns (Python)
Choose and document:
- Use an
Idempotency-Key header when supported
- Upsert using a stable
external_id
- Persist a lightweight state store:
- simplest: SQLite file (recommended for OSS)
- alternative: JSONL log + compaction
Minimum: ensure repeated runs don’t create duplicates.
Observability (Python)
Minimum logs should include:
run_id
- request: method, url/path, status_code, elapsed_ms, attempt
- record counts: processed/created/updated/skipped/failed
Also include a final summary log line.
Verification requirements
For non-trivial work, add:
- unit tests for mapping/transform logic
- at least one test for pagination or retry behavior (mocked)
- a “dry-run” CLI flag (prints intended writes)
Output format when writing code
- Provide a small directory layout
- Explain how to configure env vars
- Include exact commands to run (and test)
1---2name: superpowers-python-automation3description: "Implements reliable automations in Python for REST APIs: httpx/requests patterns, retries, timeouts, pagination, typing, config, logging, and tests. Use when writing Python scripts/services that call external APIs."4metadata:5 source: https://skillmd.com/skills/phoroth/superpowers-python-automation6---78# Python Automation Skill910This skill provides concrete Python patterns to implement robust REST API automations.1112## When to use this skill13- Python scripts that call one or more REST APIs14- ETL jobs, sync tools, webhook handlers15- CLI tools or small services that integrate external systems1617## Preferred stack (defaults)18- HTTP client: **httpx** (preferred) or requests19- Config: env vars + `.env` (optional) with pydantic-settings if appropriate20- Logging: stdlib `logging` with structured-ish fields21- Testing: pytest (+ respx for httpx mocking when useful)2223If the project already uses different tools, follow project conventions.2425---2627## Reference architecture (small but scalable)28- `client.py`: API client wrapper (auth headers, retries, pagination helpers)29- `models.py`: typed payload models (dataclasses or pydantic)30- `sync.py`: orchestration logic (fetch -> transform -> upsert)31- `main.py`: CLI entrypoint32- `tests/`: unit tests for transform + client behavior3334## HTTP rules (mandatory)35- Always set timeouts (connect + read)36- Centralize request sending in one function so retries/logging are consistent37- Never log secrets (Authorization headers, tokens)3839### Retry policy guidance40Retry on:41- network errors/timeouts42- 429 (respect Retry-After when present)43- 500–59944Optional: 408, and 409 only if operation is safe and semantics known4546Do NOT retry on:47- most 400–499 (unless explicitly safe)4849### Timeouts50- Set explicit timeouts; do not rely on defaults.51- Use smaller connect timeout; moderate read timeout.5253---5455## Pagination patterns56Support at least one helper that can handle:57- `next` URL in response58- cursor token in response59- page/limit parameters6061Add a hard stop:62- max pages OR max items OR max elapsed time6364---6566## Idempotency patterns (Python)67Choose and document:68- Use an `Idempotency-Key` header when supported69- Upsert using a stable `external_id`70- Persist a lightweight state store:71 - simplest: SQLite file (recommended for OSS)72 - alternative: JSONL log + compaction7374Minimum: ensure repeated runs don’t create duplicates.7576---7778## Observability (Python)79Minimum logs should include:80- `run_id`81- request: method, url/path, status_code, elapsed_ms, attempt82- record counts: processed/created/updated/skipped/failed8384Also include a final summary log line.8586---8788## Verification requirements89For non-trivial work, add:90- unit tests for mapping/transform logic91- at least one test for pagination or retry behavior (mocked)92- a “dry-run” CLI flag (prints intended writes)9394---9596## Output format when writing code97- Provide a small directory layout98- Explain how to configure env vars99- Include exact commands to run (and test)