Software Backend Engineering
Use this skill for backend service implementation and review: API boundaries, auth, data access, jobs, caching, observability, and production hardening. If the main question is platform selection, system topology, or API-contract design without implementation, hand off early.
Defaults
When this skill is active, prefer these defaults unless the repo or user says otherwise:
- validate at the boundary and keep types explicit
- use PostgreSQL plus pooling for relational workloads
- use structured logs, OpenTelemetry, explicit timeouts, and rate limits
- make mutations idempotent and background work retry-safe
- use RFC 9457 Problem Details for machine-readable errors
Quick Reference
| Need |
Default Direction |
| Public HTTP API |
REST with explicit contracts and timeouts |
| Internal TS monorepo API |
tRPC when end-to-end type safety matters |
| High-throughput internal RPC |
Connect or gRPC |
| Complex client-shaped reads |
GraphQL |
| Relational data |
PostgreSQL with migrations and pooling |
| Background work |
Queue plus idempotent handlers and DLQ policy |
| Browser auth |
OIDC or OAuth plus httpOnly cookies |
| Service auth |
short-lived tokens, workload identity, or signed service credentials |
| Caching |
explicit TTLs and invalidation rules |
| Observability |
correlation IDs, traces, structured logs, saturation metrics |
When to Use This Skill
- building or reviewing REST, GraphQL, tRPC, Connect, or gRPC services
- implementing auth, validation, rate limits, caching, queues, or webhook handling
- modelling schemas and running safe migrations
- hardening service behavior for retries, timeouts, and observability
- scaffolding or refactoring a backend with production defaults
Route Elsewhere
Workflow
- Confirm the real constraint: latency, team skill, runtime, compliance, data model, or delivery speed.
- Choose the transport and framework based on that constraint, not on trend-chasing.
- Define the boundary:
- request and response contracts
- auth and authorization rules
- error model
- idempotency and rate limiting
- Define the data path:
- schema and migrations
- transaction boundaries
- pooling and query budgets
- cache and invalidation rules
- Define the async path:
- queue semantics
- retry ownership
- deduplication and DLQ
- Add operability before calling it complete:
- timeouts and cancellation
- health checks
- structured logs and traces
- deploy and rollback expectations
ASCII Flow
Backend task
-> Define endpoint, job, service, or data boundary
-> Confirm runtime, framework, persistence, and integration contracts
-> Design request validation, auth, errors, and idempotency
-> Implement bounded slice with tests and observability
-> Check performance, security, and rollout risk
-> Verify behavior and document follow-up handoffs
Technology Selection
Pick based on the strongest operational constraint:
- TypeScript-heavy team -> Fastify, Hono, or NestJS plus Prisma or Drizzle
- audited SQL and predictable concurrency -> Go with
sqlc/pgx
- Python ecosystem or ML adjacency -> FastAPI plus SQLAlchemy
- enterprise .NET stack -> ASP.NET Core plus EF Core or explicit SQL access
- memory safety and explicitness -> Rust with Axum plus SQLx
- edge or serverless first -> lightweight stateless handlers with hard CPU and timeout budgets
Use software-baas-platforms first when the real requirement is "ship auth, storage, and realtime quickly with less custom service code."
Backend Non-Negotiables
| Category |
Rule |
| API |
Mutating endpoints require idempotency keys where retries are plausible |
| API |
List endpoints require explicit pagination (limit/cursor) and at least one filter |
| API |
Errors are structured and machine-readable (RFC 9457 Problem Details) |
| API |
Health endpoints separate liveness (/healthz) from readiness (/readyz) |
| Data |
No SELECT * on wide or high-volume paths |
| Data |
Transactions kept explicit; no implicit ambient transactions |
| Data |
New or changed query plans verified with EXPLAIN ANALYZE before production |
| Data |
ORM convenience layers bypassed on hot paths where auditability matters |
| Dependencies |
Every outbound call has an explicit timeout; no framework-default infinite wait |
| Dependencies |
Retries owned at exactly one layer (no double-retry across client + service) |
| Dependencies |
Cache invalidation rule documented before caching is added |
| Dependencies |
Background jobs safe to retry and observable (structured log on start/finish/failure) |
| Operations |
Every request carries a correlation ID propagated to all downstream calls |
| Operations |
Trace, log, and metric identifiers agree (no split identity) |
| Operations |
Slow paths have explicit latency budgets (p99 target, not "fast enough") |
| Operations |
Deploy procedure includes rollback step and smoke-check list |
Performance and Reliability Triage
When a service is slow or unstable, debug in this order:
| Step |
Check |
Signal |
| 1 |
Query behavior and N+1s |
EXPLAIN output, ORM query log showing repeated identical queries |
| 2 |
Indexes and execution plans |
Seq scans on large tables, missing index on FK or filter columns |
| 3 |
Connection pooling and queue depth |
Pool wait time > 10ms; idle connections exhausted |
| 4 |
Timeout and cancellation gaps |
Requests hanging past deadline; no context propagation through outbound calls |
| 5 |
Caching or read-shaping opportunities |
Same query with same result executing > 10x/s; hot read path with no invalidation |
| 6 |
Runtime or tier limits |
CPU throttling, memory pressure, rate limit headers from upstream |
Do not add caching before you understand the real bottleneck.
Operational Playbooks
- use references/operational-playbook.md for full service design and review checklists
- use qa-resilience when retries, deadlines, breakers, or degraded-mode behavior are the main question
- use dev-api-design when the contract itself is the main artifact
Production Readiness Checklist
Before marking a service production-ready:
Known Traps
- Introducing asynchronous jobs to hide a broken synchronous path instead of fixing the contract, timeout budget, or workload shape.
- Shipping retries without deadlines, jitter, and idempotency keys, then multiplying load during incidents.
- Changing API or webhook behavior without a compatibility window, replay plan, or structured error-versioning posture.
- Adding caches before proving whether the real bottleneck is query shape, pooling, lock contention, or outbound dependency latency.
- Treating background consumers as “fire and forget” even though poison-message handling, replay semantics, and observability are undefined.
Common Anti-Patterns
- Letting framework defaults define the service contract, error model, and cancellation semantics.
- Using one generic repository abstraction for every query, including hot paths that need explicit SQL, batching, or shape control.
- Mixing request handling, domain logic, external side effects, and persistence concerns in one controller or handler.
- Relying on eventual retries to clean up non-idempotent side effects.
- Calling a backend “production ready” before timeouts, readiness checks, trace correlation, and rollback smoke tests exist.
Navigation
Core references
- references/backend-best-practices.md
- references/edge-deployment-guide.md
- references/infrastructure-economics.md
- references/database-patterns.md
- references/message-queues-background-jobs.md
- references/rpc-and-transport-patterns.md
- references/go-best-practices.md
- references/rust-best-practices.md
- references/python-best-practices.md
- references/nodejs-best-practices.md
- references/csharp-best-practices.md
- data/sources.json
Shared review utilities
Templates
- assets/nodejs/template-nodejs-prisma-postgres.md
- assets/nodejs/template-nodejs-fastify-drizzle-postgres.md
- assets/go/template-go-fiber-gorm.md
- assets/go/template-go-chi-sqlc-pgx.md
- assets/rust/template-rust-axum-seaorm.md
- assets/rust/template-rust-axum-sqlx.md
- assets/python/template-python-fastapi-sqlalchemy.md
- assets/csharp/template-csharp-aspnet-efcore.md
Related Skills
Gate before invoking any foundation below: Each foundation has a When to Apply / When to Skip section. If your task matches a skip-condition, route to the foundation it names instead — don't pull in primitives the task doesn't need.
Fact-Checking
- Known bugs, regressions, framework/compiler/runtime footguns, and version-specific crash or workaround guidance must be verified against current primary web sources before being treated as current fact.
- Verify current runtime versions, support windows, framework capabilities, and cloud-platform constraints before final answers.
- Prefer official docs and release or support policy pages for version-sensitive recommendations.
- If web access is unavailable, mark version or support guidance as unverified.
Learnings Loop
Before applying this skill on a non-trivial task, read learnings.consolidated.md in this directory (and learnings.md if present).
After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.
1---
2name: software-backend
3description: "Builds backend services and APIs with durable defaults. Use when implementing REST, GraphQL, tRPC, or gRPC services with auth, queues, data, or observability."
4compatibility: Portable core. Works on Claude Code and Codex.
5version: "1.1"
6last_validated: 2026-07-11
7---
8
9# Software Backend Engineering
10
11Use this skill for backend service implementation and review: API boundaries, auth, data access, jobs, caching, observability, and production hardening. If the main question is platform selection, system topology, or API-contract design without implementation, hand off early.
12
13## Defaults
14
15When this skill is active, prefer these defaults unless the repo or user says otherwise:
16
17- validate at the boundary and keep types explicit
18- use PostgreSQL plus pooling for relational workloads
19- use structured logs, OpenTelemetry, explicit timeouts, and rate limits
20- make mutations idempotent and background work retry-safe
21- use RFC 9457 Problem Details for machine-readable errors
22
23## Quick Reference
24
25| Need | Default Direction |
26|------|-------------------|
27| Public HTTP API | REST with explicit contracts and timeouts |
28| Internal TS monorepo API | tRPC when end-to-end type safety matters |
29| High-throughput internal RPC | Connect or gRPC |
30| Complex client-shaped reads | GraphQL |
31| Relational data | PostgreSQL with migrations and pooling |
32| Background work | Queue plus idempotent handlers and DLQ policy |
33| Browser auth | OIDC or OAuth plus httpOnly cookies |
34| Service auth | short-lived tokens, workload identity, or signed service credentials |
35| Caching | explicit TTLs and invalidation rules |
36| Observability | correlation IDs, traces, structured logs, saturation metrics |
37
38## When to Use This Skill
39
40- building or reviewing REST, GraphQL, tRPC, Connect, or gRPC services
41- implementing auth, validation, rate limits, caching, queues, or webhook handling
42- modelling schemas and running safe migrations
43- hardening service behavior for retries, timeouts, and observability
44- scaffolding or refactoring a backend with production defaults
45
46## Route Elsewhere
47
48- frontend-only work -> [software-frontend](../software-frontend/SKILL.md)
49- infrastructure provisioning and cluster design -> [ops-devops-platform](../ops-devops-platform/SKILL.md)
50- API contract design without implementation -> [dev-api-design](../dev-api-design/SKILL.md)
51- BaaS platform selection (data/auth layer) -> [software-baas-platforms](../software-baas-platforms/SKILL.md)
52- PaaS hosting selection (compute layer: Vercel, Fly.io, Railway, Render, Cloudflare Workers, Deno Deploy) -> [software-paas-hosting](../software-paas-hosting/SKILL.md)
53- SQL tuning and indexing deep dives -> [data-sql-optimization](../data-sql-optimization/SKILL.md)
54- security reviews and threat modelling -> [software-security-appsec](../software-security-appsec/SKILL.md)
55- broader system architecture -> [software-architecture-design](../software-architecture-design/SKILL.md)
56
57---
58
59## Workflow
60
611. Confirm the real constraint: latency, team skill, runtime, compliance, data model, or delivery speed.
622. Choose the transport and framework based on that constraint, not on trend-chasing.
633. Define the boundary:
64 - request and response contracts
65 - auth and authorization rules
66 - error model
67 - idempotency and rate limiting
684. Define the data path:
69 - schema and migrations
70 - transaction boundaries
71 - pooling and query budgets
72 - cache and invalidation rules
735. Define the async path:
74 - queue semantics
75 - retry ownership
76 - deduplication and DLQ
776. Add operability before calling it complete:
78 - timeouts and cancellation
79 - health checks
80 - structured logs and traces
81 - deploy and rollback expectations
82
83---
84
85## ASCII Flow
86
87```text
88Backend task
89 -> Define endpoint, job, service, or data boundary
90 -> Confirm runtime, framework, persistence, and integration contracts
91 -> Design request validation, auth, errors, and idempotency
92 -> Implement bounded slice with tests and observability
93 -> Check performance, security, and rollout risk
94 -> Verify behavior and document follow-up handoffs
95```
96
97## Technology Selection
98
99Pick based on the strongest operational constraint:
100
101- TypeScript-heavy team -> Fastify, Hono, or NestJS plus Prisma or Drizzle
102- audited SQL and predictable concurrency -> Go with `sqlc/pgx`
103- Python ecosystem or ML adjacency -> FastAPI plus SQLAlchemy
104- enterprise .NET stack -> ASP.NET Core plus EF Core or explicit SQL access
105- memory safety and explicitness -> Rust with Axum plus SQLx
106- edge or serverless first -> lightweight stateless handlers with hard CPU and timeout budgets
107
108Use [software-baas-platforms](../software-baas-platforms/SKILL.md) first when the real requirement is "ship auth, storage, and realtime quickly with less custom service code."
109
110---
111
112## Backend Non-Negotiables
113
114| Category | Rule |
115|----------|------|
116| **API** | Mutating endpoints require idempotency keys where retries are plausible |
117| **API** | List endpoints require explicit pagination (`limit`/`cursor`) and at least one filter |
118| **API** | Errors are structured and machine-readable (RFC 9457 Problem Details) |
119| **API** | Health endpoints separate liveness (`/healthz`) from readiness (`/readyz`) |
120| **Data** | No `SELECT *` on wide or high-volume paths |
121| **Data** | Transactions kept explicit; no implicit ambient transactions |
122| **Data** | New or changed query plans verified with `EXPLAIN ANALYZE` before production |
123| **Data** | ORM convenience layers bypassed on hot paths where auditability matters |
124| **Dependencies** | Every outbound call has an explicit timeout; no framework-default infinite wait |
125| **Dependencies** | Retries owned at exactly one layer (no double-retry across client + service) |
126| **Dependencies** | Cache invalidation rule documented before caching is added |
127| **Dependencies** | Background jobs safe to retry and observable (structured log on start/finish/failure) |
128| **Operations** | Every request carries a correlation ID propagated to all downstream calls |
129| **Operations** | Trace, log, and metric identifiers agree (no split identity) |
130| **Operations** | Slow paths have explicit latency budgets (p99 target, not "fast enough") |
131| **Operations** | Deploy procedure includes rollback step and smoke-check list |
132
133---
134
135## Performance and Reliability Triage
136
137When a service is slow or unstable, debug in this order:
138
139| Step | Check | Signal |
140|------|-------|--------|
141| 1 | Query behavior and N+1s | EXPLAIN output, ORM query log showing repeated identical queries |
142| 2 | Indexes and execution plans | Seq scans on large tables, missing index on FK or filter columns |
143| 3 | Connection pooling and queue depth | Pool wait time > 10ms; idle connections exhausted |
144| 4 | Timeout and cancellation gaps | Requests hanging past deadline; no context propagation through outbound calls |
145| 5 | Caching or read-shaping opportunities | Same query with same result executing > 10x/s; hot read path with no invalidation |
146| 6 | Runtime or tier limits | CPU throttling, memory pressure, rate limit headers from upstream |
147
148Do not add caching before you understand the real bottleneck.
149
150---
151
152## Operational Playbooks
153
154- use [references/operational-playbook.md](references/operational-playbook.md) for full service design and review checklists
155- use [qa-resilience](../qa-resilience/SKILL.md) when retries, deadlines, breakers, or degraded-mode behavior are the main question
156- use [dev-api-design](../dev-api-design/SKILL.md) when the contract itself is the main artifact
157
158## Production Readiness Checklist
159
160Before marking a service production-ready:
161
162- [ ] All mutating endpoints have idempotency keys or safe-retry semantics
163- [ ] Every outbound call has an explicit timeout (no framework-default infinite waits)
164- [ ] Health endpoint distinguishes liveness from readiness
165- [ ] Correlation IDs propagated from inbound request to all downstream calls and logs
166- [ ] DLQ policy defined for every queue consumer (what happens to poison messages)
167- [ ] New query plans verified (`EXPLAIN ANALYZE`) before merge to main
168- [ ] Rollback procedure documented and smoke-test list exists
169
170## Known Traps
171
172- Introducing asynchronous jobs to hide a broken synchronous path instead of fixing the contract, timeout budget, or workload shape.
173- Shipping retries without deadlines, jitter, and idempotency keys, then multiplying load during incidents.
174- Changing API or webhook behavior without a compatibility window, replay plan, or structured error-versioning posture.
175- Adding caches before proving whether the real bottleneck is query shape, pooling, lock contention, or outbound dependency latency.
176- Treating background consumers as “fire and forget” even though poison-message handling, replay semantics, and observability are undefined.
177
178## Common Anti-Patterns
179
180- Letting framework defaults define the service contract, error model, and cancellation semantics.
181- Using one generic repository abstraction for every query, including hot paths that need explicit SQL, batching, or shape control.
182- Mixing request handling, domain logic, external side effects, and persistence concerns in one controller or handler.
183- Relying on eventual retries to clean up non-idempotent side effects.
184- Calling a backend “production ready” before timeouts, readiness checks, trace correlation, and rollback smoke tests exist.
185
186## Navigation
187
188### Core references
189
190- [references/backend-best-practices.md](references/backend-best-practices.md)
191- [references/edge-deployment-guide.md](references/edge-deployment-guide.md)
192- [references/infrastructure-economics.md](references/infrastructure-economics.md)
193- [references/database-patterns.md](references/database-patterns.md)
194- [references/message-queues-background-jobs.md](references/message-queues-background-jobs.md)
195- [references/rpc-and-transport-patterns.md](references/rpc-and-transport-patterns.md)
196- [references/go-best-practices.md](references/go-best-practices.md)
197- [references/rust-best-practices.md](references/rust-best-practices.md)
198- [references/python-best-practices.md](references/python-best-practices.md)
199- [references/nodejs-best-practices.md](references/nodejs-best-practices.md)
200- [references/csharp-best-practices.md](references/csharp-best-practices.md)
201- [data/sources.json](data/sources.json)
202
203### Shared review utilities
204
205- [../software-clean-code-standard/assets/checklists/backend-api-review-checklist.md](../software-clean-code-standard/assets/checklists/backend-api-review-checklist.md)
206- [../software-clean-code-standard/assets/checklists/secure-code-review-checklist.md](../software-clean-code-standard/assets/checklists/secure-code-review-checklist.md)
207- [../software-clean-code-standard/references/auth-utilities.md](../software-clean-code-standard/references/auth-utilities.md)
208- [../software-clean-code-standard/references/error-handling.md](../software-clean-code-standard/references/error-handling.md)
209- [../software-clean-code-standard/references/config-validation.md](../software-clean-code-standard/references/config-validation.md)
210- [../software-clean-code-standard/references/resilience-utilities.md](../software-clean-code-standard/references/resilience-utilities.md)
211- [../software-clean-code-standard/references/logging-utilities.md](../software-clean-code-standard/references/logging-utilities.md)
212- [../software-clean-code-standard/references/testing-utilities.md](../software-clean-code-standard/references/testing-utilities.md)
213- [../software-clean-code-standard/references/observability-utilities.md](../software-clean-code-standard/references/observability-utilities.md)
214
215### Templates
216
217- [assets/nodejs/template-nodejs-prisma-postgres.md](assets/nodejs/template-nodejs-prisma-postgres.md)
218- [assets/nodejs/template-nodejs-fastify-drizzle-postgres.md](assets/nodejs/template-nodejs-fastify-drizzle-postgres.md)
219- [assets/go/template-go-fiber-gorm.md](assets/go/template-go-fiber-gorm.md)
220- [assets/go/template-go-chi-sqlc-pgx.md](assets/go/template-go-chi-sqlc-pgx.md)
221- [assets/rust/template-rust-axum-seaorm.md](assets/rust/template-rust-axum-seaorm.md)
222- [assets/rust/template-rust-axum-sqlx.md](assets/rust/template-rust-axum-sqlx.md)
223- [assets/python/template-python-fastapi-sqlalchemy.md](assets/python/template-python-fastapi-sqlalchemy.md)
224- [assets/csharp/template-csharp-aspnet-efcore.md](assets/csharp/template-csharp-aspnet-efcore.md)
225
226## Related Skills
227
228> **Gate before invoking any foundation below:** Each foundation has a `When to Apply` / `When to Skip` section. If your task matches a skip-condition, route to the foundation it names instead — don't pull in primitives the task doesn't need.
229
230- [software-architecture-design](../software-architecture-design/SKILL.md)
231- [software-security-appsec](../software-security-appsec/SKILL.md)
232- [ops-devops-platform](../ops-devops-platform/SKILL.md)
233- [qa-resilience](../qa-resilience/SKILL.md)
234- [qa-testing-strategy](../qa-testing-strategy/SKILL.md)
235- [foundations-queueing-theory](../foundations-queueing-theory/SKILL.md) — Little's Law, M/M/c, and Kingman's formula for queue sizing in message-queue and rate-limiter design
236- [foundations-distributed-systems](../foundations-distributed-systems/SKILL.md) — CAP, consistency models, quorum sizing, and idempotency contracts for service mesh and RPC patterns
237- [foundations-reliability-theory](../foundations-reliability-theory/SKILL.md) — MTBF/MTTR, availability composition, and error-budget math for SLO-driven backend design
238- [software-code-review](../software-code-review/SKILL.md)
239- [dev-api-design](../dev-api-design/SKILL.md)
240- [data-sql-optimization](../data-sql-optimization/SKILL.md)
241
242## Fact-Checking
243
244- Known bugs, regressions, framework/compiler/runtime footguns, and version-specific crash or workaround guidance must be verified against current primary web sources before being treated as current fact.
245- Verify current runtime versions, support windows, framework capabilities, and cloud-platform constraints before final answers.
246- Prefer official docs and release or support policy pages for version-sensitive recommendations.
247- If web access is unavailable, mark version or support guidance as unverified.
248
249## Learnings Loop
250
251Before applying this skill on a non-trivial task, read `learnings.consolidated.md` in this directory (and `learnings.md` if present).
252
253After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to `learnings.md` via `agents-skills-feedback-loop/scripts/append_learning.py`. Do not modify `SKILL.md` itself.
254
gabrielmoreira