Engineering Culture and Practice for Legal-AI Products
Scope
This pack covers engineering culture, process standards, and architectural guidance for a legal-AI product team. It is opinionated — these are the practices that reduce risk and improve velocity in small teams building products where correctness and trust matter.
Engineering culture principles
Correctness over speed (in production)
Legal professionals rely on the product to be accurate and available. A misrouted skill that generates a wrong jurisdiction answer, or a deployment that causes data loss, can destroy user trust. Apply the bias toward correctness to:
- Production deployments (test-first, staged rollout)
- LLM skill outputs (structured review gates before new skills reach users)
- Schema migrations (always reversible; never destructive without a backup)
Boring technology
Prefer well-understood, battle-tested tools over the newest framework. The product's competitive advantage is in the legal domain and the AI quality, not the infrastructure novelty. Postgres is almost always the right database choice for the first three years. A monolith with well-defined modules beats premature microservices at < 20 engineers.
Shared ownership
On-call rotation is not just for ops engineers — every engineer who ships a feature owns its production behaviour. Post-mortems are blameless and mandatory for P0/P1 incidents.
Code review standards
What a review must address
- Correctness — Does the code do what the ticket says? Are edge cases (empty arrays, null values, unauthenticated requests) handled?
- Security — Is user input validated and sanitised? Are access control checks present and correct? Are secrets stored in environment variables, never in code?
- Observability — Are meaningful logs emitted? Are errors reported to the error tracker? Are metrics incremented?
- Test coverage — Is there a test for the happy path and the key failure modes? Does the PR include a test for any bug it fixes?
- Design system alignment — Does new UI use tokens and components from the library? (See [[wiki-dev-design]])
Review etiquette
- Review within one business day. Blocking a PR for more than 24 hours without comment is a culture debt.
- Mark comments as
blocking:, suggestion:, or nit: so authors know what must change vs what is optional.
- Approve with conditions rather than leaving a PR in requested-changes limbo if only nits remain.
- Leave positive comments for good patterns — normalise noticing what works.
Testing strategy
Layers
| Layer |
What it covers |
Tooling |
| Unit |
Pure functions, utilities, business logic |
Vitest (frontend/API), pytest (Python) |
| Integration |
Database queries, external API clients, skill routing |
Vitest + test containers, pytest + test DB |
| E2E |
Critical user flows end-to-end |
Playwright |
| AI output |
Skill quality, hallucination detection |
Custom eval harness (see [[wiki-data]] for eval logging) |
Rules
- New features ship with unit + integration tests. No exceptions.
- E2E tests cover the five most critical paths: signup, first skill invocation, document upload, billing, logout.
- AI skill tests are not deterministic — run them in eval mode, not in CI assertions. Capture baseline pass rates and alert on regression.
- Target: CI green on
main always. Broken main is a P1 incident.
Deployment pipelines
Environments
local dev → feature branch PR preview → staging → production
- Feature branch preview: auto-deployed on PR open via Vercel/Fly.io preview URL. Allows designer and PM review before merge.
- Staging: mirrors production config. Database is a copy of production with PII masked. All migrations run on staging first.
- Production: deploy after staging green + manual approval (or auto-deploy from
main after passing all checks, depending on team maturity).
Deployment checklist
Database migration safety rules
- Migrations are forward-only in production. Write a separate rollback migration; do not rely on
DOWN in the same file.
- Never add a
NOT NULL column without a default, or drop a column that code still reads, in the same migration.
- Large table migrations (> 1 M rows) require
CONCURRENTLY index builds and offline/online migration strategy.
- Always take a snapshot before running a migration on production.
On-call and incident management
Severity tiers
| Tier |
Definition |
Response time |
| P0 |
Platform down or data loss |
< 15 min response |
| P1 |
Core feature unavailable, billing broken |
< 1 hr response |
| P2 |
Feature degraded, elevated error rate |
< 4 hr response |
| P3 |
Minor bug, cosmetic issue |
Next sprint |
On-call rotation
- Minimum 2-person rotation to avoid single points of failure
- On-call shifts: 1 week; hand-off meeting includes: open P1/P2 issues, deployment notes, any flaky alerts
- Runbook for every recurring alert type; update after each incident
Post-mortem practice
Mandatory for P0 and P1; optional but encouraged for P2. Format:
## Post-mortem: [Incident title] — [Date]
### Summary (2–3 sentences)
### Timeline
### Root cause
### Impact (users affected, duration, data affected)
### What went well
### What went wrong
### Action items (owner, due date)
Post-mortems are blameless. The goal is system improvement, not individual accountability.
Observability
The three pillars
- Logs — structured JSON logs from all services. Key fields:
level, service, trace_id, user_id (hashed/pseudonymised), duration_ms, status_code. Ship to a log aggregator (Datadog, Better Stack, Axiom).
- Metrics — application-level counters and histograms. Key metrics: request rate, error rate, p50/p95/p99 latency, LLM token usage, skill invocation counts.
- Traces — distributed tracing for the AI pipeline is especially important: a single user request may span the router, multiple skill calls, a vector DB query, and an LLM API call. Use OpenTelemetry.
Alerting rules
- Error rate > 1% for any endpoint → P2 alert
- p95 latency > 5 s for any LLM skill → P3 alert; > 30 s → P2
- Any 5xx on
/api/billing → P1 alert
- Audit log ingestion rate drops > 50% vs 7-day baseline → P1 alert (may indicate data loss)
Monolith vs microservices
For teams under ~15 engineers building a legal-AI product: start with a modular monolith. The AI skill layer naturally decomposes into modules (router, skills, connectors, auth, billing) without needing separate deployments. Premature service extraction adds operational overhead with no benefit at this scale.
When to extract a service:
- Independent scaling requirements (e.g. a document-processing worker that must scale independently of the web tier)
- Independent deployment cadence required by organisational boundaries (e.g. a separate team owns the billing service)
- Hard security boundary required (e.g. PII processing must be isolated and audited separately)
Caveats & currency
Engineering tooling evolves rapidly. Vitest superseded Jest for most new projects but check the current Next.js recommended testing setup. OpenTelemetry instrumentation for the AI SDK tier changes as the SDK matures; check the @vercel/otel or opentelemetry-sdk-node docs for the current recommended setup.
Related skills
- [[wiki-data]]
- [[wiki-frontend]]
- [[wiki-dev-design]]
- [[wiki-haqq-product]]
1---2name: wiki-engineering3description: Use when discussing engineering culture, process, or technical decisions for a legal-AI product. Covers code review standards, on-call and incident management, post-mortem practice, deployment pipelines, testing strategy, observability, and the architectural choices specific to legal-AI platforms (monolith vs microservices, multi-tenancy, compliance-grade logging). Reach for this skill when the user asks about eng culture, deployment practices, testing, or team engineering standards.4license: MIT5---67# Engineering Culture and Practice for Legal-AI Products89## Scope1011This pack covers engineering culture, process standards, and architectural guidance for a legal-AI product team. It is opinionated — these are the practices that reduce risk and improve velocity in small teams building products where correctness and trust matter.1213---1415## Engineering culture principles1617### Correctness over speed (in production)1819Legal professionals rely on the product to be accurate and available. A misrouted skill that generates a wrong jurisdiction answer, or a deployment that causes data loss, can destroy user trust. Apply the bias toward correctness to:20- Production deployments (test-first, staged rollout)21- LLM skill outputs (structured review gates before new skills reach users)22- Schema migrations (always reversible; never destructive without a backup)2324### Boring technology2526Prefer well-understood, battle-tested tools over the newest framework. The product's competitive advantage is in the legal domain and the AI quality, not the infrastructure novelty. Postgres is almost always the right database choice for the first three years. A monolith with well-defined modules beats premature microservices at < 20 engineers.2728### Shared ownership2930On-call rotation is not just for ops engineers — every engineer who ships a feature owns its production behaviour. Post-mortems are blameless and mandatory for P0/P1 incidents.3132---3334## Code review standards3536### What a review must address37381. **Correctness** — Does the code do what the ticket says? Are edge cases (empty arrays, null values, unauthenticated requests) handled?392. **Security** — Is user input validated and sanitised? Are access control checks present and correct? Are secrets stored in environment variables, never in code?403. **Observability** — Are meaningful logs emitted? Are errors reported to the error tracker? Are metrics incremented?414. **Test coverage** — Is there a test for the happy path and the key failure modes? Does the PR include a test for any bug it fixes?425. **Design system alignment** — Does new UI use tokens and components from the library? (See [[wiki-dev-design]])4344### Review etiquette4546- Review within one business day. Blocking a PR for more than 24 hours without comment is a culture debt.47- Mark comments as `blocking:`, `suggestion:`, or `nit:` so authors know what must change vs what is optional.48- Approve with conditions rather than leaving a PR in requested-changes limbo if only nits remain.49- Leave positive comments for good patterns — normalise noticing what works.5051---5253## Testing strategy5455### Layers5657| Layer | What it covers | Tooling |58|---|---|---|59| Unit | Pure functions, utilities, business logic | Vitest (frontend/API), pytest (Python) |60| Integration | Database queries, external API clients, skill routing | Vitest + test containers, pytest + test DB |61| E2E | Critical user flows end-to-end | Playwright |62| AI output | Skill quality, hallucination detection | Custom eval harness (see [[wiki-data]] for eval logging) |6364### Rules6566- New features ship with unit + integration tests. No exceptions.67- E2E tests cover the five most critical paths: signup, first skill invocation, document upload, billing, logout.68- AI skill tests are not deterministic — run them in eval mode, not in CI assertions. Capture baseline pass rates and alert on regression.69- Target: CI green on `main` always. Broken `main` is a P1 incident.7071---7273## Deployment pipelines7475### Environments7677```78local dev → feature branch PR preview → staging → production79```8081- **Feature branch preview**: auto-deployed on PR open via Vercel/Fly.io preview URL. Allows designer and PM review before merge.82- **Staging**: mirrors production config. Database is a copy of production with PII masked. All migrations run on staging first.83- **Production**: deploy after staging green + manual approval (or auto-deploy from `main` after passing all checks, depending on team maturity).8485### Deployment checklist8687- [ ] All CI checks green (lint, test, type-check, build)88- [ ] Database migration tested on staging with rollback plan documented89- [ ] Feature flags set correctly (new feature behind a flag for staged rollout)90- [ ] Runbook updated if this changes operational behaviour91- [ ] Monitoring alert thresholds reviewed (new endpoints, new error types)9293### Database migration safety rules94951. Migrations are forward-only in production. Write a separate rollback migration; do not rely on `DOWN` in the same file.962. Never add a `NOT NULL` column without a default, or drop a column that code still reads, in the same migration.973. Large table migrations (> 1 M rows) require `CONCURRENTLY` index builds and offline/online migration strategy.984. Always take a snapshot before running a migration on production.99100---101102## On-call and incident management103104### Severity tiers105106| Tier | Definition | Response time |107|---|---|---|108| P0 | Platform down or data loss | < 15 min response |109| P1 | Core feature unavailable, billing broken | < 1 hr response |110| P2 | Feature degraded, elevated error rate | < 4 hr response |111| P3 | Minor bug, cosmetic issue | Next sprint |112113### On-call rotation114115- Minimum 2-person rotation to avoid single points of failure116- On-call shifts: 1 week; hand-off meeting includes: open P1/P2 issues, deployment notes, any flaky alerts117- Runbook for every recurring alert type; update after each incident118119### Post-mortem practice120121Mandatory for P0 and P1; optional but encouraged for P2. Format:122123```124## Post-mortem: [Incident title] — [Date]125126### Summary (2–3 sentences)127### Timeline128### Root cause129### Impact (users affected, duration, data affected)130### What went well131### What went wrong132### Action items (owner, due date)133```134135Post-mortems are blameless. The goal is system improvement, not individual accountability.136137---138139## Observability140141### The three pillars142143- **Logs** — structured JSON logs from all services. Key fields: `level`, `service`, `trace_id`, `user_id` (hashed/pseudonymised), `duration_ms`, `status_code`. Ship to a log aggregator (Datadog, Better Stack, Axiom).144- **Metrics** — application-level counters and histograms. Key metrics: request rate, error rate, p50/p95/p99 latency, LLM token usage, skill invocation counts.145- **Traces** — distributed tracing for the AI pipeline is especially important: a single user request may span the router, multiple skill calls, a vector DB query, and an LLM API call. Use OpenTelemetry.146147### Alerting rules148149- Error rate > 1% for any endpoint → P2 alert150- p95 latency > 5 s for any LLM skill → P3 alert; > 30 s → P2151- Any 5xx on `/api/billing` → P1 alert152- Audit log ingestion rate drops > 50% vs 7-day baseline → P1 alert (may indicate data loss)153154---155156## Monolith vs microservices157158For teams under ~15 engineers building a legal-AI product: **start with a modular monolith**. The AI skill layer naturally decomposes into modules (router, skills, connectors, auth, billing) without needing separate deployments. Premature service extraction adds operational overhead with no benefit at this scale.159160When to extract a service:161- Independent scaling requirements (e.g. a document-processing worker that must scale independently of the web tier)162- Independent deployment cadence required by organisational boundaries (e.g. a separate team owns the billing service)163- Hard security boundary required (e.g. PII processing must be isolated and audited separately)164165---166167## Caveats & currency168169Engineering tooling evolves rapidly. Vitest superseded Jest for most new projects but check the current Next.js recommended testing setup. OpenTelemetry instrumentation for the AI SDK tier changes as the SDK matures; check the `@vercel/otel` or `opentelemetry-sdk-node` docs for the current recommended setup.170171---172173## Related skills174175- [[wiki-data]]176- [[wiki-frontend]]177- [[wiki-dev-design]]178- [[wiki-haqq-product]]
sboghossian