tools-only DSPy Production Readiness Checklist
Use this checklist to ensure your DSPy application is production-ready.
Code Quality
Module Design
- Modules follow single responsibility principle
- Clear separation between retrieval, generation, and orchestration
- Reusable components extracted into libraries
- Type hints used throughout codebase
- Docstrings document all public APIs
Error Handling
- All external calls wrapped in try/except
- Graceful degradation implemented for failures
- Fallback responses defined for critical paths
- Circuit breakers implemented for external services
- Retry logic with exponential backoff
Validation
- Input validation for all user-facing functions
- Output validation with dspy.Assert/Suggest
- Schema validation for structured outputs
- Rate limiting implemented
- Request size limits enforced
Testing Coverage
Unit Tests
- All modules have unit tests (>80% coverage)
- LM calls mocked in unit tests
- Edge cases tested (empty inputs, malformed data)
- Error paths tested
- Fixtures created for common test data
Integration Tests
- End-to-end workflows tested
- External service integrations tested
- Database interactions tested
- API endpoints tested
- Performance requirements validated
Evaluation
- Baseline metrics established
- Test dataset representative of production
- Multiple evaluation metrics defined
- Evaluation integrated into CI/CD
- Quality thresholds enforced (e.g., >80% accuracy)
Performance
Latency
- P50 latency < 500ms
- P95 latency < 2000ms
- P99 latency < 5000ms
- Timeout configured (e.g., 10s max)
- Load testing performed (target: 100 req/s)
Optimization
- Models compiled offline (not in production)
- Compiled prompts version controlled
- Caching implemented (memory + disk)
- Database queries optimized
- Batch processing where applicable
Resource Usage
- Memory usage profiled (<2GB per instance)
- CPU usage monitored (<70% avg)
- GPU memory managed (if applicable)
- Connection pools configured
- Resource cleanup on shutdown
Monitoring & Observability
Metrics
- Request rate tracked (requests/sec)
- Latency tracked (p50, p95, p99)
- Error rate tracked (errors/total)
- Success rate tracked
- Cost per request tracked
Logging
- Structured logging implemented (JSON)
- Log levels configured (INFO in prod, DEBUG in dev)
- Sensitive data redacted from logs
- Request IDs for tracing
- Error stack traces captured
Alerting
- Alerts configured for high error rates (>5%)
- Alerts for high latency (p95 > 3s)
- Alerts for service unavailability
- On-call rotation defined
- Runbook created for common issues
Dashboards
- Real-time metrics dashboard created
- Historical trends tracked
- Cost tracking dashboard
- User-facing status page
Security
Authentication & Authorization
- API keys rotated regularly
- Environment variables for secrets
- No secrets in code or git
- Rate limiting per user/API key
- HTTPS enforced for all endpoints
Data Protection
- User data encrypted at rest
- User data encrypted in transit
- PII handling compliant (GDPR, CCPA)
- Data retention policies defined
- Audit logs for data access
Dependencies
- Dependencies scanned for vulnerabilities
- Dependency versions pinned
- Regular security updates scheduled
- License compliance verified
- Supply chain security reviewed
Deployment
Infrastructure
- Infrastructure as code (Terraform, Pulumi)
- Auto-scaling configured
- Health checks implemented
- Graceful shutdown handling
- Zero-downtime deployment strategy
Configuration
- Environment-specific configs (dev, staging, prod)
- Feature flags for gradual rollout
- Configuration changes don't require redeployment
- Secrets managed with vault/secrets manager
- Configuration validated on startup
Rollback Strategy
- Previous version deployable within 5 minutes
- Database migrations reversible
- Feature flags for instant disable
- Blue-green deployment or canary releases
- Rollback tested regularly
Documentation
Code Documentation
- README with setup instructions
- API documentation (OpenAPI/Swagger)
- Architecture diagrams
- Deployment guide
- Troubleshooting guide
Operations
- Runbook for common incidents
- Disaster recovery plan
- Backup and restore procedures
- Monitoring and alerting documentation
- On-call playbook
User Documentation
- User guide created
- API examples provided
- Rate limits documented
- Error codes explained
- Support contact information
Compliance & Legal
Regulatory
- GDPR compliance (if applicable)
- CCPA compliance (if applicable)
- HIPAA compliance (if handling health data)
- Data residency requirements met
- Terms of service defined
Model Governance
- Model card created (performance, limitations)
- Bias evaluation performed
- Fairness metrics tracked
- Model versioning implemented
- Model approval process defined
Final Checks
Pre-Launch
- Load testing completed at 2x expected traffic
- Chaos engineering tests passed
- Security audit completed
- Legal review completed
- Go-live checklist reviewed
Launch Day
- All team members available
- Monitoring dashboards visible
- Rollback plan ready
- Communication plan active
- Support team briefed
Post-Launch
- Monitor metrics for 24 hours
- Review incident logs
- Gather user feedback
- Schedule retrospective
- Update documentation with learnings
Scoring
Total items: ~120
Production Ready: 95%+ checked (114+ items) Nearly Ready: 85-94% checked (102-113 items) Needs Work: 70-84% checked (84-101 items) Not Ready: <70% checked (<84 items)
Version: 1.0 Last Updated: 2025-10-30