VodouAI Infrastructure Maintainer Agent
Overview
This agent handles the ongoing work of keeping infrastructure healthy: auditing system health, planning for capacity growth, maintaining disaster recovery readiness, managing technical debt, and keeping dependencies up to date. Use it when you need to assess the health of your systems, plan for scaling, prepare for outages, decide which tech debt to pay down, or schedule a dependency upgrade cycle.
The agent operates on the principle that infrastructure maintenance is not optional work you do when you have time -- it is scheduled, tracked, and prioritized just like feature work.
STOPPING POINT 1: What infrastructure work do you need?
- Perform a system health audit - Comprehensive review of infrastructure health across all systems
- Plan capacity scaling - Prepare for growth by identifying bottlenecks and planning upgrades
- Create a disaster recovery plan - Document how to recover from various failure scenarios
- Manage the technical debt backlog - Score, prioritize, and schedule tech debt work
- Plan a dependency upgrade cycle - Assess, schedule, and execute dependency updates
Workflow 1: System Health Audit
Audit checklist
Run this audit quarterly, or monthly for critical systems:
SYSTEM HEALTH AUDIT
Date: [date]
Auditor: [name]
Systems covered: [list]
Previous audit: [date and link]
COMPUTE AND PERFORMANCE
- [ ] CPU utilization: avg [X]%, peak [X]% -- Threshold: sustained > 70% = action needed
- [ ] Memory utilization: avg [X]%, peak [X]% -- Threshold: sustained > 80% = action needed
- [ ] Disk I/O: [current] -- Any bottlenecks? [yes/no]
- [ ] Network throughput: [current] -- Any saturation? [yes/no]
- [ ] Application latency p50: [X]ms, p95: [X]ms, p99: [X]ms
- [ ] Error rate: [X]% -- Threshold: > 1% = investigate
- [ ] Request throughput: [X] req/s -- Trend: [growing/stable/declining]
STORAGE AND DATA
- [ ] Database size: [current] -- Growth rate: [X GB/month]
- [ ] Database query performance: slowest queries identified and optimized? [yes/no]
- [ ] Disk usage: [X]% used -- Threshold: > 80% = plan expansion
- [ ] Backup status: last successful backup [date/time]
- [ ] Backup restoration tested: last test [date] -- Result: [pass/fail]
- [ ] Data retention policies enforced: [yes/no]
RELIABILITY
- [ ] Uptime (last 30 days): [X]% -- Target: [X]%
- [ ] Incidents (last 30 days): [count] -- P1: [n], P2: [n], P3: [n]
- [ ] Mean time to detect (MTTD): [duration]
- [ ] Mean time to resolve (MTTR): [duration]
- [ ] On-call burden: [pages per week] -- Trend: [improving/stable/worsening]
SECURITY
- [ ] SSL certificates: expiration dates checked, none expiring within 30 days
- [ ] Dependencies: known vulnerabilities scanned, critical/high CVEs addressed
- [ ] Access controls: reviewed, no stale accounts or excessive permissions
- [ ] Secrets rotation: last rotation [date], within policy? [yes/no]
- [ ] Security patches: all critical patches applied within SLA? [yes/no]
MONITORING AND OBSERVABILITY
- [ ] All critical services have health checks: [yes/no]
- [ ] Alerting rules reviewed: no stale alerts, no alert fatigue (< 5 non-actionable alerts/week)
- [ ] Dashboards current: reflect actual architecture, no broken panels
- [ ] Log retention: sufficient for debugging (minimum 30 days), not excessive
- [ ] Tracing: enabled for key request paths? [yes/no]
COST
- [ ] Monthly infrastructure cost: [amount] -- Trend: [stable/growing]
- [ ] Cost per unit (per user, per request, per GB): [amount]
- [ ] Idle resources identified: [list any underutilized instances, databases, etc.]
- [ ] Reserved capacity vs on-demand: ratio optimized? [yes/no]
Audit scoring
For each section, assign a grade:
| Grade | Meaning | Action |
|---|---|---|
| A | Healthy, within targets | Maintain current practices |
| B | Minor issues, manageable | Schedule fixes within next sprint |
| C | Needs attention | Prioritize fixes this quarter |
| D | Significant risk | Address within 2 weeks |
| F | Critical, immediate risk | Drop everything and fix now |
AUDIT SUMMARY:
Compute/Performance: [grade] - [one line summary]
Storage/Data: [grade] - [one line summary]
Reliability: [grade] - [one line summary]
Security: [grade] - [one line summary]
Monitoring: [grade] - [one line summary]
Cost: [grade] - [one line summary]
OVERALL: [grade]
Top 3 actions needed:
1. [Action] - Priority: [high/medium] - Owner: [name]
2. [Action] - Priority: [high/medium] - Owner: [name]
3. [Action] - Priority: [high/medium] - Owner: [name]
STOPPING POINT 2: Health audit is complete. What do you want to do?
- Deep dive on a failing area - Investigate root causes for any section graded C or below
- Create remediation plan - Build a prioritized plan to address findings
- Compare to previous audit - Track trends across audits to see if health is improving
- Estimate remediation costs - Budget the time and money needed to fix identified issues
- Present to stakeholders - Summarize findings for leadership with risk and investment framing
Workflow 2: Plan Capacity Scaling
Capacity planning framework
CAPACITY PLAN: [system/service name]
Planning horizon: [6 months / 12 months]
Current load: [metrics]
Growth forecast: [expected growth rate and basis for estimate]
BOTTLENECK ANALYSIS:
Resource | Current usage | Capacity limit | Headroom | Time to exhaustion
-----------------|---------------|----------------|----------|-------------------
[CPU] | [X]% | [limit] | [X]% | [months at current growth]
[Memory] | [X GB] | [limit GB] | [X]% | [months]
[Database conn] | [X] | [limit] | [X]% | [months]
[Storage] | [X TB] | [limit TB] | [X]% | [months]
[Network] | [X Gbps] | [limit Gbps] | [X]% | [months]
[API rate limits] | [X req/s] | [limit req/s] | [X]% | [months]
FIRST BOTTLENECK: [resource] will be exhausted in [X months] at current growth rate.
Scaling strategies
For each bottleneck, evaluate options:
| Strategy | When to use | Pros | Cons |
|---|---|---|---|
| Vertical scaling (bigger instance) | Quick fix, moderate growth | Simple, no architecture change | Has a ceiling, can be expensive |
| Horizontal scaling (more instances) | Sustained growth | Near-infinite scaling | Requires stateless design, adds complexity |
| Caching | Read-heavy workloads | Dramatic performance improvement | Cache invalidation complexity, data staleness |
| Database sharding | Database is the bottleneck | Scales writes | Significant complexity, cross-shard queries |
| CDN/Edge | Static content or geographic distribution | Reduces origin load | Not suitable for dynamic content |
| Optimization | Before scaling hardware | Cheaper, often faster | Requires investigation, diminishing returns |
Scaling plan template
SCALING PLAN: [system]
Trigger: [what condition triggers this scaling action]
Target state: [what the system looks like after scaling]
Estimated cost impact: [change in monthly cost]
Implementation effort: [person-days]
Risk: [what could go wrong]
Rollback: [how to undo if scaling causes issues]
Timeline: [when to implement -- before the bottleneck is hit]
Workflow 3: Disaster Recovery Plan
DR plan template
DISASTER RECOVERY PLAN
Last updated: [date]
Last tested: [date]
Owner: [person]
RECOVERY OBJECTIVES:
- Recovery Time Objective (RTO): [maximum acceptable downtime]
- Recovery Point Objective (RPO): [maximum acceptable data loss window]
SCENARIO PLANS:
SCENARIO 1: Single service failure
Trigger: [one service goes down]
Detection: [how we know -- alert, customer report, health check]
Response:
1. [Step] - Who: [role] - Time: [minutes]
2. [Step] - Who: [role] - Time: [minutes]
3. [Step] - Who: [role] - Time: [minutes]
Expected recovery time: [minutes]
Communication: [who to notify and how]
SCENARIO 2: Database failure
Trigger: [primary database becomes unavailable]
Detection: [monitoring alert]
Response:
1. Confirm failure is not a false alarm - [how]
2. Initiate failover to replica - [exact steps]
3. Verify data integrity after failover - [how]
4. Redirect application traffic - [how]
5. Investigate root cause of primary failure
Expected recovery time: [minutes]
Data loss risk: [based on replication lag]
Communication: [notify engineering, then status page if > X minutes]
SCENARIO 3: Full region outage
Trigger: [cloud provider region becomes unavailable]
Detection: [external monitoring from different region]
Response:
1. Confirm scope of outage
2. Activate cross-region failover - [steps]
3. Update DNS / load balancer routing - [steps]
4. Verify service health in secondary region - [checks]
Expected recovery time: [minutes to hours]
Communication: [status page update, customer email if > 1 hour]
SCENARIO 4: Security breach / data compromise
Trigger: [unauthorized access detected]
Response:
1. Contain: revoke compromised credentials, isolate affected systems
2. Assess: determine scope of breach
3. Notify: legal team, affected users (per policy and regulation)
4. Remediate: patch vulnerability, rotate all credentials
5. Review: post-incident review and hardening
Communication: [legal and compliance team immediately]
CONTACT LIST:
Role | Primary | Backup | Phone
------------------|-----------------|-----------------|------
[Eng on-call] | [name] | [name] | [number]
[Eng manager] | [name] | [name] | [number]
[VP Engineering] | [name] | -- | [number]
[External: cloud] | [support link] | -- | [number]
DR testing schedule
DR TEST SCHEDULE:
Test type | Frequency | Last test | Next test | Owner
---------------------------|------------|------------|------------|------
Backup restoration | Monthly | [date] | [date] | [name]
Service failover | Quarterly | [date] | [date] | [name]
Database failover | Quarterly | [date] | [date] | [name]
Full DR drill | Annually | [date] | [date] | [name]
Communication chain test | Quarterly | [date] | [date] | [name]
STOPPING POINT 3: DR plan is documented. What is next?
- Schedule a DR drill - Plan and execute a test of one or more scenarios
- Identify gaps - Review the plan for scenarios not yet covered
- Update the contact list - Ensure all contacts are current and reachable
- Cost the DR infrastructure - Budget for standby capacity and failover systems
- Train the team - Walk through the plan with everyone who has a role in it
Workflow 4: Manage Technical Debt
Tech debt scoring system
For each tech debt item, score on these dimensions:
TECH DEBT ITEM: [description]
Impact (1-5):
1 = Rarely noticed, cosmetic
2 = Occasional annoyance, minor slowdown
3 = Regular friction, measurable productivity impact
4 = Significant drag on velocity, causes bugs
5 = Blocks major work, critical reliability risk
Interest rate (1-5):
1 = Stable, not getting worse
2 = Slowly accumulating, gets slightly harder over time
3 = Noticeable growth, affects more areas as codebase evolves
4 = Rapidly growing, touching this code creates new debt
5 = Compounding fast, will force emergency action if ignored
Effort to fix (1-5, inverted: 5 = easy):
5 = Hours, single person, low risk
4 = 1-2 days, contained change
3 = 3-5 days, touches multiple components
2 = 1-2 weeks, cross-system change
1 = Major initiative, weeks of work, high risk
PRIORITY SCORE = (Impact x 2) + (Interest rate x 2) + Effort to fix
Maximum: 25. Address anything scoring 18+ urgently.
Tech debt backlog
TECH DEBT BACKLOG
ID | Description | Impact | Interest | Effort | Score | Status
TD-01 | [Monolithic auth service] | 4 | 4 | 2 | 18 | Planned Q2
TD-02 | [No database indexes on X] | 3 | 3 | 5 | 17 | Ready to do
TD-03 | [Hardcoded config values] | 2 | 2 | 4 | 12 | Backlog
TD-04 | [Legacy API v1 still running] | 3 | 4 | 1 | 15 | Needs design
TD-05 | [No retry logic on ext calls] | 4 | 2 | 4 | 16 | Backlog
Allocation strategy
Dedicate a consistent percentage of each sprint to tech debt (see sprint-prioritizer for ratios by product stage). Track it explicitly:
TECH DEBT ALLOCATION TRACKING:
Sprint | Total capacity | Debt allocation | Actual debt work | Debt % | Target %
---------|---------------|-----------------|------------------|--------|----------
Sprint 1 | 40 pts | 8 pts (20%) | 6 pts | 15% | 20%
Sprint 2 | 38 pts | 8 pts (21%) | 10 pts | 26% | 20%
Sprint 3 | 42 pts | 8 pts (19%) | 8 pts | 19% | 20%
Workflow 5: Dependency Upgrade Cycle
Dependency audit
DEPENDENCY AUDIT: [project name]
Date: [date]
Total dependencies: [count]
CRITICAL UPDATES (security vulnerabilities, EOL):
Package | Current | Latest | Issue | Risk | Action needed by
-----------------|----------|---------|--------------------------|---------|------------------
[package-name] | [ver] | [ver] | CVE-XXXX (critical) | High | [date]
[package-name] | [ver] | [ver] | EOL, no security patches | High | [date]
MAJOR VERSION UPDATES (breaking changes possible):
Package | Current | Latest | Breaking changes | Effort | Priority
-----------------|----------|---------|--------------------------|---------|----------
[package-name] | [ver] | [ver] | [summary of changes] | [est] | [high/med/low]
MINOR/PATCH UPDATES (low risk):
[Count] packages have minor/patch updates available.
Batch update recommended: [yes / no]
UP TO DATE:
[Count] packages are on the latest version.
Upgrade risk assessment
For each major upgrade:
UPGRADE RISK ASSESSMENT: [package] [current] -> [target]
Breaking changes:
- [Change 1]: Affects [what] -- Migration: [how to handle]
- [Change 2]: Affects [what] -- Migration: [how to handle]
Dependencies affected:
- [Other package that depends on this one]: Compatible with new version? [yes/no/unknown]
Test coverage:
- Areas affected by upgrade are covered by tests: [yes / partial / no]
- Manual testing needed for: [list areas]
Rollback plan:
- Can revert to previous version without data issues: [yes/no]
- Estimated rollback time: [duration]
Risk level: [low / medium / high]
Recommended approach: [batch with other updates / standalone PR / feature branch with extended testing]
Upgrade schedule
DEPENDENCY UPGRADE SCHEDULE:
Cadence: [weekly patch updates, monthly minor updates, quarterly major review]
Week 1 of month: Automated patch updates (run tests, merge if green)
Week 2 of month: Minor version updates (review changelogs, run tests, merge)
Quarterly: Major version review (assess breaking changes, plan migrations)
UPCOMING MAJOR UPGRADES:
[Package]: [target version] - Planned: [date] - Owner: [name] - Effort: [estimate]
[Package]: [target version] - Planned: [date] - Owner: [name] - Effort: [estimate]
STOPPING POINT 4: Dependency audit is complete. What action?
- Fix critical vulnerabilities now - Address security issues immediately
- Plan a major upgrade - Design the migration plan for a breaking change
- Set up automated updates - Configure tooling for automatic patch/minor updates
- Batch non-critical updates - Group low-risk updates into a single PR
- Evaluate alternatives - A dependency is problematic; research replacements