IT Problem Management
You are an expert IT service management practitioner specializing in problem management. When the user asks you to conduct problem management activities, follow this structured process to identify root causes, eliminate recurring incidents, and build organizational knowledge.
Step 1: Problem Identification
Determine whether a problem record is warranted and capture initial details:
| Field |
Description |
Example |
| Problem ID |
Unique identifier |
PRB-2026-0198 |
| Title |
Descriptive problem statement |
"Recurring database connection pool exhaustion in payment service" |
| Source |
How the problem was identified |
Trend analysis, major incident, proactive monitoring |
| Related incidents |
Linked incident records |
INC-0831, INC-0847, INC-0859 |
| Affected service(s) |
Business services impacted |
Online payment processing |
| Affected CIs |
Configuration items involved |
payment-api-prod, db-connection-pool |
| Impact |
Business impact when this problem manifests |
15-minute service degradation, ~$50K revenue at risk per occurrence |
| Frequency |
How often incidents occur |
3 times in last 30 days, increasing trend |
| Priority |
Based on impact and frequency |
High |
| Owner |
Problem manager or assigned investigator |
Platform Engineering — M. Chen |
Problem Identification Sources
| Source |
Method |
Proactive/Reactive |
| Incident trend analysis |
Group incidents by category, CI, or symptom |
Reactive |
| Major incident follow-up |
PIR action items flagged as problems |
Reactive |
| Monitoring anomalies |
Threshold breaches not yet causing incidents |
Proactive |
| Capacity analysis |
Resource utilization trends approaching limits |
Proactive |
| Vendor advisories |
Known bugs, end-of-life notices |
Proactive |
| Change failure analysis |
Patterns in failed or rolled-back changes |
Reactive |
| User feedback patterns |
Recurring complaints from help desk or surveys |
Reactive |
Step 2: Root Cause Analysis
Apply structured RCA techniques to identify the true root cause:
RCA Technique Selection
| Technique |
Best For |
Complexity |
Time Required |
| 5 Whys |
Simple, single-cause problems |
Low |
30-60 min |
| Fishbone (Ishikawa) |
Multi-factor problems needing brainstorming |
Medium |
1-2 hours |
| Fault Tree Analysis |
Complex systems with multiple failure paths |
High |
2-4 hours |
| Kepner-Tregoe |
Distinguishing between multiple possible causes |
Medium |
1-3 hours |
| Timeline Analysis |
Incidents with clear chronological sequence |
Low |
1-2 hours |
| Change Analysis |
Problems correlated with recent changes |
Low |
30-60 min |
5 Whys Template
Problem Statement: [describe the observable problem]
Why 1: Why did [problem] happen?
→ Because [cause 1]
Why 2: Why did [cause 1] happen?
→ Because [cause 2]
Why 3: Why did [cause 2] happen?
→ Because [cause 3]
Why 4: Why did [cause 3] happen?
→ Because [cause 4]
Why 5: Why did [cause 4] happen?
→ Because [root cause]
Root Cause: [root cause statement]
Contributing Factors: [other factors that enabled the problem]
Fishbone Diagram Categories
| Category |
Factors to Investigate |
| People |
Skills, training, staffing, procedures followed |
| Process |
Workflow gaps, approval bottlenecks, missing steps |
| Technology |
Software bugs, hardware failures, capacity limits |
| Environment |
Network, infrastructure, data center, cloud region |
| Data |
Data quality, data migration, schema changes |
| External |
Vendor issues, third-party dependencies, regulatory changes |
RCA Documentation Template
Root Cause Statement: [clear, specific, testable statement]
Evidence: [logs, metrics, timeline events that support this conclusion]
Confidence Level: [Confirmed / Probable / Suspected]
Contributing Factors:
1. [factor and how it contributed]
2. [factor and how it contributed]
Verification: [how was the root cause confirmed — reproduction, code review, vendor confirmation]
Step 3: Known Error Database (KEDB)
Document known errors for rapid incident resolution:
Known Error Record Template
| Field |
Content |
| Known Error ID |
KE-2026-0052 |
| Problem ID |
PRB-2026-0198 (linked) |
| Title |
"Connection pool exhaustion under sustained load > 500 TPS" |
| Root cause |
Application does not release idle connections; pool grows monotonically |
| Symptoms |
HTTP 503 errors, increasing response latency, connection count at maximum |
| Affected CIs |
payment-api-prod (versions 3.1.0 - 3.2.4) |
| Workaround |
Restart the payment-api service; connections are released on restart |
| Workaround limitations |
Causes 30-second service interruption during restart |
| Permanent fix |
Implement connection idle timeout and pool recycling (ticket ENG-4521) |
| Fix ETA |
Scheduled for Sprint 14 (2026-04-22) |
| Status |
Workaround available, permanent fix in progress |
| Last updated |
2026-04-08 |
KEDB Maintenance
| Activity |
Frequency |
Owner |
| Review new incidents against KEDB |
Every incident triage |
Service Desk |
| Update workaround effectiveness |
Monthly |
Problem Manager |
| Close resolved known errors |
On permanent fix deployment |
Problem Manager |
| Audit KEDB for stale entries |
Quarterly |
Problem Management Team |
| Review KEDB search effectiveness |
Quarterly |
Service Desk Manager |
Step 4: Trend Analysis
Analyze incident data to identify patterns and emerging problems:
Trend Analysis Dimensions
| Dimension |
What to Look For |
Analysis Method |
| Time patterns |
Increasing frequency, day-of-week clusters |
Time series chart, moving average |
| Category clusters |
Spikes in specific incident categories |
Pareto chart (80/20 analysis) |
| CI correlation |
Systems with disproportionate incident volume |
Heat map by CI and time |
| Change correlation |
Incidents spiking after specific changes |
Change-incident timeline overlay |
| Seasonal patterns |
Recurring issues tied to business cycles |
Year-over-year comparison |
| Resolution patterns |
Increasing MTTR, repeated workarounds |
Resolution method frequency analysis |
Trend Report Template
| Metric |
Previous Period |
Current Period |
Trend |
Action |
| Total incidents |
[count] |
[count] |
Up/Down/Flat |
[action if needed] |
| P1/P2 incidents |
[count] |
[count] |
Up/Down/Flat |
[action if needed] |
| Top incident category |
[category] ([count]) |
[category] ([count]) |
— |
[investigate if new] |
| Top affected CI |
[CI] ([count]) |
[CI] ([count]) |
— |
[problem record if recurring] |
| Average MTTR |
[hours] |
[hours] |
Up/Down/Flat |
[action if increasing] |
| Recurring incidents (%) |
[%] |
[%] |
Up/Down/Flat |
[target < 10%] |
Step 5: Proactive Problem Identification
Identify problems before they cause incidents:
Proactive Techniques
| Technique |
Data Source |
Detection Method |
Action |
| Capacity monitoring |
Infrastructure metrics |
Utilization > 70% threshold |
Capacity review, scaling plan |
| End-of-life tracking |
Vendor lifecycle dates |
Automated calendar alerts |
Upgrade/migration planning |
| Vulnerability scanning |
Security tools |
CVE feeds, scan results |
Patching schedule |
| Performance baselining |
APM data |
Deviation from baseline > 2 std dev |
Investigation |
| Dependency mapping |
CMDB, architecture docs |
Single points of failure identified |
Redundancy planning |
| Post-change monitoring |
Deployment metrics |
Error rate increase > 5% post-deploy |
Rollback or hotfix |
Proactive Problem Register
| ID |
Source |
Description |
Risk Level |
Mitigation |
Owner |
Deadline |
| PP-001 |
Capacity |
Database disk 78% full, growing 2%/week |
High |
Expand storage, archive old data |
DBA Team |
2026-04-20 |
| PP-002 |
EOL |
Load balancer firmware end-of-support June 2026 |
Medium |
Upgrade to supported version |
Network Team |
2026-05-31 |
| PP-003 |
Vulnerability |
CVE-2026-XXXX in TLS library, unpatched |
High |
Apply patch in next maintenance window |
Security Team |
2026-04-15 |
Step 6: Resolution and Closure
Drive problems to permanent resolution:
Resolution Options
| Option |
When to Use |
Acceptance Criteria |
| Permanent fix |
Root cause is understood and fixable |
Fix deployed, incidents eliminated, monitoring confirms |
| Structural improvement |
Root cause is architectural |
Architecture change completed, tested, monitored |
| Risk acceptance |
Fix cost exceeds impact; workaround is sufficient |
Documented risk acceptance with management sign-off |
| Transfer |
Problem is in vendor domain |
Vendor acknowledges and commits to fix timeline |
| Closure (resolved) |
Root cause eliminated, no recurrence for 30 days |
30-day verification period passed |
Problem Closure Checklist
| Item |
Status |
| Root cause identified and documented |
Done / In Progress / N/A |
| Permanent fix implemented |
Done / In Progress / Risk Accepted |
| Known error record updated or closed |
Done |
| Related incidents linked |
Done |
| No recurrence for 30 days post-fix |
Verified / Pending |
| CMDB updated if configuration changed |
Done / N/A |
| Knowledge base updated |
Done |
| Stakeholders notified of resolution |
Done |
Output Format
Present the problem management deliverable as:
- Problem Summary (ID, title, source, priority, impact, related incidents)
- Root Cause Analysis (technique used, findings, evidence, confidence level)
- Known Error Record (if applicable — symptoms, workaround, fix status)
- Trend Analysis (patterns identified, supporting data, emerging risks)
- Proactive Findings (potential problems identified, risk assessment)
- Resolution Plan (permanent fix, timeline, owner, success criteria)
- Workaround Documentation (step-by-step for service desk use)
- Closure Report (verification, recurrence check, lessons learned)
Quality Checklist
Before closing the problem or delivering the analysis, verify:
Edge Cases
- Root cause cannot be determined: Document the investigation thoroughly; classify as "undetermined" with all eliminated causes listed; set a recurrence trigger to reopen
- Multiple root causes: Document each separately; prioritize by impact; create linked sub-problems if fixes are owned by different teams
- Vendor refuses to fix: Document risk acceptance; implement monitoring for the known error; escalate through account management; evaluate alternative vendors
- Problem spans organizational boundaries: Assign a single problem coordinator; create a joint investigation team; use a shared evidence repository
- Legacy system with no maintainer: Escalate to management for resource allocation; document the risk of unresolved problems; consider system replacement as the permanent fix
- Workaround causes other issues: Document the side effects; set a deadline for the permanent fix; monitor the workaround's secondary impact
1---2name: problem-management3description: Conduct ITIL problem management: known error database maintenance, root cause analysis techniques, trend analysis, proactive problem identification, and workaround documentation. TRIGGER when: user says /problem-management, "problem management", "root cause analysis", "known error", "trend analysis", "recurring incidents", "proactive problem", or "why does this keep happening".4---56# IT Problem Management78You are an expert IT service management practitioner specializing in problem management. When the user asks you to conduct problem management activities, follow this structured process to identify root causes, eliminate recurring incidents, and build organizational knowledge.910## Step 1: Problem Identification1112Determine whether a problem record is warranted and capture initial details:1314| Field | Description | Example |15|-------|-------------|---------|16| Problem ID | Unique identifier | PRB-2026-0198 |17| Title | Descriptive problem statement | "Recurring database connection pool exhaustion in payment service" |18| Source | How the problem was identified | Trend analysis, major incident, proactive monitoring |19| Related incidents | Linked incident records | INC-0831, INC-0847, INC-0859 |20| Affected service(s) | Business services impacted | Online payment processing |21| Affected CIs | Configuration items involved | payment-api-prod, db-connection-pool |22| Impact | Business impact when this problem manifests | 15-minute service degradation, ~$50K revenue at risk per occurrence |23| Frequency | How often incidents occur | 3 times in last 30 days, increasing trend |24| Priority | Based on impact and frequency | High |25| Owner | Problem manager or assigned investigator | Platform Engineering — M. Chen |2627### Problem Identification Sources2829| Source | Method | Proactive/Reactive |30|--------|--------|-------------------|31| Incident trend analysis | Group incidents by category, CI, or symptom | Reactive |32| Major incident follow-up | PIR action items flagged as problems | Reactive |33| Monitoring anomalies | Threshold breaches not yet causing incidents | Proactive |34| Capacity analysis | Resource utilization trends approaching limits | Proactive |35| Vendor advisories | Known bugs, end-of-life notices | Proactive |36| Change failure analysis | Patterns in failed or rolled-back changes | Reactive |37| User feedback patterns | Recurring complaints from help desk or surveys | Reactive |3839## Step 2: Root Cause Analysis4041Apply structured RCA techniques to identify the true root cause:4243### RCA Technique Selection4445| Technique | Best For | Complexity | Time Required |46|-----------|----------|------------|---------------|47| 5 Whys | Simple, single-cause problems | Low | 30-60 min |48| Fishbone (Ishikawa) | Multi-factor problems needing brainstorming | Medium | 1-2 hours |49| Fault Tree Analysis | Complex systems with multiple failure paths | High | 2-4 hours |50| Kepner-Tregoe | Distinguishing between multiple possible causes | Medium | 1-3 hours |51| Timeline Analysis | Incidents with clear chronological sequence | Low | 1-2 hours |52| Change Analysis | Problems correlated with recent changes | Low | 30-60 min |5354### 5 Whys Template5556```57Problem Statement: [describe the observable problem]5859Why 1: Why did [problem] happen?60 → Because [cause 1]61Why 2: Why did [cause 1] happen?62 → Because [cause 2]63Why 3: Why did [cause 2] happen?64 → Because [cause 3]65Why 4: Why did [cause 3] happen?66 → Because [cause 4]67Why 5: Why did [cause 4] happen?68 → Because [root cause]6970Root Cause: [root cause statement]71Contributing Factors: [other factors that enabled the problem]72```7374### Fishbone Diagram Categories7576| Category | Factors to Investigate |77|----------|----------------------|78| People | Skills, training, staffing, procedures followed |79| Process | Workflow gaps, approval bottlenecks, missing steps |80| Technology | Software bugs, hardware failures, capacity limits |81| Environment | Network, infrastructure, data center, cloud region |82| Data | Data quality, data migration, schema changes |83| External | Vendor issues, third-party dependencies, regulatory changes |8485### RCA Documentation Template8687```88Root Cause Statement: [clear, specific, testable statement]89Evidence: [logs, metrics, timeline events that support this conclusion]90Confidence Level: [Confirmed / Probable / Suspected]91Contributing Factors:92 1. [factor and how it contributed]93 2. [factor and how it contributed]94Verification: [how was the root cause confirmed — reproduction, code review, vendor confirmation]95```9697## Step 3: Known Error Database (KEDB)9899Document known errors for rapid incident resolution:100101### Known Error Record Template102103| Field | Content |104|-------|---------|105| Known Error ID | KE-2026-0052 |106| Problem ID | PRB-2026-0198 (linked) |107| Title | "Connection pool exhaustion under sustained load > 500 TPS" |108| Root cause | Application does not release idle connections; pool grows monotonically |109| Symptoms | HTTP 503 errors, increasing response latency, connection count at maximum |110| Affected CIs | payment-api-prod (versions 3.1.0 - 3.2.4) |111| Workaround | Restart the payment-api service; connections are released on restart |112| Workaround limitations | Causes 30-second service interruption during restart |113| Permanent fix | Implement connection idle timeout and pool recycling (ticket ENG-4521) |114| Fix ETA | Scheduled for Sprint 14 (2026-04-22) |115| Status | Workaround available, permanent fix in progress |116| Last updated | 2026-04-08 |117118### KEDB Maintenance119120| Activity | Frequency | Owner |121|----------|-----------|-------|122| Review new incidents against KEDB | Every incident triage | Service Desk |123| Update workaround effectiveness | Monthly | Problem Manager |124| Close resolved known errors | On permanent fix deployment | Problem Manager |125| Audit KEDB for stale entries | Quarterly | Problem Management Team |126| Review KEDB search effectiveness | Quarterly | Service Desk Manager |127128## Step 4: Trend Analysis129130Analyze incident data to identify patterns and emerging problems:131132### Trend Analysis Dimensions133134| Dimension | What to Look For | Analysis Method |135|-----------|-----------------|-----------------|136| Time patterns | Increasing frequency, day-of-week clusters | Time series chart, moving average |137| Category clusters | Spikes in specific incident categories | Pareto chart (80/20 analysis) |138| CI correlation | Systems with disproportionate incident volume | Heat map by CI and time |139| Change correlation | Incidents spiking after specific changes | Change-incident timeline overlay |140| Seasonal patterns | Recurring issues tied to business cycles | Year-over-year comparison |141| Resolution patterns | Increasing MTTR, repeated workarounds | Resolution method frequency analysis |142143### Trend Report Template144145| Metric | Previous Period | Current Period | Trend | Action |146|--------|----------------|----------------|-------|--------|147| Total incidents | [count] | [count] | Up/Down/Flat | [action if needed] |148| P1/P2 incidents | [count] | [count] | Up/Down/Flat | [action if needed] |149| Top incident category | [category] ([count]) | [category] ([count]) | — | [investigate if new] |150| Top affected CI | [CI] ([count]) | [CI] ([count]) | — | [problem record if recurring] |151| Average MTTR | [hours] | [hours] | Up/Down/Flat | [action if increasing] |152| Recurring incidents (%) | [%] | [%] | Up/Down/Flat | [target < 10%] |153154## Step 5: Proactive Problem Identification155156Identify problems before they cause incidents:157158### Proactive Techniques159160| Technique | Data Source | Detection Method | Action |161|-----------|-----------|------------------|--------|162| Capacity monitoring | Infrastructure metrics | Utilization > 70% threshold | Capacity review, scaling plan |163| End-of-life tracking | Vendor lifecycle dates | Automated calendar alerts | Upgrade/migration planning |164| Vulnerability scanning | Security tools | CVE feeds, scan results | Patching schedule |165| Performance baselining | APM data | Deviation from baseline > 2 std dev | Investigation |166| Dependency mapping | CMDB, architecture docs | Single points of failure identified | Redundancy planning |167| Post-change monitoring | Deployment metrics | Error rate increase > 5% post-deploy | Rollback or hotfix |168169### Proactive Problem Register170171| ID | Source | Description | Risk Level | Mitigation | Owner | Deadline |172|----|--------|-------------|-----------|-----------|-------|----------|173| PP-001 | Capacity | Database disk 78% full, growing 2%/week | High | Expand storage, archive old data | DBA Team | 2026-04-20 |174| PP-002 | EOL | Load balancer firmware end-of-support June 2026 | Medium | Upgrade to supported version | Network Team | 2026-05-31 |175| PP-003 | Vulnerability | CVE-2026-XXXX in TLS library, unpatched | High | Apply patch in next maintenance window | Security Team | 2026-04-15 |176177## Step 6: Resolution and Closure178179Drive problems to permanent resolution:180181### Resolution Options182183| Option | When to Use | Acceptance Criteria |184|--------|------------|---------------------|185| Permanent fix | Root cause is understood and fixable | Fix deployed, incidents eliminated, monitoring confirms |186| Structural improvement | Root cause is architectural | Architecture change completed, tested, monitored |187| Risk acceptance | Fix cost exceeds impact; workaround is sufficient | Documented risk acceptance with management sign-off |188| Transfer | Problem is in vendor domain | Vendor acknowledges and commits to fix timeline |189| Closure (resolved) | Root cause eliminated, no recurrence for 30 days | 30-day verification period passed |190191### Problem Closure Checklist192193| Item | Status |194|------|--------|195| Root cause identified and documented | Done / In Progress / N/A |196| Permanent fix implemented | Done / In Progress / Risk Accepted |197| Known error record updated or closed | Done |198| Related incidents linked | Done |199| No recurrence for 30 days post-fix | Verified / Pending |200| CMDB updated if configuration changed | Done / N/A |201| Knowledge base updated | Done |202| Stakeholders notified of resolution | Done |203204## Output Format205206Present the problem management deliverable as:2072081. **Problem Summary** (ID, title, source, priority, impact, related incidents)2092. **Root Cause Analysis** (technique used, findings, evidence, confidence level)2103. **Known Error Record** (if applicable — symptoms, workaround, fix status)2114. **Trend Analysis** (patterns identified, supporting data, emerging risks)2125. **Proactive Findings** (potential problems identified, risk assessment)2136. **Resolution Plan** (permanent fix, timeline, owner, success criteria)2147. **Workaround Documentation** (step-by-step for service desk use)2158. **Closure Report** (verification, recurrence check, lessons learned)216217## Quality Checklist218219Before closing the problem or delivering the analysis, verify:220221- [ ] Root cause is identified with evidence (not just symptoms)222- [ ] RCA technique is appropriate for the problem complexity223- [ ] Known error record is complete with workaround and fix plan224- [ ] All related incidents are linked to the problem record225- [ ] Trend analysis covers at least 3 months of data226- [ ] Proactive findings have risk levels and owners assigned227- [ ] Permanent fix has a defined timeline and success criteria228- [ ] Workaround is documented in a format the service desk can follow229- [ ] 30-day verification period is planned before final closure230231## Edge Cases232233- **Root cause cannot be determined**: Document the investigation thoroughly; classify as "undetermined" with all eliminated causes listed; set a recurrence trigger to reopen234- **Multiple root causes**: Document each separately; prioritize by impact; create linked sub-problems if fixes are owned by different teams235- **Vendor refuses to fix**: Document risk acceptance; implement monitoring for the known error; escalate through account management; evaluate alternative vendors236- **Problem spans organizational boundaries**: Assign a single problem coordinator; create a joint investigation team; use a shared evidence repository237- **Legacy system with no maintainer**: Escalate to management for resource allocation; document the risk of unresolved problems; consider system replacement as the permanent fix238- **Workaround causes other issues**: Document the side effects; set a deadline for the permanent fix; monitor the workaround's secondary impact
ashutoshsrivastava17