1---2name: memory-leak-investigation3enabled: true4description: |5 Use when performing memory leak investigation — provides a systematic6 methodology for detecting, diagnosing, and resolving memory leaks in7 production applications. Covers heap analysis, allocation profiling, leak8 pattern identification, garbage collection tuning, and validation procedures9 across common runtime environments.10required_connections:11 - prefix: apm12 label: "APM / Profiling Platform"13 - prefix: monitoring14 label: "Monitoring Platform"15config_fields:16 - key: runtime17 label: "Application Runtime"18 required: true19 placeholder: "e.g., JVM (Java 17), Node.js 20, Python 3.12, Go 1.22, .NET 8"20 - key: symptom21 label: "Primary Symptom"22 required: true23 placeholder: "e.g., OOM kills, growing RSS, increasing GC pauses"24 - key: time_to_oom25 label: "Approximate Time to OOM"26 required: false27 placeholder: "e.g., 6 hours, 3 days"28features:29 - PERFORMANCE30 - DEBUGGING31 - MEMORY32metadata:33 source: https://skillmd.com/skills/cloudthinker-ai/memory-leak-investigation34---3536# Memory Leak Investigation3738## Phase 1: Symptom Confirmation391. Verify memory leak exists40 - [ ] Memory usage grows continuously over time (not plateauing)41 - [ ] Memory not reclaimed after GC cycles42 - [ ] OOM kills or restarts observed43 - [ ] GC pause times increasing over time44 - [ ] Application performance degrades over uptime452. Establish the leak rate (MB/hour or MB/day)463. Determine if leak correlates with specific operations474. Check when the leak started (deployment, config change, traffic change)4849### Memory Growth Profile5051| Metric | At Start | After 1hr | After 6hr | After 24hr | Trend |52|--------|---------|----------|----------|-----------|-------|53| Heap used (MB) | | | | | +MB/hr |54| RSS (MB) | | | | | |55| GC frequency | | | | | |56| GC pause (ms) | | | | | |57| Live objects | | | | | |5859## Phase 2: Heap Snapshot Analysis601. Capture heap snapshots at intervals61 - Snapshot 1: Shortly after application start62 - Snapshot 2: After several hours of operation63 - Snapshot 3: When memory is significantly elevated642. Compare snapshots to identify growing object types653. Find objects that should be garbage collected but are retained664. Identify the retention path from GC root to leaked objects6768### Top Growing Object Types6970| Object Type | Count (T1) | Count (T2) | Size (T1) | Size (T2) | Growth Rate | Suspect |71|------------|-----------|-----------|----------|----------|-------------|---------|72| | | | MB | MB | MB/hr | [ ] |7374## Phase 3: Allocation Profiling751. Enable allocation tracking in profiler762. Identify hot allocation sites (where leaked objects are created)773. Trace allocation call stacks for suspected object types784. Correlate allocations with specific request types or code paths795. Check for known leak patterns8081### Common Leak Patterns Checklist8283| Pattern | Description | Detected |84|---------|-------------|----------|85| Event listener accumulation | Listeners added but never removed | [ ] |86| Cache without eviction | Unbounded cache or map growth | [ ] |87| Connection/stream not closed | DB connections, file handles, HTTP connections | [ ] |88| Circular references | Objects preventing GC (weak references needed) | [ ] |89| Static/global collections | Collections growing without bounds | [ ] |90| Timer/interval not cleared | Scheduled tasks holding references | [ ] |91| Thread-local accumulation | Thread-local storage not cleaned | [ ] |92| Closure capturing | Closures retaining large scope references | [ ] |93| Native memory leak | JNI, native buffers, off-heap allocations | [ ] |9495## Phase 4: Root Cause Isolation961. Reproduce the leak in a controlled environment972. Narrow down to specific code module or dependency983. Write a minimal reproduction if possible994. Verify the root cause with targeted profiling1005. Document the leak mechanism101102### Root Cause Analysis103104| Finding | Code Location | Leak Mechanism | Impact | Fix Approach |105|---------|-------------|---------------|--------|-------------|106| | file:line | | MB/hr | |107108## Phase 5: Fix Implementation1091. Implement the fix110 - [ ] Add proper resource cleanup (close, dispose, remove listener)111 - [ ] Add eviction policy to caches (LRU, TTL, size limit)112 - [ ] Fix object lifecycle management113 - [ ] Use weak references where appropriate114 - [ ] Add connection pool limits and timeout1152. Add defensive measures116 - [ ] Add memory usage logging117 - [ ] Implement resource tracking in development mode118 - [ ] Add unit tests for resource cleanup119120## Phase 6: Validation1211. Deploy fix and monitor memory profile1222. Verify memory stabilizes at expected level1233. Run extended soak test (48+ hours)1244. Compare memory profile against pre-leak baseline1255. Set up ongoing memory monitoring and alerts126127### Validation Results128129| Metric | Before Fix | After Fix | Target | Status |130|--------|-----------|----------|--------|--------|131| Heap growth rate | MB/hr | MB/hr | ~0 MB/hr | Pass/Fail |132| Stable heap size | N/A (growing) | MB | < MB | |133| GC pause P95 | ms | ms | < ms | |134| OOM incidents/week | | 0 | 0 | |135136## Counter-Rationalizations137138| Shortcut | Counter | Why |139|----------|---------|-----|140| "We can skip some steps for this case" | Adapt the workflow steps, don't skip them | Skipped steps are where incidents and oversights originate |141| "The user seems to already know what to do" | Complete all workflow phases with the user | The workflow catches blind spots that experience alone misses |142| "This is a minor case, full process is overkill" | Scale the process down, don't turn it off | Minor cases become major when unstructured; the process scales, not disappears |143| "I'll fill in the details later" | Complete each section before moving on | Deferred details are forgotten; real-time capture is more accurate |144| "The template output isn't necessary" | Always produce the structured output format | Structured output enables comparison, audit trails, and handoff to other teams |145146## Output Format147- **Leak Confirmation**: Growth rate and correlation analysis148- **Heap Analysis**: Object type growth and retention paths149- **Root Cause Report**: Code location and leak mechanism150- **Fix Description**: Changes made with rationale151- **Validation Report**: Post-fix memory stability proof152153## Action Items154- [ ] Confirm memory leak with growth rate measurement155- [ ] Capture and compare heap snapshots156- [ ] Identify leaking object types and retention paths157- [ ] Isolate root cause to specific code158- [ ] Implement and test fix159- [ ] Validate with extended soak test160- [ ] Set up memory growth alerting to detect future leaks