cloudthinker-ai Memory Leak Investigation
Phase 1: Symptom Confirmation
- Verify memory leak exists
- Memory usage grows continuously over time (not plateauing)
- Memory not reclaimed after GC cycles
- OOM kills or restarts observed
- GC pause times increasing over time
- Application performance degrades over uptime
- Establish the leak rate (MB/hour or MB/day)
- Determine if leak correlates with specific operations
- Check when the leak started (deployment, config change, traffic change)
Memory Growth Profile
| Metric | At Start | After 1hr | After 6hr | After 24hr | Trend |
|---|---|---|---|---|---|
| Heap used (MB) | +MB/hr | ||||
| RSS (MB) | |||||
| GC frequency | |||||
| GC pause (ms) | |||||
| Live objects |
Phase 2: Heap Snapshot Analysis
- Capture heap snapshots at intervals
- Snapshot 1: Shortly after application start
- Snapshot 2: After several hours of operation
- Snapshot 3: When memory is significantly elevated
- Compare snapshots to identify growing object types
- Find objects that should be garbage collected but are retained
- Identify the retention path from GC root to leaked objects
Top Growing Object Types
| Object Type | Count (T1) | Count (T2) | Size (T1) | Size (T2) | Growth Rate | Suspect |
|---|---|---|---|---|---|---|
| MB | MB | MB/hr | [ ] |
Phase 3: Allocation Profiling
- Enable allocation tracking in profiler
- Identify hot allocation sites (where leaked objects are created)
- Trace allocation call stacks for suspected object types
- Correlate allocations with specific request types or code paths
- Check for known leak patterns
Common Leak Patterns Checklist
| Pattern | Description | Detected |
|---|---|---|
| Event listener accumulation | Listeners added but never removed | [ ] |
| Cache without eviction | Unbounded cache or map growth | [ ] |
| Connection/stream not closed | DB connections, file handles, HTTP connections | [ ] |
| Circular references | Objects preventing GC (weak references needed) | [ ] |
| Static/global collections | Collections growing without bounds | [ ] |
| Timer/interval not cleared | Scheduled tasks holding references | [ ] |
| Thread-local accumulation | Thread-local storage not cleaned | [ ] |
| Closure capturing | Closures retaining large scope references | [ ] |
| Native memory leak | JNI, native buffers, off-heap allocations | [ ] |
Phase 4: Root Cause Isolation
- Reproduce the leak in a controlled environment
- Narrow down to specific code module or dependency
- Write a minimal reproduction if possible
- Verify the root cause with targeted profiling
- Document the leak mechanism
Root Cause Analysis
| Finding | Code Location | Leak Mechanism | Impact | Fix Approach |
|---|---|---|---|---|
| file:line | MB/hr |
Phase 5: Fix Implementation
- Implement the fix
- Add proper resource cleanup (close, dispose, remove listener)
- Add eviction policy to caches (LRU, TTL, size limit)
- Fix object lifecycle management
- Use weak references where appropriate
- Add connection pool limits and timeout
- Add defensive measures
- Add memory usage logging
- Implement resource tracking in development mode
- Add unit tests for resource cleanup
Phase 6: Validation
- Deploy fix and monitor memory profile
- Verify memory stabilizes at expected level
- Run extended soak test (48+ hours)
- Compare memory profile against pre-leak baseline
- Set up ongoing memory monitoring and alerts
Validation Results
| Metric | Before Fix | After Fix | Target | Status |
|---|---|---|---|---|
| Heap growth rate | MB/hr | MB/hr | ~0 MB/hr | Pass/Fail |
| Stable heap size | N/A (growing) | MB | < MB | |
| GC pause P95 | ms | ms | < ms | |
| OOM incidents/week | 0 | 0 |
Counter-Rationalizations
| Shortcut | Counter | Why |
|---|---|---|
| "We can skip some steps for this case" | Adapt the workflow steps, don't skip them | Skipped steps are where incidents and oversights originate |
| "The user seems to already know what to do" | Complete all workflow phases with the user | The workflow catches blind spots that experience alone misses |
| "This is a minor case, full process is overkill" | Scale the process down, don't turn it off | Minor cases become major when unstructured; the process scales, not disappears |
| "I'll fill in the details later" | Complete each section before moving on | Deferred details are forgotten; real-time capture is more accurate |
| "The template output isn't necessary" | Always produce the structured output format | Structured output enables comparison, audit trails, and handoff to other teams |
Output Format
- Leak Confirmation: Growth rate and correlation analysis
- Heap Analysis: Object type growth and retention paths
- Root Cause Report: Code location and leak mechanism
- Fix Description: Changes made with rationale
- Validation Report: Post-fix memory stability proof
Action Items
- Confirm memory leak with growth rate measurement
- Capture and compare heap snapshots
- Identify leaking object types and retention paths
- Isolate root cause to specific code
- Implement and test fix
- Validate with extended soak test
- Set up memory growth alerting to detect future leaks