Sre Principles

SRE Investigation Principles

These principles guide how to investigate and communicate findings.

Intellectual Honesty

State Confidence Clearly

• High confidence: Direct evidence, deterministic relationship
• Likely: Strong circumstantial evidence
• Possible: Some supporting evidence, alternatives exist
• Uncertain: Insufficient evidence

Acknowledge Limitations

• Say "I don't know" when you don't know
• Identify what would be needed to find out
• Don't speculate beyond available evidence

Distinguish Facts from Hypotheses

Facts (observed directly):

• "Pod restarted 5 times in the last hour"
• "Error rate increased from 0.1% to 5%"
• "Memory usage was 450MB when OOMKilled"

Hypotheses (inferred):

• "The memory leak is likely caused by..."
• "This suggests the deployment introduced..."
• "The correlation implies..."

Evidence-Based Reasoning

Every Claim Needs Evidence

Don't say: "The service is having problems" Do say: "The service is returning 500 errors (5% of requests per metrics)"

Quote Specific Data

• Timestamps: "Starting at 14:32 UTC"
• Values: "CPU spiked to 95%"
• Error messages: "ConnectionRefusedError: host:5432"

Build a Timeline

14:30 - Deployment v1.2.3 completed
14:32 - Error rate increased from 0.1% to 2%
14:35 - Pod restarts began
14:40 - Alert triggered

Falsification

Look for Contradicting Evidence

For each hypothesis, ask:

• What would disprove this?
• Have I looked for that evidence?

Update Beliefs

When evidence contradicts your hypothesis:

• Acknowledge the contradiction
• Revise or discard the hypothesis
• Form new hypotheses

Consider Alternatives

Before concluding, consider:

• Could another cause produce the same symptoms?
• What's the simplest explanation?

Communication Standards

Structure

**Summary** (1-2 sentences)
What happened and the root cause.

**Impact**
- Users affected
- Duration
- Services impacted

**Timeline**
Chronological events with timestamps.

**Root Cause**
Specific, technical explanation.

**Evidence**
Data supporting the root cause.

**Actions Taken**
What was done to resolve.

**Recommendations**
Prevent recurrence.

Lead with Conclusions

Don't bury the answer. Start with:

• What the root cause is
• What action is recommended

Then provide supporting evidence.

Thoroughness

Don't Stop at Symptoms

When to Stop Investigating

Stop when:

• You've identified specific, actionable cause
• You've exhausted available diagnostic tools
• Further investigation requires access you don't have
• The user has asked you to stop

Don't stop just because:

• You found "an error"
• Investigation is taking time
• First hypothesis seemed plausible

Example Investigation Summary

**Root Cause**: Memory leak in payment-service causing OOMKilled restarts

**Evidence**:
- Memory usage: Increased 400MB/hour (metric: container_memory_working_set_bytes)
- Events: 23 OOMKilled events in last 6 hours (get_pod_events)
- Correlation: Restarts started after deploy of commit abc123 (git_log)
- Change point: Memory trend changed at 14:32 UTC (find_change_point)

**Confidence**: High
- Memory trend and OOM events are deterministic
- Direct correlation with deployment timestamp

**Hypothesis Testing**:
- Ruled out: Traffic increase (requests stable per metrics)
- Ruled out: External dependency (no correlation)
- Confirmed: Memory growth rate constant regardless of load

**Recommendation**:
1. Immediate: Rollback to previous version
2. Follow-up: Profile memory in staging
3. Prevention: Add memory alerts at 70% threshold

**Caveat**: Did not identify the specific code causing the leak