Keepit Backup Failure Root Cause Analyzer

Keepit Backup Failure Root Cause Analyzer

Overview

The Backup Failure Root Cause Analyzer is a diagnostic skill that systematically investigates why Keepit backup jobs have failed, correlates failures with environmental changes, classifies root causes, and provides actionable remediation guidance. It combines job history analysis, error parsing, audit log correlation, health checks, and snapshot gap assessment to deliver comprehensive failure investigations.

Target Persona: Backup Administrators, IT Managers, Support Engineers

Primary Use Cases:

• Investigating why a specific backup job failed
• Diagnosing recurring backup failures across connectors
• Correlating failures with configuration or environmental changes
• Assessing data risk from backup gaps caused by failures
• Generating remediation steps and post-fix verification checklists

Trigger Keywords:

• "why did backup fail", "root cause", "backup error"
• "failure investigation", "troubleshoot backup"
• "diagnose failure", "backup not working"
• "backup keeps failing", "recurring failure"
• "what caused the backup error"

Core Workflow

Step 1: Identify the Failure

Retrieve job history for the target connector and isolate failed jobs.

# Get recent job history for the connector
jobs = get_job_history(guid=connector['guid'], duration="P7D")

# Filter to failed jobs — jobs with a 'failed' timestamp
failed_jobs = [j for j in jobs if j.get('failed')]

# Sort by failure time, most recent first
failed_jobs.sort(key=lambda j: j['failed'], reverse=True)

# If user asked about a specific failure, match by time or description
# Otherwise, investigate the most recent failure
target_failure = failed_jobs[0] if failed_jobs else None

If no failed jobs are found in the requested window, expand the search:

# Expand to 30 days if no failures in 7 days
if not failed_jobs:
    jobs = get_job_history(guid=connector['guid'], duration="P30D")
    failed_jobs = [j for j in jobs if j.get('failed')]

Step 2: Parse Error Details

Extract structured error information from the execsummary XML field.

# Parse the execsummary XML for error details
from scripts.rootcause_utils import parse_execsummary

execsummary = parse_execsummary(target_failure.get('execsummary', ''))

# Key fields extracted:
# - execsummary['errors']        → List of error messages
# - execsummary['exit_reason']   → Why the job terminated
# - execsummary['warnings']      → Warning messages preceding failure
# - execsummary['details']       → Additional context fields

Step 3: Gather Surrounding Context

Pull audit log events around the failure time window for correlation.

# Get audit events from the past 30 days
audit_logs = get_audit_log_history(duration="P30D")

# Correlate events within +/- 24 hours of the failure
from scripts.rootcause_utils import correlate_failure_with_audit

correlation = correlate_failure_with_audit(
    failure_time=target_failure['failed'],
    audit_events=audit_logs,
    window_hours=24
)

# correlation['events_before']   → Events leading up to failure
# correlation['events_after']    → Events after the failure
# correlation['config_changes']  → Configuration modifications
# correlation['auth_events']     → Authentication-related events
# correlation['total_correlated'] → Count of correlated events

Step 4: Check Current State

Verify whether the connector is still in a failed/unhealthy state.

# Check connector health status
health = get_connector_health(guid=connector['guid'])

# Possible states:
# - "healthy"    → Connector recovered on its own
# - "unhealthy"  → Still broken, needs attention
# - "critical"   → Critical issues, needs immediate attention
# - "unknown"    → Cannot determine state

Step 5: Check Active Recovery

Determine if a retry or recovery job is already running.

# Check for active jobs (retry in progress?)
active = get_active_jobs(guid=connector['guid'])

# If active jobs exist, a retry may already be underway
# Report this to avoid duplicate remediation efforts
if active:
    print(f"Recovery job detected: {active[0].get('description', 'Unknown')}")
    print(f"Started: {active[0].get('started', 'Unknown')}")

Step 6: Assess Data Impact

Calculate the backup gap and data at risk since the last successful backup.

# Get the latest successful snapshot
latest = get_latest_snapshot(guid=connector['guid'])

# Get snapshot history for gap analysis
snapshots = get_snapshot_range(guid=connector['guid'], timespan="P30D", count=99)

# Assess data risk
from scripts.rootcause_utils import assess_data_risk

risk = assess_data_risk(latest_snapshot=latest, snapshot_range=snapshots)

# risk['gap_hours']               → Hours since last good backup
# risk['gap_description']         → Human-readable gap description
# risk['risk_level']              → "low" / "medium" / "high" / "critical"
# risk['last_good_backup']        → Timestamp of last successful backup
# risk['data_at_risk_description'] → What data may be unprotected

Step 7: Correlate Failure with Changes

Determine if configuration changes caused or contributed to the failure.

# Get all connectors for cross-reference
connectors = get_cloud_connectors(connectorType="cloud")

# Check if config changes caused the failure
from scripts.rootcause_utils import detect_config_change_causality

causality = detect_config_change_causality(
    failure_time=target_failure['failed'],
    audit_events=audit_logs,
    connectors=connectors
)

# causality['causal_likely']  → True/False
# causality['changes_found']  → List of relevant changes
# causality['time_delta']     → Time between change and failure
# causality['confidence']     → "high" / "medium" / "low"
# causality['description']    → Human-readable causality assessment

Step 8: Classify and Attribute

Categorize the root cause and assign confidence level.

from scripts.rootcause_utils import classify_failure_type, classify_recurrence

# Classify the failure type
classification = classify_failure_type(
    job=target_failure,
    execsummary_parsed=execsummary
)

# classification['type']        → "auth_expired", "config_change", "api_limit",
#                                  "connectivity", "permission_denied",
#                                  "quota_exceeded", "service_unavailable", "unknown"
# classification['confidence']  → "high" / "medium" / "low"
# classification['indicators']  → Evidence supporting classification
# classification['description'] → Human-readable explanation

# Determine if this is a recurring issue
recurrence = classify_recurrence(
    job_history=jobs,
    current_failure=target_failure
)

# recurrence['classification']      → "first" / "recurring" / "chronic"
# recurrence['occurrence_count']     → Number of similar failures
# recurrence['first_seen']           → When this pattern first appeared
# recurrence['pattern_description']  → Description of the failure pattern

Step 9: Generate Resolution Steps

Produce type-specific remediation guidance.

from scripts.rootcause_utils import generate_resolution_steps

# Build context for resolution generation
context = {
    'connector': connector,
    'health': health,
    'execsummary': execsummary,
    'causality': causality,
    'recurrence': recurrence,
    'risk': risk
}

steps = generate_resolution_steps(
    failure_type=classification['type'],
    context=context
)

# Each step contains:
# - step['number']    → Step order
# - step['action']    → Brief action title
# - step['detail']    → Detailed instructions
# - step['priority']  → "critical" / "high" / "medium" / "low"

Step 10: Provide Verification Checklist

Generate post-fix validation steps to confirm the issue is resolved.

from scripts.rootcause_utils import generate_verification_checklist

checklist = generate_verification_checklist(
    failure_type=classification['type']
)

# Each checklist item:
# - item['check']           → What to verify
# - item['how_to_verify']   → How to perform the check
# - item['expected_result'] → What success looks like

Output Formatting Guidelines

Failure Investigation Header

# Backup Failure Root Cause Analysis

**Connector:** {connector_name} ({connector_type})
**Failure Time:** {failure_timestamp}
**Investigation Time:** {now}
**Status:** {current_health_status}

Root Cause Classification Display

Use severity icons to indicate urgency:

• ❌ Critical — Backup completely non-functional, data at risk
• ⚠️ Warning — Backup degraded, intermittent failures
• ✅ Resolved — Issue has been corrected, backup operational

Use priority icons for resolution steps:

• 🔴 High Priority — Address immediately
• 🟡 Medium Priority — Address within 24 hours
• 🟢 Low Priority — Schedule for next maintenance window

Summary Table Format

| Field | Value |
|-------|-------|
| Root Cause | {classification_type} |
| Confidence | {confidence_level} |
| Recurrence | {first / recurring / chronic} |
| Backup Gap | {gap_hours}h ({gap_description}) |
| Risk Level | {risk_icon} {risk_level} |
| Active Recovery | {yes/no} |

Timeline Display

## Event Timeline

| Time | Event | Relevance |
|------|-------|-----------|
| {t-2h} | Retention policy updated | 🔴 Likely cause |
| {t-0} | Backup job failed | ❌ Target failure |
| {t+1h} | Retry job started | ✅ Auto-recovery |

Resolution Steps Format

## Resolution Steps

### 🔴 Step 1: {action_title}
{detailed_instructions}

### 🟡 Step 2: {action_title}
{detailed_instructions}

### 🟢 Step 3: {action_title}
{detailed_instructions}

Verification Checklist Format

## Post-Fix Verification Checklist

- [ ] {check_description}
  - **How:** {how_to_verify}
  - **Expected:** {expected_result}

Advanced Features

Automatic Time-Window Correlation

The skill automatically correlates audit events with failure timing:

• -24h to failure: Events that may have caused the failure
• Failure to +24h: Events that may indicate recovery or further degradation
• Config changes within 4h: High-confidence causal attribution
• Auth events within 24h: Potential token/credential issues

Config Change Attribution

When audit logs show configuration changes near the failure time:

# Example: "Retention policy changed 2h before first failure"
if causality['causal_likely'] and causality['confidence'] == 'high':
    report += f"⚠️ Configuration change detected {causality['time_delta']} before failure\n"
    report += f"   Change: {causality['changes_found'][0]['description']}\n"
    report += f"   Recommendation: Review and potentially revert this change\n"

Failure Classification Categories

Recurring vs. New Failure Detection

# Classification thresholds
# 1 occurrence        → "first"     — Investigate root cause
# 2-4 occurrences     → "recurring" — Establish pattern, prioritize fix
# 5+ occurrences      → "chronic"   — Escalate, systemic issue

Data Risk Assessment Thresholds

Error Handling

No Failed Jobs Found

✅ No backup failures detected for {connector_name} in the past {duration}.

All recent jobs completed successfully. If you believe a failure occurred,
try expanding the search window:
- "Check for failures in the last 30 days"
- "Show all job history for this connector"

Execsummary Parse Failure

⚠️ Could not parse error details from job execution summary.

The job record indicates a failure at {failure_time}, but the error details
could not be extracted. This may indicate a truncated or malformed record.

**Available information:**
- Job failed: {failure_timestamp}
- Job description: {description}
- Connector: {connector_name}

**Recommendation:** Check Keepit admin console for detailed error logs.

Insufficient Audit Data

⚠️ Limited audit log data available for correlation.

Only {count} audit events found in the correlation window.
Root cause analysis may be incomplete without full audit context.

**Proceeding with available data...**

Connector Not Found

❌ Connector "{connector_name}" not found.

Please verify the connector name or GUID. Available connectors:
{list_of_connectors}

Best Practices

Investigation Approach

1. Start with the most recent failure — It has the freshest context and audit data
2. Always check current health first — The issue may already be resolved
3. Correlate before concluding — Never attribute root cause without checking audit logs
4. Assess recurrence — A first-time failure is different from a chronic one
5. Quantify data risk — Stakeholders need to know impact, not just cause

Escalation Guidelines

• First occurrence, low risk: Standard remediation, monitor
• Recurring, medium risk: Escalate to team lead, create tracking ticket
• Chronic, high risk: Escalate to management, schedule dedicated investigation
• Any critical risk: Immediate notification to data protection officer

Communication Guidelines

• Lead with impact: "Your Exchange backup has a 48-hour gap" before "Token expired"
• Provide time context: "This started failing 2 hours after the retention policy change"
• Be specific about remediation: "Re-authorize the Microsoft 365 connector" not "Fix auth"
• Include verification: "After fixing, run a manual backup and confirm it succeeds"

Common User Requests

"Why did my backup fail?"

1. Identify the connector and most recent failure
2. Parse error details from execsummary
3. Classify the failure type
4. Provide root cause with confidence level
5. Give resolution steps

"Is this a recurring issue?"

1. Pull 30-day job history
2. Classify recurrence pattern
3. Show failure timeline with dates
4. Identify if pattern exists (e.g., every Monday)
5. Recommend schedule or configuration changes

"What data am I missing?"

1. Get latest snapshot timestamp
2. Calculate backup gap
3. Assess risk level
4. Describe what data may be unprotected
5. Provide urgency context

"Was it caused by the config change?"

1. Pull audit logs around failure time
2. Run config change causality analysis
3. Show timeline of changes vs. failures
4. Provide confidence assessment
5. Recommend revert if causal

"What do I need to fix?"

1. Complete root cause analysis
2. Generate type-specific resolution steps
3. Prioritize by urgency
4. Provide verification checklist
5. Estimate time to resolution

"Give me a full RCA report"

1. Run complete 10-step workflow
2. Format as structured report with all sections
3. Include timeline, classification, impact, resolution
4. Add verification checklist
5. Provide executive summary

Cross-Skill Investigation Workflows

With Job History Analyzer

When failure investigation requires broader job trend context:

# Use job-history-analyzer patterns for trend detection
# Then drill into specific failures with this skill
jobs = get_job_history(guid=connector['guid'], duration="P30D")
# Analyze success/failure trends, then investigate individual failures

With Account Health Skill

When failure may be part of a broader account health issue:

# Check overall account health for systemic problems
health = get_connector_health(guid=connector['guid'])
# If multiple connectors are unhealthy, this may be an account-level issue
connectors = get_cloud_connectors(connectorType="cloud")

With Compliance Audit Log

When investigation requires detailed audit trail:

# Get comprehensive audit history for compliance-grade investigation
audit_logs = get_audit_log_history(duration="P30D")
# Use for change management correlation and compliance reporting

With Snapshot Analyzer

When failure impact requires detailed snapshot gap analysis:

# Deep dive into snapshot history for precise gap measurement
snapshots = get_snapshot_range(guid=connector['guid'], timespan="P30D", count=99)
latest = get_latest_snapshot(guid=connector['guid'])
# Combine with failure timeline for complete impact picture

With Retention Policy Auditor

When config change causality involves retention policies:

# Cross-reference retention policy changes with failure timing
connectors = get_cloud_connectors(connectorType="cloud")
# Check retention_updated fields against failure timestamps

Skill Integration Points

Technical Notes

MCP Functions Used

Time Calculations

• Failure time parsing: ISO 8601 format (2025-01-15T08:30:00Z)
• Correlation window: Default +/- 24 hours from failure time
• Config change window: High confidence within 4 hours, medium within 24 hours
• Backup gap: Difference between now and last successful snapshot
• Recurrence window: 30 days for pattern detection
• Duration formats: ISO 8601 durations (P7D = 7 days, P30D = 30 days)

Error Resilience

• Missing execsummary: Fall back to job-level metadata (description, timestamps)
• Empty audit logs: Proceed with job-only analysis, note limited correlation
• Connector health unavailable: Skip health check, note in report
• Snapshot data missing: Estimate gap from job history timestamps
• Parse failures: Catch XML parse errors, return raw data with error flag
• API timeouts: Retry once, then proceed with partial data and note gaps

Success Metrics