Azure Diagnostics

Azure Diagnostics

Troubleshoot Azure problems systematically: narrow scope, gather evidence, form hypotheses, validate fixes. Use structured root cause analysis to prevent recurrence.

Use This Skill When

• The user reports application downtime, errors, or performance regression
• Deployments succeed but service behavior is broken or degraded
• There are connectivity, identity, DNS, scaling, or quota issues
• Configuration drift or unexpected resource state is observed
• Error messages are cryptic or originating from dependent services

Context: Incident Classification

Availability (0% traffic passing): Database down, network cut, authentication broken
Degradation (partial traffic/slowness): High latency, connection pool exhaustion, throttling
Configuration drift: Resource settings changed outside IaC, policy violation, quota exceeded
Data integrity: Corruption, replication lag, backup failure
Security: Unauthorized access, malicious activity, compliance violation

Required Inputs

• Affected resources: Service name, region, resource group
• Symptom timeline: Start time, duration, frequency (one-off or recurring)
• Error signals: HTTP status codes, error messages, exception stack traces, correlation IDs
• Recent changes: Deployment, config update, DNS change, certificate rotation, network topology
• Expected vs. observed: What should happen vs. what actually happened
• User impact: How many users? Business tier? SLA breach?

Decision Tree

Is the service completely unavailable (0% traffic)?
├─ Yes → Check Azure Service Health, then check compute (app/function) status
│         ├─ Resource deleted/stopped? → Restart or redeploy
│         ├─ Resource failed to start? → Check startup logs, quotas
│         └─ Resource healthy but unreachable? → Check network, DNS, firewall
└─ No (partial/slow) → Proceed to performance analysis

Did this occur after a recent deployment or change?
├─ Yes → Suspect new code, config, or infrastructure
│         ├─ Rollback or undo change → Verify symptom resolves
│         └─ If resolved: Root cause is the change; analyze diff
└─ No → Suspect environmental change, quota, or dependent service

Is the issue isolated to one region/subscription or widespread?
├─ Isolated → Check resource-specific logs, networking, IAM
└─ Widespread → Check Azure platform status, quotas, throttling

Workflow

Phase 1: Triage (First 2 minutes)

1. Check Azure Service Health:

   # Portal: Service Health blade
   # Or search "Service Health" in Azure portal
   # Look for: Incidents, Advisories, Planned Maintenance affecting your region
   

2. Verify resource status:

   az resource show --ids /subscriptions/$SUBID/resourceGroups/$RG/providers/$TYPE/$NAME
   # Check: provisioningState, powerState, healthStatus fields
   

3. Check resource limits:

   • Has quota been exceeded? (CPU, memory, concurrent connections, API calls/sec)
   • Is the resource in a failed provisioning state?

Phase 2: Evidence Gathering (5-15 minutes)

1. Collect logs from all layers:

   Application Logs (Application Insights or Log Analytics):

   // Find errors in last hour
   traces | where timestamp > ago(1h) and severityLevel >= 2
   | summarize count() by tostring(severityLevel), message
   

   Dependency Tracking (what services called what):

   dependencies | where timestamp > ago(1h) and success == false
   | summarize count() by target, type, resultCode
   

   Network Flow (connectivity issues):

   az network watcher flow-log show --resource-group $RG --nsg-name $NSG
   # Download NSG flow logs from storage account
   

   Resource Logs (Azure Monitor Diagnostics):

   # For Key Vault:
   az monitor diagnostic-settings list --resource $KV_ID
   # Check: logs enabled, destination correct, retention policy
   

2. Extract correlation IDs and trace context:

   - Application Insights: Operation ID (shows request chain across services)
   - HTTP requests: x-correlation-id, x-request-id headers
   - Logs: transaction ID linking app → dependency → database
   

3. Timeline of events:

   10:30 AM: User reports 500 error
   10:31 AM: Error rate spikes to 95%
   10:32 AM: Database logs show connection timeout
   10:33 AM: DBA identifies firewall rule was deleted
   

Phase 3: Root Cause Analysis

Check each layer bottom-up:

1. Infrastructure Layer:

   • Is the resource running? (VM status, App Service plan load, Function runtime)
   • Are quotas exceeded? (vCPU, disk, network bandwidth, RPS limits)
   • Is there a network path? (ping, tracert, NSG rules, firewall)

   # Check VM status
   az vm get-instance-view --resource-group $RG --name $VM --query 'instanceView.statuses'
   
   # Check App Service plan utilization
   az monitor metrics list --resource $PLAN_ID --metric "CpuPercentage,MemoryPercentage"
   

2. Network Layer:

   • Can DNS resolve? (nslookup, dig)
   • Is there routing? (ip route show, traceroute)
   • Are NSGs blocking traffic? (check inbound/outbound rules)
   • Is private endpoint configured correctly? (DNS name resolution, IP routing)

   # Test DNS from within the resource:
   nslookup myapp.azurewebsites.net
   # If fails: Check private DNS zones, conditional forwarders
   

3. Identity Layer:

   • Does managed identity have permissions? (RBAC role assignments)
   • Is Key Vault access allowed? (network rules, access policies, RBAC)
   • Has credential rotated or expired? (service principal secret)

   # Check role assignment
   az role assignment list --assignee $PRINCIPAL_ID --scope $SCOPE
   
   # Verify Key Vault access
   az keyvault key list --vault-name $KV --query '[].name'
   

4. Data Layer:

   • Is the database accessible? (connection string, firewall, availability)
   • Are credentials valid? (username, password, connection string format)
   • Is there replication lag or transaction lock? (database health check)

   # Check SQL database connectivity
   sqlcmd -S $SERVER.database.windows.net -U $USER -P $PWD -Q "SELECT 1"
   

5. Application Layer:

   • Is there a code bug? (recent deployment, null pointer, logic error)
   • Are resource limits exhausted? (connection pool, memory, threads)?
   • Is there an unhandled exception? (check exception logs)

Phase 4: Hypothesis Testing

Form and test one hypothesis at a time:

Hypothesis 1: Network connectivity broken
  → Test: Can ping the endpoint? Can telnet to port?
  → Result: YES/NO → Adjust hypothesis or confirm

Hypothesis 2: Identity permissions missing
  → Test: Run az role assignment list for managed identity
  → Result: Missing "Key Vault Secrets User" role → CONFIRMED

Hypothesis 3: Quota exceeded
  → Test: az vm list-usage --location eastus --query "[?name.value=='cores'].currentValue"
  → Result: 100 cores used, limit 98 → CONFIRMED

Phase 5: Implement & Verify Fix

1. Minimal reversible fix:

   • Do NOT make multiple changes at once
   • Prefer configuration change over code deployment
   • Have rollback plan ready

2. Verify the fix:

   # Run post-fix validation
   # 1. Service health check passes
   # 2. Error rate returns to baseline
   # 3. Latency within SLA
   # 4. No cascading failures in dependent services
   

3. Monitor for 5-10 minutes:

   • Is error rate stable and low?
   • Are there any new errors in logs?
   • Have dependent services recovered?

Output Contract

1. Problem Statement

   • What users experience (HTTP 500, timeout, latency)
   • When it started, how long it lasted
   • Blast radius (% of users, which regions/tiers)

2. Root Cause (with Evidence)

   • Most likely cause with 2-3 supporting log entries
   • Alternative causes if root cause is 70-80% confidence

3. Supporting Evidence

   • Log excerpts showing the failure
   • Correlation IDs linking events across services
   • Timeline of events

4. Fix Actions (step-by-step, reversible)

   • Exact commands or portal steps
   • Expected wait time for fix to take effect
   • Rollback procedure if fix doesn't work

5. Verification Checklist

   • Smoke test (e.g., GET /health returns 200)
   • Full transaction test (e.g., POST /order, verify in database)
   • Dependent service test (e.g., can call downstream API)

6. Prevention & Lessons Learned

   • Alert that should have caught this earlier
   • Configuration or IaC change to prevent recurrence
   • Post-incident runbook for next time

Guardrails

• Separate facts from assumptions: Show evidence for each claim.
• Prefer reversible fixes: Change config before redeploying code.
• Avoid blame: Focus on systems, not people ("firewall rule was deleted" not "DBA made a mistake").
• Keep timeline explicit: Timestamps matter for correlation.
• Do not guess: If logs don't show the cause, keep investigating; don't assume.
• Test rollback: Never assume rollback works without testing.
• Create post-incident runbook: Document for faster diagnosis next time.