# 1525 07 Operations Runbook 5edc8070 > 📖 Operations Runbook: terraform-e2e - Skill: `tools-only/1525-07-operations-runbook-5edc8070` (Agent Skill, multi-file: 3 files) - Install (CLI): `npx skillmds@latest add tools-only/1525-07-operations-runbook-5edc8070` - Raw SKILL.md: https://api.skillmd.com/api/skills/tools-only/1525-07-operations-runbook-5edc8070/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: tools-only (https://skillmd.com/u/tools-only) - Updated: 2026-09-22 - Page: https://skillmd.com/skills/tools-only/1525-07-operations-runbook-5edc8070 --- # 📖 Operations Runbook: terraform-e2e ![Step](https://img.shields.io/badge/Step-7c-blue?style=for-the-badge) ![Status](https://img.shields.io/badge/Status-Draft-orange?style=for-the-badge) ![Agent](https://img.shields.io/badge/Agent-As--Built-purple?style=for-the-badge)
📑 Runbook Contents - [⚡ Quick Reference](#-quick-reference) - [📋 1. Daily Operations](#-1-daily-operations) - [🚨 2. Incident Response](#-2-incident-response) - [🔧 3. Common Procedures](#-3-common-procedures) - [🕐 4. Maintenance Windows](#-4-maintenance-windows) - [📞 5. Contacts & Escalation](#-5-contacts--escalation) - [📝 6. Change Log](#-6-change-log) - [References](#references)
> Generated by as-built agent | 2026-02-26 | ⬅️ Previous | 📑 Index | Next ➡️ | | ---------------------------------------------- | ------------------- | ---------------------------------------------------- | | [07-design-document.md](07-design-document.md) | [README](README.md) | [07-resource-inventory.md](07-resource-inventory.md) | **Version**: 1.0 **Date**: 2026-02-26 **Environment**: dev **Region**: swedencentral --- ## ⚡ Quick Reference | Item | Value | | ---- | ----- | | **Primary Region** | swedencentral | | **Resource Group** | rg-terraform-e2e-dev | | **Frontend Endpoint** | https://app-terraform-e2e-fe-dev-3hpu.azurewebsites.net | | **Backend Endpoint** | https://app-terraform-e2e-dev-3hpu.azurewebsites.net | | **Support Contact** | team-terraform | | **Escalation Path** | On-call engineer → Platform lead → Subscription owner | ### Critical Resources | Resource | Name | Resource Group | Severity | | -------- | ---- | -------------- | -------- | | App Service (Frontend) | app-terraform-e2e-fe-dev-3hpu | rg-terraform-e2e-dev | 🟠 P2 | | App Service (Backend) | app-terraform-e2e-dev-3hpu | rg-terraform-e2e-dev | 🔴 P1 | | SQL Database | sqldb-terraform-e2e-dev | rg-terraform-e2e-dev | 🔴 P1 | | Key Vault | kv-tfe2dev-3hpu | rg-terraform-e2e-dev | 🟠 P2 | --- ## 📋 1. Daily Operations ### 1.1 Health Checks **Morning Health Check:** 1. ✅ Verify frontend and backend return HTTP response 2. ✅ Confirm App Services are `Running` 3. ✅ Confirm SQL server state is `Ready` and DB state is `Online` 4. ✅ Check last 24h smart detector alerts for anomalies **KQL Query - System Health Overview:** ```kusto requests | where timestamp > ago(24h) | summarize RequestCount = count(), FailedRequests = countif(success == false), P95DurationMs = percentile(duration, 95) by cloud_RoleName, bin(timestamp, 1h) | order by timestamp desc ``` ### 1.2 Log Review **Priority Logs to Review:** | Log Source | Query Focus | Action Threshold | | ---------- | ----------- | ---------------- | | Application Insights | Failed requests and dependencies | >2% failures over 15 min | | App Service platform logs | Availability and restart events | Any repeated restart event | | Azure SQL diagnostics | Connectivity/authentication issues | >5 failures in 15 min | --- ## 🚨 2. Incident Response ### 2.1 Severity Definitions | Severity | Definition | Response Time | | -------- | ---------- | ------------- | | 🔴 P1 | Backend unavailable or SQL outage | 15 min | | 🟠 P2 | Partial degradation (high latency, intermittent errors) | 30 min | | 🟢 P3 | Non-critical issue (single feature impact) | 4 hours | ### 2.2 Runbooks by Alert | Alert | Runbook | Owner | | ----- | ------- | ----- | | Failure Anomalies smart detector | Triage App Insights failures, then app logs and SQL connectivity | Platform on-call | | App unavailable | Validate app runtime state, restart app if needed, rollback recent config | Platform on-call | | SQL connectivity failures | Validate SQL server status and AAD auth configuration | Platform + DBA | --- ## 🔧 3. Common Procedures ### 3.1 Restart Services ```bash az webapp restart --name app-terraform-e2e-dev-3hpu --resource-group rg-terraform-e2e-dev az webapp restart --name app-terraform-e2e-fe-dev-3hpu --resource-group rg-terraform-e2e-dev ``` ### 3.2 Validate Runtime State ```bash az webapp show --name app-terraform-e2e-dev-3hpu --resource-group rg-terraform-e2e-dev --query "{name:name,state:state,httpsOnly:httpsOnly,host:defaultHostName}" -o table az webapp show --name app-terraform-e2e-fe-dev-3hpu --resource-group rg-terraform-e2e-dev --query "{name:name,state:state,httpsOnly:httpsOnly,host:defaultHostName}" -o table az sql server show --name sql-terraform-e2e-dev-3hpu --resource-group rg-terraform-e2e-dev --query "{name:name,state:state,publicNetworkAccess:publicNetworkAccess,version:version}" -o table ``` ### 3.3 Terraform Drift Check ```bash cd infra/terraform/terraform-e2e terraform plan -detailed-exitcode ``` ### 3.4 Secret/RBAC Validation ```bash az keyvault show --name kv-tfe2dev-3hpu --resource-group rg-terraform-e2e-dev --query "{name:name,enableRbacAuthorization:properties.enableRbacAuthorization,publicNetworkAccess:properties.publicNetworkAccess}" -o table az role assignment list --scope /subscriptions/00858ffc-dded-4f0f-8bbf-e17fff0d47d9/resourceGroups/rg-terraform-e2e-dev/providers/Microsoft.KeyVault/vaults/kv-tfe2dev-3hpu -o table ``` --- ## 🕐 4. Maintenance Windows | Task | Schedule | Duration | | ---- | -------- | -------- | | Terraform patching/updates | Saturdays 09:00 UTC | 1 hour | | App dependency refresh | Monthly, first business Monday | 1 hour | | DR rehearsal | Quarterly | 2 hours | --- ## 📞 5. Contacts & Escalation | Role | Contact | Phone | On-Call Rotation | | ---- | ------- | ----- | ---------------- | | L1 On-Call Engineer | team-terraform | N/A | Weekly | | L2 Platform Lead | team-terraform lead | N/A | Weekly backup | | L3 Subscription Owner | subscription admin | N/A | Escalation only | ### Escalation Path 1. L1 triages within SLA window. 2. If unresolved in 30 minutes (P1/P2), escalate to L2. 3. If platform-wide or policy-related blocker persists, escalate to L3. --- ## 📝 6. Change Log | Date | Change | Author | | ---- | ------ | ------ | | 2026-02-26 | Initial as-built runbook generated from deployed Terraform state and deployment summary | as-built agent | --- ## References | Topic | Link | | ----- | ---- | | Azure Monitor Alerts | [Alerting Best Practices](https://learn.microsoft.com/azure/azure-monitor/best-practices-alerts) | | KQL Queries | [KQL Reference](https://learn.microsoft.com/azure/azure-monitor/logs/get-started-queries) | | Azure Service Health | [Overview](https://learn.microsoft.com/azure/service-health/overview) | | App Service Operations | [Operations guide](https://learn.microsoft.com/azure/app-service/overview) | ---
| ⬅️ [07-design-document.md](07-design-document.md) | 🏠 [Project Index](README.md) | ➡️ [07-resource-inventory.md](07-resource-inventory.md) | | ------------------------------------------------- | ----------------------------- | ------------------------------------------------------- |