Sap Ha Cluster Health

Environment Configuration

All environment-specific values (subscription ID, AMS workspace ID, proxy URLs, API keys, SAP landscape) are provided via the Team Onboarding instructions. The agent reads these from the onboarding context at runtime. Do not hardcode environment values in this skill.

Authentication: Use the agent's built-in tools (RunAzCliReadCommands, GetArmResourceAsJson, QueryLogAnalyticsByWorkspaceId, GetMetricTimeSeriesElementsForAzureResource) for Azure API calls. These authenticate automatically via the agent's Managed Identity.

Data Reuse (AAU Optimization): Before calling any API or proxy, check if the data was already retrieved earlier in this conversation. Reuse landscape registry, VM power states, config files, and AMS query results from context. Do not re-fetch data that is already available.

Config reads & proxy fallback: Stored SAP/OS configs are read directly from the sap-configs blob container using the agent's own Managed Identity (--auth-mode login) — there is no config proxy. The MCP command proxy is optional and runs only live VM commands; if it is not deployed or errors (timeout, 5xx, unreachable), continue with stored blob configs + Azure-native sources (AMS, ARM API, Azure Monitor). Never block the skill on the proxy.

Infrastructure Requirements

This section covers what deployed infrastructure (Storage Account / MCP command proxy) enhances this skill — not to be confused with the operational tier modes (T1 / T3) in the ## Mode Selection section below. The skill adapts automatically based on what is listed in the ## Deployed Infrastructure section of Team Onboarding.

• No infrastructure listed — Cluster state is inferred only from Internal Load Balancer backend pool health probes + AMS Prometheus_HaClusterExporter_CL + Activity Log (fencing = VM deallocate events). Cannot inspect corosync.conf, SAPHanaSR-showAttr output, SBD status, or SR hooks in global.ini. Always disclose in the report header: "Cluster diagnosis is approximate (ILB probes + AMS only). Deploy a config store for corosync / SBD / SR-hook visibility, or add the MCP command proxy for live crm_mon."
• Storage Account listed — Also reads collected crm-status.txt, saphanasr-showattr.txt, corosync.conf, sbd-config.txt, and global.ini from the sap-configs blob container. Full HSR sync state + SOK / SFAIL visibility.
• MCP command proxy also listed — Also pulls live crm_mon, live SAPHanaSR-showAttr, and live HSR state through the command proxy. Best fidelity for "what is happening right now" questions.

Mode Selection

• User asks "Show cluster status for HSO" → T1 mode (read-only, display status)
• User asks "Why did HSO fail over?" → T1 forensic mode (read-only, timeline reconstruction)
• Alert fires "Pacemaker node offline" or scheduled (every 15 min) → T3 mode (diagnose, recommend, await approval)
• SFAIL detected even in T1 → auto-escalate to T3 (immediate alert)

When to Use

• "Show Pacemaker status for HSO" / "Cluster status"
• "Is HSR in sync?" / "Replication lag?"
• "Why did HSO fail over?" / "Fencing event investigation"
• "Takeover readiness?" / "Is HA working?"
• Auto-triggered by alerts on cluster/HSR state

Scope

Cluster-bearing systems only — apply to any system whose deployment in the landscape inventory is high-availability or disaster-recovery (these have Pacemaker + HSR). Skip systems whose deployment is standalone or distributed (no cluster). Works for both scale-up and scale-out architectures; for scale-out, also check standby-node readiness and that every worker node is present in the cluster. If asked about a non-cluster system, reply that cluster health doesn't apply to a <type> system rather than erroring.

Data Sources

Authentication

IMPORTANT — Azure API Access: Do NOT use IMDS tokens (169.254.169.254) or ManagedIdentityCredential — they are not available in the agent sandbox. Instead:

• For Azure Resource Manager queries: Use the built-in GetArmResourceAsJson or RunAzCliReadCommands tools
• For Log Analytics queries: Use the built-in QueryLogAnalyticsByWorkspaceId tool
• For metrics: Use the built-in GetMetricTimeSeriesElementsForAzureResource tool
• For live VM commands: invoke the SAP Command Runner skill (the sap-sre-proxy MCP connector) — never make direct HTTP or proxy calls

# Use the built-in tools above for Azure API access.
import json, re
from datetime import datetime, timedelta, timezone

# SUB_ID / AMS_WORKSPACE_ID: from Team Onboarding (AMS only if it is listed)

# Live cluster/HSR data comes ONLY from the MCP command proxy, invoked via the SAP Command
# Runner skill (run_batch). Never call the proxy directly. When the proxy is not configured,
# fall back to blob configs + AMS (see the Data Sources table). When T3 mode needs to
# remediate, invoke the SAP Command Runner skill.

# Allowlisted read-only commands to request via SAP Command Runner (run_batch):
HA_COMMANDS = [
    {"id": "crm_status", "cmd": "crm_mon -1 --output-as=text 2>/dev/null || crm status"},
    {"id": "saphanasr_showattr", "cmd": "SAPHanaSR-showAttr 2>/dev/null || echo 'N/A'"},
    {"id": "global_ini_hooks", "cmd": "grep -A5 '\\[ha_dr_provider' /hana/shared/*/global/hdb/custom/config/global.ini 2>/dev/null || echo 'N/A'"},
    {"id": "hsr_state", "cmd": "su - $(ls /hana/shared/ | head -1 | tr '[:upper:]' '[:lower:]')adm -c 'python /usr/sap/*/HDB*/exe/python_support/systemReplicationStatus.py' 2>/dev/null || echo 'N/A'"},
    {"id": "crm_config", "cmd": "cibadmin --query --scope crm_config 2>/dev/null | grep -E 'stonith-enabled|stonith-action|stonith-timeout|concurrent-fencing' || echo 'N/A'"},
]

# Stored fallback configs (crm-status.txt, saphanasr-showattr.txt, corosync.conf, global.ini)
# are read directly from the sap-configs blob with the agent MI:
#   az storage blob download --auth-mode login --account-name <sa> --container-name sap-configs \
#     --name <SID>/<host>/latest/cluster/<file>

T1 Mode: Cluster + HSR Status

Run getschema FIRST (mandatory). Before the queries below, run Prometheus_HaClusterExporter_CL | getschema and SapHana_SystemReplication_CL | getschema. The ha_cluster_* and HSR column names/suffixes vary by AMS collector version — the KQL below is a template; bind it to the real column names from getschema, never assume them from memory or another table.

Pacemaker Checks

Prometheus_HaClusterExporter_CL
| where TimeGenerated > ago(15m)
| summarize arg_max(TimeGenerated, *) by ha_cluster_pacemaker_nodes_s
| project Node=ha_cluster_pacemaker_nodes_s, Maintenance=ha_cluster_pacemaker_nodes_maintenance_d

HSR Checks (10 total)

Data source tagging: Always include in output which source was used:

• Source: ✅ Live — MCP command proxy (run_batch) succeeded
• Source: ⚠️ Blob (stale, <timestamp>) — fallback to cron-collected data

T1 Forensic Mode: Failure Timeline

1. Query Resource Health for platform events in time window
2. Query Activity Log for VM deallocate/restart (= STONITH fencing)
3. Query AMS for state transitions (node online→offline, resource Master→Slave)
4. Query VM Availability metric for dips
5. Check blob crm-status.txt for failed actions
6. Build root cause timeline

T3 Mode: Remediation with Approval

When SFAIL, quorum loss, or increasing fail-counts detected:

1. Diagnose: Gather full cluster + HSR state

2. Classify severity:

   • SFAIL with both nodes online → HIGH (takeover won't work)
   • Node offline, resources running on surviving node → MEDIUM (degraded but functional)
   • Quorum lost → CRITICAL (cluster may fence unpredictably)

3. Recommend specific action:

   • SFAIL → "Re-register secondary: hdbnsutil -sr_register --remoteHost=..."
   • Failed resource → "Cleanup: crm resource cleanup <rsc>"
   • Node in standby → "Bring online: crm node online <node>"

4. Approval: Send Teams card with diagnosis + recommended action

5. Execute: On approval, invoke the SAP Command Runner skill:

   • For failed resource cleanup: command_id=crm_cleanup, vm=, rg=
   • For node standby: command_id=crm_maintenance_on, vm=, rg=
   • For node online: command_id=crm_maintenance_off, vm=, rg=

   Do NOT call the command proxy directly from this skill.

6. Validate: Re-check cluster + HSR state after action

Output Format

T1 mode:

HSO — HA & DR Status: 🟢 HEALTHY

Pacemaker:
  Nodes: vm01 ✅ online, vm02 ✅ online
  Resources: SAPHana (Master on vm01), SAPHanaTopology (Started on both)
  Fail-counts: 0
  Maintenance: OFF

HSR:
  Replication: ✅ ACTIVE (sync mode, logreplay)
  SAPHanaSR: SOK (both sites)
  SR Hook: SAPHanaSR registered in global.ini
  AUTOMATED_REGISTER: true
  PREFER_SITE_TAKEOVER: true

Takeover Readiness: ✅ READY

T3 mode:

🔴 HSO — HA ALERT: HSR SFAIL detected

  SAPHanaSR sync_state: SFAIL (secondary not in sync)
  Both nodes online — but takeover WILL FAIL if primary crashes.

  RECOMMENDED ACTION: Re-register secondary
    Command: hdbnsutil -sr_register --remoteHost=vm01 ...
    Risk: LOW (secondary only, no primary impact)

  [Approve] [Deny] [Show Details]