Huawei Cloud Cce Alarm Correlation Engine

Huawei Cloud CCE Alarm Correlation Engine

Overview

Correlate Huawei Cloud AOM active and historical alarms for CCE operations, then convert raw alarm streams into prioritized investigation leads. This skill also manages AOM alarm rules and notification action rules through a strict preview-and-confirm workflow.

Architecture: python3 scripts/huawei-cloud.py dispatcher -> local hcloud (KooCLI) -> AOM/CCE/IAM cloud service operations -> alarm query, alarm correlation, alarm-rule management, notification-rule management, and CCE alarm health inspection.

Execution Method: All Huawei Cloud operations must go through the bundled Python dispatcher. The dispatcher invokes the local hcloud CLI. Direct SDK imports, hand-written API signing, curl IAM flows, openstack commands, or out-of-band cloud access paths are prohibited.

Related Skills:

• huawei-cloud-cce-metric-analyzer - CCE, AOM, and cloud-resource metric checks
• huawei-cloud-cce-kubernetes-event-analyzer - Kubernetes warning event analysis
• huawei-cloud-cce-pod-failure-diagnoser - Pod failure diagnosis
• huawei-cloud-cce-node-failure-diagnoser - Node failure diagnosis
• huawei-cloud-cce-auto-remediation-runner - Remediation preview and execution

Capabilities:

• Query active AOM alarms, historical AOM alarms, and merged active+historical views
• Analyze alarm deduplication, severity grouping, burst alarms, attention alarms, and chronic alarms
• Inspect CCE alarm health and produce risk items for follow-up diagnosis
• Query AOM alarm rules with optional cluster_id filtering
• Resolve the AOM Prometheus instance bound to a CCE cluster
• Create metric alarm rules and event alarm rules
• Create notification action rules from a user-provided SMN topic
• Batch configure CCE recommended alarm rules from the cloud-side CCE alarm template
• Batch clean CCE alarm rules matching the cloud-side CCE alarm template
• Update, delete, enable, and disable AOM alarm rules
• Query AOM action rules and mute rules

Typical Use Cases:

• "List AOM alarms for this CCE cluster"
• "Analyze active and historical alarms for alarm storms"
• "Show current alarm rules for cluster <cluster_id>"
• "Create CCE recommended alarm rules with this notification rule"
• "Clean CCE template alarm rules for this cluster"
• "Create a notification action rule using this SMN topic"
• "Disable this noisy alarm rule after I confirm"

Prerequisites

1. Runtime Dependencies

• Python 3.8+ for the dispatcher and result processing
• hcloud (KooCLI) 7.2.2+ in PATH
• A local hcloud profile configured with hcloud configure
• Run environment checks before first use when available in the skill package

2. Credential Configuration

• Valid Huawei Cloud credentials via explicit tool parameters, local hcloud profile, or environment variable fallback
• Credential priority for hcloud calls is: explicit tool parameters > local hcloud profile > environment variables
• Tools that need project_id resolve it internally where possible: explicit project_id parameter first, then active hcloud profile/IAM project lookup for the target region, then environment fallback

Security Rules:

• Never expose AK/SK, tokens, or credential-derived secrets in code, commands, logs, or responses
• Never run echo $HUAWEI_AK or echo $HUAWEI_SK
• Never write credentials to files
• Prefer hcloud profile for normal use
• Use IAM users with least privilege

Optional Environment Fallback:

export HUAWEI_AK=<your-ak>
export HUAWEI_SK=<your-sk>
export HUAWEI_REGION=cn-north-4
export HUAWEI_PROJECT_ID=<project-id>
export HUAWEI_SECURITY_TOKEN=<security-token>

3. IAM Permission Requirements

Permission Failure Handling:

1. Show the failed operation and required permission.
2. Ask the user to grant the missing IAM permission.
3. Pause mutation work until the user confirms permissions are ready.

Core Commands

All commands use the Python dispatcher script:

python3 scripts/huawei-cloud.py <action> <key=value>...

KooCLI命令格式标准

This skill does not ask users to run raw hcloud commands directly. All cloud operations must use the dispatcher command format:

python3 scripts/huawei-cloud.py <tool-name> key=value key=value

The dispatcher converts tool parameters into standard KooCLI calls:

hcloud <service> <operation> --cli-region=<region> --cli-output=json [--cli-jsonInput=<file>]

Follow these rules for every command:

• Use key=value parameters; quote values that contain spaces, >, <, |, JSON, or PromQL.
• Do not print or persist AK/SK, security tokens, or generated JSON input files.
• Use dispatcher preview output first for R2/R1/R0 tools; add confirm=true only after explicit user confirmation.
• Prefer hcloud profile credentials; explicit tool parameters override profile and environment fallback.
• See CLI Installation Guide for setup and validation.

1. Alarm Query And Correlation

# Query active + historical alarms in a region
python3 scripts/huawei-cloud.py huawei_list_aom_alarms \
  region=cn-north-4

# Query active + historical alarms for a cluster
python3 scripts/huawei-cloud.py huawei_list_aom_alarms \
  region=cn-north-4 cluster_id=<cluster-id>

# Query current active alarms only
python3 scripts/huawei-cloud.py huawei_list_aom_current_alarms \
  region=cn-north-4 cluster_id=<cluster-id>

# Analyze deduplicated, burst, attention, and chronic alarm groups
python3 scripts/huawei-cloud.py huawei_analyze_aom_alarms \
  region=cn-north-4 cluster_id=<cluster-id>

Do not conclude "no issue" from absence of active alarms alone. Always consider historical alarms when diagnosing a recent or recovered problem.

2. Alarm Rule Query

# Query all alarm rules in a region
python3 scripts/huawei-cloud.py huawei_list_aom_alarm_rules \
  region=cn-north-4

# Query alarm rules related to one CCE cluster
python3 scripts/huawei-cloud.py huawei_list_aom_alarm_rules \
  region=cn-north-4 cluster_id=<cluster-id>

# Resolve the cluster AOM Prometheus instance
python3 scripts/huawei-cloud.py huawei_resolve_cce_aom_prom_instance \
  region=cn-north-4 cluster_id=<cluster-id>

huawei_list_aom_alarm_rules supports cluster filtering by cluster_id only. Do not use cluster_name as a filter.

3. Notification Rules

# List existing action/notification rules
python3 scripts/huawei-cloud.py huawei_list_aom_action_rules \
  region=cn-north-4

# Preview creating a notification action rule from an SMN topic
python3 scripts/huawei-cloud.py huawei_create_aom_notification_action_rule \
  region=cn-north-4 rule_name=auto-cluster-xxx \
  notification_topic_name=<smn-topic-name> \
  notification_topic_urn=<smn-topic-urn>

# Confirm creation
python3 scripts/huawei-cloud.py huawei_create_aom_notification_action_rule \
  region=cn-north-4 rule_name=auto-cluster-xxx \
  notification_topic_name=<smn-topic-name> \
  notification_topic_urn=<smn-topic-urn> \
  confirm=true

Never choose a notification rule automatically. If bind_notification_rule_id is missing for batch alarm creation, list available action rules and wait for explicit user selection, or ask the user to provide an SMN topic and create a notification rule first.

4. Alarm Rule Creation

# Preview creating a CCE template metric alarm rule
python3 scripts/huawei-cloud.py huawei_create_aom_alarm_rule \
  region=cn-north-4 cluster_id=<cluster-id> \
  alarm_item=NodeCPUUsageHigherThanEightyPercent

# Confirm creating a CCE template metric alarm rule
python3 scripts/huawei-cloud.py huawei_create_aom_alarm_rule \
  region=cn-north-4 cluster_id=<cluster-id> \
  alarm_item=NodeCPUUsageHigherThanEightyPercent \
  bind_notification_rule_id=<action-rule-id> \
  confirm=true

# Preview creating an event alarm rule
python3 scripts/huawei-cloud.py huawei_create_aom_event_alarm_rule \
  region=cn-north-4 cluster_id=<cluster-id> \
  event_name="<event-name>"

# Confirm creating an event alarm rule
python3 scripts/huawei-cloud.py huawei_create_aom_event_alarm_rule \
  region=cn-north-4 cluster_id=<cluster-id> \
  event_name="<event-name>" \
  bind_notification_rule_id=<action-rule-id> \
  confirm=true

Metric rules with cluster_id + alarm_item and event rules both use the same CCE template payload path and default naming as batch creation. Pass rule_name only when the user explicitly requests a custom name. Single-rule creation requires explicit bind_notification_rule_id during confirmed execution, same as batch CCE alarm rule configuration. Event alarm rule event_name should reference references/cce-event-list.md; metric alarm_item should use the CCE template alias or rule name.

5. CCE Template Alarm Rules

# Preview batch creating CCE recommended alarm rules
python3 scripts/huawei-cloud.py huawei_configure_cce_aom_alarm_rules \
  region=cn-north-4 cluster_id=<cluster-id> \
  bind_notification_rule_id=<action-rule-id>

# Confirm batch creation
python3 scripts/huawei-cloud.py huawei_configure_cce_aom_alarm_rules \
  region=cn-north-4 cluster_id=<cluster-id> \
  bind_notification_rule_id=<action-rule-id> confirm=true

# Preview cleanup of CCE template alarm rules for a cluster
python3 scripts/huawei-cloud.py huawei_cleanup_cce_aom_alarm_rules \
  region=cn-north-4 cluster_id=<cluster-id>

# Confirm cleanup
python3 scripts/huawei-cloud.py huawei_cleanup_cce_aom_alarm_rules \
  region=cn-north-4 cluster_id=<cluster-id> \
  delete_auto_notification_rule=true confirm=true

huawei_configure_cce_aom_alarm_rules requires explicit bind_notification_rule_id. If not provided, do not return or choose available_notification_rules from this tool; call huawei_list_aom_action_rules separately and present choices to the user.

6. Alarm Rule Mutation

# Preview update
python3 scripts/huawei-cloud.py huawei_update_aom_alarm_rule \
  region=cn-north-4 rule_name=my-rule threshold=80

# Confirm update
python3 scripts/huawei-cloud.py huawei_update_aom_alarm_rule \
  region=cn-north-4 rule_name=my-rule threshold=80 confirm=true

# Preview delete
python3 scripts/huawei-cloud.py huawei_delete_aom_alarm_rule \
  region=cn-north-4 rule_name=my-rule

# Confirm delete
python3 scripts/huawei-cloud.py huawei_delete_aom_alarm_rule \
  region=cn-north-4 rule_name=my-rule confirm=true

# Preview disable or enable
python3 scripts/huawei-cloud.py huawei_disable_aom_alarm_rule \
  region=cn-north-4 rule_id=<rule-id>

python3 scripts/huawei-cloud.py huawei_enable_aom_alarm_rule \
  region=cn-north-4 rule_id=<rule-id>

7. Action And Mute Rule Visibility

# Query mute rules
python3 scripts/huawei-cloud.py huawei_list_aom_mute_rules \
  region=cn-north-4

# Preview deleting an action rule
python3 scripts/huawei-cloud.py huawei_delete_aom_action_rule \
  region=cn-north-4 rule_name=<rule-name>

8. Cluster Alarm Inspection

python3 scripts/huawei-cloud.py huawei_aom_alarm_inspection \
  region=cn-north-4 cluster_id=<cluster-id>

Risk Levels

This skill includes read-only query tools and mutation tools. Mutation tools must be previewed first and must wait for explicit user confirmation before using confirm=true.

Parameter Reference

Common Parameters

Alarm Query Parameters

Alarm Rule Parameters

Output Format

See Output Schema for the complete JSON response structure.

Key output fields:

• success: command success status
• error: failure reason when success=false
• preview: mutation preview when confirm=true is not provided
• requires_confirmation: whether user confirmation is required
• report: alarm-correlation summary
• issues: inspection risk items
• alarms, rules, action_rules, mute_rules: queried resources

Workflow

See Workflow for the detailed alarm-correlation flow.

Recommended workflow:

1. Read the alarm name, resource, time window, and severity level from the user.
2. Default to the last 1 hour unless the user gives a different incident window.
3. Query active + historical alarms with huawei_list_aom_alarms.
4. Analyze alarm groups with huawei_analyze_aom_alarms.
5. Query alarm rules, action rules, and mute rules when notification gaps or suppression are suspected.
6. Group findings by resource, namespace, node, workload, and alarm type.
7. Hand off to diagnosis skills for Pod, Node, Network, Storage, or Workload root cause analysis.

Verification

Run read-only checks first:

python3 scripts/huawei-cloud.py huawei_list_aom_alarms region=cn-north-4
python3 scripts/huawei-cloud.py huawei_list_aom_alarm_rules region=cn-north-4
python3 scripts/huawei-cloud.py huawei_list_aom_action_rules region=cn-north-4

For cluster-scoped checks:

python3 scripts/huawei-cloud.py huawei_aom_alarm_inspection \
  region=cn-north-4 cluster_id=<cluster-id>

Mutation verification:

1. Run the command without confirm=true.
2. Review preview, affected resources, and expected impact.
3. Wait for explicit user confirmation.
4. Re-run with confirm=true.
5. Verify with the corresponding list/query command.

Best Practices

1. Use active + historical alarms for diagnosis - active-only views can miss recently recovered issues.
2. Filter by cluster_id - use exact cluster ID for cluster-scoped alarm rule and alarm analysis tasks.
3. Preview every mutation - never skip the preview step for R2/R1/R0 tools.
4. Do not auto-select notification rules - always wait for user selection of bind_notification_rule_id.
5. Keep notification setup explicit - create notification action rules only from a user-provided SMN topic.
6. Separate analysis from remediation - recommend remediation, then hand off to remediation skills.
7. Protect observability - avoid disabling or deleting rules unless impact is understood and confirmed.

Notes

• Alarm diagnosis must consider both active and historical alarms; do not conclude the cluster is healthy only because there are no active alarms.
• R2/R1/R0 tools must use preview-first execution and wait for explicit user confirmation before applying changes.
• Do not automatically select notification rules, SMN topics, clusters, or templates for the user.
• Do not modify alarm resources outside the tools provided by this skill.
• Use cluster_id for cluster-scoped filtering; cluster names are not accepted by huawei_list_aom_alarm_rules.

Troubleshooting

Limitations

• This skill does not modify CCE, ECS, ELB, EIP, VPC, security group, node, workload, or Kubernetes resources.
• This skill must not create, update, or delete mute rules.
• This skill must not use out-of-band cloud commands to modify alarm resources.
• This skill cannot safely infer a notification rule for the user.
• huawei_list_aom_alarm_rules filters by cluster_id, not cluster name.

References