Huawei Cloud CCE Metric Analyzer
Overview
Query and analyze metrics for CCE clusters (Pod/Node CPU/memory/disk) and cloud resources (ECS, ELB, EIP, NAT). Supports threshold-based anomaly detection, status classification (critical/warning/normal), and full-cluster monitoring aggregation.
Architecture: python3 scripts/huawei-cloud.py dispatcher → hcloud (KooCLI) cloud service queries + signed AOM Prometheus HTTP queries + limited kubectl reads only when Kubernetes resource relationships are required → Pod/Node metrics, ECS/ELB/EIP/NAT metrics → Threshold classification → Anomaly detection
Execution method: Cloud service queries are executed through the local hcloud CLI. AOM Prometheus query_range calls are the only exception and use signed HTTPS requests because the required Prometheus range-query path is not compatible with hcloud. Do not call Huawei Cloud SDKs, curl IAM flows, openstack, or hand-written cloud APIs outside the bundled dispatcher.
Related Skills: use pod/node diagnosers, Kubernetes event analyzer, capacity/cost skills, or auto-remediation runner for follow-up diagnosis or explicitly requested remediation.
Capabilities:
- Pod CPU/memory TopN ranking and single Pod time-series metrics
- Node CPU/memory/disk TopN ranking and single Node time-series metrics
- Node GPU and xGPU metrics, including GPU utilization, memory, temperature, power, schedule policy, xGPU allocation, usage, and health
- CoreDNS QPS, error rate excluding NXDOMAIN, NXDOMAIN rate, P95 latency, replica count, and per-Pod CPU/memory metrics
- nginx-ingress QPS, 4xx/5xx rate, success rate, P95 latency, active connections, per-Pod CPU/memory, and Ingress TLS certificate expiration status
- Autoscaler unschedulable Pods, node state count, scale-up/down events, errors, node groups, HPA current/desired replicas, and per-Pod CPU/memory metrics
- Kubernetes control-plane metrics for apiserver, etcd, controller-manager, and scheduler
- ECS instance CPU/memory/disk/network metrics
- ELB connection, bandwidth, QPS metrics
- EIP bandwidth, traffic, packet loss metrics
- NAT Gateway SNAT connection metrics
- Full-cluster monitoring aggregation with anomaly detection (80% threshold)
- Threshold-based status classification (critical/warning/normal/unknown)
Typical Use Cases: query Pod/Node TopN, GPU/xGPU, CoreDNS, nginx-ingress, autoscaler, control-plane, ECS/ELB/EIP/NAT metrics, full-cluster aggregation, and threshold-based anomaly detection.
Prerequisites
1. Runtime Dependencies
- Python 3.8+ for the dispatcher and result processing
- hcloud (KooCLI) 7.2.2+ for CCE/ECS/ELB/VPC/EIP/NAT/CES/IAM cloud service queries
kubectl only for Kubernetes resource reads that cannot be derived from AOM/hcloud, such as Pod label_selector filtering, Ingress TLS certificate checks, and LoadBalancer Service discovery for ELB/EIP association; clusters without external EIP require the kubectl cce plugin from kubectl-cce-plugin/README.md
- Prometheus-related monitoring data is queried from AOM Prometheus with signed HTTPS requests; the cluster must have the Prometheus add-on integrated with AOM, otherwise these tools may return empty metric series
- Controller-manager, scheduler, and etcd metrics require the
kube-controller-manager, kube-scheduler, and etcd-server ServiceMonitors to be enabled separately in AOM; otherwise these tools may return empty metric series
- Autoscaler, ingress-controller, and NVIDIA GPU metrics require the corresponding
autoscaler, ingress-controller, and nvidia-gpu-device-plugin PodMonitors to be enabled separately in AOM; ingress request metrics also require nginx_ingress_controller_requests to be explicitly allowed in the ingress-controller PodMonitor
- Run environment check before first use (see Verification section)
- Install and use
kubectl-cce according to references/kubectl-cce.md
2. Credential Configuration
- Valid Huawei Cloud credentials via hcloud profile or AK/SK mode
- Security Rules:
- 🚫 Never expose AK/SK values in code, conversation, or commands
- 🚫 Never use
echo $HUAWEI_AK or echo $HUAWEI_SK to check credentials
- ✅ Credential priority for hcloud calls is: explicit tool parameters > local hcloud profile > environment variables
- ✅ AOM Prometheus signed HTTP and Kubernetes certificate setup cannot use encrypted hcloud profile material, so they use explicit tool parameters first and environment variables as the signing fallback
- ✅ Prefer IAM users over root account for cloud operations
- ✅ Enable MFA for sensitive operations
Configuration Method:
hcloud configure list
export HUAWEI_AK=<your-ak>
export HUAWEI_SK=<your-sk>
export HUAWEI_REGION=cn-north-4
3. IAM Permission Requirements
| API Action |
Permission |
Purpose |
cce:cluster:get |
Get cluster |
View CCE cluster details |
aom:instance:list |
List AOM instances |
Discover AOM Prometheus instance for metrics |
aom:metricsData:get |
Get metrics data |
Query Pod/Node CPU/memory/disk metrics |
ces:metricsData:get |
Get CES metrics |
Query ECS/ELB/EIP/NAT cloud resource metrics |
ecs:cloudServers:list |
List ECS servers |
Correlate ECS instance IDs |
elb:loadbalancers:list |
List ELB instances |
Correlate ELB IDs |
vpc:eips:list |
List EIPs |
Correlate EIP IDs |
nat:natGateways:list |
List NAT Gateways |
Correlate NAT Gateway IDs |
Permission Failure Handling:
- When any command fails due to IAM permission errors, display the required permission list
- Guide the user to create a custom policy in the IAM console and grant authorization
- Pause execution and wait for user confirmation that permissions have been granted
Core Commands
All commands use the Python dispatcher script: python3 scripts/huawei-cloud.py <action> <key=value>...
KooCLI命令格式标准
Do not ask users to run raw hcloud commands directly. Use the dispatcher format:
python3 scripts/huawei-cloud.py <tool-name> key=value key=value
The dispatcher converts cloud service queries to KooCLI calls. AOM Prometheus range queries use signed HTTPS requests because that path is not compatible with hcloud. Avoid Kubernetes resource reads unless the tool explicitly needs Pod labels, Ingress TLS Secrets, or LoadBalancer Services. Quote values containing spaces, >, <, |, JSON, or PromQL; never print or persist AK/SK, security tokens, kubeconfig files, or temporary payloads; keep Kubernetes/AOM PromQL scoped with cluster="<cluster_id>".
1. CCE Pod Metrics
# Pod TopN — cluster-wide CPU/memory ranking
python3 scripts/huawei-cloud.py huawei_get_cce_pod_metrics_topN \
region=cn-north-4 cluster_id=<cluster-id> \
namespace=default top_n=10 hours=1
# Pod TopN with label selector
python3 scripts/huawei-cloud.py huawei_get_cce_pod_metrics_topN \
region=cn-north-4 cluster_id=<cluster-id> \
namespace=default label_selector="app=nginx,version=v1" top_n=10 hours=1
# Single Pod time-series
python3 scripts/huawei-cloud.py huawei_get_cce_pod_metrics \
region=cn-north-4 cluster_id=<cluster-id> \
pod_name=my-app-xxx namespace=default hours=1
# Single Pod GPU and xGPU metrics
python3 scripts/huawei-cloud.py huawei_get_cce_pod_gpu_metrics \
region=cn-north-4 cluster_id=<cluster-id> \
pod_name=my-gpu-app-xxx namespace=default hours=1
2. CCE Node Metrics
# Node TopN — cluster-wide CPU/memory/disk ranking
python3 scripts/huawei-cloud.py huawei_get_cce_node_metrics_topN \
region=cn-north-4 cluster_id=<cluster-id> \
top_n=10 hours=1
# Single Node time-series
python3 scripts/huawei-cloud.py huawei_get_cce_node_metrics \
region=cn-north-4 cluster_id=<cluster-id> \
node_ip=10.0.0.1 hours=1
# Node GPU and xGPU metrics
python3 scripts/huawei-cloud.py huawei_get_cce_node_gpu_metrics \
region=cn-north-4 cluster_id=<cluster-id> \
node_ip=10.0.0.1 hours=1
3. CCE CoreDNS Metrics
# CoreDNS key metrics: QPS, error rate excluding NXDOMAIN, NXDOMAIN rate, P95 latency, replicas, CPU, and memory
python3 scripts/huawei-cloud.py huawei_get_cce_coredns_metrics \
region=cn-north-4 cluster_id=<cluster-id> \
namespace=kube-system pod_regex=".*coredns.*" hours=1
4. CCE nginx-ingress Metrics
# nginx-ingress request processing and Ingress TLS certificate expiration
python3 scripts/huawei-cloud.py huawei_get_cce_nginx_ingress_metrics \
region=cn-north-4 cluster_id=<cluster-id> \
namespace=kube-system pod_regex=".*nginx.*ingress.*|.*ingress.*nginx.*" \
ingress_namespace=default cert_expire_warning_days=30 hours=1
5. CCE Autoscaler Metrics
# Cluster Autoscaler and HPA metrics
python3 scripts/huawei-cloud.py huawei_get_cce_autoscaler_metrics \
region=cn-north-4 cluster_id=<cluster-id> \
namespace=kube-system pod_regex=".*cluster.*autoscaler.*|.*autoscaler.*" \
include_hpa=true hours=1
6. Kubernetes Control Plane Metrics
python3 scripts/huawei-cloud.py huawei_get_cce_apiserver_metrics \
region=cn-north-4 cluster_id=<cluster-id> hours=1
python3 scripts/huawei-cloud.py huawei_get_cce_etcd_metrics \
region=cn-north-4 cluster_id=<cluster-id> hours=1
python3 scripts/huawei-cloud.py huawei_get_cce_controller_manager_metrics \
region=cn-north-4 cluster_id=<cluster-id> namespace=kube-system hours=1
python3 scripts/huawei-cloud.py huawei_get_cce_scheduler_metrics \
region=cn-north-4 cluster_id=<cluster-id> namespace=kube-system hours=1
7. Cloud Resource Metrics
# ECS instance metrics
python3 scripts/huawei-cloud.py huawei_get_ecs_metrics \
region=cn-north-4 instance_id=<instance-id>
# ELB metrics
python3 scripts/huawei-cloud.py huawei_get_elb_metrics \
region=cn-north-4 elb_id=<loadbalancer-id> hours=1
# EIP metrics
python3 scripts/huawei-cloud.py huawei_get_eip_metrics \
region=cn-north-4 eip_id=<eip-id> hours=1
# NAT Gateway metrics
python3 scripts/huawei-cloud.py huawei_get_nat_gateway_metrics \
region=cn-north-4 nat_gateway_id=<nat-gateway-id> hours=1
8. Cluster Monitoring Aggregation
# Aggregate all monitoring data with anomaly detection
python3 scripts/huawei-cloud.py huawei_cce_cluster_monitoring_aggregation \
region=cn-north-4 cluster_id=<cluster-id> \
start_time="2026-05-30 00:00:00" end_time="2026-05-30 23:59:59" \
namespace=default top_n=10
This tool aggregates: Pod TopN CPU/memory, Node TopN CPU/memory/disk, ELB metrics (matched through LoadBalancer Services fetched by kubectl), NAT Gateway metrics, EIP metrics (bandwidth, packet loss), and anomaly detection using 80% threshold.
It also includes CoreDNS, nginx-ingress, and autoscaler summaries. Cloud resources are scoped to the current cluster when an association can be proven: ELB is matched through LoadBalancer Service IP/EIP, NAT Gateway is filtered by the cluster VPC, and EIP is limited to associated ELB/NAT/Service IPs.
LoadBalancer Service discovery uses kubectl with generated kubeconfig through the cluster EIP when external access is available. If the cluster has no EIP, it uses the kubectl cce plugin. If neither path works, aggregation fails.
Risk Levels
This skill is read-only. It does not create, update, delete, restart, scale, or modify Huawei Cloud or Kubernetes resources.
| Level |
Meaning |
Execution Guidance |
| R3 |
No-risk read-only query or local analysis |
May run automatically |
| R2 |
Low-risk change, such as creating monitoring configuration without deleting resources or increasing service capacity/cost |
Not used by current tools |
| R1 |
Risky operation, such as restart-like impact, disabling protection, or changes that may increase cost or reduce observability |
Not used by current tools |
| R0 |
Critical operation, such as deleting clusters, applications, or broad-impact monitoring protections |
Not used by current tools |
| Tool |
Operation Type |
Risk Level |
Description |
huawei_get_cce_pod_metrics_topN |
Query |
R3 |
Read Pod CPU/memory/disk TopN metrics from AOM Prometheus |
huawei_get_cce_pod_metrics |
Query |
R3 |
Read single Pod CPU/memory/disk time-series metrics |
huawei_get_cce_node_metrics_topN |
Query |
R3 |
Read Node CPU/memory/disk TopN metrics from AOM Prometheus |
huawei_get_cce_node_metrics |
Query |
R3 |
Read single Node CPU/memory/disk time-series metrics |
huawei_get_cce_node_gpu_metrics |
Query |
R3 |
Read single Node GPU and xGPU metrics from AOM Prometheus |
huawei_get_cce_pod_gpu_metrics |
Query |
R3 |
Read single Pod GPU and xGPU metrics from AOM Prometheus |
huawei_get_cce_coredns_metrics |
Query |
R3 |
Read CoreDNS QPS, error rate excluding NXDOMAIN, NXDOMAIN rate, P95 latency, replicas, and per-Pod CPU/memory metrics |
huawei_get_cce_nginx_ingress_metrics |
Query |
R3 |
Read nginx-ingress request-processing metrics and Ingress TLS certificate expiration status; QPS falls back to nginx process request counters when request-dimension metrics are absent |
huawei_get_cce_autoscaler_metrics |
Query |
R3 |
Read Cluster Autoscaler scaling metrics, HPA replica state, and autoscaler Pod CPU/memory metrics |
huawei_get_cce_apiserver_metrics |
Query |
R3 |
Read kube-apiserver QPS, error rate, latency, and inflight request metrics |
huawei_get_cce_etcd_metrics |
Query |
R3 |
Read etcd leader, proposal, DB size, disk latency, CPU, and memory metrics |
huawei_get_cce_controller_manager_metrics |
Query |
R3 |
Read control-plane workqueue depth, adds, retries, queue latency, and work duration metrics |
huawei_get_cce_scheduler_metrics |
Query |
R3 |
Read scheduler attempts, pending Pods, scheduling latency, and queue metrics |
huawei_get_ecs_metrics |
Query |
R3 |
Read ECS monitoring data through hcloud/CES |
huawei_get_elb_metrics |
Query |
R3 |
Read ELB monitoring data through hcloud/CES |
huawei_get_eip_metrics |
Query |
R3 |
Read EIP monitoring data through hcloud/CES |
huawei_get_nat_gateway_metrics |
Query |
R3 |
Read NAT Gateway monitoring data through hcloud/CES |
huawei_cce_cluster_monitoring_aggregation |
Query + local analysis |
R3 |
Aggregate Pod/Node/cloud-resource metrics and classify anomalies locally |
Parameter Reference
Common Parameters
| Parameter |
Required/Optional |
Description |
Default |
region |
Required |
Huawei Cloud region |
HUAWEI_REGION |
cluster_id |
Required |
CCE cluster ID |
N/A |
namespace |
Recommended |
Kubernetes namespace |
default |
ak |
Optional |
Explicit AK; highest priority for all calls |
profile/env fallback |
sk |
Optional |
Explicit SK; highest priority for all calls |
profile/env fallback |
project_id |
Optional |
Explicit Project ID; hcloud uses profile before env fallback |
Auto from IAM/profile |
huawei_get_cce_pod_metrics_topN Parameters
| Parameter |
Required |
Description |
Default |
namespace |
No |
Namespace filter |
all |
label_selector |
No |
Label selector (e.g. app=web) |
N/A |
top_n |
No |
Number of top items |
10 |
hours |
No |
Metrics lookback hours |
1 |
node_ip |
No |
Filter Pods on specific node |
N/A |
cpu_query |
No |
Custom CPU PromQL |
Auto |
memory_query |
No |
Custom memory PromQL |
Auto |
disk_query |
No |
Custom disk PromQL |
Auto |
huawei_get_cce_pod_metrics Parameters
| Parameter |
Required |
Description |
Default |
pod_name |
Yes |
Target Pod name |
N/A |
namespace |
No |
Namespace |
default |
hours |
No |
Metrics lookback hours |
1 |
cpu_query |
No |
Custom CPU PromQL |
Auto |
memory_query |
No |
Custom memory PromQL |
Auto |
disk_query |
No |
Custom disk PromQL |
Auto |
huawei_get_cce_pod_gpu_metrics Parameters
| Parameter |
Required |
Description |
Default |
pod_name |
Yes |
Target Pod name |
N/A |
namespace |
No |
Target Pod namespace |
all |
hours |
No |
Metrics lookback hours |
1 |
gpu_selector |
No |
Custom GPU metric label selector. Use this when GPU metrics do not use the pod or namespace labels |
pod="<pod_name>",namespace="<namespace>" |
Optional custom PromQL overrides are supported for GPU utilization, memory, schedule policy, xGPU allocation/usage, and xGPU health metrics.
huawei_get_cce_node_metrics_topN Parameters
| Parameter |
Required |
Description |
Default |
top_n |
No |
Number of top items |
10 |
hours |
No |
Metrics lookback hours |
1 |
huawei_get_cce_node_metrics Parameters
| Parameter |
Required |
Description |
Default |
node_ip |
Yes |
Target Node IP |
N/A |
hours |
No |
Metrics lookback hours |
1 |
huawei_get_cce_node_gpu_metrics Parameters
| Parameter |
Required |
Description |
Default |
node_ip |
Yes |
Target Node IP or node name |
N/A |
hours |
No |
Metrics lookback hours |
1 |
gpu_selector |
No |
Custom GPU metric label selector. Use this when GPU metrics do not use the node label |
`node=~" |
Optional custom PromQL overrides are supported for GPU utilization, memory, temperature, power, schedule policy, xGPU allocation/usage, and xGPU health metrics.
huawei_get_cce_coredns_metrics Parameters
| Parameter |
Required |
Description |
Default |
namespace |
No |
CoreDNS namespace |
kube-system |
pod_regex |
No |
Regex used to match CoreDNS Pods |
.*coredns.* |
hours |
No |
Metrics lookback hours |
1 |
Optional custom PromQL overrides are supported for QPS, error rate, NXDOMAIN rate, P95 latency, CPU, memory, and replica count.
huawei_get_cce_nginx_ingress_metrics Parameters
| Parameter |
Required |
Description |
Default |
namespace |
No |
Namespace of nginx-ingress controller Pods. Use an empty value to query all namespaces |
kube-system |
pod_regex |
No |
Regex used to match nginx-ingress controller Pods |
`.*nginx.ingress. |
ingress_namespace |
No |
Namespace filter for Ingress TLS certificate checks |
all |
hours |
No |
Metrics lookback hours |
1 |
cert_expire_warning_days |
No |
Days before expiry to mark certificates as warning |
30 |
check_certificates |
No |
Whether to inspect Ingress TLS Secrets for expiration status |
true |
Ingress-controller metrics depend on the corresponding AOM PodMonitor. The nginx_ingress_controller_requests metric must be explicitly allowed in the ingress-controller PodMonitor; otherwise request-dimension metrics such as 4xx/5xx QPS, success rate, and latency may be empty, and QPS may only use the nginx_ingress_controller_nginx_process_requests_total fallback when available.
Optional custom PromQL overrides are supported for QPS, 4xx/5xx, success rate, P95 latency, active connections, CPU, and memory.
huawei_get_cce_autoscaler_metrics Parameters
| Parameter |
Required |
Description |
Default |
namespace |
No |
Namespace of Cluster Autoscaler Pods. Use an empty value to query all namespaces |
kube-system |
pod_regex |
No |
Regex used to match autoscaler Pods |
`.*cluster.autoscaler. |
hpa_namespace |
No |
Namespace filter for HPA replica metrics |
all |
hours |
No |
Metrics lookback hours |
1 |
include_hpa |
No |
Whether to query HPA current/desired replica metrics |
true |
Optional custom PromQL overrides are supported for unschedulable Pods, node states, scale events, errors, node groups, HPA replicas, CPU, and memory.
Kubernetes Control Plane Tool Parameters
Applies to huawei_get_cce_apiserver_metrics, huawei_get_cce_etcd_metrics, huawei_get_cce_controller_manager_metrics, and huawei_get_cce_scheduler_metrics.
huawei_get_cce_apiserver_metrics defaults to cluster="<cluster_id>",component="apiserver" and does not add namespace or Pod labels. Its default P95 latency excludes WATCH|CONNECT requests and also returns latency_p95_by_verb_ms for diagnosis. Use metric_selector only when the Prometheus labels differ.
huawei_get_cce_etcd_metrics defaults to cluster="<cluster_id>" and does not add namespace or Pod labels. Use metric_selector only when the Prometheus labels differ.
huawei_get_cce_controller_manager_metrics defaults to cluster="<cluster_id>" because CCE AOM workqueue metrics may not expose stable controller-manager Pod labels. It returns both aggregate workqueue metrics and per-queue name breakdowns.
huawei_get_cce_scheduler_metrics defaults to cluster="<cluster_id>" and returns aggregate metrics plus result, profile/result, and queue breakdowns.
Controller-manager, scheduler, and etcd metrics depend on AOM ServiceMonitor collection being enabled for the corresponding kube-controller-manager, kube-scheduler, and etcd-server endpoints. If ServiceMonitor is not enabled, the tools can run successfully but return empty series.
| Parameter |
Required |
Description |
Default |
namespace |
No |
Namespace of control-plane Pods. Use an empty value to query all namespaces |
kube-system |
pod_regex |
No |
Regex used to match target component Pods |
component-specific |
metric_selector |
No |
Custom apiserver/etcd/controller-manager/scheduler metric label selector |
apiserver: cluster="<cluster_id>",component="apiserver"; etcd/controller-manager/scheduler: cluster="<cluster_id>" |
hours |
No |
Metrics lookback hours |
1 |
Cloud Resource Tool Parameters
| Tool |
Required ID Parameter |
Optional Parameters |
huawei_get_ecs_metrics |
instance_id |
none |
huawei_get_elb_metrics |
elb_id |
hours |
huawei_get_eip_metrics |
eip_id |
hours |
huawei_get_nat_gateway_metrics |
nat_gateway_id |
hours |
huawei_cce_cluster_monitoring_aggregation Parameters
| Parameter |
Required |
Description |
Default |
start_time |
Yes |
Start time (YYYY-MM-DD HH:MM:SS) |
N/A |
end_time |
Yes |
End time (YYYY-MM-DD HH:MM:SS) |
N/A |
namespace |
No |
Namespace filter |
default |
top_n |
No |
Number of top items |
10 |
security_token |
No |
Temporary security token for AK/SK session credentials |
env fallback |
Output Format
See Output Schema for the complete JSON response structure.
Key output fields:
success — boolean, true if query completed
region — Huawei Cloud region
cluster_id / cluster_name — CCE cluster identity
aom_instance_id — AOM Prometheus instance used for metric queries
metrics — Dict with cpu/memory/disk data per resource, including status classification
certificate_check — nginx-ingress Ingress TLS certificate expiration summary when certificate checking is enabled
time_series — Historical data points with timestamp, time, average, min, max
status — Threshold classification: critical (>80% CPU, >85% memory/disk), warning (>50% CPU/memory, >70% disk), normal (below warning), unknown (no data)
Workflow
- Resolve region, cluster ID, and credentials using the documented priority.
- Discover the AOM Prometheus instance from the CCE cluster add-on binding.
- Start with Pod/Node TopN or aggregation, then drill into a Pod, Node, component, or cloud resource.
- Keep PromQL scoped by
cluster="<cluster_id>"; add namespace, pod, or resource filters only to reduce noise.
- Use status classification as an investigation lead, then correlate anomalies with events or alarm history.
Verification
- Run
python3 scripts/huawei-cloud.py huawei_get_cce_pod_metrics_topN region=cn-north-4 cluster_id=<cluster-id> namespace=default top_n=5 to verify Pod metric queries
- Run
python3 scripts/huawei-cloud.py huawei_get_cce_node_metrics_topN region=cn-north-4 cluster_id=<cluster-id> top_n=5 to verify Node metric queries
- Run
python3 scripts/huawei-cloud.py huawei_get_ecs_metrics region=cn-north-4 instance_id=<instance-id> to verify CES metric connectivity
Best Practices
- Start with Pod/Node TopN before drilling into individual resources.
- Keep
hours small (1-4) for recent analysis; cap historical reviews at 24 hours.
- Provide
namespace to reduce Pod noise while preserving the cluster filter.
- Focus on
critical and warning resources first.
- Use
huawei_cce_cluster_monitoring_aggregation for full-cluster health checks.
- Correlate metric anomalies with
huawei-cloud-cce-kubernetes-event-analyzer.
- Do not expose production Pod names, node IPs, or cluster IDs in public summaries.
Notes
- This skill is strictly read-only and never modifies resources or configurations.
- Thresholds are predefined baselines; tune them against workload SLOs before making operational decisions.
- AK/SK must never be hardcoded; use hcloud profile for normal hcloud calls or environment fallback for signed AOM/Kubernetes calls.
scripts/huawei-cloud.py is the only user-facing execution method
- AOM Prometheus instance is auto-discovered; no need to manually specify
aom_instance_id
- Cloud resource metrics (ECS/ELB/EIP/NAT) use CES (Cloud Eye Service), not AOM
- Do not make automatic scaling or remediation decisions based solely on metric analysis.
Troubleshooting
| Pitfall |
Symptom |
Quick Fix |
Missing cluster_id |
Action fails immediately |
Provide cluster_id from cluster listing |
| AOM Prometheus instance not found |
Metric queries return empty results |
Ensure AOM Prom instance is created for the cluster; check aom:instance:list permission |
| Large time window without namespace filter |
Slow response, too many results |
Narrow hours to 1-4 and add namespace filter |
| Cloud resource ID not found |
ECS/ELB/EIP/NAT query returns error |
Verify resource ID and CES IAM permission |
| Custom PromQL syntax error |
Custom query returns empty |
Use default PromQL unless familiar with AOM PromQL |
| Aggregation missing time range |
start_time / end_time missing |
Provide both time boundaries |
Limitations
- AOM Prometheus data requires the cluster Prometheus add-on to be integrated with AOM.
- Control-plane ServiceMonitors and component PodMonitors must be enabled before related metrics appear.
- Query results reflect collected monitoring data only; missing series are not proof that the workload is healthy.
- This skill does not remediate, scale, restart, create, update, or delete cloud or Kubernetes resources.
References
| Document |
Description |
| Workflow |
Metric query sequence, threshold detection, next-step handoff |
| Risk Rules |
Read-only constraints, data redaction, time-bounding, threshold caveats |
| Output Schema |
JSON response schema for metric and status output |
| CLI Installation Guide |
hcloud, kubectl, kubectl-cce, and dispatcher setup |
| IAM Policies |
Required read-only Huawei Cloud and Kubernetes permissions |
| Verification Method |
Static checks and smoke tests |
| Acceptance Criteria |
Functional, security, documentation, and quality gates |
1---
2id: huawei-cloud-cce-metric-analyzer
3name: huawei-cloud-cce-metric-analyzer
4description: |
5 Huawei Cloud CCE Metric analysis skill using the Python dispatcher with hcloud-backed cloud service queries.
6 Use this skill when the user wants to: (1) query Pod/Node/CoreDNS/nginx-ingress/autoscaler/control-plane CPU, memory, disk, QPS, latency, request, connection, certificate, scaling, or error-rate metrics, (2) get resource usage TopN rankings, (3) query ECS/ELB/EIP/NAT cloud resource metrics, (4) aggregate cluster monitoring data with anomaly detection, (5) detect threshold-based resource anomalies.
7 Trigger: user mentions "metric analysis", "指标分析", "CCE metrics", "CCE 指标", "AOM metrics", "AOM 指标", "CoreDNS metrics", "CoreDNS 指标", "nginx ingress metrics", "nginx-ingress 指标", "autoscaler metrics", "autoscaler 指标", "HPA metrics", "HPA 指标", "apiserver metrics", "etcd metrics", "controller manager metrics", "scheduler metrics", "control plane metrics", "控制面指标", "certificate expiration", "证书过期", "resource metrics", "资源指标", "CPU usage", "CPU 使用率", "memory usage", "内存使用率", "performance monitoring", "性能监控", "TopN", "resource ranking", "资源排名"
8tags: [cce, metrics, aom, observability, analysis]
9---
10
11# Huawei Cloud CCE Metric Analyzer
12
13## Overview
14
15Query and analyze metrics for CCE clusters (Pod/Node CPU/memory/disk) and cloud resources (ECS, ELB, EIP, NAT). Supports threshold-based anomaly detection, status classification (critical/warning/normal), and full-cluster monitoring aggregation.
16
17**Architecture**: `python3 scripts/huawei-cloud.py` dispatcher → hcloud (KooCLI) cloud service queries + signed AOM Prometheus HTTP queries + limited kubectl reads only when Kubernetes resource relationships are required → Pod/Node metrics, ECS/ELB/EIP/NAT metrics → Threshold classification → Anomaly detection
18
19> **Execution method**: Cloud service queries are executed through the local `hcloud` CLI. AOM Prometheus `query_range` calls are the only exception and use signed HTTPS requests because the required Prometheus range-query path is not compatible with hcloud. Do not call Huawei Cloud SDKs, curl IAM flows, openstack, or hand-written cloud APIs outside the bundled dispatcher.
20
21**Related Skills**: use pod/node diagnosers, Kubernetes event analyzer, capacity/cost skills, or auto-remediation runner for follow-up diagnosis or explicitly requested remediation.
22
23**Capabilities**:
24- Pod CPU/memory TopN ranking and single Pod time-series metrics
25- Node CPU/memory/disk TopN ranking and single Node time-series metrics
26- Node GPU and xGPU metrics, including GPU utilization, memory, temperature, power, schedule policy, xGPU allocation, usage, and health
27- CoreDNS QPS, error rate excluding NXDOMAIN, NXDOMAIN rate, P95 latency, replica count, and per-Pod CPU/memory metrics
28- nginx-ingress QPS, 4xx/5xx rate, success rate, P95 latency, active connections, per-Pod CPU/memory, and Ingress TLS certificate expiration status
29- Autoscaler unschedulable Pods, node state count, scale-up/down events, errors, node groups, HPA current/desired replicas, and per-Pod CPU/memory metrics
30- Kubernetes control-plane metrics for apiserver, etcd, controller-manager, and scheduler
31- ECS instance CPU/memory/disk/network metrics
32- ELB connection, bandwidth, QPS metrics
33- EIP bandwidth, traffic, packet loss metrics
34- NAT Gateway SNAT connection metrics
35- Full-cluster monitoring aggregation with anomaly detection (80% threshold)
36- Threshold-based status classification (critical/warning/normal/unknown)
37
38**Typical Use Cases**: query Pod/Node TopN, GPU/xGPU, CoreDNS, nginx-ingress, autoscaler, control-plane, ECS/ELB/EIP/NAT metrics, full-cluster aggregation, and threshold-based anomaly detection.
39
40## Prerequisites
41
42### 1. Runtime Dependencies
43
44- Python 3.8+ for the dispatcher and result processing
45- hcloud (KooCLI) 7.2.2+ for CCE/ECS/ELB/VPC/EIP/NAT/CES/IAM cloud service queries
46- `kubectl` only for Kubernetes resource reads that cannot be derived from AOM/hcloud, such as Pod `label_selector` filtering, Ingress TLS certificate checks, and LoadBalancer Service discovery for ELB/EIP association; clusters without external EIP require the `kubectl cce` plugin from `kubectl-cce-plugin/README.md`
47- Prometheus-related monitoring data is queried from AOM Prometheus with signed HTTPS requests; the cluster must have the Prometheus add-on integrated with AOM, otherwise these tools may return empty metric series
48- Controller-manager, scheduler, and etcd metrics require the `kube-controller-manager`, `kube-scheduler`, and `etcd-server` ServiceMonitors to be enabled separately in AOM; otherwise these tools may return empty metric series
49- Autoscaler, ingress-controller, and NVIDIA GPU metrics require the corresponding `autoscaler`, `ingress-controller`, and `nvidia-gpu-device-plugin` PodMonitors to be enabled separately in AOM; ingress request metrics also require `nginx_ingress_controller_requests` to be explicitly allowed in the ingress-controller PodMonitor
50- Run environment check before first use (see Verification section)
51- Install and use `kubectl-cce` according to [references/kubectl-cce.md](references/kubectl-cce.md)
52
53### 2. Credential Configuration
54
55- Valid Huawei Cloud credentials via hcloud profile or AK/SK mode
56- **Security Rules**:
57 - 🚫 Never expose AK/SK values in code, conversation, or commands
58 - 🚫 Never use `echo $HUAWEI_AK` or `echo $HUAWEI_SK` to check credentials
59 - ✅ Credential priority for hcloud calls is: explicit tool parameters > local hcloud profile > environment variables
60 - ✅ AOM Prometheus signed HTTP and Kubernetes certificate setup cannot use encrypted hcloud profile material, so they use explicit tool parameters first and environment variables as the signing fallback
61 - ✅ Prefer IAM users over root account for cloud operations
62 - ✅ Enable MFA for sensitive operations
63
64**Configuration Method**:
65
66```bash
67hcloud configure list
68
69export HUAWEI_AK=<your-ak>
70export HUAWEI_SK=<your-sk>
71export HUAWEI_REGION=cn-north-4
72```
73
74### 3. IAM Permission Requirements
75
76| API Action | Permission | Purpose |
77| ------------------------------- | ------------------- | ------------------------------------------ |
78| `cce:cluster:get` | Get cluster | View CCE cluster details |
79| `aom:instance:list` | List AOM instances | Discover AOM Prometheus instance for metrics |
80| `aom:metricsData:get` | Get metrics data | Query Pod/Node CPU/memory/disk metrics |
81| `ces:metricsData:get` | Get CES metrics | Query ECS/ELB/EIP/NAT cloud resource metrics |
82| `ecs:cloudServers:list` | List ECS servers | Correlate ECS instance IDs |
83| `elb:loadbalancers:list` | List ELB instances | Correlate ELB IDs |
84| `vpc:eips:list` | List EIPs | Correlate EIP IDs |
85| `nat:natGateways:list` | List NAT Gateways | Correlate NAT Gateway IDs |
86
87**Permission Failure Handling**:
88
891. When any command fails due to IAM permission errors, display the required permission list
902. Guide the user to create a custom policy in the IAM console and grant authorization
913. Pause execution and wait for user confirmation that permissions have been granted
92
93## Core Commands
94
95All commands use the Python dispatcher script: `python3 scripts/huawei-cloud.py <action> <key=value>...`
96
97## KooCLI命令格式标准
98
99Do not ask users to run raw `hcloud` commands directly. Use the dispatcher format:
100
101```bash
102python3 scripts/huawei-cloud.py <tool-name> key=value key=value
103```
104
105The dispatcher converts cloud service queries to KooCLI calls. AOM Prometheus range queries use signed HTTPS requests because that path is not compatible with hcloud. Avoid Kubernetes resource reads unless the tool explicitly needs Pod labels, Ingress TLS Secrets, or LoadBalancer Services. Quote values containing spaces, `>`, `<`, `|`, JSON, or PromQL; never print or persist AK/SK, security tokens, kubeconfig files, or temporary payloads; keep Kubernetes/AOM PromQL scoped with `cluster="<cluster_id>"`.
106
107### 1. CCE Pod Metrics
108
109```bash
110# Pod TopN — cluster-wide CPU/memory ranking
111python3 scripts/huawei-cloud.py huawei_get_cce_pod_metrics_topN \
112 region=cn-north-4 cluster_id=<cluster-id> \
113 namespace=default top_n=10 hours=1
114
115# Pod TopN with label selector
116python3 scripts/huawei-cloud.py huawei_get_cce_pod_metrics_topN \
117 region=cn-north-4 cluster_id=<cluster-id> \
118 namespace=default label_selector="app=nginx,version=v1" top_n=10 hours=1
119
120# Single Pod time-series
121python3 scripts/huawei-cloud.py huawei_get_cce_pod_metrics \
122 region=cn-north-4 cluster_id=<cluster-id> \
123 pod_name=my-app-xxx namespace=default hours=1
124
125# Single Pod GPU and xGPU metrics
126python3 scripts/huawei-cloud.py huawei_get_cce_pod_gpu_metrics \
127 region=cn-north-4 cluster_id=<cluster-id> \
128 pod_name=my-gpu-app-xxx namespace=default hours=1
129```
130
131### 2. CCE Node Metrics
132
133```bash
134# Node TopN — cluster-wide CPU/memory/disk ranking
135python3 scripts/huawei-cloud.py huawei_get_cce_node_metrics_topN \
136 region=cn-north-4 cluster_id=<cluster-id> \
137 top_n=10 hours=1
138
139# Single Node time-series
140python3 scripts/huawei-cloud.py huawei_get_cce_node_metrics \
141 region=cn-north-4 cluster_id=<cluster-id> \
142 node_ip=10.0.0.1 hours=1
143
144# Node GPU and xGPU metrics
145python3 scripts/huawei-cloud.py huawei_get_cce_node_gpu_metrics \
146 region=cn-north-4 cluster_id=<cluster-id> \
147 node_ip=10.0.0.1 hours=1
148```
149
150### 3. CCE CoreDNS Metrics
151
152```bash
153# CoreDNS key metrics: QPS, error rate excluding NXDOMAIN, NXDOMAIN rate, P95 latency, replicas, CPU, and memory
154python3 scripts/huawei-cloud.py huawei_get_cce_coredns_metrics \
155 region=cn-north-4 cluster_id=<cluster-id> \
156 namespace=kube-system pod_regex=".*coredns.*" hours=1
157```
158
159### 4. CCE nginx-ingress Metrics
160
161```bash
162# nginx-ingress request processing and Ingress TLS certificate expiration
163python3 scripts/huawei-cloud.py huawei_get_cce_nginx_ingress_metrics \
164 region=cn-north-4 cluster_id=<cluster-id> \
165 namespace=kube-system pod_regex=".*nginx.*ingress.*|.*ingress.*nginx.*" \
166 ingress_namespace=default cert_expire_warning_days=30 hours=1
167```
168
169### 5. CCE Autoscaler Metrics
170
171```bash
172# Cluster Autoscaler and HPA metrics
173python3 scripts/huawei-cloud.py huawei_get_cce_autoscaler_metrics \
174 region=cn-north-4 cluster_id=<cluster-id> \
175 namespace=kube-system pod_regex=".*cluster.*autoscaler.*|.*autoscaler.*" \
176 include_hpa=true hours=1
177```
178
179### 6. Kubernetes Control Plane Metrics
180
181```bash
182python3 scripts/huawei-cloud.py huawei_get_cce_apiserver_metrics \
183 region=cn-north-4 cluster_id=<cluster-id> hours=1
184
185python3 scripts/huawei-cloud.py huawei_get_cce_etcd_metrics \
186 region=cn-north-4 cluster_id=<cluster-id> hours=1
187
188python3 scripts/huawei-cloud.py huawei_get_cce_controller_manager_metrics \
189 region=cn-north-4 cluster_id=<cluster-id> namespace=kube-system hours=1
190
191python3 scripts/huawei-cloud.py huawei_get_cce_scheduler_metrics \
192 region=cn-north-4 cluster_id=<cluster-id> namespace=kube-system hours=1
193```
194
195### 7. Cloud Resource Metrics
196
197```bash
198# ECS instance metrics
199python3 scripts/huawei-cloud.py huawei_get_ecs_metrics \
200 region=cn-north-4 instance_id=<instance-id>
201
202# ELB metrics
203python3 scripts/huawei-cloud.py huawei_get_elb_metrics \
204 region=cn-north-4 elb_id=<loadbalancer-id> hours=1
205
206# EIP metrics
207python3 scripts/huawei-cloud.py huawei_get_eip_metrics \
208 region=cn-north-4 eip_id=<eip-id> hours=1
209
210# NAT Gateway metrics
211python3 scripts/huawei-cloud.py huawei_get_nat_gateway_metrics \
212 region=cn-north-4 nat_gateway_id=<nat-gateway-id> hours=1
213```
214
215### 8. Cluster Monitoring Aggregation
216
217```bash
218# Aggregate all monitoring data with anomaly detection
219python3 scripts/huawei-cloud.py huawei_cce_cluster_monitoring_aggregation \
220 region=cn-north-4 cluster_id=<cluster-id> \
221 start_time="2026-05-30 00:00:00" end_time="2026-05-30 23:59:59" \
222 namespace=default top_n=10
223```
224
225This tool aggregates: Pod TopN CPU/memory, Node TopN CPU/memory/disk, ELB metrics (matched through LoadBalancer Services fetched by `kubectl`), NAT Gateway metrics, EIP metrics (bandwidth, packet loss), and anomaly detection using 80% threshold.
226
227It also includes CoreDNS, nginx-ingress, and autoscaler summaries. Cloud resources are scoped to the current cluster when an association can be proven: ELB is matched through LoadBalancer Service IP/EIP, NAT Gateway is filtered by the cluster VPC, and EIP is limited to associated ELB/NAT/Service IPs.
228
229LoadBalancer Service discovery uses `kubectl` with generated kubeconfig through the cluster EIP when external access is available. If the cluster has no EIP, it uses the `kubectl cce` plugin. If neither path works, aggregation fails.
230
231## Risk Levels
232
233This skill is read-only. It does not create, update, delete, restart, scale, or modify Huawei Cloud or Kubernetes resources.
234
235| Level | Meaning | Execution Guidance |
236| ----- | ------- | ------------------ |
237| R3 | No-risk read-only query or local analysis | May run automatically |
238| R2 | Low-risk change, such as creating monitoring configuration without deleting resources or increasing service capacity/cost | Not used by current tools |
239| R1 | Risky operation, such as restart-like impact, disabling protection, or changes that may increase cost or reduce observability | Not used by current tools |
240| R0 | Critical operation, such as deleting clusters, applications, or broad-impact monitoring protections | Not used by current tools |
241
242| Tool | Operation Type | Risk Level | Description |
243| ---- | -------------- | ---------- | ----------- |
244| `huawei_get_cce_pod_metrics_topN` | Query | R3 | Read Pod CPU/memory/disk TopN metrics from AOM Prometheus |
245| `huawei_get_cce_pod_metrics` | Query | R3 | Read single Pod CPU/memory/disk time-series metrics |
246| `huawei_get_cce_node_metrics_topN` | Query | R3 | Read Node CPU/memory/disk TopN metrics from AOM Prometheus |
247| `huawei_get_cce_node_metrics` | Query | R3 | Read single Node CPU/memory/disk time-series metrics |
248| `huawei_get_cce_node_gpu_metrics` | Query | R3 | Read single Node GPU and xGPU metrics from AOM Prometheus |
249| `huawei_get_cce_pod_gpu_metrics` | Query | R3 | Read single Pod GPU and xGPU metrics from AOM Prometheus |
250| `huawei_get_cce_coredns_metrics` | Query | R3 | Read CoreDNS QPS, error rate excluding NXDOMAIN, NXDOMAIN rate, P95 latency, replicas, and per-Pod CPU/memory metrics |
251| `huawei_get_cce_nginx_ingress_metrics` | Query | R3 | Read nginx-ingress request-processing metrics and Ingress TLS certificate expiration status; QPS falls back to nginx process request counters when request-dimension metrics are absent |
252| `huawei_get_cce_autoscaler_metrics` | Query | R3 | Read Cluster Autoscaler scaling metrics, HPA replica state, and autoscaler Pod CPU/memory metrics |
253| `huawei_get_cce_apiserver_metrics` | Query | R3 | Read kube-apiserver QPS, error rate, latency, and inflight request metrics |
254| `huawei_get_cce_etcd_metrics` | Query | R3 | Read etcd leader, proposal, DB size, disk latency, CPU, and memory metrics |
255| `huawei_get_cce_controller_manager_metrics` | Query | R3 | Read control-plane workqueue depth, adds, retries, queue latency, and work duration metrics |
256| `huawei_get_cce_scheduler_metrics` | Query | R3 | Read scheduler attempts, pending Pods, scheduling latency, and queue metrics |
257| `huawei_get_ecs_metrics` | Query | R3 | Read ECS monitoring data through hcloud/CES |
258| `huawei_get_elb_metrics` | Query | R3 | Read ELB monitoring data through hcloud/CES |
259| `huawei_get_eip_metrics` | Query | R3 | Read EIP monitoring data through hcloud/CES |
260| `huawei_get_nat_gateway_metrics` | Query | R3 | Read NAT Gateway monitoring data through hcloud/CES |
261| `huawei_cce_cluster_monitoring_aggregation` | Query + local analysis | R3 | Aggregate Pod/Node/cloud-resource metrics and classify anomalies locally |
262
263## Parameter Reference
264
265### Common Parameters
266
267| Parameter | Required/Optional | Description | Default |
268| ------------ | ----------------- | -------------------- | --------------- |
269| `region` | Required | Huawei Cloud region | `HUAWEI_REGION` |
270| `cluster_id` | Required | CCE cluster ID | N/A |
271| `namespace` | Recommended | Kubernetes namespace | `default` |
272| `ak` | Optional | Explicit AK; highest priority for all calls | profile/env fallback |
273| `sk` | Optional | Explicit SK; highest priority for all calls | profile/env fallback |
274| `project_id` | Optional | Explicit Project ID; hcloud uses profile before env fallback | Auto from IAM/profile |
275
276### `huawei_get_cce_pod_metrics_topN` Parameters
277
278| Parameter | Required | Description | Default |
279| --------------- | -------- | ------------------------------- | -------- |
280| `namespace` | No | Namespace filter | all |
281| `label_selector`| No | Label selector (e.g. app=web) | N/A |
282| `top_n` | No | Number of top items | 10 |
283| `hours` | No | Metrics lookback hours | 1 |
284| `node_ip` | No | Filter Pods on specific node | N/A |
285| `cpu_query` | No | Custom CPU PromQL | Auto |
286| `memory_query` | No | Custom memory PromQL | Auto |
287| `disk_query` | No | Custom disk PromQL | Auto |
288
289### `huawei_get_cce_pod_metrics` Parameters
290
291| Parameter | Required | Description | Default |
292| ------------ | -------- | -------------------------- | -------- |
293| `pod_name` | Yes | Target Pod name | N/A |
294| `namespace` | No | Namespace | `default`|
295| `hours` | No | Metrics lookback hours | 1 |
296| `cpu_query` | No | Custom CPU PromQL | Auto |
297| `memory_query` | No | Custom memory PromQL | Auto |
298| `disk_query` | No | Custom disk PromQL | Auto |
299
300### `huawei_get_cce_pod_gpu_metrics` Parameters
301
302| Parameter | Required | Description | Default |
303| --------- | -------- | ----------- | ------- |
304| `pod_name` | Yes | Target Pod name | N/A |
305| `namespace` | No | Target Pod namespace | all |
306| `hours` | No | Metrics lookback hours | 1 |
307| `gpu_selector` | No | Custom GPU metric label selector. Use this when GPU metrics do not use the `pod` or `namespace` labels | `pod="<pod_name>",namespace="<namespace>"` |
308
309Optional custom PromQL overrides are supported for GPU utilization, memory, schedule policy, xGPU allocation/usage, and xGPU health metrics.
310
311### `huawei_get_cce_node_metrics_topN` Parameters
312
313| Parameter | Required | Description | Default |
314| ----------- | -------- | --------------------------- | -------- |
315| `top_n` | No | Number of top items | 10 |
316| `hours` | No | Metrics lookback hours | 1 |
317
318### `huawei_get_cce_node_metrics` Parameters
319
320| Parameter | Required | Description | Default |
321| ---------- | -------- | --------------------------- | -------- |
322| `node_ip` | Yes | Target Node IP | N/A |
323| `hours` | No | Metrics lookback hours | 1 |
324
325### `huawei_get_cce_node_gpu_metrics` Parameters
326
327| Parameter | Required | Description | Default |
328| --------- | -------- | ----------- | ------- |
329| `node_ip` | Yes | Target Node IP or node name | N/A |
330| `hours` | No | Metrics lookback hours | 1 |
331| `gpu_selector` | No | Custom GPU metric label selector. Use this when GPU metrics do not use the `node` label | `node=~"<node_ip>|<node_name>"` |
332
333Optional custom PromQL overrides are supported for GPU utilization, memory, temperature, power, schedule policy, xGPU allocation/usage, and xGPU health metrics.
334
335### `huawei_get_cce_coredns_metrics` Parameters
336
337| Parameter | Required | Description | Default |
338| --------- | -------- | ----------- | ------- |
339| `namespace` | No | CoreDNS namespace | `kube-system` |
340| `pod_regex` | No | Regex used to match CoreDNS Pods | `.*coredns.*` |
341| `hours` | No | Metrics lookback hours | 1 |
342
343Optional custom PromQL overrides are supported for QPS, error rate, NXDOMAIN rate, P95 latency, CPU, memory, and replica count.
344
345### `huawei_get_cce_nginx_ingress_metrics` Parameters
346
347| Parameter | Required | Description | Default |
348| --------- | -------- | ----------- | ------- |
349| `namespace` | No | Namespace of nginx-ingress controller Pods. Use an empty value to query all namespaces | `kube-system` |
350| `pod_regex` | No | Regex used to match nginx-ingress controller Pods | `.*nginx.*ingress.*|.*ingress.*nginx.*` |
351| `ingress_namespace` | No | Namespace filter for Ingress TLS certificate checks | all |
352| `hours` | No | Metrics lookback hours | 1 |
353| `cert_expire_warning_days` | No | Days before expiry to mark certificates as warning | 30 |
354| `check_certificates` | No | Whether to inspect Ingress TLS Secrets for expiration status | true |
355
356Ingress-controller metrics depend on the corresponding AOM PodMonitor. The `nginx_ingress_controller_requests` metric must be explicitly allowed in the ingress-controller PodMonitor; otherwise request-dimension metrics such as 4xx/5xx QPS, success rate, and latency may be empty, and QPS may only use the `nginx_ingress_controller_nginx_process_requests_total` fallback when available.
357
358Optional custom PromQL overrides are supported for QPS, 4xx/5xx, success rate, P95 latency, active connections, CPU, and memory.
359
360### `huawei_get_cce_autoscaler_metrics` Parameters
361
362| Parameter | Required | Description | Default |
363| --------- | -------- | ----------- | ------- |
364| `namespace` | No | Namespace of Cluster Autoscaler Pods. Use an empty value to query all namespaces | `kube-system` |
365| `pod_regex` | No | Regex used to match autoscaler Pods | `.*cluster.*autoscaler.*|.*autoscaler.*` |
366| `hpa_namespace` | No | Namespace filter for HPA replica metrics | all |
367| `hours` | No | Metrics lookback hours | 1 |
368| `include_hpa` | No | Whether to query HPA current/desired replica metrics | true |
369
370Optional custom PromQL overrides are supported for unschedulable Pods, node states, scale events, errors, node groups, HPA replicas, CPU, and memory.
371
372### Kubernetes Control Plane Tool Parameters
373
374Applies to `huawei_get_cce_apiserver_metrics`, `huawei_get_cce_etcd_metrics`, `huawei_get_cce_controller_manager_metrics`, and `huawei_get_cce_scheduler_metrics`.
375
376`huawei_get_cce_apiserver_metrics` defaults to `cluster="<cluster_id>",component="apiserver"` and does not add namespace or Pod labels. Its default P95 latency excludes `WATCH|CONNECT` requests and also returns `latency_p95_by_verb_ms` for diagnosis. Use `metric_selector` only when the Prometheus labels differ.
377
378`huawei_get_cce_etcd_metrics` defaults to `cluster="<cluster_id>"` and does not add namespace or Pod labels. Use `metric_selector` only when the Prometheus labels differ.
379
380`huawei_get_cce_controller_manager_metrics` defaults to `cluster="<cluster_id>"` because CCE AOM workqueue metrics may not expose stable controller-manager Pod labels. It returns both aggregate workqueue metrics and per-queue `name` breakdowns.
381
382`huawei_get_cce_scheduler_metrics` defaults to `cluster="<cluster_id>"` and returns aggregate metrics plus `result`, `profile/result`, and `queue` breakdowns.
383
384Controller-manager, scheduler, and etcd metrics depend on AOM ServiceMonitor collection being enabled for the corresponding `kube-controller-manager`, `kube-scheduler`, and `etcd-server` endpoints. If ServiceMonitor is not enabled, the tools can run successfully but return empty series.
385
386| Parameter | Required | Description | Default |
387| --------- | -------- | ----------- | ------- |
388| `namespace` | No | Namespace of control-plane Pods. Use an empty value to query all namespaces | `kube-system` |
389| `pod_regex` | No | Regex used to match target component Pods | component-specific |
390| `metric_selector` | No | Custom apiserver/etcd/controller-manager/scheduler metric label selector | apiserver: `cluster="<cluster_id>",component="apiserver"`; etcd/controller-manager/scheduler: `cluster="<cluster_id>"` |
391| `hours` | No | Metrics lookback hours | 1 |
392
393### Cloud Resource Tool Parameters
394
395| Tool | Required ID Parameter | Optional Parameters |
396| ---- | --------------------- | ------------------- |
397| `huawei_get_ecs_metrics` | `instance_id` | none |
398| `huawei_get_elb_metrics` | `elb_id` | `hours` |
399| `huawei_get_eip_metrics` | `eip_id` | `hours` |
400| `huawei_get_nat_gateway_metrics` | `nat_gateway_id` | `hours` |
401
402### `huawei_cce_cluster_monitoring_aggregation` Parameters
403
404| Parameter | Required | Description | Default |
405| ------------- | -------- | ------------------------------- | -------- |
406| `start_time` | Yes | Start time (YYYY-MM-DD HH:MM:SS)| N/A |
407| `end_time` | Yes | End time (YYYY-MM-DD HH:MM:SS) | N/A |
408| `namespace` | No | Namespace filter | `default`|
409| `top_n` | No | Number of top items | 10 |
410| `security_token` | No | Temporary security token for AK/SK session credentials | env fallback |
411
412## Output Format
413
414See [Output Schema](references/output-schema.md) for the complete JSON response structure.
415
416**Key output fields**:
417- `success` — boolean, true if query completed
418- `region` — Huawei Cloud region
419- `cluster_id` / `cluster_name` — CCE cluster identity
420- `aom_instance_id` — AOM Prometheus instance used for metric queries
421- `metrics` — Dict with cpu/memory/disk data per resource, including status classification
422- `certificate_check` — nginx-ingress Ingress TLS certificate expiration summary when certificate checking is enabled
423- `time_series` — Historical data points with `timestamp`, `time`, `average`, `min`, `max`
424- `status` — Threshold classification: `critical` (>80% CPU, >85% memory/disk), `warning` (>50% CPU/memory, >70% disk), `normal` (below warning), `unknown` (no data)
425
426## Workflow
427
4281. Resolve region, cluster ID, and credentials using the documented priority.
4292. Discover the AOM Prometheus instance from the CCE cluster add-on binding.
4303. Start with Pod/Node TopN or aggregation, then drill into a Pod, Node, component, or cloud resource.
4314. Keep PromQL scoped by `cluster="<cluster_id>"`; add namespace, pod, or resource filters only to reduce noise.
4325. Use status classification as an investigation lead, then correlate anomalies with events or alarm history.
433
434## Verification
435
4361. Run `python3 scripts/huawei-cloud.py huawei_get_cce_pod_metrics_topN region=cn-north-4 cluster_id=<cluster-id> namespace=default top_n=5` to verify Pod metric queries
4372. Run `python3 scripts/huawei-cloud.py huawei_get_cce_node_metrics_topN region=cn-north-4 cluster_id=<cluster-id> top_n=5` to verify Node metric queries
4383. Run `python3 scripts/huawei-cloud.py huawei_get_ecs_metrics region=cn-north-4 instance_id=<instance-id>` to verify CES metric connectivity
439
440## Best Practices
441
4421. Start with Pod/Node TopN before drilling into individual resources.
4432. Keep `hours` small (1-4) for recent analysis; cap historical reviews at 24 hours.
4443. Provide `namespace` to reduce Pod noise while preserving the cluster filter.
4454. Focus on `critical` and `warning` resources first.
4465. Use `huawei_cce_cluster_monitoring_aggregation` for full-cluster health checks.
4476. Correlate metric anomalies with `huawei-cloud-cce-kubernetes-event-analyzer`.
4487. Do not expose production Pod names, node IPs, or cluster IDs in public summaries.
449
450## Notes
451
452- This skill is strictly read-only and never modifies resources or configurations.
453- Thresholds are predefined baselines; tune them against workload SLOs before making operational decisions.
454- AK/SK must never be hardcoded; use hcloud profile for normal hcloud calls or environment fallback for signed AOM/Kubernetes calls.
455- `scripts/huawei-cloud.py` is the only user-facing execution method
456- AOM Prometheus instance is auto-discovered; no need to manually specify `aom_instance_id`
457- Cloud resource metrics (ECS/ELB/EIP/NAT) use CES (Cloud Eye Service), not AOM
458- Do not make automatic scaling or remediation decisions based solely on metric analysis.
459
460## Troubleshooting
461
462| Pitfall | Symptom | Quick Fix |
463| ------------------------------------------ | ------------------------------------- | -------------------------------------------- |
464| Missing `cluster_id` | Action fails immediately | Provide `cluster_id` from cluster listing |
465| AOM Prometheus instance not found | Metric queries return empty results | Ensure AOM Prom instance is created for the cluster; check `aom:instance:list` permission |
466| Large time window without namespace filter | Slow response, too many results | Narrow `hours` to 1-4 and add `namespace` filter |
467| Cloud resource ID not found | ECS/ELB/EIP/NAT query returns error | Verify resource ID and CES IAM permission |
468| Custom PromQL syntax error | Custom query returns empty | Use default PromQL unless familiar with AOM PromQL |
469| Aggregation missing time range | `start_time` / `end_time` missing | Provide both time boundaries |
470
471## Limitations
472
473- AOM Prometheus data requires the cluster Prometheus add-on to be integrated with AOM.
474- Control-plane ServiceMonitors and component PodMonitors must be enabled before related metrics appear.
475- Query results reflect collected monitoring data only; missing series are not proof that the workload is healthy.
476- This skill does not remediate, scale, restart, create, update, or delete cloud or Kubernetes resources.
477
478## References
479| Document | Description |
480| -------- | ----------- |
481| [Workflow](references/workflow.md) | Metric query sequence, threshold detection, next-step handoff |
482| [Risk Rules](references/risk-rules.md) | Read-only constraints, data redaction, time-bounding, threshold caveats |
483| [Output Schema](references/output-schema.md) | JSON response schema for metric and status output |
484| [CLI Installation Guide](references/cli-installation-guide.md) | hcloud, kubectl, kubectl-cce, and dispatcher setup |
485| [IAM Policies](references/iam-policies.md) | Required read-only Huawei Cloud and Kubernetes permissions |
486| [Verification Method](references/verification-method.md) | Static checks and smoke tests |
487| [Acceptance Criteria](references/acceptance-criteria.md) | Functional, security, documentation, and quality gates |