Huawei Cloud Skill Tester

Huawei Cloud Skill Tester — Three-Track Eight-Phase E2E Testing Pipeline

Independent, repeatable Huawei Cloud Skill testing framework. Does not depend on skill-creator; can test any existing Huawei Cloud Skill. Focuses on real-environment functional testing and multi-skill orchestration scenarios.

Overview

This Skill provides a three-track, eight-phase standardized testing pipeline (three-tier layout × 8 phase total: Phase 0~6 执行 + Phase 7 终报告):

Core Design Principles

1. Chain Verification — Before each Phase, check that the previous phase's JSON exists; if missing, refuse to execute
2. Agent-proof — Write operations require user confirmation for each item; automatic gate bypassing is not allowed
3. Three-Track Layering — Clear gates between Tiers; Tier 1 must be completed before entering Tier 2
4. Batch Repeatable — Supports --skills "skill-a,skill-b" or --all-installed
5. Fallback Strategy — When only 1 skill, Phase 5/6 automatically downgrade to single-skill lifecycle testing. Sibling auto-scan is ON by default (Phase 5/6 自动从被测 skill 同目录找其他 huawei-cloud-* skill 做编排组合测试). Use --no-siblings to opt-out.
6. Standardized JSON Output — All phases output in a unified schema; Phase 7 merges into a single report
7. Real-Environment First — All Tier 2 orchestrations execute against real Huawei Cloud; no mocks or simulations

Data Flow Diagram

User Input (--skills or --all-installed)
    │
    ├── Tier 1 ──── Iterate over each skill ────
    │   Phase 0 → 1 → 2 → 3 → 4
    │   (phase-N-summary.json chain validation)
    │
    ├── Tier 2 ──── Integration ────
    │   Phase 5 (orchestration scenario derivation + real-environment execution)
    │   Phase 6 (e2e full-flow: create→query→update→delete lifecycle)
    │   Only 1 skill → Downgrade single-skill closed loop
    │
    └── Tier 3 ──── Final ────
        Phase 7 (merge phase-0~6 JSON into consolidated report)

Prerequisites

1. hcloud CLI installed and authenticated (for Tier 2 CLI mode testing) — Reference: https://support.huaweicloud.com/qs-hcli/hcli_02_003.html
2. Python 3.8+ + huaweicloudsdk packages (for SDK mode testing) — SDK Reference: https://console.huaweicloud.com/apiexplorer/#/sdkcenter
3. Huawei Cloud AK/SK — 自动扫描所有以 HUAWEI / HW / HWC 开头的环境变量（键名含 AK/SK 标记的键值对）。If missing, the framework emits the env-var setup template to stderr and exits 77. NEVER ask the user to type AK/SK in chat; user must set env vars in their shell profile out-of-band and re-run.（完整协议见 references/agent-protocol.md）
4. Target Skill must be under $SKILL_INSTALL_DIR/ (auto-detected: ~/.agents/skills/ → ~/.hermes/skills/ → default) or a user-specified path
5. jq command (all JSON processing depends on it)
6. API Reference: https://console.huaweicloud.com/apiexplorer/#/openapi

Workflow — Three-Track Eight-Phase

Tier 1: Single-Skill Unit Testing
   Phase 0: Installation Verification (install/uninstall/reinstall)
   Phase 1: Feature Extraction (metadata + commands + resource types + doc checks)
   Phase 2: Technical Research (CLI→SDK→API three-level availability)
   Phase 3: Test Case Generation (functional TC-F + boundary + negative + API TC-A)
   Phase 4: Real-Environment Execution (read-only automatic + write ops require confirmation)

Tier 2: Integration Testing — Real-Environment Orchestration
   Phase 5: Multi-Skill Orchestration (conflict scan / scenario derivation / self-check)
   Phase 6: End-to-End Flow (resource lifecycle: create→query→update→delete)

Tier 3: Final Report
   Phase 7: Consolidated Report (merge phase-0~6 JSON, verdict + issues + reasons)

各 Phase 的完整实现规范（步骤、判定标准、JSON 字段）见 references/phase-details.md。

Phases at a Glance

详细步骤、链式验证规则、JSON schema 见 references/phase-details.md 与 references/output-schema-spec.md。

KooCLI Command Format Standard

This testing framework uses bash scripts as the primary execution mode, not direct hcloud CLI commands. However, when executing test cases, the framework constructs hcloud CLI commands in the following format:

Format template — illustrative, NOT an executable command: hcloud <Service> <Operation> --cli-region={region} [--param1=value1 ...]

Format Rules:

For OBS service, the framework uses hcloud obs (obsutil) subsystem:

Format template — illustrative, NOT an executable command: hcloud obs <command> [args...] [options...]

Core Commands

Full Pipeline Run

# Specify skills
bash scripts/run-test-pipeline.sh --skills "huawei-cloud-bss-voucher-manage"

# Specify multiple skills (comma-separated)
bash scripts/run-test-pipeline.sh --skills "huawei-cloud-bss-voucher-manage, huawei-cloud-ecs-manage"

# Scan all installed
bash scripts/run-test-pipeline.sh --all-installed

# Start from a specific phase (recovery scenarios only)
bash scripts/run-test-pipeline.sh --skills "bss-voucher" --phase 4

# Fresh mode
bash scripts/run-test-pipeline.sh --skills "bss-voucher" --fresh

Single Phase Run (Debug)

bash scripts/tier1/phase-0-install-check.sh --skill "huawei-cloud-bss-voucher-manage"
bash scripts/tier1/phase-1-skill-analysis.sh --skill "huawei-cloud-bss-voucher-manage"
bash scripts/tier2/phase-5-orchestration.sh --skills "skill-a, skill-b"
bash scripts/tier3/phase-7-final-report.sh --skills "skill-a, skill-b"

Run Multi-Skill Orchestration

# Derive and execute orchestration scenarios for 3 skills
bash scripts/tier2/phase-5-orchestration.sh --skills "ecs-manage, vpc-manage, eip-manage"

# Run E2E lifecycle test for a single skill
bash scripts/tier2/phase-6-full-flow.sh --skill "huawei-cloud-rds-intelligent-service"

Parameters

Environment Variables (Advanced)

Quality Reporting

This Skill integrates skill_quality_sdk.py (vendored, zero third-party dependency) for execution quality reporting. Every run-test-pipeline.sh run automatically reports one record — skill name (huawei-cloud-skill-tester), status (success / sys_fail), error code, cost (seconds), skills under test, and output directory — to the skillsopr operations console, enabling usage/statistics counting of the tester itself.

Integration

• Bash entry point (scripts/run-test-pipeline.sh): the report_quality function is wired into the EXIT/INT/TERM trap, so every pipeline exit path reports:
  • exit 0 → status=success
  • exit 77 → status=sys_fail, error_code=C01 (AK/SK credentials missing)
  • any other non-zero exit → status=sys_fail, error_code=B01
• The report is fire-and-forget (background process, 3s HTTP timeout): reporting failure or latency never blocks, changes, or fails the testing pipeline.
• The SDK is Python 3 stdlib only; python3 is already a hard prerequisite.

Authentication & degraded reporting

• The SDK reads AK/SK with the same fallback chain as the tester itself (lib/utils.sh): HUAWEI_ACCESS_KEY / HUAWEI_SECRET_KEY first, then SKILL_QUALITY_AK/SK and other HUAWEI* / HW* / HWC* variants — so a tester run that has valid credentials always reports authenticated.
• When no AK/SK is available (e.g. the exit-77 credential-missing path), the SDK degrades to a direct anonymous POST to the APIG public endpoint and marks the payload unauthenticated=1 (default SKILL_QUALITY_ALLOW_ANONYMOUS=1). This keeps C01 / sys_fail events structurally reachable — they would otherwise be silently dropped because the IAM token (which itself needs credentials) can never be obtained. Set SKILL_QUALITY_ALLOW_ANONYMOUS=0 to restore the legacy skip-without-credentials behavior.

Error Code Convention

Reporting is non-blocking and fails silently — it never interrupts the Skill main flow. Disable via SKILL_QUALITY_DISABLE=1 for local testing.

References

核心文档（必读）

• references/architecture.md — Three-track eight-phase architecture diagram (Mermaid)
• references/output-schema-spec.md — Complete JSON field specification for each phase (including Phase 7 final report schema)
• references/phase-transition-rules.md — Phase transition / fallback / skip rules
• references/acceptance-criteria.md — Quality gates + 17-item report acceptance checklist
• references/verification-method.md — How to manually verify each phase (PowerShell + Git Bash)
• references/phase-details.md — 各 Phase 完整实现规范（步骤、判定标准、JSON 字段）
• references/agent-protocol.md — 凭证请求协议（AK/SK 缺失时的完整处理流程）
• scripts/skill_quality_sdk.py — Vendored execution-quality reporting SDK (see Quality Reporting)

配套参考

• references/cli-installation-guide.md — How to install and configure hcloud CLI (prerequisite for Phase 2/4)
• references/iam-policies.md — Minimum IAM permissions required to run the tester

模板（JSON Schema）

• templates/phase-report-schema.json — JSON Schema for phase-N-summary.json (N=0..6)
• templates/test-case-schema.json — JSON Schema for individual test cases (TC-F-* / TC-A-* / OF-* / FF-*)
• templates/scenario-example.json — Reference example for Phase 6 multi-skill scenario derivation

Output Format

All test artifacts go to a sibling directory of the tested skill, named <skill-name>-test-files/. This keeps the skill source dir clean and preserves run history for diff/regression.

skills/
├── huawei-cloud-rds-query/                  ← skill source (untouched)
└── huawei-cloud-rds-query-test-files/       ← test artifacts (created on first run, kept across runs)
    ├── phases/
    │   ├── phase-0-summary.json
    │   ├── phase-1-summary.json
    │   ├── ...
    │   └── phase-7-summary.json
    └── reports/
        ├── report-20260724-152309/          ← one subdir per run (timestamped)
        │   ├── test-report.json
        │   └── test-report.md
        ├── report-20260725-090000/
        │   ├── test-report.json
        │   └── test-report.md
        └── ...

• Phase 0~6 output phase-N-summary.json to <skill-name>-test-files/phases/
• Phase 7 merges them into <skill-name>-test-files/reports/report-<timestamp>/test-report.{json,md}
• Test artifacts are preserved across runs (no auto-cleanup, ever). --fresh does NOT delete — it archives old phases/*.json to phases/archive/<timestamp>/ so the chain check resets while history stays intact. reports/ is always kept.
• The test-installed copy of the skill in $SKILL_INSTALL_DIR/ is still uninstalled on exit, so the next run sees a clean install state.

See references/output-schema-spec.md for the JSON schema. Phase 5 and 6 additionally output scenario execution logs with real CLI/SDK responses for auditability.

Best Practices

• Complete Tier 1 before entering Tier 2 to ensure skills are individually functional before orchestration
• Confirm write operations one by one in Phase 4 and Phase 5/6; do not batch-confirm to avoid misoperations
• With only 1 skill, Phase 5/6 automatically downgrade to single-skill closed loop; no need to manually skip
• When using --fresh to reset and rerun, confirm there are no uncleaned test resources
• Review orchestration scenarios before execution to ensure resource dependency order is correct

Notes

• Three-track eight-phase strictly follows sequential order; chain verification prevents skipping
• API endpoints are strictly prohibited from being inferred; only obtain from SDK _http_info or API Explorer
• Credentials are read from environment variables; hardcoding is prohibited
• If AK/SK is missing, the framework emits the env-var setup template to stderr and exits 77. The Agent MUST output that template to the user verbatim and instruct them to set env vars out-of-band (in their shell / PowerShell $PROFILE). The Agent MUST NEVER ask the user to type or paste AK/SK in chat. Strictly prohibited from silently skipping any step that requires credentials.（完整协议见 references/agent-protocol.md）
• Resources created during testing must be tracked; if any are left behind, output manual cleanup instructions
• Orchestration scenarios are auto-derived; user should review and confirm before execution
• Write operations in orchestration scenarios require per-step user confirmation

Edge Cases

Agent Protocol — Credential Request

When Phase 4 or Phase 6 needs to call live Huawei Cloud APIs but cannot find credentials in the environment, the framework does not silently skip. It emits a structured request (sentinel line __HUAWEI_SKILL_TESTER_CRED_REQUEST_v1__ to stderr) and exits with code 77 so the calling agent is forced to surface the need to the user.

核心规则（MUST follow）:

1. 检测到 exit 77 或 sentinel → 暂停流水线,不得跳过 Phase 5/7
2. 将 stderr 中的 env-var 设置模板原样输出给用户,引导用户在 shell profile / PowerShell $PROFILE 中带外配置(占位符 <your-access-key-id> / <your-secret-access-key>)
3. 禁止在对话中索要 AK/SK 明文(ask_user / read -p / 剪贴板回传 / 读取 ~/.hcloud/config.json 等路径)
4. 用户带外设置后,重跑失败 phase;用户拒绝则明确标记"live phases skipped — no credentials",不得标记 pass

完整协议(6 步响应流程、直接终端模式、示例行为)见 references/agent-protocol.md。

Design Principles

• Chain Verification — Each Phase checks the previous phase's JSON to prevent skipping
• Agent-proof — Write operations must be confirmed by the user; fake confirmations are not allowed
• Data-Driven — All phases output in JSON format; Phase 7 merges
• Batch Repeatable — The same set of skills can be tested repeatedly; --fresh resets
• Real-Environment First — All orchestrations and E2E flows execute against real Huawei Cloud; no mocks
• Degrade Without Losing Value — Single skill does not run empty orchestration phases; degrades to meaningful single-skill lifecycle tests
• Resource Safety — Resources created during testing must be tracked; if any remain, output clear manual cleanup instructions
• Credentials Mandatory — If AK/SK is missing, the framework emits the env-var setup template to stderr and exits 77. The Agent MUST output that template to the user and instruct them to set env vars out-of-band. The Agent MUST NEVER ask the user to type or paste AK/SK in chat. Strictly prohibited from silently skipping any step that requires credentials.