| name | huawei-cloud-skill-tester |
| version | 1.5.0 |
| description | 1. Six-phase: Install (Phase 1), Basic (Phase 2), Combination (Phase 3), Solution (Phase 4), Functional (Phase 5) — with Phase 0 Feature Verification upfront (CLI→SDK→API→prompt user)
2. Lifecycle mode (--lifecycle): Create→Poll→Delete with AK/SK prompt, cost confirm, auto cleanup
3. SDK/API fallback: non-CLI skills auto-parse Core Commands table, generate SDK/API test cases
4. Bash script commands: parse bash scripts/ from Core Commands, substitute {path}, execute
5. Interactive questioning test: auto-detect Socratic 4-dimension flow, validate questioning patterns, one-at-a-time, summary table, user confirmation
6. Capability detection: read workflow section, detect scaffold/create, test with temp skill from templates
7. Phase 0 Feature Verification: read target skill features, list for user confirmation before testing
Triggers include: "测试Skill","Skill测试","华为云Skill测试","功能测试","functional test","生命周期测试","lifecycle test","六步测试","回归测试","帮我测试Skill","test skill quality","检测交互式Skill","interactive test","Socratic测试","socratic test","测试交互式Skill"
|
| tags | ["huawei-cloud","skill-testing","functional-testing","lifecycle-testing","hallucination-detection","interactive-testing","devops"] |
Huawei Cloud Skill Tester
Execute six-phase quality testing on Huawei Cloud AI Shell Skills, covering the full pipeline from feature verification through functional execution, with built-in hallucination detection and multi-Skill combination compatibility verification.
Test Spec: references/test-spec.md — Complete six-phase test specification.
Test Guide: references/skill-test-guide.md — Industry best practices and optimization directions.
概述
Current AIShell Skill testing only covers "install → run → uninstall", lacking structured validation, multi-Skill combination testing, hallucination detection, regression, and functional execution mechanisms. This Skill provides a complete six-phase test framework (Phase 0–5) to ensure Skills meet quality standards at every level — feature verification, installation, single functionality, multi-Skill collaboration, end-to-end solutions, and real API execution.
Design Principles
Principle 1: Test Pyramid — Installation verification (many) → Basic functionality (moderate) → Combination compatibility (some) → Solution-level (few). Fast feedback at the bottom, real-world coverage at the top.
Principle 2: Hallucination Defense — Apply structured assertions to Agent output, verify Skill routing correctness, detect responsibility confusion and parameter fabrication.
Principle 3: Combination First — Multiple Skills under the same cloud service must undergo unified combination testing to verify no conflicts or gaps.
前置条件
-
hcloud CLI installed with AK/SK authentication configured
hcloud --version
hcloud configure list
-
Target Skill installable via npx
npx skills add <repo> --skill <target-skill-name>
-
AIShell running, Agent can load Skills
-
Test case configuration prepared (or generated from template)
cp templates/test-cases.yaml.template ./test-cases.yaml
核心命令
| Command | Description |
|---|
bash scripts/verify-skill-features.sh <skill-path> | Phase 0: Feature verification (user confirms before testing) |
bash scripts/validate-skill.sh <skill-path> --phase all-install | Phase 1: Full installation verification (includes install, frontmatter, dependency, sections, i18n, security, lifecycle) |
bash scripts/validate-skill.sh <skill-path> --phase uninstall --install-mode local | Phase 1: Install/uninstall lifecycle (5-step) |
bash scripts/validate-skill.sh <skill-path> --phase i18n | Phase 1: i18n directory validation |
bash scripts/test-skill.sh <name> --skill-path <path> --phase all-basic | Phase 2: All basic functionality tests |
bash scripts/test-skill.sh <name> --skill-path <path> --phase security | Phase 2: Security pattern scan (hardcoded credentials) |
bash scripts/test-skill.sh <name> --skill-path <path> --phase all-combination --related <s2,s3> | Phase 3: All combination tests + hallucination detection |
bash scripts/test-skill.sh <name> --skill-path <path> --phase identify-related | Phase 3: Identify related skills |
bash scripts/detect-hallucination.sh <name> --skill-path <path> --related <s2,s3> | Hallucination detection |
bash scripts/test-skill.sh <name> --skill-path <path> --phase solution --scenario <name> | Phase 4: Solution-level testing |
bash scripts/functional-test.sh <name> --skill-path <path> --region cn-north-4 | Phase 5: Full six-step functional test |
bash scripts/functional-test.sh <name> --skill-path <path> --dry-run | Phase 5: Dry run (parse only, no execution) |
bash scripts/functional-test.sh <name> --skill-path <path> --regression-base <prev> --output <report> | Phase 5: Regression comparison |
bash scripts/test-skill.sh <name> --skill-path <path> --phase full --related <s2,s3> --output ./report.yaml | Full six-phase pipeline (Phase 0–5) |
参数确认
| Parameter | Required | Description | Example |
|---|
<skill-name> | Yes | Target Skill name | your-skill-name |
<skill-path> | Yes | Target Skill directory path | ./skills/<domain>/<your-skill-name> |
--phase | Yes | Test phase to execute | all-install, uninstall, i18n, all-basic, security, all-combination, identify-related, solution, functional, all-functional, full |
--related | No | Related Skill names for combination testing | skill2,skill3 |
--scenario | No | Solution scenario name for Phase 4 | build-network-env |
--output | No | Test report output path | ./test-report.yaml |
--install-mode | No | Installation mode for lifecycle test: local (default) or online | local |
--region | No | Huawei Cloud region for functional tests | cn-north-4 |
KooCLI命令格式标准
hcloud <Service> <Operation> --cli-region=<region> --param1=value1 --param2=value2
hcloud ECS ListServers --cli-region=cn-north-4
hcloud ECS ShowServer --cli-region=cn-north-4 --server_id={instance_id}
| Feature | Description | Example |
|---|
| Service name | Uppercase PascalCase | ECS, VPC, IAM |
| Operation name | PascalCase | ListServers, ShowServer |
| Region param | --cli-region=<value> | --cli-region=cn-north-4 |
| Simple param | --key=value | --server_id=xxx |
| Index param | --key.1=value1 | --servers.1.id=xxx |
| Nested param | --key.sub_key=value | --config.protocol=vnc |
工作流
Phase 0: Feature Verification (NEW)
Verify target Skill's main features are correctly understood before starting any tests. The user reviews and confirms the parsed feature list.
bash scripts/verify-skill-features.sh <skill-path>
Verification Items:
| Item | Description | Level |
|---|
| Description points | 5-point description correctly parsed | Required |
| Core Operations | CLI/SDK/API commands correctly extracted | Required |
| Triggers | Trigger keywords correctly identified | Required |
| Workflow scenarios | Business scenarios correctly detected | Recommended |
Phase 1: Installation Verification
Verify Skill installs correctly, structure is complete, and specification is compliant.
npx skills add <skill-repo> --skill <target-skill-name>
bash scripts/validate-skill.sh <skill-path> --phase install
bash scripts/validate-skill.sh <skill-path> --phase frontmatter
bash scripts/validate-skill.sh <skill-path> --phase dependency
bash scripts/validate-skill.sh <skill-path> --phase sections
bash scripts/validate-skill.sh <skill-path> --phase i18n
bash scripts/validate-skill.sh <skill-path> --phase security
bash scripts/validate-skill.sh <skill-path> --phase uninstall --install-mode local
bash scripts/validate-skill.sh <skill-path> --phase all-install
Verification Items:
| Item | Description | Level |
|---|
| SKILL.md | SKILL.md exists and is readable | Required |
| Frontmatter | name, description (5-point), version, tags compliant | Required |
| references/ | Directory with reference documents | Recommended |
| scripts/ | Directory with test/validate scripts | Recommended |
| templates/ | Directory with templates | Recommended |
| demo/ | Directory with demo examples | Recommended |
| i18n/ | Directory with internationalization translations | Recommended |
| i18n locale format | Locale dirs follow BCP 47 (xx-XX) format | Recommended |
| i18n SKILL files | Each locale has SKILL translation file (e.g., SKILL_CN.md) | Recommended |
| i18n frontmatter | Translation files have valid frontmatter | Recommended |
| i18n name match | Translation name field matches original SKILL.md | Recommended |
| validate script | validate-skill.sh executable and passes | Recommended |
| CLI dependency | hcloud available and authenticated | Recommended |
| Reference docs | Key files exist under references/ | Recommended |
| Required sections | Bilingual match: 概述/Overview, 前置条件/Prerequisites, 工作流/Workflow, 核心命令/Core Commands, 参数确认/Parameters, 参考文档/References | Required |
| KooCLI section | KooCLI命令格式标准 / KooCLI Command Format (required if CLI operations present) | Required |
| Cross-Skill refs | No direct Skill cross-references (Agent orchestrates) | Recommended |
| Install lifecycle | Install → Verify installed → Uninstall → Verify uninstalled → Reinstall | Required |
| Install mode local | npx skills add <path> --skill <name> local install succeeds | Required |
| Install mode online | npx skills add <repo> --skill <name> online install succeeds | Required |
| Verify installation | Skill appears in npx skills list after install | Required |
| Uninstall | npx skills remove <name> succeeds | Required |
| Verify uninstallation | Skill no longer in npx skills list after remove | Required |
| Reinstall | Skill reinstalled after lifecycle for subsequent tests | Required |
Section Bilingual Matching Rules
Section detection supports bilingual matching: a section is considered present if any variant (Chinese or English) is found as a ##-level heading.
| Section | Chinese Variant | English Variant(s) | Required |
|---|
| 概述 | ## 概述 | ## Overview | Yes |
| 前置条件 | ## 前置条件 | ## Prerequisites | Yes |
| 工作流 | ## 工作流 | ## Workflow, ## Main Steps | Yes |
| 核心命令 | ## 核心命令 | ## Core Commands | Yes |
| 参数确认 | ## 参数确认 | ## Parameters | Yes |
| 参考文档 | ## 参考文档 | ## References, ## Reference Documents | Yes |
| KooCLI命令格式标准 | ## KooCLI命令格式标准 | ## KooCLI Command Format, ## Huawei Cloud CLI Command Format | Yes (CLI ops) |
i18n Directory Testing
Verify internationalization directory structure and translation file compliance.
Expected structure: skillname/i18n/{locale}/SKILL_{suffix}.md
Example:
skillname/
├── SKILL.md
└── i18n/
├── zh-CN/
│ └── SKILL_CN.md
├── ja-JP/
│ └── SKILL_JP.md
└── en-US/
└── SKILL_US.md
bash scripts/validate-skill.sh <skill-path> --phase i18n
bash scripts/test-skill.sh <skill-name> --skill-path <skill-path> --phase i18n
i18n Verification Items:
| Item | Description | Level |
|---|
| i18n/ directory | Directory exists | Recommended |
| Locale format | Locale dir names follow BCP 47 (xx-XX) | Recommended |
| SKILL files | Each locale has SKILL translation file | Recommended |
| Frontmatter | Translation files have valid frontmatter | Recommended |
| Name match | Translation name matches original SKILL.md | Recommended |
Phase 2: Basic Functionality Testing
Verify Skill is correctly triggered in AICLI and core functionality outputs correctly. Note: when running as part of --phase full, i18n and uninstall checks are already covered by Phase 1 and skipped here to avoid duplication.
bash scripts/test-skill.sh <skill-name> --skill-path <skill-path> --phase basic
bash scripts/test-skill.sh <skill-name> --skill-path <skill-path> --phase trigger
bash scripts/test-skill.sh <skill-name> --skill-path <skill-path> --phase boundary
bash scripts/test-skill.sh <skill-name> --skill-path <skill-path> --phase compare
bash scripts/test-skill.sh <skill-name> --skill-path <skill-path> --phase i18n
bash scripts/test-skill.sh <skill-name> --skill-path <skill-path> --phase security
bash scripts/test-skill.sh <skill-name> --skill-path <skill-path> --phase uninstall
bash scripts/test-skill.sh <skill-name> --skill-path <skill-path> --phase all-basic
Test Types:
| Type | Method | Target |
|---|
| Trigger identification | Input trigger words, verify Skill is correctly loaded | Trigger accuracy >= 90% |
| Core functionality | Execute typical use cases, verify output contains expected content | Output structure complete |
| Boundary/exception | Input invalid/boundary parameters | Error handling reasonable, no fabrication |
| No false trigger | Input unrelated requests | Skill not incorrectly activated |
| With/Without | Compare same task with/without Skill | Quantified value delta > 0 |
Phase 3: Combination Compatibility Testing
Verify multiple Skills under the same cloud service work together without conflicts or hallucination.
bash scripts/test-skill.sh <skill-name> --skill-path <skill-path> --phase identify-related
bash scripts/test-skill.sh <skill-name> --skill-path <skill-path> --phase combination --related <skill2,skill3>
bash scripts/test-skill.sh <skill-name> --skill-path <skill-path> --phase competition --related <skill2,skill3>
bash scripts/test-skill.sh <skill-name> --skill-path <skill-path> --phase isolation --related <skill2,skill3>
bash scripts/detect-hallucination.sh <skill-name> --skill-path <skill-path> --related <skill2,skill3>
bash scripts/test-skill.sh <skill-name> --skill-path <skill-path> --phase all-combination --related <skill2,skill3>
Hallucination Detection Items:
| Hallucination Type | Detection Method | Criterion |
|---|
| Responsibility confusion | Verify Agent invokes the Skill matching the request | invoked Skill == expected Skill |
| Parameter fabrication | Verify output parameter values within valid range | parameter value in known service/resource whitelist |
| Workflow stitching error | Verify multi-step workflow step completeness | actual steps == expected step sequence |
| Context pollution | Verify subsequent tasks contain no prior residue | Task B output intersect Task A entities = empty |
| Format hallucination | Verify output structure matches specification | output matches JSON Schema |
Phase 4: Solution-Level Testing
Verify Skill performance in real business solution end-to-end scenarios.
bash scripts/test-skill.sh <skill-name> --skill-path <skill-path> --phase solution --scenario <scenario-name>
bash scripts/test-skill.sh <skill-name> --skill-path <skill-path> --phase performance
bash scripts/test-skill.sh <skill-name> --skill-path <skill-path> --phase report --output <report-path>
Performance Metrics:
| Metric | Description | Suggested Threshold |
|---|
| total_time_seconds | End-to-end total time | < 300s |
| total_tokens | Total token consumption | < 50000 |
| accuracy_rate | Output accuracy rate | > 0.9 |
| hallucination_rate | Hallucination occurrence rate | < 0.05 |
| trigger_accuracy | Trigger accuracy rate | > 0.9 |
Phase 5: Functional Testing (NEW)
Verify Skill functionality through a six-step lifecycle: parse SKILL.md commands, auto-generate test cases, define scoring criteria, execute hcloud CLI commands against real APIs, classify bugs by severity, and support regression testing.
Lifecycle Mode (--lifecycle): For Skills with resource Create/Delete operations, chains end-to-end workflow with AK/SK prompt, cost confirmation, variable passing, resource polling, and automatic cleanup.
Executor Backend (--executor): Three-tier execution backend that auto-detects the best available tool — CLI → SDK → API.
Interactive Questioning Test: For meta-skills with Socratic questioning flow (e.g., skill-creator), auto-detects 4 questioning dimensions (Target service, Function scope, CLI operations, Trigger scenarios) and validates questioning patterns: one-at-a-time, summary table after 5 questions, user confirmation before proceeding, and mandatory interactive phase declaration.
Interactive E2E Creation Test (NEW): For skills with both Socratic questioning AND template scaffolding, runs a complete end-to-end creation pipeline — takes preset 4-dimension answers, scaffolds a complete skill from templates (SKILL.md, references, scripts, i18n), validates the created skill structure, tests its CLI commands, and verifies the output matches the input answers.
bash scripts/functional-test.sh <skill-name> --skill-path <skill-path> --region cn-north-4
bash scripts/functional-test.sh <skill-name> --skill-path <skill-path> --executor auto
bash scripts/functional-test.sh <skill-name> --skill-path <skill-path> --executor cli
bash scripts/functional-test.sh <skill-name> --skill-path <skill-path> --executor sdk
bash scripts/functional-test.sh <skill-name> --skill-path <skill-path> --executor api
bash scripts/functional-test.sh <skill-name> --skill-path <skill-path> --lifecycle --region cn-north-4
bash scripts/functional-test.sh <skill-name> --skill-path <skill-path> --lifecycle --dry-run
bash scripts/functional-test.sh <skill-name> --skill-path <skill-path> \
--regression-base <previous-report.yaml> --output <report.yaml>
bash scripts/test-skill.sh <skill-name> --skill-path <skill-path> --phase all-functional
bash scripts/functional-test.sh <skill-name> --skill-path <skill-path> \
--lifecycle --region cn-north-4 --yes --test-vars references/test-vars.json
bash scripts/functional-test.sh <skill-name> --skill-path <skill-path> --region cn-north-4
Six-Step Functional Test Flow:
| Step | Description | Verification |
|---|
| Step 1: 准备与配置 | Parse SKILL.md metadata and extract service commands | Service detected, commands parsed |
| Step 2: 设计测试用例 | Auto-generate CLI, trigger, boundary, workflow, interactive, and lifecycle test cases | ≥1 test case generated |
| Step 3: 评分与评估标准 | Define weights (execution 30%, accuracy 35%, completeness 20%, format 15%) | Scoring criteria defined |
| Step 4: 自动化执行与验证 | Execute hcloud commands, verify exit code and output content | Execution rate ≥90% |
| Step 5: 结果分析与分类 | Classify bugs by severity (critical/major/minor/suggestion), RCA | Zero critical bugs |
| Step 6: 迭代优化 | Compare with previous baseline, detect regressions | No regression spikes |
Lifecycle Mode (--lifecycle): Replaces Step 4 with a chained workflow:
| Lifecycle Step | Description | Security Checks |
|---|
| 0. Authentication | Check hcloud auth, prompt AK/SK if missing | Credentials never stored in scripts |
| 1. Create resource | Execute Create command, capture resource ID | Cost confirmation required |
| 2. Poll until ACTIVE | Poll Show/List every 10s until ready (timeout: 120s) | Read-only, no side effects |
| 3. Delete resource | Delete created resource, poll until DELETED |
⚠ 强制性约束：生命周期测试必须执行实际功能操作
生命周期模式直接从 SKILL.md 提取命令文本并逐字执行。如果命令中包含 {vpc_id}、{image_id}、{subnet_id} 等占位符，会被作为字面字符串传给 hcloud CLI 导致参数验证错误。
解决方案： 被测试的 Skill 必须提供 references/test-vars.json，包含真实资源 ID。格式示例：
{
"vpc_id": "0b4171c1-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"subnet_id": "fa5fb1a9-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"image_id": "74d6130f-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"flavor": "kc1.small.1",
"region": "cn-north-4"
}
运行生命周期测试时加 --test-vars 参数：
bash scripts/functional-test.sh <skill-name> --skill-path <path> \
--lifecycle --region cn-north-4 --yes --test-vars references/test-vars.json
脚本会自动将命令中的 {key} 和 <key> 占位符替换为 JSON 中的值。只读命令（ListFlavors、ShowServer 等）不受影响。
Scoring Dimensions:
| Dimension | Weight | Threshold | Description |
|---|
| Execution | 30 pts | ≥90% | Did the command execute without error? |
| Accuracy | 35 pts | ≥85% | Does output contain all expected content? |
| Completeness | 20 pts | ≥80% | Are all required fields present? |
| Format | 15 pts | ≥90% | Is output structure well-formed? |
Bug Severity Levels:
| Severity | Score | Description | Action |
|---|
| CRITICAL | 0.0 | Command fails, wrong API, security issue | Blocks release |
| MAJOR | 0.5 | Missing required fields, incorrect output | Must fix before release |
| MINOR | 0.8 | Format deviation, non-critical missing info | Should fix if time |
| SUGGESTION | 1.0 | Performance optimization, best practice | Nice to have |
One-Command Full Pipeline (Phase 0–5)
bash scripts/test-skill.sh <skill-name> \
--skill-path <skill-path> \
--phase full \
--related <skill2,skill3> \
--scenario <scenario-name> \
--output ./test-report.yaml
Edge Cases
| Scenario | Handling |
|---|
| Target Skill installation fails | Log error, abort subsequent phases, report marks INSTALL_FAIL |
| hcloud CLI not installed | Prompt install command, abort test |
| AK/SK not configured | Prompt configuration steps, abort test |
| Test case config missing | Use default template from templates/ |
| Related Skill not installed | Skip combination test, report marks COMBINATION_SKIP |
| Hallucination detected | Mark HALLUCINATION_DETECTED, log details, continue subsequent tests |
| Agent timeout no response | Mark TIMEOUT, log timeout phase, continue subsequent tests |
| Network unreachable | Mark NETWORK_ERROR, abort tests requiring cloud API |
| hcloud command fails (functional test) | Classify as CRITICAL bug, log command + output, continue remaining tests |
| No hcloud commands in SKILL.md | Mark FUNCTIONAL_SKIP (non-CLI skill), skip functional execution |
| Regression: new failures detected | Mark REGRESSION_DETECTED, list delta, flag for investigation |
| Functional test dry run | Parse and generate test cases only, no API calls |
| Interactive questioning skill detected | Auto-generate Socratic pattern tests: 4 dimensions, one-at-a-time, summary table, user confirm |
| Interactive skill missing dimension | Mark DIMENSION_MISSING for each of 4 questioning dimensions not found in SKILL.md |
| Interactive + scaffold skill detected | Auto-run interactive_e2e test: scaffold skill from templates, validate, test CLI, verify answers |
Verification Method
Phase 1 Verification
bash scripts/validate-skill.sh <skill-path> --phase all-install
Phase 2 Verification
bash scripts/test-skill.sh <skill-name> --skill-path <skill-path> --phase all-basic
Phase 3 Verification
bash scripts/test-skill.sh <skill-name> --skill-path <skill-path> --phase all-combination --related <skill2>
Phase 4 Verification
bash scripts/test-skill.sh <skill-name> --skill-path <skill-path> --phase solution --scenario <name>
Phase 5 Verification
bash scripts/functional-test.sh <skill-name> --skill-path <skill-path> --region cn-north-4
bash scripts/functional-test.sh <skill-name> --skill-path <skill-path> --dry-run
bash scripts/functional-test.sh <skill-name> --skill-path <skill-path> --regression-base <prev-report>
Test Report
After testing completes, a YAML format report is generated at the specified path. Structure details in templates/test-report.yaml.template.
Report contains:
- PASS/FAIL status for all test cases across four phases
- Hallucination detection results and details
- Performance metric data
- With/Without comparison delta values
- Overall assessment and recommendations
参考文档
Script Usage
| Script | Purpose | Example |
|---|
scripts/validate-skill.sh | Phase 1 installation verification | bash scripts/validate-skill.sh ./path --phase all-install |
scripts/validate-skill.sh | i18n directory validation | bash scripts/validate-skill.sh ./path --phase i18n |
scripts/test-skill.sh | Phase 2/3/4 test execution | bash scripts/test-skill.sh <name> --skill-path <path> --phase full |
scripts/test-skill.sh | i18n test execution | bash scripts/test-skill.sh <name> --skill-path <path> --phase i18n |
scripts/test-skill.sh | Functional test execution | bash scripts/test-skill.sh <name> --skill-path <path> --phase functional |
scripts/functional-test.sh | Phase 5 six-step functional test | bash scripts/functional-test.sh <name> --skill-path <path> --region cn-north-4 |
scripts/detect-hallucination.sh | Hallucination detection | bash scripts/detect-hallucination.sh <name> --skill-path <path> --related <s2> |
scripts/generate-report.sh | Test report generation | bash scripts/generate-report.sh <result-dir> --output report.yaml |
Typical Use Cases
Scenario 1: Test Single Skill Basic Functionality
User: Help me test the <target-skill-name> Skill
Agent:
1. Phase 1: npx install + structure validation -> PASS
2. Phase 2: AICLI trigger + core use cases + boundary cases -> PASS
3. Generate Phase 1+2 test report
Scenario 2: Test Multi-Skill Combination Compatibility
User: Test compatibility between ECS manage and ECS diagnosis Skills
Agent:
1. Phase 1+2: Test each Skill's basic functionality -> PASS
2. Phase 3: Combination loading + cross-scenario cases + competition test + hallucination detection -> PASS
3. Generate combination compatibility test report
Scenario 3: End-to-End Solution Testing
User: Run end-to-end test on VPC networking solution Skills
Agent:
1. Phase 1+2+3: Basic and combination tests -> PASS
2. Phase 4: Execute "build complete network environment" solution scenario
3. Collect performance metrics, generate complete test report
Scenario 4: Detect Skill Hallucination Issues
User: Detect hallucination in your-skill-name under multi-Skill environment
Agent:
1. Identify related Skill set
2. Run hallucination detection: responsibility confusion, parameter fabrication, context pollution
3. Output hallucination detection report, mark HALLUCINATION_DETECTED or PASS
Scenario 5: Run Six-Step Functional Test (NEW)
User: Do a functional test on the huawei-cloud-ecs-manage Skill
Agent:
1. Step 1: Parse SKILL.md, extract ECS service commands (ListServers, ShowServer, etc.)
2. Step 2: Auto-generate test cases (CLI commands + trigger + boundary + workflow)
3. Step 3: Apply scoring criteria (execution 30%, accuracy 35%, completeness 20%, format 15%)
4. Step 4: Execute hcloud CLI commands against cn-north-4, verify exit codes and output
5. Step 5: Classify any failures by severity (critical/major/minor/suggestion)
6. Step 6: Compare with previous run (if --regression-base provided)
7. Generate functional test report
Scenario 6: Functional Dry Run (NEW)
User: Run a functional test dry run on the vpc-manage Skill (don't execute commands)
Agent:
1. Step 1: Parse SKILL.md, extract VPC commands
2. Step 2: Generate test cases: TC-FUNC-001 (VPC ListVpcs), TC-FUNC-002 (VPC ShowVpc), etc.
3. Step 3: Display scoring criteria and thresholds
4. Report: "Dry run completed — 5 test cases ready. Run without --dry-run to execute."
Scenario 7: Lifecycle E2E: Create ECS → Verify → Delete (NEW)
User: Run lifecycle test on the huawei-cloud-ecs-manage Skill
Agent:
1. Check hcloud authentication → prompt for AK/SK if needed
2. Parse SKILL.md → detect CreateServer, ShowServer, DeleteServer
3. Display cost warning for ECS CreateServer (s6.small.1, cn-north-4)
4. After user confirmation: execute CreateServer → capture server_id
5. Poll ShowServer every 10s until status == "ACTIVE" (timeout: 120s)
6. Display cost warning for ECS DeleteServer
7. After user confirmation: execute DeleteServer
8. Poll ShowServer until 404/not found (confirm deletion)
9. Generate lifecycle test report
10. IF ANY STEP FAILS: cleanup_resources() trap ensures server is deleted
Scenario 8: Test Interactive Socratic Skill (NEW)
User: Run an interactive test on the huawei-cloud-skill-creator Skill
Agent:
1. Step 1: Parse SKILL.md — detect Socratic questioning flow, 4 dimensions
2. Step 2: Auto-generate interactive test cases:
- socratic_pattern: Socratic questioning pattern detected?
- one_at_a_time: One question at a time?
- summary_table: Summary table requirement?
- user_confirm: User confirmation requirement?
- dimension_1~4: Each of 4 questioning dimensions covered?
- mandatory_phase: Mandatory interactive phase declared?
- overall: Overall interactive quality score
3. Step 3-4: Validate each interactive pattern against SKILL.md content
4. Step 5: Report which dimensions are covered/missing
5. Generate interactive test report with PASS/FAIL for each pattern
Scenario 9: Interactive E2E — Create Then Test a Skill (NEW)
User: Test the full creation pipeline of huawei-cloud-skill-creator
Agent:
1. Step 1: Detect Socratic questioning + template scaffolding → interactive_e2e capability
2. Step 2: Generate interactive_e2e test case: full creation pipeline
3. Step 4: Run interactive-e2e-test.sh with preset 4-dimension answers:
- Dimension 1 (Target service): ECS
- Dimension 2 (Function scope): Inspect and diagnose ECS instances
- Dimension 3 (CLI operations): ListServers, ShowServer
- Dimension 4 (Trigger scenarios): 检查ECS, inspect ecs
4. Scaffold: Create skill from templates → render SKILL.md + references/ + scripts/ + i18n/
5. Validate: Run validate-skill.sh on created skill → verify structure, frontmatter, sections
6. Test: Run test-cli-commands.sh on created skill → verify CLI commands work
7. Verify: Confirm created SKILL.md references correct service, triggers, and operations
8. Report: Pass/fail summary of the full interactive creation pipeline
User-Agent Identification
Add User-Agent identification when calling hcloud CLI during testing, for platform Skill usage tracking:
export HCLOUD_USER_AGENT=HuaweiCloud-Agent-Skills
Key Principles
- Test Pyramid — Installation verification (many) → Basic functionality (moderate) → Combination compatibility (some) → Solution-level (few)
- Hallucination Defense — Structured assertions + routing verification + whitelist validation, reject format-abnormal output
- Combination First — Same-service multi-Skill must undergo unified combination testing
- Idempotent First — Prefer read-only/idempotent operations in testing, avoid side effects
- Isolation — Each test case independent, no dependency on other case results
- Repeatability — Same input produces same output, eliminate randomness
- Incremental Regression — Auto-run affected test subset after each change
- Security First — No hardcoded AK/SK in tests, destructive operations require confirmation
- User-Agent — Add User-Agent identification in CLI calls for tracking
Critical Answer Reference — Must Surface When Asked
When asked about any of the following topics, always provide the explicit answer below. Do not defer to referenced documents or give vague summaries.
AK/SK Security Rule (qa_004)
Never hardcode AK/SK credentials in any Skill code, test script, or configuration file. Always authenticate via hcloud configure (interactive) or environment variables (HCLOUD_AK / HCLOUD_SK). Any test or script that requires authentication must obtain credentials exclusively through these runtime mechanisms.
Huawei Cloud CLI Command Format (qa_011)
Format: hcloud <Service> <Operation> --cli-region=<region> [--key=value ...]
- Service: PascalCase, uppercase (e.g.,
ECS, VPC, IAM, RDS, CCE)
- Operation: PascalCase (e.g.,
ListServers, ShowServer, CreateServer)
- Region: always
--cli-region=<value> (e.g., --cli-region=cn-north-4)
- Simple params:
--key=value
- Indexed params:
--key.1=value1 --key.2=value2
- Nested params:
--key.sub_key=value
Test Result Categorization (qa_020)
Test results are categorized into two levels:
- Required checks: result is PASS or FAIL. Failure means the Skill does not meet mandatory quality standards.
- Recommended checks: result is WARN if not met. Warning means the Skill lacks an optional but strongly recommended element (e.g., references/, scripts/, templates/, demo/, i18n/).
Cost Confirmation for Resource-Mutating Operations (qa_007)
Any operation that creates, modifies, or deletes cloud resources (e.g., ECS CreateServer, VPC DeleteVpc) must: (1) display the estimated cost or resource impact to the user, and (2) obtain explicit user confirmation before execution. Read-only operations (List*, Show*) are exempt. In test environments, prefer idempotent read-only operations; if a mutating test is necessary, use --dry-run where available or wrap in a confirmation prompt.
Skill Versioning Scheme (qa_009)
Skills MUST follow Semantic Versioning (SemVer): MAJOR.MINOR.PATCH (e.g., 1.0.0).
- MAJOR: breaking changes to Skill interface or workflow
- MINOR: new functionality added backward-compatibly
- PATCH: bug fixes or documentation corrections
The version field in SKILL.md frontmatter must be updated on every release.
iam-policies.md Required Contents (qa_014)
The references/iam-policies.md file must include:
- Minimum IAM permissions (policy actions) required for each operation the Skill performs
- Grouped by cloud service (e.g., ECS, VPC, IAM)
- Recommended policy name or custom policy JSON template
- Distinction between read-only permissions (needed for basic testing) and write permissions (needed for resource-mutating operations)
SKILL.md Body Line Limit (qa_015)
The recommended body line limit for SKILL.md is 500 lines (excluding frontmatter). Skills exceeding this limit should split detailed content into reference documents under references/ and keep SKILL.md as a concise entry point with links.
Workflow Steps for Creating a Skill (qa_013)
The 10 standard workflow steps for creating a Huawei Cloud AI Shell Skill are:
- Define Skill scope and target cloud service operations
- Create SKILL.md with frontmatter (name, description, version, tags)
- Write required sections: 概述/Overview, 前置条件/Prerequisites, 工作流/Workflow, 核心命令/Core Commands, 参数确认/Parameters, 参考文档/References
- Add KooCLI命令格式标准 section if the Skill uses CLI operations
- Create reference documents under references/ (test-spec, iam-policies, quality-checklist)
- Create test/validate scripts under scripts/ (validate-skill.sh, test-skill.sh)
- Create templates under templates/ for test cases and reports
- Add i18n translations under i18n/{locale}/SKILL_{suffix}.md
- Run Phase 1 installation verification:
bash scripts/validate-skill.sh <path> --phase all-install
- Run full six-phase test pipeline:
bash scripts/test-skill.sh <name> --skill-path <path> --phase full and fix any failures
