robustmq-chaos-test

# RobustMQ Chaos Test Skill ## When to Use **Cron trigger** (every 4 hours): the system message will say > "按 P0 跑一轮 RobustMQ 故障场景" **Manual CLI**: user says something like > "帮我跑一轮 RobustMQ chaos 测试" / "run a chaos test round" In both cases execute the **Full Run** below. If the user says "按 P1 跑一轮" or names a specific scenario, execute the **Single Scenario** flow for that scenario only. --- ## Pre-check Before starting any run: 1. Call `cluster_manage(action=status)`. - If `status` is NOT `stopped`, call `cluster_manage(action=stop)` to clear any leftover state from a previous run. 2. Verify `chaos-test/config.yml` has `cluster.binary` and `cluster.project_root` filled in correctly (the cluster tool fails fast if binary is missing — surface that error immediately and stop). --- ## Scenario Catalogue | Scenario name | fault_type | target | params | Core? | |---|---|---|---|---| | broker-kill-single | broker-kill | robustmq-server | — | ✅ | | network-delay-100ms | network-delay | eth0 | delay_ms=100, jitter_ms=10 | — | | leader-transfer | broker-kill | robustmq-server | — | ✅ | > **Note:** Update this table when new scenarios are added. Target names and > interface names depend on the deployment environment — verify before running. Core scenarios: **broker-kill-single**, **leader-transfer**. Run passed = all core pass AND non-core pass rate ≥ 75%. --- ## Single Scenario — 5-Step Flow Execute these steps sequentially. Do NOT skip steps. ### Step 1 — Baseline Snapshot ``` observability(action=snapshot, data_dirs=<from cluster start>) ``` Record the snapshot as `baseline`. Proceed even if some metrics are unavailable; log a warning but do not abort. ### Step 2 — Inject Fault ``` chaos(action=inject, fault_type=<type>, target=<target>, params=<params>) ``` Save the returned `fault_id`. If inject returns an error, mark the scenario `passed=False` with `status=inject_error` and skip to Step 5 (skip recover). ### Step 3 — Fault-Period SDK Observation (record only) ``` client(action=run, scenario=<scenario>, cluster_endpoint=<endpoint>) ``` Record all results. **Do NOT use these results to determine pass/fail.** Their only purpose is observability — they show what clients experienced during the fault. A high loss rate here is expected and normal. ### Step 4 — Recover ``` chaos(action=recover, fault_id=<fault_id>) ``` If recover returns an error, log it and continue — attempt self-healing validation anyway. ### Step 5 — Self-Healing Validation (sole pass/fail basis) Wait **60 seconds** after recovery, then run: ``` client(action=run, scenario=<scenario>, cluster_endpoint=<endpoint>) ``` **Pass criteria** (ALL must hold): - `exit_code == 0` - `lost == 0` - `p99_ms < 500` If any criterion fails → scenario `passed=False`. If `status=script_format_error` → scenario `passed=False`, note the format error separately (this is a test-infrastructure issue, not a RobustMQ bug). --- ## Full Run Flow 1. **Pre-check** (see above). 2. **Start cluster**: `cluster_manage(action=start)` → save `endpoint` and `data_dirs`. 3. **Run each scenario** using the Single Scenario flow. - Run scenarios sequentially (not in parallel) to avoid interference. - If a scenario crashes the cluster (all brokers dead), restart it before continuing: `cluster_manage(action=stop)` → `cluster_manage(action=start)`. 4. **Stop cluster**: `cluster_manage(action=stop)`. 5. **Generate report**: `report(action=generate_and_push, run_data={...})`. - `run_data` must include: `run_id`, `started_at`, `finished_at`, `scenarios`. - Each scenario entry: scenario name, sdk, passed, sent, received, lost, p99_ms, duration_seconds, errors, status. 6. **Send Feishu notification**: - If `run_passed=True`: send brief pass message with `github_url`. - If `run_passed=False`: send failure alert listing failed scenarios and `github_url`. - If `consecutive_failures >= 3`: prepend `🚨 连续 {n} 轮失败，请人工介入`. --- ## Circuit Breaker Track `consecutive_failures` across runs (persist in your memory or state): - Increment on `run_passed=False`. - Reset to 0 on `run_passed=True`. - If `consecutive_failures >= 3`: send an urgent Feishu alert and **pause** the cron schedule. Do NOT continue running automatically until a human acknowledges and resets the counter. --- ## Feishu Message Templates **Pass:** ``` ✅ RobustMQ 故障测试通过 Run ID: {run_id} 时间: {finished_at} 核心场景: 全部通过 总通过率: {pass_rate}% 报告: {github_url} ``` **Fail:** ``` ❌ RobustMQ 故障测试失败 Run ID: {run_id} 时间: {finished_at} 失败场景: {failed_scenario_list} 报告: {github_url} ``` **Circuit breaker:** ``` 🚨 连续 {consecutive_failures} 轮失败，请人工介入 最后失败: {run_id} {finished_at} 报告: {github_url} ``` --- ## Pitfalls - **Never judge pass/fail on fault-period results** (Step 3). Only Step 5 post-recovery validation counts. - **`script_format_error` ≠ RobustMQ bug.** Report it separately; do not inflate the failure count. Fix the script first. - **ROBUSTMQ_HOME must be set** before any run. The cluster tool returns an error immediately if it is not — surface it and stop rather than retrying. - **Consecutive failures count whole runs**, not individual scenarios. One run with two failed scenarios = 1 failure, not 2. - **eth0 is not universal.** The network-delay target interface name varies by host. Verify it before running in a new environment. - **Deploy Key permissions.** If `report` returns `push_error`, the reports are still written locally at `json_path` / `markdown_path`. Investigate the key before declaring the run lost. - **60-second wait is mandatory.** Do not skip or shorten it — RobustMQ leader election and connection re-establishment take time.