run-e2e

# Run SkyWalking E2E Test Run an E2E test case using `skywalking-infra-e2e`. The user provides a test case path (e.g., `simple/jdk`, `storage/banyandb`, `alarm`). ## Prerequisites All tools require **Go** installed. Check `.github/workflows/` for the exact `e2e` commit used in CI. ### e2e CLI Built from [apache/skywalking-infra-e2e](https://github.com/apache/skywalking-infra-e2e), pinned by commit in CI: ```bash # Install the pinned commit go install github.com/apache/skywalking-infra-e2e/cmd/e2e@<commit-id> # Or clone and build locally (useful when debugging the e2e tool itself) git clone https://github.com/apache/skywalking-infra-e2e.git cd skywalking-infra-e2e git checkout <commit-id> make build # binary is in bin/e2e — add to PATH or copy to $GOPATH/bin ``` ### swctl, yq, and other tools E2E test cases run pre-install steps (see `setup.steps` in each `e2e.yaml`) that install tools into `/tmp/skywalking-infra-e2e/bin`. When running locally, you need these tools on your PATH. **swctl** — SkyWalking CLI, used in verify cases to query OAP's GraphQL API. Pinned at `SW_CTL_COMMIT` in `test/e2e-v2/script/env`: ```bash # Option 1: Use the install script (same as CI) bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl export PATH=/tmp/skywalking-infra-e2e/bin:$PATH # Option 2: Build from source go install github.com/apache/skywalking-cli/cmd/swctl@<SW_CTL_COMMIT> ``` **yq** — YAML processor, used in verify cases: ```bash # Option 1: Use the install script bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq export PATH=/tmp/skywalking-infra-e2e/bin:$PATH # Option 2: brew install yq (macOS) ``` **Other tools** (only needed for specific test cases): | Tool | Install script | Used by | |------|---------------|---------| | `kubectl` | `install.sh kubectl` | Kubernetes-based tests | | `helm` | `install.sh helm` | Helm chart tests | | `istioctl` | `install.sh istioctl` | Istio/service mesh tests | | `etcdctl` | `install.sh etcdctl` | etcd cluster tests | All install scripts are at `test/e2e-v2/script/prepare/setup-e2e-shell/`. ## Steps ### 1. Determine the test case Resolve the user's argument to a full path under `test/e2e-v2/cases/`. If ambiguous, list matching directories and ask. ```bash ls test/e2e-v2/cases/<argument>/e2e.yaml ``` ### 2. Check if rebuild is needed Compare source file timestamps against the last build: ```bash # OAP server changes since last build find oap-server apm-protocol -type f \( \ -name "*.java" -o -name "*.yaml" -o -name "*.yml" -o \ -name "*.json" -o -name "*.xml" -o -name "*.properties" -o \ -name "*.proto" \ \) -newer dist/apache-skywalking-apm-bin.tar.gz 2>/dev/null | head -5 # Test service changes since last build find test/e2e-v2/java-test-service -type f \( \ -name "*.java" -o -name "*.xml" -o -name "*.yaml" -o -name "*.yml" \ \) -newer test/e2e-v2/java-test-service/e2e-service-provider/target/*.jar 2>/dev/null | head -5 ``` If files are found, warn the user and suggest rebuilding before running. ### 3. Rebuild if needed (only with user confirmation) ```bash # Rebuild OAP ./mvnw clean flatten:flatten package -Pall -Dmaven.test.skip && make docker # Rebuild test services ./mvnw -f test/e2e-v2/java-test-service/pom.xml clean flatten:flatten package ``` ### 4. Run the E2E test Set required environment variables and run: ```bash export SW_AGENT_JDK_VERSION=8 e2e run -c test/e2e-v2/cases/<case-path>/e2e.yaml ``` ### 5. If the test fails Do NOT run cleanup immediately. Instead: 1. Check container logs: ```bash docker compose -f test/e2e-v2/cases/<case-path>/docker-compose.yml logs oap docker compose -f test/e2e-v2/cases/<case-path>/docker-compose.yml logs provider ``` 2. Run verify separately (can retry after investigation): ```bash e2e verify -c test/e2e-v2/cases/<case-path>/e2e.yaml ``` 3. Only cleanup when done debugging: ```bash e2e cleanup -c test/e2e-v2/cases/<case-path>/e2e.yaml ``` ### 6. Manually fire each verify query (fast triage) The `e2e verify` retry loop runs in sequence and stops at the first failing case, so a single bad query hides every case after it. When a verify fails, **run each verify case directly against the still-running OAP** before editing anything — you'll see the real error (bad flag, missing data, wrong expected), not the progress spinner. This is also the right way to author new verify cases: craft the query against live OAP, confirm the actual YAML, then write the expected file. ```bash # Find the host-side port that infra-e2e bound to OAP's container port 12800. # (Each run picks a new random port; the trigger log prints it too.) docker ps --filter "name=skywalking_e2e-oap" --format "{{.Ports}}" \ | grep -oE "[0-9]+->12800" | head -1 # => e.g. 56381->12800 URL=http://localhost:56381/graphql SWCTL=/tmp/skywalking-infra-e2e/bin/swctl # Copy the query from e2e.yaml verbatim, then substitute ${oap_host} → localhost # and ${oap_12800} → the port you just found: $SWCTL --display yaml --base-url=$URL service ly IOS $SWCTL --display yaml --base-url=$URL logs list --service-name=MyiOSApp $SWCTL --display yaml --base-url=$URL metrics exec --expression=service_cpm --service-name=MyiOSApp ``` When a `swctl` subcommand rejects a flag (`Incorrect Usage: flag provided but not defined: -layer`), the e2e config is using syntax the pinned `swctl` commit doesn't support. Find the right syntax with `swctl <cmd> --help` and update the e2e config. Common cases encountered: | Broken flag/form | Working form | |---|---| | `service ls --layer IOS` | `service ly IOS` | | `metrics exec ... --is-normal=true` | drop `--is-normal` (default behavior) | For queries that *don't* use `swctl` (raw `curl` against `/loki/...`, Zipkin, PromQL), hit the matching exposed port: ```bash curl "http://localhost:$(docker ps --filter name=skywalking_e2e-oap --format '{{.Ports}}' | grep -oE '[0-9]+->3100' | head -1 | cut -d'-' -f1)/loki/api/v1/labels" ``` ### 7. UI template changes require a fresh DB `UITemplateInitializer.initTemplate()` (in `oap-server/server-core`) calls `uiTemplateManagementService.addIfNotExist(setting)` — keyed by the `id` field in each `ui-initialized-templates/**/*.json`. Same ID → skipped. So edits to an *existing* template JSON (adding widgets, relabeling, changing expressions) will **not** be applied on an already-initialized OAP, even after a container restart, because the old copy still lives in storage. To pick up dashboard JSON changes: ```bash # Remove both containers — BanyanDB stores state inside the container FS in the # e2e compose (no named volume), so removing the container wipes state cleanly. docker rm -f skywalking_e2e-oap-1 skywalking_e2e-banyandb-1 # For compose setups that use a named volume, also: # docker volume rm <volume-name> # Then re-run — OAP sees empty storage, loads the new template JSON. e2e run -c test/e2e-v2/cases/<case>/e2e.yaml ``` Symptom to watch for: you edit the JSON, rebuild, redeploy — dashboard in the UI still shows the pre-edit layout. That's not a caching bug; that's `addIfNotExist` doing exactly what its name says. ### 8. Author the expected YAML from live output For a new verify case, the workflow is: 1. Fire the query manually (see step 6) and capture the YAML. 2. Pick which fields are *meaningful domain values* (must match exactly) vs *dynamic runtime values* (`notEmpty` / `gt` / `ge`). See `test/e2e-v2/CLAUDE.md` for the decision guide. 3. Write the expected file. If the response is a list, wrap the items in `{{- contains . }} ... {{- end }}` so ordering and extra actual items don't fail the match. 4. Re-run `e2e verify` alone (the containers are still up from the previous run); iterate on the expected file without rebuilding. ### 9. Expected-file authoring traps These burn CI cycles and pass locally. Each was learned the hard way. **Unquoted `content: {{ notEmpty .content }}` with `:` inside the value.** Sim-generated or real log content routinely includes colons (`POST https://api.example.com/cart failed: 500`, `HTTP/1.1 500: Internal Server Error`). Without quoting, the template renders to invalid YAML (snakeyaml parses `failed:` as a nested key) and the *whole log entry* marshals to `nil`. Symptom: diff shows `- nil` at *every* position in the expected logs list vs real maps in actual. Fix: wrap in single quotes — `content: '{{ notEmpty .content }}'`. Single-quoted YAML preserves `:` in the scalar; only fails on embedded `'`. Double quotes also work unless the content has `"`. **Nested `contains` with multiple per-element pattern assertions against a varied stream.** The template renders the block body once per actual element; when the outer block body has *multiple* inner `contains` patterns asserting specific tag key/value pairs, and only *some* actuals satisfy all the inner patterns, `go-cmp` with `contains` can end up comparing `[rendered_for_A0, nil, nil, ...]` vs `[A0, A1, A2, ...]` and fail despite `contains` being permissive on extras. Specifically: outer `contains .logs` with a single log pattern + inner `contains .tags` asserting two distinct key/value pairs. On a simulator emitting heterogeneous errors (js + promise + ajax + pageNotFound), only a subset satisfy the inner assertion. Passes locally with 1–2 logs, fails in CI with 6+. - Keep the outer `contains` body lenient: field-shape checks (`notEmpty`, `gt`), one discriminator tag that *every* element in the stream carries. - Cover per-category assertions via separate labeled-metric verify cases, not inside the nested template. - Rule of thumb: "at least one log exists with the right layer routing" inside the logs expected; per-category coverage via `meter_*_count{label=X}` verify cases. **Hand-crafted OTLP curl payloads drift from real SDK output.** When the upstream SDK ships a published simulator image (mini-program-monitor's `sim-wechat` / `sim-alipay`, browser-client-js sim, etc.), prefer driving the e2e with that image in `MODE=timed` with a bounded `DURATION_MS` over hand-rolling the OTLP JSON. Hand-crafted payloads miss real-world shape issues: delta-vs-cumulative temporality, label-cardinality surprises, stacktrace formatting variance, attribute key names that changed between SDK versions. Pin to a released tag (`v0.4.0`), not `:latest` or HEAD SHA — reproducibility. **`timeUnixNano: "0"` in an OTLP metric datapoint.** The receiver propagates this into MAL's bucket computation and the metric lands in the 1970 time bucket — `swctl metrics exec` over the "last N minutes" window won't find it. Either use `$(date +%s)000000000` at setup time or omit the field if the receiver accepts "now" as default. **Setup-step curl loop with `|| sleep` pattern.** The shell line `for ... do curl && break || sleep 5; done` exits 0 when every attempt connection-refused because the final `sleep 5` returns 0. OAP takes ~50 s to start in CI, so all attempts fail before OAP is ready, and the setup step silently succeeds with zero traffic ingested. Fix: `curl -sS -f --retry 30 --retry-delay 5 --retry-connrefused --retry-all-errors --max-time 10 ...` + `set -e` at step top. **`swctl` flag rejected.** If a verify case uses a flag the pinned `swctl` commit doesn't support (`service ls --layer` vs `service ly`), the whole case fails 20× before CI gives up. Fire each verify query by hand once before pushing (step 6 above). **Published image cache miss in CI.** `docker compose pull` sometimes hits rate limits or unreachable registries; the test spins until timeout with "dependency failed to start". Look at the CI log for `Error response from daemon: pull access denied` or `manifest unknown`. If you see that, pin a different image tag that's definitely published (check `docker manifest inspect <tag>` locally), not a floating one. ## Common test cases | Shorthand | Path | |-----------|------| | `simple/jdk` | `test/e2e-v2/cases/simple/jdk/` | | `storage/banyandb` | `test/e2e-v2/cases/storage/banyandb/` | | `storage/elasticsearch` | `test/e2e-v2/cases/storage/elasticsearch/` | | `alarm` | `test/e2e-v2/cases/alarm/` | | `log` | `test/e2e-v2/cases/log/` | | `profiling/trace` | `test/e2e-v2/cases/profiling/trace/` |