validate-side-by-side

# Validate side by side **Audience:** operators of the published `obs-migrate` CLI (PyPI/`uvx`), using public docs and their real source + Elastic/Kibana — not a repo lab harness. Goal: run `obs-migrate compare` to check per-panel parity against the source — numerically where the native PROMQL oracle applies, and as a structural-only row (never hidden) for Datadog, non-PromQL, or no-oracle panels. The command issues read-only `_query` requests against the **target** Elasticsearch cluster; it writes nothing to Grafana or Datadog and does not re-run migration. ## Prerequisites (install) These skills help **operators** of the published CLI (not a repo checkout). If `obs-migrate` is missing or `doctor` is not **Ready**, follow `install-obs-migrate` first — that skill owns PyPI/`uvx`/pip, extras, and Python/`uv` gotchas. Do not invent alternate install commands here. ```bash uvx --from 'elastic-observability-migration[all]' obs-migrate doctor # After a persistent install, the same check is: obs-migrate doctor ``` Source/Elastic credentials: `connect-to-o11y-source` (and your env exports). ## Command ```bash obs-migrate compare \ --artifact-dir <output-dir>/dashboards \ --es-url "$ELASTICSEARCH_ENDPOINT" \ --api-key "$KEY" \ --report-out <output-dir>/dashboards/comparison_report.json ``` `--es-url` / `--api-key` default to `ELASTICSEARCH_ENDPOINT|ES_URL` and `KEY`. Repeat `--artifact-dir` to merge multiple migrate runs. Add `--ca-cert` / `--insecure` for TLS, and tune `--index`, `--step-seconds`, `--window-minutes`, or `--report-out` when the default oracle window does not match the dashboard. Full flag list and defaults: `https://github.com/elastic/observability-migration-platform/blob/main/docs/command-contract.md`. Optional package-native scorecard that also runs emitted-query acceptance (and optionally compare in-process): ```bash obs-migrate verify \ --artifact-dir <output-dir>/dashboards \ --es-url "$ELASTICSEARCH_ENDPOINT" \ --api-key "$KEY" \ --compare \ --report-out <output-dir>/dashboards/verify_report.json ``` ## What "verified" means (honest table) | Source / cluster | Mode | Verdicts | What it proves | |---|---|---|---| | PromQL / Grafana on a cluster with native PROMQL | Numeric (`mode=native_oracle`) | `STRICT_PASS` (≤1% max relative error), `FUZZY_PASS` (≤5%), `SHAPE_PASS` (5% < error ≤ 25%), `FAIL` (no overlap, or relative error above 25%), `SKIP`, `ERROR` | Translated ES|QL buckets match Elasticsearch's native `PROMQL(<source query>)` oracle over the same index and time window. Multi-target panels verify one row per target (`target` = refId); mirrorable stat reductions (window `MAX` / latest-bucket `LAST`) compare as scalars. **Never trust `SHAPE_PASS` by name — read `max_relative_error`.** | | Panels whose packets carry live source-vs-target verdicts (`obs-migrate migrate --source datadog --source-execution --validate`) | Live source (`mode=live_source`) | `SOURCE_PASS`, `SOURCE_DRIFT`, `SOURCE_FAIL` (fails the run), `ERROR` (target broken) | The source API's own numbers vs the target ES|QL over the same window — only meaningful when both ingest the same telemetry. Without matching data you commonly see `SOURCE_DRIFT` (does **not** alone fail the run) | | Datadog panels without live comparison, non-PromQL panels, or clusters without native PROMQL | Structural (`mode=structural`) | `STRUCTURAL` | Semantic gate only — **not numerically verified**; the command checked shape/metadata, not bucket-by-bucket numbers | Never describe a `STRUCTURAL` row as numeric proof. Never hide the structural fallback behind exit code `0`. ## Deterministic flow (optional) When live telemetry is sparse or mismatched, seed synthetic data both sides can read, compare, then clean up: ```bash obs-migrate seed-sample-data \ --artifact-dir <output-dir>/dashboards \ --es-url "$ELASTICSEARCH_ENDPOINT" \ --api-key "$KEY" obs-migrate compare \ --artifact-dir <output-dir>/dashboards \ --es-url "$ELASTICSEARCH_ENDPOINT" \ --api-key "$KEY" \ --report-out <output-dir>/dashboards/comparison_report.json obs-migrate remove-sample-data \ --artifact-dir <output-dir>/dashboards \ --es-url "$ELASTICSEARCH_ENDPOINT" \ --api-key "$KEY" \ --confirm ``` `seed-sample-data` and `remove-sample-data` are **ES-only** (they touch the target cluster, not the source). `remove-sample-data` is **dry-run by default** — pass `--confirm` to actually delete seeder-owned streams. Auth flag is **`--api-key`** (not `--es-api-key`). ## Reading the result The command writes **`comparison_report.json`** (machine-readable) and a sibling **`comparison_report.md`** with a panel-by-panel table: dashboard, panel, mode, verdict, max relative error, native/translated/common series counts, reason. Numeric JSON rows also carry `native_series`/`translated_series`/`common_series` and `notes`, and every FAIL or SKIP has a populated reason. **`--report-out` defaults to `comparison_report.json` in the current working directory** — pass an explicit path under the artifact dir to keep the report beside the migration artifacts. **Exit codes:** - **`2`** — Elasticsearch unreachable or invalid input (missing credentials, bad/missing `verification_packets.json`). - **`1`** — at least one panel parity check returned `FAIL` or `ERROR` (or a live source comparison returned `SOURCE_FAIL`). - **`0`** — otherwise (including runs where every row is `STRUCTURAL`, `SOURCE_DRIFT`, `SKIP`, or non-`FAIL` numeric verdicts). Besides **`FAIL`** / **`SOURCE_FAIL`** / **`ERROR`** (which set exit `1`), verdicts **`SKIP`**, **`SHAPE_PASS`**, and **`SOURCE_DRIFT`** do not fail the run but still warrant a look — route them to **`explain-migration-gaps`** or re-check `--window-minutes` / `--step-seconds` / target telemetry before trusting an all-green exit code. **`SHAPE_PASS` is bounded at 25% relative error; above that the row is `FAIL`.** Still read `max_relative_error` on every `SHAPE_PASS` row — a 20% miss is a pass-shaped name, not numeric proof. Route panels with verdict **`FAIL`** / **`SOURCE_FAIL`** or structural rows the user expected to be numerically verified to the **`explain-migration-gaps`** skill for rebuild guidance. Note that **`STRUCTURAL`** can also hide panels that migrated with **accepted approximations** (`migrated_with_warnings` / Datadog `warning`) — structural shape ≠ semantic fidelity; use `explain-migration-gaps` when the user expected numeric proof. For a shareable headline scorecard (not per-panel parity), use **`report-migration-coverage`**. ### Beyond compare (operator follow-ups) `obs-migrate compare` proves oracle / live-source parity where applicable; it does **not** prove Lens UI render. When an operator asks "will it show up correctly in Kibana?": - `obs-migrate verify` (emitted-query acceptance + optional `--compare`) - `grafana-validate-uploaded` and/or open the dashboard in Kibana view mode - `debug-uploaded-kibana-dashboard` for a single broken panel - Optional lab/repo gates listed by `obs-migrate verify` and in `https://github.com/elastic/observability-migration-platform/blob/main/docs/testing.md` — not required for the normal operator path ## Honest limits / Do NOT - **Exit `0` with all-`STRUCTURAL` rows is NOT numeric proof** — you only confirmed structural compatibility, not that numbers match. Datadog without `--source-execution` typically lands here. - **`SHAPE_PASS` is not numeric proof.** It means labels overlap and relative error is between 5% and 25%. Read `max_relative_error`. Above 25% the row is `FAIL`. - **A `FAIL` / `SOURCE_DRIFT` may be a data-window or step mismatch**, not a translation bug — re-run with `--window-minutes` and `--step-seconds` aligned to the dashboard (and consider `seed-sample-data`) before declaring a translation defect. - **Do not claim Datadog panels were numerically verified via the PROMQL oracle** — without live source packets they degrade to `STRUCTURAL`; with `--source-execution --validate` they use `SOURCE_*` verdicts instead. - **Do not write to the source** — compare is read-only on the target cluster; it does not prove the uploaded Kibana dashboard renders in the UI (empty panels may still be missing telemetry). ## See also - `install-obs-migrate` — install/doctor when the CLI is missing or not Ready. - `report-migration-coverage` skill — shareable coverage summary from migrate artifacts. - `explain-migration-gaps` skill — why a panel did not migrate cleanly and how to rebuild it. - `debug-uploaded-kibana-dashboard` skill — UI render failures after upload. - `prepare-production-cutover` skill — go/no-go using compare + render-audit + coverage. - `obs-migrate seed-sample-data` / `obs-migrate remove-sample-data` — optional deterministic data setup and teardown (`https://github.com/elastic/observability-migration-platform/blob/main/docs/command-contract.md`). - For Datadog, `obs-migrate migrate --source datadog --source-execution --validate` fills the verification packets' `source_execution`/`comparison` blocks so a later `obs-migrate compare` can emit `SOURCE_*` verdicts (needs DD creds and comparable telemetry on both sides). - `https://github.com/elastic/observability-migration-platform/blob/main/docs/command-contract.md` — full compare, seed, and remove flag reference for the installed version. - `https://github.com/elastic/observability-migration-platform/blob/main/docs/testing.md` — layered verifier and render-audit gates.