Validate Side By Side

Validate side by side

Audience: operators of the published obs-migrate CLI (PyPI/uvx), using public docs and their real source + Elastic/Kibana — not a repo lab harness.

Goal: run obs-migrate compare to check per-panel parity against the source — numerically where the native PROMQL oracle applies, and as a structural-only row (never hidden) for Datadog, non-PromQL, or no-oracle panels. The command issues read-only _query requests against the target Elasticsearch cluster; it writes nothing to Grafana or Datadog and does not re-run migration.

Prerequisites (install)

These skills help operators of the published CLI (not a repo checkout). If obs-migrate is missing or doctor is not Ready, follow install-obs-migrate first — that skill owns PyPI/uvx/pip, extras, and Python/uv gotchas. Do not invent alternate install commands here.

uvx --from 'elastic-observability-migration[all]' obs-migrate doctor
# After a persistent install, the same check is: obs-migrate doctor

Source/Elastic credentials: connect-to-o11y-source (and your env exports).

Command

obs-migrate compare \
  --artifact-dir <output-dir>/dashboards \
  --es-url "$ELASTICSEARCH_ENDPOINT" \
  --api-key "$KEY" \
  --report-out <output-dir>/dashboards/comparison_report.json

--es-url / --api-key default to ELASTICSEARCH_ENDPOINT|ES_URL and KEY. Repeat --artifact-dir to merge multiple migrate runs. Add --ca-cert / --insecure for TLS, and tune --index, --step-seconds, --window-minutes, or --report-out when the default oracle window does not match the dashboard. Full flag list and defaults: https://github.com/elastic/observability-migration-platform/blob/main/docs/command-contract.md.

Optional package-native scorecard that also runs emitted-query acceptance (and optionally compare in-process):

obs-migrate verify \
  --artifact-dir <output-dir>/dashboards \
  --es-url "$ELASTICSEARCH_ENDPOINT" \
  --api-key "$KEY" \
  --compare \
  --report-out <output-dir>/dashboards/verify_report.json

What "verified" means (honest table)

Never describe a STRUCTURAL row as numeric proof. Never hide the structural fallback behind exit code 0.

Deterministic flow (optional)

When live telemetry is sparse or mismatched, seed synthetic data both sides can read, compare, then clean up:

obs-migrate seed-sample-data \
  --artifact-dir <output-dir>/dashboards \
  --es-url "$ELASTICSEARCH_ENDPOINT" \
  --api-key "$KEY"

obs-migrate compare \
  --artifact-dir <output-dir>/dashboards \
  --es-url "$ELASTICSEARCH_ENDPOINT" \
  --api-key "$KEY" \
  --report-out <output-dir>/dashboards/comparison_report.json

obs-migrate remove-sample-data \
  --artifact-dir <output-dir>/dashboards \
  --es-url "$ELASTICSEARCH_ENDPOINT" \
  --api-key "$KEY" \
  --confirm

seed-sample-data and remove-sample-data are ES-only (they touch the target cluster, not the source). remove-sample-data is dry-run by default — pass --confirm to actually delete seeder-owned streams. Auth flag is --api-key (not --es-api-key).

Reading the result

The command writes comparison_report.json (machine-readable) and a sibling comparison_report.md with a panel-by-panel table: dashboard, panel, mode, verdict, max relative error, native/translated/common series counts, reason. Numeric JSON rows also carry native_series/translated_series/common_series and notes, and every FAIL or SKIP has a populated reason. --report-out defaults to comparison_report.json in the current working directory — pass an explicit path under the artifact dir to keep the report beside the migration artifacts.

Exit codes:

• 2 — Elasticsearch unreachable or invalid input (missing credentials, bad/missing verification_packets.json).
• 1 — at least one panel parity check returned FAIL or ERROR (or a live source comparison returned SOURCE_FAIL).
• 0 — otherwise (including runs where every row is STRUCTURAL, SOURCE_DRIFT, SKIP, or non-FAIL numeric verdicts).

Besides FAIL / SOURCE_FAIL / ERROR (which set exit 1), verdicts SKIP, SHAPE_PASS, and SOURCE_DRIFT do not fail the run but still warrant a look — route them to explain-migration-gaps or re-check --window-minutes / --step-seconds / target telemetry before trusting an all-green exit code. SHAPE_PASS is bounded at 25% relative error; above that the row is FAIL. Still read max_relative_error on every SHAPE_PASS row — a 20% miss is a pass-shaped name, not numeric proof.

Route panels with verdict FAIL / SOURCE_FAIL or structural rows the user expected to be numerically verified to the explain-migration-gaps skill for rebuild guidance. Note that STRUCTURAL can also hide panels that migrated with accepted approximations (migrated_with_warnings / Datadog warning) — structural shape ≠ semantic fidelity; use explain-migration-gaps when the user expected numeric proof. For a shareable headline scorecard (not per-panel parity), use report-migration-coverage.

Beyond compare (operator follow-ups)

obs-migrate compare proves oracle / live-source parity where applicable; it does not prove Lens UI render. When an operator asks "will it show up correctly in Kibana?":

• obs-migrate verify (emitted-query acceptance + optional --compare)
• grafana-validate-uploaded and/or open the dashboard in Kibana view mode
• debug-uploaded-kibana-dashboard for a single broken panel
• Optional lab/repo gates listed by obs-migrate verify and in https://github.com/elastic/observability-migration-platform/blob/main/docs/testing.md — not required for the normal operator path

Honest limits / Do NOT

• Exit 0 with all-STRUCTURAL rows is NOT numeric proof — you only confirmed structural compatibility, not that numbers match. Datadog without --source-execution typically lands here.
• SHAPE_PASS is not numeric proof. It means labels overlap and relative error is between 5% and 25%. Read max_relative_error. Above 25% the row is FAIL.
• A FAIL / SOURCE_DRIFT may be a data-window or step mismatch, not a translation bug — re-run with --window-minutes and --step-seconds aligned to the dashboard (and consider seed-sample-data) before declaring a translation defect.
• Do not claim Datadog panels were numerically verified via the PROMQL oracle — without live source packets they degrade to STRUCTURAL; with --source-execution --validate they use SOURCE_* verdicts instead.
• Do not write to the source — compare is read-only on the target cluster; it does not prove the uploaded Kibana dashboard renders in the UI (empty panels may still be missing telemetry).

See also

• install-obs-migrate — install/doctor when the CLI is missing or not Ready.
• report-migration-coverage skill — shareable coverage summary from migrate artifacts.
• explain-migration-gaps skill — why a panel did not migrate cleanly and how to rebuild it.
• debug-uploaded-kibana-dashboard skill — UI render failures after upload.
• prepare-production-cutover skill — go/no-go using compare + render-audit + coverage.
• obs-migrate seed-sample-data / obs-migrate remove-sample-data — optional deterministic data setup and teardown (https://github.com/elastic/observability-migration-platform/blob/main/docs/command-contract.md).
• For Datadog, obs-migrate migrate --source datadog --source-execution --validate fills the verification packets' source_execution/comparison blocks so a later obs-migrate compare can emit SOURCE_* verdicts (needs DD creds and comparable telemetry on both sides).
• https://github.com/elastic/observability-migration-platform/blob/main/docs/command-contract.md — full compare, seed, and remove flag reference for the installed version.
• https://github.com/elastic/observability-migration-platform/blob/main/docs/testing.md — layered verifier and render-audit gates.