understand-source-schema

name

understand-source-schema

description

Use when the user asks how their schema/fields/metric names/labels map or translate to Elastic, why migrated panels can't find data, what fields they need, or how to customize/override the field mapping (rule pack / field profile) — explains how a source observability schema (Prometheus metric/label names, Datadog dotted metrics/tags) maps to Kibana/Elastic field names, shows the concrete source-to-target field mapping for the user's own dashboards, and what to change so migrated dashboards find data.

Understand the source schema (source → Elastic field mapping)

Goal: help the user see exactly how their source field names become Elastic field names, get that mapping for their dashboards, and know how to override it. Source schemas (Prometheus instance/job/node_cpu_seconds_total, Datadog system.cpu.user/host) usually do not match Elastic field names, so this gap is expected, not a bug.

How the mapping works (Grafana)

SchemaResolver (adapters/source/grafana/schema.py) first auto-detects how your Prometheus data actually landed in Elastic by reading _field_caps from --es-url, then rewrites metric names, labels, and metric types to match that layout. There are three layouts (schema profiles):

Schema profile	How the data got into Elastic	Metric http_requests_total →	Label service →
prometheus_remote_write	Elastic Fleet/Agent Prometheus integration	prometheus.http_requests_total.counter / .value / .rate (suffix by role)	prometheus.labels.service
prometheus_native	Native ES /_prometheus/api/v1/write endpoint	metrics.http_requests_total	labels.service
generic / OTel (none detected)	OTel collector, custom mapping, or no data found	http_requests_total (pass-through)	exact field → OTel candidate (service.name) → as-is

Label resolution order (resolve_label): ignored labels → rule-pack label_rewrites → exact field match (source-faithful) → profile-namespaced field (prometheus.labels.<l> / labels.<l>) → discovered OTel mapping from _field_caps → built-in candidate (e.g. instance → service.instance.id/host.name, job → service.name) → pass-through.

Metric type matters too: rate()/irate() only work if the metric is stored as a counter. is_counter() decides from rule-pack metric_kinds → counter_suffixes → the field's time_series_metric capability → the profile's counter field. A counter ingested as a gauge breaks rate math even when the field name is right.

Hard dependency — ingest first, then migrate with --es-url and --preflight. Profile detection only works when the data is already in Elastic and --es-url is reachable. If it is not (wrong/missing key, TLS failure, or migrating before pointing Prometheus at Elastic), detection finds nothing, the profile is none, and the resolver falls back to OTel candidates + pass-through — dashboards look migrated but query the wrong fields. Confirm the profile via Grafana required_target_contract.json (schema_profile, field_capabilities_discovery, field status) from a preflight run before trusting any mapping.

Datadog uses field profiles (--field-profile): metric_map (explicit metric overrides), tag_map (tag → ES field), plus metric_prefix/tag_prefix for unmapped names. Built-ins: otel (default), prometheus, elastic_agent, passthrough. See docs/sources/grafana.md and docs/sources/datadog.md for the full tables.

Get the mapping for the user's own dashboards

Assume the user installed the package (obs-migrate on PATH); prefix .venv/bin/ only for a repo checkout. Run a migration to an artifact dir with a live --es-url so the resolver can confirm which target fields actually exist:

export GRAFANA_URL="https://grafana.example.com" GRAFANA_USER="..." GRAFANA_PASS="..."
export ELASTICSEARCH_ENDPOINT="https://...es..." KEY="<api-key>"

obs-migrate migrate \
  --source grafana --input-mode api \
  --output-dir migration_output \
  --assets dashboards --preflight \
  --es-url "$ELASTICSEARCH_ENDPOINT" --es-api-key "$KEY"

(Have exported JSON instead of API access? Use --input-mode files --input-dir <their-dashboards-dir>.) --es-url is what makes the field-existence (confirmed/missing) check meaningful; --preflight writes the contract artifacts below.

Get the purpose-built per-panel mapping table (start here)

The most direct answer to "how do my fields map?" is the schema-change report, a per-panel dashboard │ panel │ source_fields │ target_stream │ target_fields table. Dashboard migration writes it automatically at <output-dir>/dashboards/schema_change_report.md, alongside <output-dir>/dashboards/telemetry_contract.json.

To regenerate the report, or to merge several source outputs into one table, use the installed package command (no source checkout, no scripts/ directory needed):

obs-migrate schema-report \
  --artifact-dir migration_output/dashboards \
  --output schema_change_report.md

Point --artifact-dir at the per-source dashboards/ output (the dir containing yaml/ and verification_packets.json). Repeat --artifact-dir to merge several sources into one report. Add --contract-out telemetry_contract.json to also emit the machine-readable telemetry producer contract. Open schema_change_report.md and read the table.

Where else the same information lives

These artifacts are also written by the migration run itself, under migration_output/dashboards/:

What	File	Notes
Grafana required target fields + whether they exist	required_target_contract.json	includes schema_profile, field_capabilities_discovery, and each resolved target field's status (e.g. confirmed/missing/unknown) when --es-url was used.
Datadog required target fields + whether they exist	target_readiness_contract.json	includes the active field_profile, metric/log index patterns, source fields, resolved target fields, and status.
Per-panel translation detail (source vs. translated query)	verification_packets.json	Open the file to read the exact key names rather than assuming them — packet shape varies.
Must-fix worklist	migration_summary.md	human-readable verdict + actions

Customize / override the mapping

Grafana — emit a starter rule pack, edit, re-run with --rules-file:

obs-migrate extensions --source grafana --format yaml --template-out custom-rule-pack.yaml

query:
  label_rewrites:
    instance: my.host.field
    job: my.service.field
  label_candidates:
    datacenter: [cloud.region, cloud.availability_zone]
  ignored_labels: [__name__]
controls:
  field_overrides:
    instance: service.instance.id

obs-migrate migrate --source grafana ... --rules-file custom-rule-pack.yaml

The CLI can also suggest a starter pack from validation failures via --suggest-rule-pack-out <path> (writes auto-detected label candidates). extensions and --suggest-rule-pack-out are shipped in the package.

Datadog — pick a built-in --field-profile {otel,prometheus,elastic_agent,passthrough} or pass a custom YAML profile path (metric_map/tag_map). Emit a starter with obs-migrate extensions --source datadog --template-out custom-field-profile.yaml.

Do NOT

• Do not assert verification_packets.json field/key names from memory — open the file and read them. Packet keys are easy to get subtly wrong.
• Do not invent metric-name transformation rules (e.g. exact prometheus.<metric>.value forms) without confirming against the emitted YAML/packets for the actual run.
• Do not trust field mappings from a run where --es-url was unreachable or the target had no data yet — with no detected schema profile the resolver guesses OTel candidates and passes names through. Ingest first, then re-run with a reachable --es-url.
• Do not treat a source-vs-Elastic naming difference as a migration bug — it is the schema gap this skill exists to map and resolve.
• Do not reach for repo-only scripts for the schema report: migration writes schema_change_report.md automatically, and obs-migrate schema-report is the package-native regeneration/merge command. (scripts/generate_telemetry_contract.py is the same thing in a source checkout.)

See also

• obs-migrate schema-report --help — the per-panel source→target table command (shipped in the package).
• docs/sources/grafana.md (SchemaResolver + rule packs) and docs/sources/datadog.md (field profiles) — the full mapping tables (online docs / repo).
• assess-migration-readiness skill — missing fields/metrics show up there as blockers/actions.
• obs-migrate extensions --help and grafana-migrate --help — rule-pack and --rules-file options for the installed version.