netdata-migration

# Netdata migration This skill guides a migration from another observability vendor's metric pipeline to Netdata. The goal is OpenTelemetry-native instrumentation that exports to Netdata over OTLP/gRPC, with existing dashboards and alerts rebuilt on the Netdata side where parity is possible. ## When to use this skill - The user is actively cutting over from a vendor SDK (Datadog, New Relic, Dynatrace) to OpenTelemetry + Netdata. - The user has a Prometheus scrape pipeline and wants to reuse it while adding Netdata ingestion. - The user wants to know which signals from the old vendor have a one-to-one analogue in Netdata. ## Key facts - Netdata accepts metrics, logs, and traces via OTLP/gRPC. Traces need a Netdata Agent built from `master` after 2026-08-17 (nightly) or the first stable release after v2.11.1. Stable v2.11.x has no trace receiver; on it, keep traces on the old backend until the Agent is upgraded. - Netdata's Traces tab covers trace search, lookup by trace ID, the slowest traces, a duration overview, and attribute facets. It is not a full APM suite; see [`rules/parity-expectations.md`](./rules/parity-expectations.md) for the gaps. - Most vendor SDKs (Datadog, New Relic, Dynatrace) now have an OpenTelemetry-compatible mode or a first-party exporter. Prefer that path over writing a second agent. - Prometheus metrics can be ingested without migration code: run the OTel Collector's `prometheus` receiver, or use Netdata's native Prometheus endpoint scraping. - Dashboard parity is rarely 1:1. Netdata charts are auto-generated by default, shaped by mapping files. Rebuilding a Datadog-style curated dashboard is a separate task. - Alerts do not migrate. Each vendor's rule language is different. Expect to rewrite alert expressions against Netdata's alert engine. ## Step-by-step 1. Inventory what the current stack emits: - Traces (a migration target when the Agent accepts traces; check the version first). - Metrics (the main migration target). - Logs (second migration target). - Vendor-specific events (sometimes unportable). 2. Pick a strategy: - **Parallel run**: keep the vendor pipeline, add an OTel Collector that exports to Netdata in parallel. Compare for 1-4 weeks. Then cut over. - **Cutover**: replace the vendor SDK in one go. Faster but no side-by-side validation. 3. Replace the SDK. See the per-vendor rule files in [References](#references). 4. Reconfigure the exporter. OTLP/gRPC at the Netdata receiver port (default 4317). Set `service.name`, `service.version`, `deployment.environment` as resource attributes. 5. Rebuild alerts against Netdata. Prioritize by current alert firing frequency; low-volume alerts can wait. 6. Decommission the vendor agent once the parallel-run window shows no regressions. ## Common mistakes - Leaving the vendor SDK loaded alongside the OTel SDK in the same process. They often patch the same libraries and conflict. Remove the vendor SDK fully. - Forwarding traces to a stable v2.11.x (or older) Agent. It has no trace receiver; every export fails. Check trace support first (see `skills/netdata-otel-setup/rules/trace-ingestion.md`). - Leaving trace sampling at the vendor's defaults. Netdata keeps every span it receives, bounded by `traces.retention` (1GB or 7 days by default). Size retention, or sample in the Collector, before cutover. - Expecting metric names to carry over. OTel semantic conventions differ from Datadog/New Relic metric names. Use OTel names going forward; rebuild dashboards against them. - Trying to replay historical vendor data into Netdata. There is no backfill tool, and Netdata rejects spans that started more than 24 hours ago. Cutover's retention starts at the cutover date. ## Migration sequencing A reliable migration order for a medium-sized fleet (20-100 services, multiple teams): 1. **Week 0: tooling in place.** Deploy a Netdata Parent, an OTel Collector gateway, and confirm a single smoke-test service flows end to end. No production services migrated. 2. **Weeks 1-2: easiest services.** Pick 1-3 services with minimal custom instrumentation (pure auto-instrumentation wins). Migrate. Run parallel for at least one week. 3. **Weeks 3-6: medium services.** Services with some custom metrics and straightforward alerts. Rebuild alerts as you go; this is where the bulk of the alert-rewriting happens. 4. **Weeks 7-10: hard services.** Custom SDK extensions, bespoke dashboards, cross-service SLOs. Budget extra time for rebuilding alert dependencies. 5. **Final weeks: decommission.** Stop the vendor agents. Archive any historical data that must be retained for compliance. A cutover under two weeks is possible for a small shop. A fleet migration under six weeks usually means something was skipped. ## Verification Run the MCP integration skill's telemetry-flow check against the service that was just migrated. Before cutover, also run the same check against the old vendor. A parallel-run comparison: ```text For service <name>, list the top 10 metrics by volume in Netdata in the last 30 minutes. List the top 10 metrics by volume in the old vendor for the same window. Identify any metric present in the old vendor but absent in Netdata. ``` Gaps revealed by that check usually trace back to missing instrumentation packages (for example, the Datadog SDK wrapped a library that the OTel auto-instrumentation set does not yet cover). A second check to run a week after cutover: have every on-call acknowledge that the runbooks they actually used in the last 30 days still work against Netdata. A runbook that no longer matches the observability tool is a silent hazard. Plan an explicit "metric inventory" diff, not just a spot check. Export the metric list from the old vendor (most have an API for this), and compare to Netdata's `list_metrics` output. The diff is what tells you whether something was quietly dropped. ## References - [`rules/from-datadog.md`](./rules/from-datadog.md) - [`rules/from-new-relic.md`](./rules/from-new-relic.md) - [`rules/from-dynatrace.md`](./rules/from-dynatrace.md) - [`rules/from-prometheus.md`](./rules/from-prometheus.md) - [`rules/parity-expectations.md`](./rules/parity-expectations.md) - OpenTelemetry semantic conventions: https://opentelemetry.io/docs/specs/semconv/