signoz-reducing-telemetry-cost

name	signoz-reducing-telemetry-cost
description	Investigate and reduce SigNoz telemetry ingestion cost and metric cardinality across metrics, logs, and traces. Find what drives SigNoz spend (via the Cost Meter), which metrics have runaway or unbounded label cardinality, and safe, dashboard-, alert-, and Infra-page-aware ways to cut volume. Make sure to use this skill whenever the user asks "why is my SigNoz bill so high", "what's driving my ingestion cost", "reduce telemetry volume", "which metrics cost the most", "cardinality health check", or "what can I safely drop", or otherwise asks about telemetry spend, ingestion volume, or metric cardinality, even if they don't say "cost" or "optimize" explicitly.
argument-hint	<investigation focus, such as metrics cost or cardinality health>

Prerequisites

This skill calls SigNoz MCP server tools heavily (signoz_list_metrics, signoz_get_field_keys, signoz_execute_builder_query, signoz_get_top_metrics, signoz_check_metric_usage, signoz_check_metric_cardinality, signoz_aggregate_logs, signoz_aggregate_traces, signoz_search_logs, signoz_list_alert_rules, signoz_get_alert, signoz_get_service_top_operations). Before running the workflow, confirm the signoz_* tools are available. If they are not, the SigNoz MCP server is not installed or configured; run signoz-mcp-setup first. The whole investigation is grounded in these queries; without the server there is nothing to analyze.

Read both reference files before drawing conclusions:

• references/otel-attribute-cardinality.md: classify any metric label you encounter.
• references/infra-do-not-drop.md: the metrics that power the built-in Infrastructure page and the APM/Services page (span-derived signoz_* RED metrics); never present these as "safe to drop" even when usage shows them unused.

Workflow

Always start with the Cost Meter snapshot (Step 1). For a full cost investigation, run the metrics, logs, and traces steps for every signal with data, ordered by current cost contribution (primary, secondary, tertiary). Finish with the report (Step 5).

Step 1: Cost Meter snapshot

Establish the cross-signal cost picture first. Always call signoz_list_metrics with source: "meter"; treat its returned metric names, types, temporalities, and units as the live source of truth because the meter set evolves. Then query each relevant discovered metric with signoz_execute_builder_query (source: "meter", requestType: "time_series", stepInterval: 3600, the discovered temporality, and timeAggregation: "sum"); see references/cost-meter-queries.md for the full tool-argument template. Sum complete hourly buckets and exclude every datapoint marked partial: true. Do not use signoz_query_metrics for Cost Meter totals or grouped total attribution. Report only values returned by successful queries. If a query fails or returns no usable values after the MCP tools are available, show the intended query and say that the total could not be computed; never invent a total. If the tools are unavailable, follow the prerequisite instead.

For a rolling 7-day window (end = now, start = end − 7 days), get the per-signal totals (span size, log size, metric datapoints), then compute and report:

• Primary cost driver. The signal with the highest dollar weight: traces/logs at $0.30/GB, metrics at $0.10/M samples (orientation only; never quote dollar savings). This picks by cost, not by raw volume.
• Bytes per record. span.size ÷ span.count and log.size ÷ log.count, which tells you whether a signal is a payload-size problem or a volume problem.

Then break the primary signal down by environment and service. First call signoz_get_field_keys with signal: "metrics" and source: "meter"; use only keys it returns and copy each key's name, fieldDataType, fieldContext, and signal into the raw builder groupBy without translating or dropping fields. Run the same meter query with that complete groupBy and report the top ~10 per group with their share. If a non-prod environment (staging, dev, test, qa, sandbox, preview, uat, …) is > 40% of volume, recommend Ingestion Limits on that key before any signal-level change: https://signoz.io/docs/ingestion/signoz-cloud/keys/

Step 2: Metrics

Run when the Cost Meter shows metric data, ordered by its cost contribution, or when the user explicitly asks about metric cost or cardinality.

2a. Rank by volume with signoz_get_top_metrics. Returns the top 100 metrics by ingested samples with percentages pre-computed and totalValue sample counts (pass start/end). This is the volume-ranked worklist. Histogram metrics (.bucket suffix) are usually the top contributors, since each bucket boundary is a separate sample per scrape.

2b. Check usage with signoz_check_metric_usage. Pass the top metric names (batch of ≤ 50 per call). Returns { dashboards, alerts, error } per metric. A metric is a drop candidate only when its error is empty and both dashboards and alerts are empty. If error is non-empty the lookup is incomplete (a timeout, or an older SigNoz that lacks the endpoint) and the returned lists are unreliable, so never treat that metric as unused; mark it Needs one check first (verify its usage manually). A clean lookup with both lists empty is a drop candidate, except for the guard below.

Do-not-drop guard (mandatory). Before calling any empty-usage metric a "safe drop", check it against references/infra-do-not-drop.md. The Infrastructure page (Hosts / Kubernetes) queries system.* and many k8s.* / container.* metrics through built-in queries, not dashboards, so usage-check reports them empty even though dropping them breaks that page. If a candidate matches the do-not-drop set, present it as "Infra-page dependency: breaks the Hosts/Kubernetes view; confirm you don't use that view before dropping," never as "safe to drop." This overrides the empty usage result. Also exclude internal signoz_ / signoz. metrics (auto-generated RED metrics that power the APM page, not customer-controlled).

2c. Inspect cardinality with signoz_check_metric_cardinality. Run this for metrics that are not drop candidates and for any drop candidate the user chooses to retain. Cardinality analysis adds no value for a metric the user has agreed to drop. The tool returns attribute keys sorted highest-cardinality first, each with valueCount and sample values. Classify each with references/otel-attribute-cardinality.md:

• UNBOUNDED (url.full, http.target, db.query.text, client.port, trace.id, exception.stacktrace, …): grow without ceiling; flag regardless of current count.
• ACCUMULATING (container.id, k8s.pod.uid, k8s.pod.name, k8s.pod.start_time): valueCount reflects historical pod churn, not active series; explain the distinction.
• HIGH but bounded (valueCount ≳ 100): check whether dashboards/alerts actually filter on that label before recommending aggregation.

Infra identity override (mandatory). For a metric protected by references/infra-do-not-drop.md, preserve the identity attributes and page metadata used by that metric's Infra entity/view. Do not aggregate or remove entity UID/name attributes when they resolve that entity. Keep k8s.pod.start_time on Pod metrics because the Pods page uses it for Pod Age. This overrides the generic ACCUMULATING fixes in the cardinality reference.

To reduce cardinality use the metricstransform processor's aggregate_labels action to merge series (samples are the billable cost, so merging is what actually cuts it), not the transform processor's delete_key, which leaves the same sample count and creates colliding series. If a label is essential to the metric's identity, drop the whole metric or fix it at the SDK instead. For histograms, reducing bucket boundaries cuts samples with little P99 impact. Docs: https://signoz.io/docs/userguide/drop-metrics/ · https://signoz.io/docs/metrics-management/dropping-metric-labels/

2d. Review the collection interval. For a high-volume metric that must be kept, identify how it is produced and its current interval before recommending a change. A longer interval reduces ingested datapoints but also lowers time resolution, so preserve the resolution required by its dashboards and alerts. Use the source's own control: a receiver collection_interval for Collector-generated metrics, the scrape interval for Prometheus-scraped metrics, or OTEL_METRIC_EXPORT_INTERVAL for SDK push metrics when that SDK supports it. Never recommend switching a metric between delta and cumulative temporality; changing temporality for the same metric can break SigNoz queries.

Step 3: Logs

Run when the Cost Meter shows log data, ordered by its cost contribution, or when the user explicitly asks about log cost.

3a. Total + attribution decides the path.

• Total log GB (the absolute cost figure): use signoz_execute_builder_query with the discovered meter metric whose live unit and meaning represent log bytes, summed as in Step 1.
• Attribution: run the same meter query grouped by service.name. This returns one group per service plus an unset/empty-service.name group for logs with no attribution. Compute the ratio entirely from THIS grouped result so numerator and denominator share one basis; a grouped sum can differ from the ungrouped total, so never divide the grouped attributed sum by the ungrouped total:
  • attributed GB = sum of the groups with a non-empty service.name.
  • grouped total = sum of all groups (including the empty one).
  • Attribution % = attributed ÷ grouped total.
• This is a hard branch:
  • ≥ 10% → Path A (service mode).
  • < 10% → Path B (namespace mode). Logs come from an infra forwarder (Fluent Bit / Fluentd / Vector), not OTel SDKs, so service.name isn't set. Path B is a less common setup, so sanity-check its numbers.

3b. Analyze the selected attribution path and identify candidate fixes. The fixes in this step are candidates only. Complete the alert check in Step 3c before recommending any log-volume reduction, including a source log-level change.

Path A, service mode (≥ 10%). Get the severity mix with signoz_aggregate_logs, aggregation: count, groupBy: "service.name,severity_text". Classify severities: REDUCIBLE = INFO, INFORMATION, DEBUG, TRACE, VERBOSE; HIGH-SIGNAL = ERROR, FATAL, CRITICAL, WARN, WARNING. For each top service by GB:

• Reducible-dominant + own service code → candidate: set LOG_LEVEL=WARN (stops generation at source).
• Reducible-dominant + third-party library → candidate: Collector filter on instrumentation_scope.name (read the scope from a signoz_search_logs sample's scope_name).
• No source access → candidate: Collector filter on severity_text matching only the reducible set (INFO/DEBUG/TRACE), never a range that also catches WARN+.
• High-signal-dominant (WARN/ERROR > 50%) → the logs are worth keeping; a high error rate may be a real problem, so flag it separately, do not recommend filtering it away.

Service severity guard. The candidate fixes above (LOG_LEVEL=WARN, or a Collector filter scoped to INFO/DEBUG only) preserve WARN+ by construction, so the severity mix does not block them; Step 3c still determines whether an alert depends on the records. The guard applies only to actions that would also drop high-signal logs: a blanket service drop, or a severity_text filter whose range includes WARN/ERROR/FATAL/CRITICAL. Before recommending one of those, compute high-signal % = (all HIGH-SIGNAL severities, including ERROR, FATAL, CRITICAL, WARN, and WARNING, matched case-insensitively) ÷ service total. If it is > 1% (or a non-trivial absolute count), do not take the blanket action: keep the reduction scoped to INFO/DEBUG and flag the errors as a real signal.

Path B, namespace mode (< 10%).

• Top namespaces: signoz_aggregate_logs, count, groupBy: "k8s.namespace.name,deployment.environment", order by count desc, limit ~10. Use a 24h window (7-day namespace scans time out on large tenants; count is the proxy, since the Cost Meter has no namespace dimension).
• Severity per namespace: signoz_aggregate_logs, count, groupBy: "k8s.namespace.name,severity_text".
• Samples: signoz_search_logs, filter: "k8s.namespace.name = '<ns>'", small limit; read body, severity_text, scope_name.

Namespace severity guard. high-signal % = (all HIGH-SIGNAL severities, including ERROR, FATAL, CRITICAL, WARN, and WARNING, matched case-insensitively) ÷ namespace total. If it is > 1% (or a non-trivial absolute count), do not consider dropping or filtering the whole namespace. Scope the filter to the specific noisy pattern (a log category, a severity_text match, or a component). Never drop a namespace that carries active errors. Only namespaces that are essentially all INFO/DEBUG are wholesale-filter candidates, subject to Step 3c.

• Empty severity_text on samples → the forwarder ships raw lines unparsed; the fix is a json/regex parser in the OTel Collector log pipeline (without it, severity filtering is impossible).
• k8s.event.* logs are often high-volume / low-value → droppable if not alerted on.

3c. Log alerts: check before any log-reduction recommendation. signoz_list_alert_rules, paginating through every page (follow pagination.nextOffset until pagination.hasMore is false; do not stop at the first page, or an alert on a later page is missed and a filter looks safe when it isn't). Keep alertType == "LOGS_BASED_ALERT". For each, signoz_get_alert(id) and read condition.compositeQuery.queries[].spec.filter.expression + groupBy to see which service / severity / namespace it guards. If a filter would blind an alert, mark it Will break alert coverage and name the alert. Docs: https://signoz.io/docs/logs-management/guides/drop-logs/

Step 4: Traces

Run when the Cost Meter shows span data, ordered by its cost contribution, or when the user explicitly asks about span cost.

4a. Global operation-name view, read this first. signoz_aggregate_traces, aggregation: count, groupBy: "name", orderBy: "count() desc", limit 20, no service filter. This surfaces auto-instrumentation noise (health checks, SQL, cache, sidecars) across all services at once.

4b. Cost per service + ops per service. Span GB by service: use signoz_execute_builder_query with the discovered meter metric whose live unit and meaning represent span bytes, summed as in Step 1 and grouped by the service.name field returned by signoz_get_field_keys (copying its name, fieldDataType, fieldContext, and signal). Compute each service's % against the grouped total (sum of all service groups from this same query), not a top-N sum and not the separately-fetched ungrouped total; keep numerator and denominator on one grouped basis. For each top-3 service by GB, get its dominant operations with signoz_get_service_top_operations (or signoz_aggregate_traces with service: "<svc>" + groupBy: "name"). The op breakdown is top-3 for brevity, but the error-rate check (4c) and the APM guard below apply to every service you consider reducing.

4c. Error rate per service. signoz_aggregate_traces, count, groupBy: "service.name" (total), then again with error: true. Error rate = errors ÷ total, per service.

4d. Trace alerts. signoz_list_alert_rules, paginating through every page (follow pagination.nextOffset until pagination.hasMore is false). Keep alertType == "TRACES_BASED_ALERT"; signoz_get_alert(id) for what each guards.

Classify the dominant operations.

• Common noise candidates: health/liveness (/health, /ping, /ready, grpc.health.v1.Health/Check); proxy/sidecar (envoy.*, istio.*, linkerd.*). Treat these as candidates, not automatically safe removals; the APM guard below still applies.
• Research before concluding, and do not assume: SQL fragments (the language decides the library: Java → JDBC, Python → SQLAlchemy, Node → pg/mysql2/sequelize, .NET → EF/Dapper); cache commands (HMGET/GET/SET… → ioredis, redis-py, Jedis, go-redis…); unfamiliar gRPC methods. Search first; if still unidentified after searching, say so.

Fix layer. For a confirmed noise operation, prefer the deployed SDK or instrumentation library's documented disable/exclusion control so generation stops at source. Identify the language and library before naming a setting; Java, Python, Node.js, and .NET use different controls. If no SDK control exists, use a Collector filter on the operation name.

Span → APM guard (mandatory). Never recommend or configure head, probabilistic, tail, or any other trace sampling as a cost-reduction lever. When a user asks for sampling, state both effects: the built-in APM/Services metrics cover only retained traces, so absolute request counts and rates undercount real traffic; latency trends and error spikes may remain useful, but the APM page no longer represents all requests. This limited usefulness does not make sampling an allowed lever, and processor placement is not an exception.

Use SDK exclusions and Collector filters for confirmed noise. State that the removed operation will disappear from APM before giving a configuration. Docs: https://signoz.io/docs/traces-management/guides/drop-spans/

Step 5: Report what you found

Lead with a TL;DR: concise, no headers, two parts:

1. Cost orientation: which signal is the primary driver, how much, and what generates it (name the service, environment, or metric). If several things together dominate, name them with individual shares and combined impact.
2. Prioritized action list: every actionable finding in priority order, each with its status inline: Safe to implement / Needs one check first (state what) / Will break dashboards (name them) / Will break alert coverage (name the alert) / Infra-page dependency (breaks the Hosts/K8s view). For metric drops, include the volume % so the reader knows the cost impact.

After the TL;DR, add only evidence or context that was not already stated. Rank the actions using the decision order in Steps 2–4, but state each finding and guardrail once instead of repeating the action list or the per-signal playbooks.

Keep it conversational: a prioritized triage list, not a formatted report with headers and tables. Cover every signal with Cost Meter data in that single list; do not ask whether to inspect a signal the workflow already analyzed.

Guardrails

• Never declare a drop or filter "safe" on a partial check. Paginate signoz_list_alert_rules fully (through pagination.hasMore) before ruling out alert impact, and treat a non-empty error from signoz_check_metric_usage as unknown usage (needs a manual check), not as "unused." An incomplete lookup is not a green light.
• Volume comes from discovered Cost Meter metrics and their live units: bytes for span/log volume or samples for metric volume. Never cite span/log record counts as volume.
• Cost totals and grouped total attribution use signoz_execute_builder_query with raw timeAggregation: sum, hourly stepInterval: 3600, and complete datapoints only, never signoz_query_metrics.
• Every Cost Meter builder_query sent through that raw escape hatch includes limit: 100 and Query Builder v5 order: [{key:{name:"__result"}, direction:"desc"}]. This is wire order, not dashboard orderBy; preserve it in grouped queries. The limit ranks groups over the whole window, so call out the possibility that a short-lived group falls outside the top N.
• Never present an Infra-page or APM metric as safe to drop (see references/infra-do-not-drop.md), even when usage-check shows no dashboards or alerts.
• Logs: never recommend dropping/filtering a whole service or namespace that carries active ERROR/WARN (> 1% high-signal, or a non-trivial absolute count); scope the filter to the noisy pattern. Fix order: at source (LOG_LEVEL=WARN) → Collector scope filter (instrumentation_scope.name) → severity_text filter.
• Logs attribution is a hard threshold: ≥ 10% → Path A; < 10% → Path B. Compute it.
• Traces: never recommend or configure head, probabilistic, tail, or any other sampling. Sampling limits the built-in APM/Services metrics to retained traces, so absolute request counts and rates no longer represent all traffic.
• UNBOUNDED and IDENTIFIER labels are always worth flagging: the problem is trajectory, not just current count.
• Never suggest changing retention. Never estimate dollar savings. Never call a dashboard or alert unused/noisy/redundant; report counts only.
• Anchor claims to query results. If a signal has no Cost Meter data, say so; do not substitute count-based proxies. Don't re-list metrics already shown in the breakdown; metrics outside the top ~20 by volume are individually negligible.

Additional resources

• references/cost-meter-queries.md: Cost Meter discovery, the full signoz_execute_builder_query template, unit conversion, and the grouped-vs-ungrouped caveat.
• references/infra-do-not-drop.md: the metrics behind the built-in Infrastructure and APM/Services pages that must never be recommended for dropping.
• references/otel-attribute-cardinality.md: reference for classifying metric labels (UNBOUNDED / ACCUMULATING / BOUNDED / IDENTIFIER).
• signoz-generating-queries skill: for the ad-hoc follow-up queries this investigation points to.