Intercom Observability
Overview
Comprehensive observability for Intercom integrations covering Prometheus metrics,
OpenTelemetry traces, structured logging, and alert rules for error rates, latency,
and rate-limit usage. Read this page for the workflow and shape of each layer, then
drill into references/implementation.md for the full,
copy-pasteable code and references/examples.md for
end-to-end worked scenarios.
Prerequisites
- Prometheus or compatible metrics backend
- OpenTelemetry SDK (optional, for tracing)
- Pino or similar structured logger
- Grafana or alerting system
Instructions
Build the six observability layers in order. Each step below is the summary and the
essential skeleton โ the complete implementation for every step lives in
references/implementation.md.
Step 1: Prometheus metrics
Define five instruments on a shared Registry: a request counter, a duration
histogram, an error counter, a rate-limit gauge, and a webhook counter. Label by
endpoint/method/status (never by unbounded IDs โ see Error Handling).
import { Registry, Counter, Histogram, Gauge } from "prom-client";
const registry = new Registry();
const intercomRequests = new Counter({
name: "intercom_api_requests_total",
help: "Total Intercom API requests",
labelNames: ["endpoint", "method", "status"] as const,
registers: [registry],
});
Full metric set โ references/implementation.md, Step 1.
Step 2: Instrumented client wrapper
Wrap IntercomClient in a Proxy that times every service method, increments the
success/error counters, records error/status codes on IntercomError, and zeros the
rate-limit gauge on a 429 โ so instrumentation is automatic for all endpoints.
Full proxy โ references/implementation.md, Step 2.
Step 3: Structured logging
Configure Pino with a contact serializer that emits only id/role and never
logs email, name, or phone. Add logIntercomOp and logWebhook helpers for
consistent operation/webhook log lines.
Full logger โ references/implementation.md, Step 3.
Step 4: OpenTelemetry tracing
Wrap calls in tracedIntercomCall, which opens a per-operation intercom.* span, sets
OK/ERROR status, records exceptions, and attaches status_code/error_code/request_id
attributes on Intercom errors.
Full tracer โ references/implementation.md, Step 4.
Step 5: Alert rules
Ship the Prometheus rule group with five alerts: high error rate (>5%), high P95
latency (>3s), low rate limit (<1000), auth failures (401s), and webhook failures.
Full YAML โ references/implementation.md, Step 5.
Step 6: Metrics endpoint
Expose the registry on GET /metrics for Prometheus to scrape.
Full route โ references/implementation.md, Step 6.
Output
Applying this skill produces:
- Instrumented client โ an
IntercomClient proxy that emits metrics on every call, with zero per-call changes to existing code.
- Metrics โ
intercom_api_requests_total, intercom_api_request_duration_seconds, intercom_api_errors_total, intercom_rate_limit_remaining, intercom_webhooks_processed_total, scraped at GET /metrics.
- Traces โ one per-operation
intercom.* span per call, with Intercom error attributes on failures.
- Structured logs โ PII-redacted JSON operation and webhook log lines.
- Alerts โ a Prometheus rule group covering error rate, latency, rate limit, auth, and webhooks.
Key metrics summary
| Metric | Type | Alert Threshold |
|---|
intercom_api_requests_total | Counter | N/A (baseline) |
intercom_api_request_duration_seconds | Histogram | P95 > 3s |
intercom_api_errors_total | Counter | > 5% error rate |
intercom_rate_limit_remaining | Gauge | < 1000 |
intercom_webhooks_processed_total | Counter | Failed > 10% |
Error Handling
| Issue | Cause | Solution |
|---|
| High cardinality | Too many unique labels | Use endpoint groups, not IDs |
| Missing metrics | Uninstrumented calls | Wrap client with proxy |
| Alert storms | Wrong thresholds | Tune based on baseline data |
| Log volume too high | Debug logging in prod | Set LOG_LEVEL=info |
Examples
The following scenarios are covered in full in
references/examples.md:
- Contact lookup end-to-end โ one
contacts.find call producing a counter increment, a histogram sample, a span, and a PII-redacted log line.
- Rate-limit (429) event โ how the proxy zeros the rate-limit gauge and which alerts fire.
- Webhook success/failure accounting โ counting processed vs. failed webhooks per topic.
- Scraping
/metrics โ the raw Prometheus exposition and how it feeds Grafana and the alert rules.
Minimal end-to-end skeleton:
const client = instrumentedClient(new IntercomClient({ token: process.env.INTERCOM_ACCESS_TOKEN! }));
const contact = await tracedIntercomCall(
"contacts.find",
{ "intercom.contact_id": contactId },
() => client.contacts.find({ contactId })
);
Resources
Next Steps
For incident response once these signals are firing, see the intercom-incident-runbook
skill, which turns these alerts into a triage-and-mitigation procedure.
