name	LAYER_11_APM
description	Expert knowledge for APM (Observability) Layer modeling in Documentation Robotics
triggers	["APM","observability","monitoring","metrics","spans","traces","logs","OpenTelemetry","telemetry"]
version	0.8.3

APM Layer Skill

Layer Number: 11 Specification: Metadata Model Spec v0.8.3 Purpose: Defines observability using OpenTelemetry 1.0+, specifying traces, metrics, logs, and instrumentation.

Layer Overview

The APM Layer captures application performance monitoring:

TRACES - Distributed tracing with spans
METRICS - Performance and business metrics
LOGS - Structured logging
INSTRUMENTATION - Code instrumentation configuration
RESOURCES - Telemetry resource attributes
ALERTING - Alert rules and monitoring dashboards

This layer uses OpenTelemetry 1.0+ (industry standard for observability).

Central Entity: The Span (unit of work in trace) is the core modeling unit.

Pre-Extraction Verification

Before extracting APM elements, verify what level of observability the codebase ACTUALLY implements. Do not model aspirational instrumentation.

Step 1: Check for OTel SDK

grep "@opentelemetry" package.json

If NO OTel SDK found:

Do NOT add MetricInstrument, Span, TraceConfiguration, or ExporterConfig elements as if they are implemented
ONLY model what exists: browser console logging (LogConfiguration), UI-displayed metrics (describe in element description that export is absent)
Mark all elements provenance: inferred and add a note in the description: "Metric concept identified; no OTel export implemented."

If OTel SDK IS found:

Proceed with full extraction of all APM types

Step 2: Identify what observability actually exists

Even without OTel, codebases may have:

Browser console error logging → apm.logconfiguration
UI-visible token/usage counters → apm.metricinstrument (with note: display-only, not exported)
Error tracking utilities → apm.instrumentationconfig (type: manual)
In-memory state tracking → apm.metricinstrument (inferred, note: in-memory only)

In-UI Observability vs. OTel Instrumentation

Pattern	Model as	provenance
`console.error()` logging	`logconfiguration`	extracted
UI badge showing token count	`metricinstrument` (note: display-only)	inferred
Store tracks reconnection count	`metricinstrument` (note: in-memory only)	inferred
`opentelemetry.createCounter()`	`metricinstrument`	extracted
Prometheus export endpoint	`exporterconfig`	extracted

Entity Types

CLI Introspection: Run dr schema types apm for the authoritative, always-current list of node types. Run dr schema node <type-id> for full attribute details on any type.

Core APM Entities (15 entities)

Entity Type	Description
TraceConfiguration	Distributed tracing configuration
Span	Unit of work in distributed trace
SpanEvent	Timestamped event within span
SpanLink	Link between spans from different traces
MetricConfiguration	Metrics collection configuration
MetricInstrument	Specific metric instrument (Counter, Gauge, Histogram, etc.)
LogConfiguration	Structured logging configuration
LogRecord	Individual log record
LogProcessor	Log processing pipeline step (simple, batch, custom)
Resource	Telemetry resource attributes
InstrumentationScope	Scope of instrumentation
InstrumentationConfig	Code instrumentation configuration (library, auto/manual type)
ExporterConfig	Telemetry exporter configuration (OTLP, Jaeger, Prometheus, etc.)
Alert	Alert rule with severity, condition, and notification channels
Dashboard	Monitoring dashboard definition (Grafana, Datadog, etc.)

Type Decision Tree

Use this decision tree before assigning a type to any APM element.

Is this an alert rule with severity and condition? → apm.alert
Is this a monitoring dashboard definition (Grafana, Datadog, etc.)? → apm.dashboard
Is this an exporter configuration (OTLP, Jaeger, Prometheus endpoint)? → apm.exporterconfig
Is this a code instrumentation configuration (library name, auto/manual)? → apm.instrumentationconfig
Is this the instrumentation scope (library version, schema URL)? → apm.instrumentationscope
Is this logging pipeline configuration (service name, minimum severity)? → apm.logconfiguration
Is this a log processor step (batch, simple, custom)? → apm.logprocessor
Is this an individual structured log record with body and severity? → apm.logrecord
Is this a metrics collection configuration (export interval, cardinality limit)? → apm.metricconfiguration
Is this a specific metric instrument (counter, gauge, histogram, observable)? → apm.metricinstrument
Is this a telemetry resource (service name, deployment environment attributes)? → apm.resource
Is this a unit of work/operation in a distributed trace? → apm.span
Is this an event that occurred within a span? → apm.spanevent
Is this a link between spans from different traces? → apm.spanlink
Is this distributed tracing configuration (sampler type, propagators)? → apm.traceconfiguration

When to Use This Skill

Activate when the user:

Mentions "APM", "observability", "monitoring", "telemetry"
Wants to add tracing, metrics, or logging
Asks about performance monitoring or SLOs
Needs to instrument code or track business metrics
Wants to link observability to application services

Cross-Layer Relationships

Outgoing (APM → Other Layers):

instrumented-service → Application Layer (which service is being monitored?)
business-metrics → Business Layer (business KPIs)

Incoming (Other Layers → APM):

Application Layer → APM (services reference metrics)
API Layer → APM (operations set SLA targets)
Business Layer → APM (processes reference business metrics)
Data Model Layer → APM (metric instruments tracking data quality KPIs)
Data Store Layer → APM (query performance metrics)

Observability Best Practices

Traces - Add spans for critical operations
Metrics - Track both technical and business metrics
Logs - Use structured logging (JSON)
Context - Propagate trace context across services
Sampling - Configure appropriate sampling rates
SLOs - Define service level objectives
Alerting - Set up alerts for critical metrics
Cardinality - Avoid high-cardinality attributes

Common Commands

# Add span
dr add apm span "Process Order"

# Add metric instrument
dr add apm metricinstrument "order_rate" --description "Order processing rate counter"

# Add log record
dr add apm logrecord "Error Log"

# List APM elements
dr list apm --type span

# Validate APM layer
dr validate --layers apm

# Export APM configuration
dr export markdown --layers apm

Example: Order Processing Span

id: apm.span.process-order
name: "Process Order Span"
type: span
properties:
  traceId: "4bf92f3577b34da6a3ce929d0e0e4736"
  spanId: "00f067aa0ba902b7"
  traceState: ""
  parentSpanId: "b9c7c989f97918e1"
  spanKind: INTERNAL
  startTimeUnixNano: "1544712660000000000"
  endTimeUnixNano: "1544712661000000000"
  droppedAttributesCount: 0
  droppedEventsCount: 0
  droppedLinksCount: 0
  statusCode: OK
  attributes:
    order.id: "ord-12345"
    order.total: 99.99
    customer.id: "cust-67890"

Example: Business Metrics

id: apm.metricinstrument.order-rate
name: "Order Rate Metric"
type: metricinstrument
properties:
  type: Counter
  unit: orders
  description: "Number of orders processed per minute"
  enabled: true

Coverage Completeness Checklist

Before declaring APM layer extraction complete, verify each type was considered:

If any type has ZERO elements, explicitly decide: "This type doesn't apply to this codebase" with reasoning.

Pitfalls to Avoid

❌ Not propagating trace context across services
❌ High-cardinality attributes (e.g., unique IDs in tags)
❌ Missing business metrics (only technical metrics)
❌ Not setting SLOs/SLAs
❌ Over-instrumentation (too many spans)
❌ Missing cross-layer links to instrumented services