rust-observability

| Use when this capability is needed.

Zur Installation springen

Repository: tomevault-io/skills-registry
Letzte Quellaktivität: 3. Juli 2026 um 19:45
Erkannte Sprache von SKILL.md: Englisch
Sterne: 0
Forks: 0

Installationsoptionen

Standardmäßig ist der Prompt ausgewählt, der zuerst die Quelle prüft. Sie können zu einem direkten Befehl wechseln oder eine lokale Kopie herunterladen.

Quelldateien prüfen

Lesen Sie SKILL.md und alle von SkillsMP angezeigten Begleitdateien, bevor Sie sich für eine Installation entscheiden.

Zurück zur Suche Zurück zur Installation

name	rust-observability
description	\| Use when this capability is needed.

Rust Observability

Practical guide for making Rust systems diagnosable in production: structured logs, spans, metrics, traces, and safe telemetry boundaries.

Quick Navigation

references/tracing.md - tracing setup, spans, subscribers, context propagation
references/metrics.md - RED/USE metrics, histograms, labels, Prometheus/OpenTelemetry

Golden Rules

Emit structured fields, not interpolated strings.
Put correlation IDs and domain IDs on spans once, then inherit them.
Measure latency with histograms, not averages.
Keep label cardinality bounded.
Never log secrets, credentials, bearer tokens, or raw sensitive payloads.

Tracing Setup

use tracing_subscriber::{fmt, EnvFilter, layer::SubscriberExt, util::SubscriberInitExt};

pub fn init_tracing() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
    let filter = EnvFilter::try_from_default_env()
        .unwrap_or_else(|_| EnvFilter::new("info,tower_http=info"));

    tracing_subscriber::fmt()
        .(filter)
        .()
        .()
        .()
        .();

    (())
}

use tracing_subscriber::{fmt, registry, EnvFilter, layer::SubscriberExt};

pub fn init_observability() -> Result<(), Box<dyn std::error::Error>> {
    let filter = EnvFilter::try_from_default_env()
        .unwrap_or_else(|_| EnvFilter::new("info"));

    let subscriber = registry()
        .with(filter)
        .with(fmt::Layer::default().json());

    // Add OpenTelemetry layer when configured
    // .with(tracing_opentelemetry::layer().with_tracer(otlp_tracer));

    subscriber.try_init()?;
    Ok(())
}

use tracing_subscriber::{fmt, reload, EnvFilter, Registry};

pub struct LogLevelHandle {
    handle: reload::Handle<EnvFilter, Registry>,
}

impl LogLevelHandle {
    pub fn set_level(&self, level: &str) -> Result<(), Box<dyn std::error::Error>> {
        let new_filter = EnvFilter::try_new(level)?;
        self.handle.reload(new_filter)?;
        Ok(())
    }
}

pub fn init_dynamic_logging() -> LogLevelHandle {
    let filter = EnvFilter::new("info");
    let (filter_layer, handle) = reload::Layer::new(filter);

    tracing_subscriber::registry()
        .with(filter_layer)
        .with(fmt::Layer::default().json())
        .init();

    LogLevelHandle { handle }
}

use tracing::{info_span, Instrument};

pub async fn handle_job(job: Job) -> Result<(), Error> {
    let span = info_span!(
        "job.process",
        job_id = %job.id,
        tenant_id = %job.tenant_id,
        kind = %job.kind,
    );

    async move {
        validate(&job)?;
        process(job).await?;
        Ok(())
    }
    .instrument(span)
    .await
}

use tracing::Span;

#[derive(thiserror::Error, Debug)]
pub enum AppError {
    #[error("job {job_id} failed in tenant {tenant_id}: {source}")]
    JobFailed {
        job_id: String,
        tenant_id: String,
        source: Box<dyn std::error::Error + Send + Sync>,
    },
}

impl AppError {
    pub fn with_span_context(self, span: &Span) -> Self {
        // Copy span fields into the error for structured logging
        self
    }
}

use tower_http::trace::{DefaultMakeSpan, DefaultOnResponse, TraceLayer};
use tracing::Level;

let app = Router::new()
    .route("/health", get(health))
    .layer(
        TraceLayer::new_for_http()
            .make_span_with(
                DefaultMakeSpan::new()
                    .level(Level::INFO)
                    .include_headers(false),
            )
            .on_response(
                DefaultOnResponse::new()
                    .level(Level::INFO)
                    .latency_unit(tower_http::LatencyUnit::Micros),
            ),
    );

use opentelemetry::{
    global,
    sdk::{propagation::TraceContextPropagator, trace as sdktrace},
};
use opentelemetry_otlp::WithExportConfig;

pub fn init_otlp(endpoint: &str) -> Result<(), Box<dyn std::error::Error>> {
    global::set_text_map_propagator(TraceContextPropagator::new());

    let tracer = opentelemetry_otlp::new_pipeline()
        .tracing()
        .with_exporter(
            opentelemetry_otlp::new_exporter()
                .tonic()
                .with_endpoint(endpoint),
        )
        .with_trace_config(
            sdktrace::config().with_resource(opentelemetry::sdk::Resource::new([
                opentelemetry::KeyValue::new("service.name", "my-service"),
            ])),
        )
        .install_batch(opentelemetry::runtime::Tokio)?;

    tracing_opentelemetry::layer().with_tracer(tracer);
    Ok(())
}

Strategy	Use Case	Effect
AlwaysSample	Dev, critical paths	Every trace sent
TraceIdRatioBased	High-traffic production	Sample N% of all traces
ParentBased	Distributed tracing	Inherit sampling decision from parent span
RateLimiting	Cost control	Cap spans per second

use opentelemetry::sdk::trace::Sampler;

let config = sdktrace::config()
    .with_sampler(Sampler::TraceIdRatioBased(0.1)) // 10% sampling
    .with_resource(/* ... */);

use metrics::counter;
use metrics_exporter_prometheus::PrometheusBuilder;

pub fn init_metrics() -> Result<(), Box<dyn std::error::Error>> {
    let builder = PrometheusBuilder::new();
    builder
        .with_http_listener("0.0.0.0:9001") // separate metrics port
        .install()?;
    Ok(())
}

use metrics::{counter, gauge, histogram, lazy_static};
use std::sync::LazyLock;

static REQUEST_COUNT: LazyLock<counter::Counter> = LazyLock::new(|| {
    counter!("http_requests_total", "method" => "GET", "route" => "/users/:id")
});

static REQUEST_DURATION: LazyLock<histogram::Histogram> = LazyLock::new(|| {
    histogram!("http_request_duration_seconds", "route" => "/users/:id")
});

// Counter: things that only go up
metrics::counter!("http_requests_total", "method" => "GET", "route" => "/users/:id").increment(1);

// Histogram: latency distributions
metrics::histogram!("http_request_duration_seconds", "route" => "/users/:id").record(elapsed.as_secs_f64());

// Gauge: current state
metrics::gauge!("worker_queue_depth", "queue" => "emails").set(depth as f64);

Metric	Type	Labels	Purpose
`http_requests_total`	Counter	method, route, status	Throughput
`http_request_duration_seconds`	Histogram	route, status	Latency (p50/p95/p99)
`http_requests_in_flight`	Gauge	route	Saturation
`db_query_duration_seconds`	Histogram	query_name	Dependency latency
`worker_queue_depth`	Gauge	queue_name	Backlog
`errors_total`	Counter	error_type, domain	Error rate
`pool_connections_used`	Gauge	pool_name	Resource usage

// Use stable field names across your codebase
tracing::info!(
    user_id = %uid,
    tenant_id = %tenant,
    order_id = %order,
    amount = amount_minor,
    currency = %currency,
    "order created"
);

// Bad: error lost in a generic log
tracing::warn!("operation failed: {:?}", err);

// Good: capture typed error as a field
tracing::warn!(
    error_type = %err.kind(),
    error = %err,
    "operation failed"
);

# alert-rules.yml
groups:
  - name: rust-service
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.01
        for: 5m
        labels: { severity: critical }
        annotations:
          summary: "Error rate {{ $value | humanizePercentage }}"

      - alert: HighLatency
        expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 2
        for: 5m
        labels: { severity: warning }
        annotations:
          summary: "p95 latency {{ $value }}s"

      - alert: PoolExhaustion
        expr: pool_connections_used / pool_connections_max > 0.8
        for: 2m
        labels: { severity: warning }

// Bad: unstructured and hard to query.
println!("user {} failed: {:?}", user_id, err);

// Good: structured fields with stable keys.
tracing::warn!(%user_id, error = %err, "user operation failed");

// Bad: raw path label explodes cardinality.
metrics::counter!("requests", "path" => request.uri().path().to_owned()).increment(1);

// Good: route template has bounded values.
metrics::counter!("requests", "route" => "/users/:id").increment(1);

// Bad: blocking the async runtime with synchronous I/O
// inside `tracing::instrument` on a hot path.

// Good: keep span creation cheap; defer expensive formatting
// to subscriber layers (which run on their own thread).

// Bad: logging the same error at multiple levels
tracing::error!("operation failed");
// ... then re-thrown and caught again
tracing::warn!("operation also failed");

// Good: log at the boundary where recovery decision happens.
// Inner layers should propagate, not log.