iii-observability

# iii-observability The `iii-observability` worker provides OpenTelemetry-backed observability for the iii engine: distributed tracing, structured logs, metrics with rollups, alert rules, sampling configuration, and baggage propagation. Every surface is a callable `engine::*` function, plus a single `log` reactive trigger that fires on every ingested log entry. Functions span nine sub-namespaces — emit, query stored telemetry, inspect configuration, and propagate context. The worker is on by default (`enabled: true`). When disabled, the emit and read functions still register but become no-ops and the trigger never fires. Core config: `service_name`/`service_version`/`service_namespace` (OTel resource attributes), `exporter` (`memory` | `otlp` | `both`), `endpoint` (OTLP collector URL, default `http://localhost:4317`), `sampling_ratio` (`0.0`–`1.0`), `memory_max_spans`, and per-pillar toggles/limits (`metrics_enabled`, `logs_enabled`, retention and count caps). Most fields accept `OTEL_*` env overrides. The in-memory query functions (`logs`, `traces`) require the `memory` or `both` exporter. Traces and metrics use OTLP/gRPC by default; `https://` endpoints enable TLS and `http://` endpoints use cleartext. Set `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` to use OTLP/HTTP protobuf for traces and metrics, or use `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` / `OTEL_EXPORTER_OTLP_METRICS_PROTOCOL` for signal-specific overrides. The HTTP/protobuf path is appended automatically (`/v1/traces`, `/v1/metrics`) when the configured endpoint is a base collector URL. Logs export over OTLP/HTTP to `/v1/logs`. Use `OTEL_EXPORTER_OTLP_HEADERS` for collector headers and the signal-specific `OTEL_EXPORTER_OTLP_TRACES_HEADERS`, `OTEL_EXPORTER_OTLP_METRICS_HEADERS`, or `OTEL_EXPORTER_OTLP_LOGS_HEADERS` when needed. The logs exporter reads `OTEL_EXPORTER_OTLP_LOGS_HEADERS` first and falls back to `OTEL_EXPORTER_OTLP_HEADERS`. Keep credentials in environment variables or a secret manager, not in config files. ## When to Use - Emit structured logs or read back stored logs/spans/metrics from inside a function instead of shelling out to a collector. - React in real time to log entries (page on errors, archive everything) without polling. - Inspect engine health, active sampling rules, or alert state operationally. - Propagate OpenTelemetry baggage across calls for cross-cutting context. ## Boundaries - `engine::baggage::set` does not propagate back to the caller — baggage propagation happens at the SDK/invocation level via headers. - The in-memory query functions return nothing useful unless the `memory` (or `both`) exporter is configured; with `otlp`-only the data lives in your collector. - When `logs_enabled` is off the log pipeline is dormant and the `log` trigger never fires; the ingest `level` sets the minimum severity stored. - This worker observes telemetry — it is not a general event bus (`pubsub`) or durable queue (`queue`); those are standalone Compose workers. ## Functions - `engine::log::info`, `engine::log::warn`, `engine::log::error`, `engine::log::debug`, `engine::log::trace` — emit a log entry at the named severity; same input shape, only the level differs. - `engine::logs::list` — read stored OTel logs, filtered by time, trace correlation, or severity. - `engine::logs::clear` — wipe the in-memory log store. - `engine::traces::list` — list compact trace summaries with aggregate status/counts and optional attribute projection. - `engine::traces::spans` — list full span records for detail/timeline consumers. - `engine::traces::tree` — walk a single trace as a parent/child tree. - `engine::traces::group_by` — aggregate stored spans by an attribute value (counts, duration, errors per group). - `engine::traces::clear` — wipe stored spans. - `engine::metrics::list` — list metrics with aggregated stats and optional time bucketing. - `engine::rollups::list` — list metric rollup aggregations across 1-minute, 5-minute, and 1-hour windows. - `engine::baggage::get`, `engine::baggage::get_all` — read one or all OpenTelemetry baggage keys. - `engine::baggage::set` — set a baggage key locally (does not propagate to the caller). - `engine::sampling::rules` — list the active sampling rules. - `engine::health::check` — return engine health (status, per-component breakdown, version). - `engine::alerts::list` — inspect configured alert rules and current state. - `engine::alerts::evaluate` — manually run an alert evaluation pass. ## Reactive triggers Bind a `log` trigger when a function should run every time a log entry lands in the engine's OTel log pipeline — emitted via `engine::log::*`, ingested via OTLP, or recorded by any worker using structured logging. Each subscription receives the same OTel-shaped record, so handlers can route by severity, attribute, or trace correlation. Reach for it when: - A specific severity (typically `error`) should page a human, post to Slack, or open a ticket. - You want real-time fan-out of log entries to a downstream sink (archive, analytics, transformer) without polling `engine::logs::list`. Use `engine::logs::list` instead when you need to query stored entries on demand rather than react to each as it arrives. ### How to bind 1. Register a handler: `iii.registerFunction('monitoring::on-error', handler)`. 2. Register the trigger: ```typescript iii.registerTrigger({ type: "log", function_id: "monitoring::on-error", config: { level: "error", // optional. trace|debug|info|warn|error. Omit to fire on every level. }, }); ``` The `log` trigger only fires when the logs pipeline is enabled, and `level` filters to that minimum severity. The handler's return value is ignored; invocations are spawned asynchronously after each entry is stored. For the OTel log record shape, call `iii get function info` on the trigger type or handler function id.