observability-stack

name	observability-stack
description	Generates Pulumi TypeScript observability infrastructure for Grafana Alloy 1.13, Prometheus 3.5+, Grafana 12.3 across K8s/Docker. Use when deploying metrics stacks, provisioning alert rules/dashboards, or migrating cloud↔selfhosted observability.

[H1][OBSERVABILITY-STACK]

Dictum: Infrastructure generation follows canonical deploy.ts patterns.

Stack: Alloy -> Prometheus -> Grafana (metrics only). Logs/traces -> otelcol.exporter.debug (discarded). Runtime: OTEL_LOGS_EXPORTER=none, OTEL_TRACES_EXPORTER=none, OTEL_METRICS_EXPORTER=otlp (deploy.ts:95-97). Versions: Alloy 1.13.0 (OTel Collector v0.142.0), Prometheus 3.5.1+ (native histograms stable in 3.8+, flag no-op in 3.9+), Grafana 12.3.2.

[INDEX]	[USE_THIS_SKILL]	[USE_OTHER_SKILL]
[1]	Deploy observability via Pulumi.	pulumi-k8s-generator: Non-observability K8s resources.
[2]	OTEL collector configs (Alloy).	k8s-debug: Debug deployed observability pods.
[3]	Scrape/alert rules, dashboards.	pulumi-k8s-validator: Audit existing infra code.
[4]	Mode migration cloud<->selfhosted.	promql-generator: PromQL query construction.

Tasks:

1. Read all three references/ files before generating.
2. Gather: mode (cloud/selfhosted), retention, HA, TLS via AskUserQuestion (ambiguous only).
3. Generate Pulumi TypeScript resources following deploy.ts patterns.
4. Validate: pnpm exec nx run infrastructure:typecheck then pulumi preview --diff.

---

[1][CANONICAL_IMPLEMENTATION]

Dictum: deploy.ts is the single source of truth for resource patterns.

Canonical: infrastructure/src/deploy.ts (207 LOC).

[INDEX]	[LINES]	[SYMBOL]	[PURPOSE]
[1]	14	_CONFIG.images	Image tags (:latest -- pin for production).
[2]	22	_CONFIG.ports	alloyGrpc:4317, alloyHttp:4318, alloyMetrics:12345, prom:9090.
[3]	29-36	_Ops.alloy(promUrl)	Alloy config: OTLP receiver, prometheus.remote_write, debug.
[4]	59	_Ops.grafana(promUrl)	Prometheus datasource provisioning YAML.
[5]	63	_Ops.meta(ns,component,n?)	Metadata factory with stack: 'parametric', tier: 'observe'.
[6]	71	_Ops.prometheus(alloyHost)	Scrape config: 15s interval, alloy:12345 + localhost:9090.
[7]	120-126	_k8sObserve(ns, items)	Array-driven factory: PVC + ConfigMap + Deployment + Service.

[1.1][DATA_FLOW]

App (Effect OTEL) --OTLP--> Alloy(:4317/:4318) --remote_write--> Prometheus(:9090)
                                    |--debug sink--> (logs discarded)
                                    \--debug sink--> (traces discarded)
Grafana(:3000) --PromQL--> Prometheus(:9090)

---

[2][GENERATION]

Dictum: Cloud and selfhosted share topology; differ in resource types.

[INDEX]	[MODE]	[INFRASTRUCTURE]	[LINES]	[USE_CASE]
[1]	cloud	K8s (EKS/GKE/AKS).	131-176	Production, multi-node, HA.
[2]	selfhosted	Docker containers.	178-194	Dev, single-node, staging.

Cloud (deploy.ts:147-154):

• Alloy: ConfigMap + DaemonSet + Service (3 ports: grpc:4317, http:4318, metrics:12345). Resources: 200m/256Mi limits, 100m/128Mi requests.
• _k8sObserve array: prometheus (cmd flags, 9090, PVC) + grafana (datasource provisioning, 3000, PVC).

Selfhosted (deploy.ts:187-189):

• 3 docker.Container resources with uploads (config) + named volumes via _Ops.dockerVol.
• Grafana: GF_SECURITY_ADMIN_PASSWORD from _Ops.secret(). Shared network via _Ops.dockerNets(networkId).

All resources: Pulumi TypeScript. Never raw YAML, Helm, or docker-compose.

---

[3][VALIDATION]

Dictum: Validation gates prevent broken observability pipelines.

[INDEX]	[CHECK]	[SCOPE]
[1]	Alloy OTLP ports 4317/4318 exposed.	All.
[2]	Alloy metrics port 12345 exposed.	All.
[3]	Prometheus scrapes all components.	All.
[4]	Grafana datasource provisioned.	All.
[5]	Resource requests/limits set.	Cloud.
[6]	Named volumes for stateful data.	Selfhosted.
[7]	TLS for inter-component traffic.	Cloud (prod).

---

[4][PRODUCTION]

Dictum: Production gaps require explicit acknowledgment.

[INDEX]	[CONCERN]	[STATUS]	[RECOMMENDATION]
[1]	TLS	Not configured.	cert-manager or AWS ACM for inter-component comms.
[2]	Security contexts	Missing on observe pods.	runAsNonRoot: true, readOnlyRootFilesystem: true.
[3]	Image tags	:latest (deploy.ts:14).	Pin to specific versions for production.
[4]	NetworkPolicies	Not defined.	Restrict Alloy<-App, Prometheus<-Alloy, Grafana->Prom.
[5]	Resource limits	Alloy only.	Add to Prometheus/Grafana via _k8sObserve extension.

---

[5][DEPRECATIONS]

Dictum: Deprecated components require migration paths.

[INDEX]	[DEPRECATED]	[REPLACEMENT]	[STATUS]
[1]	Promtail	Alloy.	EOL March 2, 2026.
[2]	lokiexporter	otlphttp.	Removed OTEL 0.140+.
[3]	Standalone OTEL Collector	Alloy (includes v0.142.0).	Superseded.
[4]	River syntax (name)	"Alloy configuration syntax".	Renamed Alloy 1.8+.

---

[6][EXTENSION_LOKI_TEMPO]

Dictum: Full 3-signal integration extends the metrics-only default.

[INDEX]	[SIGNAL]	[IMAGE]	[ALLOY_CHANGE]	[RUNTIME_ENV]
[1]	Logs (Loki)	grafana/loki:3.6.x.	debug -> loki exporter.	OTEL_LOGS_EXPORTER=otlp.
[2]	Traces (Tempo)	grafana/tempo:2.7.x.	debug -> otlp exporter to Tempo.	OTEL_TRACES_EXPORTER=otlp.

[REFERENCE] references/stack_architecture.md#extension-adding-loki--tempo.

---

[7][REFERENCES]

Dictum: Reference files provide detailed implementation patterns.

• references/stack_architecture.md -- Topology, sizing, Prometheus version matrix, Loki/Tempo extension.
• references/alert_rules.md -- Alert templates, recording rules (classic + native), ConfigMap provisioning.
• references/grafana_dashboards.md -- Dashboard panels (Grafana 12.3+, schema v2), native histogram heatmap.