monitoring-observability

Monitoring and observability patterns for Prometheus metrics, Grafana dashboards, Langfuse v4 LLM tracing (as_type, score_current_span, should_export_span, LangfuseMedia), and drift detection. Use when adding logging, metrics, distributed tracing, LLM cost tracking, or quality drift monitoring.

Jump to install

Repository: yonatangross/orchestkit
Last source activity: August 5, 2026 at 16:40
Detected SKILL.md language: English
Stars: 219
Forks: 20

Install options

The review-first prompt is selected by default. You can switch to a direct command or download a local copy.

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

A direct command skips the review prompt. Inspect the source before running it.

npx skills add https://github.com/yonatangross/orchestkit --skill monitoring-observability

The command stays on one line. Scroll horizontally to inspect it before copying.

Prefer a local copy? Download the files currently available to SkillsMP.

Review the source files

Read SKILL.md and any companion files shown by SkillsMP before deciding whether to install.

File Explorer

27 files

SKILL.md

Source instructions · Read-only preview

Back to search Back to installation

chain-patterns

Chain patterns for multi-phase pipelines: MCP detection, handoff files, checkpoint-resume, worktree agents, CronCreate monitoring. Use when building or debugging a pipeline skill.

2026-08-12

doctor

OrchestKit doctor for health diagnostics across manifest integrity, hook configuration, skill validation, agent frontmatter, MCP server connectivity, CC version compatibility, and permission rules. Reports issues with severity levels and auto-remediation suggestions. Validates component counts, detects orphaned entries, and checks CC version matrix compliance. Use when diagnosing plugin health, troubleshooting configuration issues, or running pre-release checks.

2026-08-12

telemetry-inspect

Inspects the OrchestKit telemetry pipeline for the current project — lists all known telemetry files with write counts, sizes, schema status, growth trend, and orphan detection. Use when verifying the observability pipeline is healthy, debugging a missing writer, or auditing which files have schema locks vs. which are drift-vulnerable. Read-only — never modifies telemetry files.

2026-08-12

View all from this repository

Monitoring & Observability

A wrap around Prometheus, Grafana, OpenTelemetry and Langfuse, not a re-teaching of them. This skill carries OrchestKit's delta (version floors, house decisions, scars) and points at the vendor for everything else. Start at references/ork-delta.md.

Upstream coverage (do not restate)

These topics are fully covered first-party. Read the source, do not add a local copy.

Topic

First-party source

Prometheus metric types, RED method, cardinality, PromQL

https://prometheus.io/docs/practices/

Alertmanager grouping, inhibition, escalation, runbooks

https://prometheus.io/docs/alerting/latest/configuration/

Grafana dashboards, Loki and LogQL, Promtail

https://grafana.com/docs/

OpenTelemetry spans, sampling, context propagation

https://opentelemetry.io/docs/

Langfuse Python SDK (@observe, as_type, score_current_span, should_export_span, LangfuseMedia)

https://langfuse.com/docs/sdk/python

Langfuse v2 to v4 Python and v3 to v5 JS migration paths

https://langfuse.com/docs/sdk/python/v4-migration

Langfuse self-hosting (ClickHouse, Redis, S3, Helm)

https://langfuse.com/docs/deployment/self-host

Langfuse cost tracking, model pricing, Metrics API v2

https://langfuse.com/docs/model-usage-and-cost

Langfuse scores, online evaluators, annotation queues, prompt management

https://langfuse.com/docs/scores/overview

Langfuse framework integrations (LangChain, LangGraph, CrewAI, Pydantic AI, Bedrock, LiveKit)

https://langfuse.com/docs/integrations

Agent Graphs, observation types, rendered tool calls

Monitoring & Observability

Upstream coverage (do not restate)

These topics are fully covered first-party. Read the source, do not add a local copy.

Topic

First-party source

Prometheus metric types, RED method, cardinality, PromQL

https://prometheus.io/docs/practices/

Alertmanager grouping, inhibition, escalation, runbooks

https://prometheus.io/docs/alerting/latest/configuration/

Grafana dashboards, Loki and LogQL, Promtail

https://grafana.com/docs/

OpenTelemetry spans, sampling, context propagation

https://opentelemetry.io/docs/

Langfuse Python SDK (@observe, as_type, score_current_span, should_export_span, LangfuseMedia)

https://langfuse.com/docs/sdk/python

Langfuse v2 to v4 Python and v3 to v5 JS migration paths

https://langfuse.com/docs/sdk/python/v4-migration

Langfuse self-hosting (ClickHouse, Redis, S3, Helm)

https://langfuse.com/docs/deployment/self-host

Langfuse cost tracking, model pricing, Metrics API v2

https://langfuse.com/docs/model-usage-and-cost

Langfuse scores, online evaluators, annotation queues, prompt management

https://langfuse.com/docs/scores/overview

Langfuse framework integrations (LangChain, LangGraph, CrewAI, Pydantic AI, Bedrock, LiveKit)

https://langfuse.com/docs/integrations

Agent Graphs, observation types, rendered tool calls

name	monitoring-observability
license	MIT
compatibility	Claude Code 2.1.220+.
description	Monitoring and observability patterns for Prometheus metrics, Grafana dashboards, Langfuse v4 LLM tracing (as_type, score_current_span, should_export_span, LangfuseMedia), and drift detection. Use when adding logging, metrics, distributed tracing, LLM cost tracking, or quality drift monitoring.
tags	["monitoring","observability","prometheus","grafana","langfuse","tracing","metrics","drift-detection","logging"]
context	fork
version	3.0.0
author	OrchestKit
user-invocable	false
disable-model-invocation	true
complexity	medium
persuasion-type	reference
targets	[{"library":"langfuse","version":">=4.0.0"}]
upstream-version-tested	4.14.2
metadata	{"category":"document-asset-creation"}
allowed-tools	["Read","Glob","Grep","WebFetch","WebSearch"]
path_patterns	["/metrics/","/tracing/","prometheus.","grafana/*"]

name	monitoring-observability
license	MIT
compatibility	Claude Code 2.1.220+.
description	Monitoring and observability patterns for Prometheus metrics, Grafana dashboards, Langfuse v4 LLM tracing (as_type, score_current_span, should_export_span, LangfuseMedia), and drift detection. Use when adding logging, metrics, distributed tracing, LLM cost tracking, or quality drift monitoring.
tags	["monitoring","observability","prometheus","grafana","langfuse","tracing","metrics","drift-detection","logging"]
context	fork
version	3.0.0
author	OrchestKit
user-invocable	false
disable-model-invocation	true
complexity	medium
persuasion-type	reference
targets	[{"library":"langfuse","version":">=4.0.0"}]
upstream-version-tested	4.14.2
metadata	{"category":"document-asset-creation"}
allowed-tools	["Read","Glob","Grep","WebFetch","WebSearch"]
path_patterns	["/metrics/","/tracing/","prometheus.","grafana/*"]

Category	Rules	Impact	When to Use
Infrastructure Monitoring	1	CRITICAL	Grafana dashboards, Golden Signals, SLO/SLI
LLM Observability	1	HIGH	Langfuse tracing, observation types, agent graphs
Silent Failures	3	HIGH	Tool skipping, quality degradation, loop/token spike alerting

Rule	File	Key Pattern
Tool Skipping	`rules/silent-tool-skipping.md`	Expected vs actual tool calls, Langfuse traces
Quality Degradation	`rules/silent-degraded-quality.md`	Heuristics + LLM-as-judge, z-score baselines
Silent Alerting	`rules/silent-alerting.md`	Loop detection, token spikes, escalation workflow

Decision	Recommendation	Rationale
Metric methodology	RED method (Rate, Errors, Duration)	Industry standard, covers essential service health
Log format	Structured JSON	Machine-parseable, supports log aggregation
Tracing	OpenTelemetry	Vendor-neutral, auto-instrumentation, broad ecosystem
LLM observability	Langfuse (not LangSmith)	Open-source, self-hosted, built-in prompt management
LLM tracing API	`@observe(as_type=...)` + `score_current_span()`	v4: semantic types, inline scoring, span filtering
Langfuse APIs	Observations API v2 + Metrics API v2	v4 (Mar 2026): faster querying, aggregations at scale
Hook telemetry transport	JSONL under `~/.claude/analytics/`, never an SDK in-process	Hooks are per-event processes; SDK init would be paid on every spawn (`references/ork-delta.md`)

monitoring-observability

Install options

Review the source files