grafana-query

name	grafana-query
description	Query Grafana for logs, metrics, traces, and alerts from an osparc-simcore deployment. Use when: debugging service errors, checking logs in Loki, querying Prometheus metrics, investigating incidents, finding error patterns, viewing dashboards, checking alert rules, or analyzing service performance.
argument-hint	Describe what you want to investigate (e.g., 'errors in webserver last hour', 'memory usage of director-v2')

Grafana Query

When to Use

• Investigating errors or warnings in simcore services
• Checking service logs for specific events or patterns
• Querying Prometheus metrics (request rates, latencies, resource usage)
• Viewing distributed traces via Tempo
• Checking alert rule state
• Analyzing service performance or resource consumption

IMPORTANT: If you discover that the actual Grafana data (datasource types, label names, log field structures, dashboard names, service naming patterns) is inconsistent with what this skill documents, update the skill files to reflect the correct state. This ensures the skill stays accurate for future invocations.

Datasources

Datasource UIDs differ per deployment. Always discover them dynamically at the start of a query session:

1. List all datasources and their UIDs
2. Filter by type when needed: type: "loki", type: "prometheus", type: "tempo"

Expected datasource types in a typical deployment:

Name pattern	Type	Purpose
loki	Loki	Log aggregation (all service logs)
prometheus-federation	Prometheus	Federated metrics (usually the default datasource)
prometheus-catchall	Prometheus	Catchall Prometheus metrics
tempo	Tempo	Distributed tracing
cloudwatch	CloudWatch	AWS infrastructure metrics (RDS, etc.)

Dashboards

Dashboard UIDs differ per deployment. Always discover them dynamically:

Typical dashboard folders and their contents:

simcore folder

Title	Purpose
Metrics	Core simcore service metrics
Services	Service-level overview
Requests load overview	HTTP request load
autoscaling overview	Autoscaling status
API server log streaming queues	API server queues
s4l-lite admin overview	Sim4Life Lite admin

The Metrics dashboard contains panels for core simcore services. It contains metrics related to resources usage (e.g. RAM and CPU), HTTP request handling statistics and RPC request handling statistics. This is a very useful dashboard to investigate once it has been determined that an error happened within a specific simcore service.

ops folder

Title	Purpose
Traefik2	Reverse proxy metrics
RabbitMQ Overview	Message broker
Redis Overview	Cache/queue
Postgres Overview	Database metrics
Docker Registries	Registry health
AWS RDS Metrics	RDS database

system folder

Title	Purpose
Docker Swarm Overview	Cluster overview
Nodes - Overview	Host nodes
Nodes - Detailed Insights	Per-node deep dive

Procedure

0. Discover Datasources

Before any query, resolve the datasource UIDs for this deployment.

1. Identify the Target

Determine which service(s) to investigate. The simcore platform services use the naming pattern <stack>_<prefix>_<service> (e.g., simcore_staging_webserver). The exact stack/prefix varies by deployment. Discover the naming pattern by listing Loki label values.

Common service name patterns:

• Core services: <stack>_<prefix>_<service> (e.g., *_webserver, *_api-server, *_director-v2)
• Dynamic proxies: dy-proxy_<node-uuid>
• Dynamic sidecars: dy-sidecar_<node-uuid>
• Dask: dask_stack_<component>

Core simcore services: webserver, api-server, director, director-v2, catalog, dynamic-schdlr, agent, autoscaling, clusters-keeper, resource-usage-tracker, wb-api-server, wb-auth, wb-db-event-listener, wb-garbage-collector, payments, invitations, efs-guardian, datcore-adapter, traefik

2. Query Logs (Loki)

Use the Loki datasource UID discovered in step 0. Available Loki labels:

• service_name — service identifier (see naming patterns above)
• container_name — Docker container name with replica info
• host — host node name
• source — log pipeline source (typically vector)

See Log Structure Reference for JSON field details.

3. Query Metrics (Prometheus)

Use the prometheus-federation datasource UID discovered in step 0. Common metric patterns:

• HTTP request metrics from gunicorn/aiohttp
• Service-specific business metrics
• Docker/container resource metrics

4. Query Traces (Tempo)

Use the Tempo datasource UID discovered in step 0. Traces are correlated with logs via log_trace_id and log_span_id fields in log entries.

Common Query Patterns

See Query Patterns Reference for ready-to-use LogQL and PromQL queries.

---

Last updated: 2026-05-04