observability

name	observability
description	Use when diagnosing operation failures, stuck or slow operations, querying Jaeger traces, working with Grafana dashboards, debugging distributed system issues, or investigating worker selection and service communication problems.

Observability & Debugging

Load this skill when:

• Diagnosing operation failures, stuck operations, or slow operations
• Working with Jaeger traces or Grafana dashboards
• Debugging distributed system issues
• Investigating worker selection or service communication problems

---

First Rule: Check Observability Before Logs

When users report issues with operations, use Jaeger first — not logs. KTRDR has comprehensive OpenTelemetry instrumentation that provides complete visibility into distributed operations.

This enables first-response diagnosis instead of iterative detective work.

---

When to Query Jaeger

Query Jaeger when user reports:

Symptom	What Jaeger Shows
"Operation stuck"	Which phase is stuck and why
"Operation failed"	Exact error with full context
"Operation slow"	Bottleneck span immediately
"No workers selected"	Worker selection decision
"Missing data"	Data flow from IB to cache
"Service not responding"	HTTP call attempt and result

---

Quick Start Workflow

Step 1: Get operation ID

From CLI output or API response (e.g., op_training_20251113_123456_abc123)

Step 2: Query Jaeger API

OPERATION_ID="op_training_20251113_123456_abc123"
curl -s "http://localhost:16686/api/traces?tag=operation.id:$OPERATION_ID&limit=1" | jq

Step 3: Analyze trace structure

# Get span summary with durations
curl -s "http://localhost:16686/api/traces?tag=operation.id:$OPERATION_ID" | jq '
  .data[0].spans[] |
  {
    span: .operationName,
    service: .process.serviceName,
    duration_ms: (.duration / 1000),
    error: ([.tags[] | select(.key == "error" and .value == "true")] | length > 0)
  }' | jq -s 'sort_by(.duration_ms) | reverse'

Step 4: Extract relevant attributes

curl -s "http://localhost:16686/api/traces?tag=operation.id:$OPERATION_ID" | jq '
  .data[0].spans[] |
  {
    span: .operationName,
    attributes: (.tags | map({key: .key, value: .value}) | from_entries)
  }'

---

Common Diagnostic Patterns

Pattern 1: Operation Stuck

# Check for worker selection and dispatch
curl -s "http://localhost:16686/api/traces?tag=operation.id:$OP_ID" | jq '
  .data[0].spans[] |
  select(.operationName == "worker_registry.select_worker") |
  .tags[] |
  select(.key | startswith("worker_registry.")) |
  {key: .key, value: .value}'

Look for:

• worker_registry.total_workers: 0 → No workers started
• worker_registry.capable_workers: 0 → No capable workers
• worker_registry.selection_status: NO_WORKERS_AVAILABLE → All busy

Pattern 2: Operation Failed

# Extract error details
curl -s "http://localhost:16686/api/traces?tag=operation.id:$OP_ID" | jq '
  .data[0].spans[] |
  select(.tags[] | select(.key == "error" and .value == "true")) |
  {
    span: .operationName,
    service: .process.serviceName,
    exception_type: (.tags[] | select(.key == "exception.type") | .value),
    exception_message: (.tags[] | select(.key == "exception.message") | .value)
  }'

Common errors:

• ConnectionRefusedError → Service not running (check http.url)
• ValueError → Invalid input parameters
• DataNotFoundError → Data not loaded (check data.symbol, data.timeframe)

Pattern 3: Operation Slow

# Find bottleneck span (longest duration)
curl -s "http://localhost:16686/api/traces?tag=operation.id:$OP_ID" | jq '
  .data[0].spans[] |
  {
    span: .operationName,
    duration_ms: (.duration / 1000)
  }' | jq -s 'sort_by(.duration_ms) | reverse | .[0]'

Common bottlenecks:

• training.training_loop → Check training.device (GPU vs CPU)
• data.fetch → Check ib.latency_ms
• ib.fetch_historical → Check data.bars_requested

Pattern 4: Service Communication Failure

# Check HTTP calls between services
curl -s "http://localhost:16686/api/traces?tag=operation.id:$OP_ID" | jq '
  .data[0].spans[] |
  select(.operationName | startswith("POST") or startswith("GET")) |
  {
    http_call: .operationName,
    url: (.tags[] | select(.key == "http.url") | .value),
    status: (.tags[] | select(.key == "http.status_code") | .value),
    error: (.tags[] | select(.key == "error.type") | .value)
  }'

Look for:

• http.status_code: null → Connection failed
• error.type: ConnectionRefusedError → Target service not running
• http.url → Shows which service was being called

---

Key Span Attributes Reference

Operation Attributes

• operation.id — Operation identifier
• operation.type — TRAINING, BACKTESTING, DATA_DOWNLOAD
• operation.status — PENDING, RUNNING, COMPLETED, FAILED

Worker Selection

• worker_registry.total_workers — Total registered workers
• worker_registry.available_workers — Available workers
• worker_registry.capable_workers — Capable workers for this operation
• worker_registry.selected_worker_id — Which worker was chosen
• worker_registry.selection_status — SUCCESS, NO_WORKERS_AVAILABLE, NO_CAPABLE_WORKERS

Progress Tracking

• progress.percentage — Current progress (0-100)
• progress.phase — Current execution phase
• operations_service.instance_id — OperationsService instance (check for mismatches)

Error Context

• exception.type — Python exception class
• exception.message — Error message
• exception.stacktrace — Full stack trace
• error.symbol, error.strategy — Business context

Performance

• http.status_code — HTTP response status
• http.url — Target URL for HTTP calls
• ib.latency_ms — IB Gateway latency
• training.device — cuda:0 or cpu
• gpu.utilization_percent — GPU usage

---

Response Template

When diagnosing with observability, use this structure:

🔍 **Trace Analysis for operation_id: {operation_id}**

**Trace Summary**:
- Trace ID: {trace_id}
- Total Duration: {duration_ms}ms
- Services: {list of services}
- Status: {OK/ERROR}

**Execution Flow**:
1. {span_name} ({service}) - {duration_ms}ms
2. {span_name} ({service}) - {duration_ms}ms
...

**Diagnosis**:
{identified_issue_with_evidence_from_spans}

**Root Cause**:
{root_cause_explanation_with_span_attributes}

**Solution**:
{recommended_fix_with_commands}

---

Grafana Dashboards

Check Grafana for quick diagnostics before diving into traces.

URL: http://localhost:3000

Dashboard	Path	Use Case
System Overview	/d/ktrdr-system-overview	Service health, error rates, latency
Worker Status	/d/ktrdr-worker-status	Worker capacity, resource usage
Operations	/d/ktrdr-operations	Operation counts, success rates

Quick Workflows

• "Is it working?" → System Overview: Healthy Services count
• "Why is it slow?" → System Overview: P95 Latency panel
• "Workers missing?" → Worker Status: Healthy Workers and Health Matrix
• "Operations failing?" → Operations: Success Rate and Status Distribution

---

Benefits of Observability-First Debugging

• Diagnosis in FIRST response (not 10+ messages later)
• Complete context (all services, all phases, all attributes)
• Objective evidence (no guessing or assumptions)
• Distributed visibility (Backend → Worker → Host Service)
• Performance insights (identify bottlenecks immediately)
• Root cause analysis (trace error from source to root)

---

Full Documentation

For comprehensive workflows and scenarios: docs/debugging/observability-debugging-workflows.md