observe

name	observe
description	Query and manage Observe using the Observe CLI. Use when the user wants to access metrics, logs, traces, or other telemetry data about Laurel infrastructure.

You are an expert at using the Observe CLI (observe) to interact with Observe tenants. The binary is located at ~/go/bin/observe.

Setup & Authentication

Before running any commands, check if the user has a config file:

cat ~/.config/observe.yaml

If not configured, help them log in:

~/go/bin/observe --customerid <CUSTOMER_ID> --site observeinc.com login <EMAIL> --sso

Login methods:

• --sso (recommended): Opens a browser-based approval flow. Best for SSO/SAML users (Okta, Azure AD, Google, PingOne).
• --read-password: Reads password from stdin (no echo). Best for email/password users.
• Direct password on CLI: observe login [e-mail] mypassword (less secure, avoid in interactive shells).

The login command saves the auth token to ~/.config/observe.yaml under the active profile (default: default). Use --no-save to prevent saving.

Configuration File

The config file at ~/.config/observe.yaml supports multiple profiles:

profile:
  default:
    customerid: "133742069123"
    site: "observeinc.com"
    authtoken: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  staging:
    customerid: "180316196377"
    site: "observe-staging.com"
    authtoken: "YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY"
    workspace: MyWorkspace

Select a profile with --profile=<name> or the OBSERVE_PROFILE environment variable.

Running OPAL Queries

The primary command for querying data:

~/go/bin/observe query -q '<OPAL_QUERY>' -i '<INPUT_DATASET>'

Key Flags

Flag	Short	Description
--query	-q	OPAL query text
--input	-i	Input datasets: ID or Workspace.Folder/Name (comma-separated for joins)
--start-time	-s	Start of query time window
--end-time	-e	End of query time window
--relative	-r	Duration of query window (e.g., 4h, 30m, 1d)
--csv	-c	Output in CSV format
--json	-j	Output in nd-JSON format
--extended	-x	Print one column value per row (useful for long values)
--col-width	-w	Max column width for table format; 0 for unlimited (default: 64)
--file	-f	Read OPAL query from a file
--literal-strings	-l	Print embedded control characters literally
--format		Specify output format: table, extended, csv, ndjson

Input Datasets

Inputs can be specified as:

• Dataset ID: -i '41021818'
• Workspace.Name: -i 'Default.kubernetes/Container Logs'
• Multiple inputs for joins: -i '41021818,right=41012929' or -i 'Default.System,right=Default.network/Flow Logs'

If the default workspace is set in config, you can omit the workspace prefix.

Time Windows

Default: last hour, truncated to full minutes. Specify at most two of: --start-time, --end-time, --relative.

Time formats supported:

• RFC 3339: 2023-04-20T16:20:00Z
• With timezone: 2023-04-20T16:20:00.123-08:00
• Relative: -1d, -4h, -30m
• With snap: -1d@1h (1 day ago, snapped to start of hour)
• Epoch seconds: 1682007600
• Epoch milliseconds: 1682007600000
• Epoch nanoseconds: 1682007600000000000

Query Examples

Simple query:

~/go/bin/observe query -q 'limit 10' -i 'Default.System'

Query with time window and extended output:

~/go/bin/observe query -q 'pick_col timestamp, log | limit 10' -i 'Default.kubernetes/Container Logs' -r 4h -x

Query with join:

~/go/bin/observe query \
    -q 'leftjoin host_ip=@right.dst_ip, source:@right.src_ip' \
    -i '41021818,right=41012929' \
    -r 4h

Query from file with CSV output:

~/go/bin/observe query -f my_query.opal -i 'Default.System' --csv -r 24h

JSON output (useful for piping to jq):

~/go/bin/observe query -q 'limit 5' -i 'Default.System' --json

OPAL Query Patterns

Filtering

filter body ~ /error|fail|exception/i          # regex match (case insensitive)
filter service_name = "payment"                 # exact match
filter pod ~ /payment/                          # partial regex match
filter error = true                             # boolean filter
filter body ~ /error/i and pod ~ /payment/      # combined filters

Aggregation with statsby

statsby count:count(1), group_by(body)                        # count by message
statsby count:count(1), group_by(pod)                         # count by pod
statsby count:count(1), group_by(loyalty_level, error)        # multi-dimensional grouping

Time series with timechart

timechart 1h, count:count(1)                                  # hourly counts
timechart 5m, count:count(1)                                  # 5-minute buckets
filter error = true | timechart 1h, count:count(1)            # error rate over time

Column selection and extraction

pick_col timestamp, body, pod, namespace                      # select specific columns
make_col loyalty_level: string(attributes["app.loyalty.level"])  # extract from JSON
make_col card_type: string(attributes["app.payment.card_type"])  # extract nested attribute

Conditional logic

make_col failure_rate: case(body = "error msg", count, true, 0)

Common Dataset Types

Observe workspaces typically contain these dataset categories:

Kubernetes Logs

• Columns: timestamp, body, stream, cluster, namespace, container, pod, node, attributes, resource_attributes, instrumentation_scope, fields, meta
• Use for: Log search, error analysis, pod-level debugging

Tracing / Spans

• Columns: start_time, end_time, duration, service_name, span_name, error, response_status, status_code, status_message, span_type, attributes, resource_attributes, parent_span_id, span_id, trace_id
• Use for: Distributed trace analysis, service dependency mapping, latency investigation
• IMPORTANT: When using pick_col on span datasets, you MUST always include start_time and end_time columns, or the query will fail with: need to pick 'valid from' column "start_time" / need to pick 'valid to' column "end_time"

Metrics (with metric-sma-for-* companion datasets)

• Use for: Resource utilization, SLI/SLO tracking, capacity planning

Reference Tables

• Reference tables (e.g., Products, Customer Records) are regular datasets that hold enrichment data (product catalog, user info, etc.)
• Use for: Joining with trace/log data to enrich with human-readable names, prices, categories
• Query them like any other dataset: ~/go/bin/observe query -q 'limit 20' -i '<REF_TABLE_ID>' --json -w 0

Investigation Workflows

Error Investigation Playbook

When users report failures, follow this multi-step approach:

1. Get an overview — sample recent records to understand schema:

   ~/go/bin/observe query -q 'limit 5' -i '<DATASET_ID>' -r 1h -x
   

2. Quantify the problem — count errors by message type:

   ~/go/bin/observe query -q 'filter body ~ /error|fail|exception/i | statsby count:count(1), group_by(body)' -i '<LOG_DATASET>' -r 24h
   

3. Identify the blast radius — break down by pod/service/namespace:

   ~/go/bin/observe query -q 'filter body ~ /error/i | statsby count:count(1), group_by(pod)' -i '<LOG_DATASET>' -r 24h
   

4. Check for time patterns — trend errors over time:

   ~/go/bin/observe query -q 'filter body ~ /error/i | timechart 1h, count:count(1)' -i '<LOG_DATASET>' -r 24h
   

5. Inspect full error details — get stack traces and attributes:

   ~/go/bin/observe query -q 'filter body ~ /error/i | pick_col timestamp, body, attributes | limit 5' -i '<LOG_DATASET>' -r 24h --json -w 0
   

6. Correlate with traces — find error spans in the tracing dataset:

   ~/go/bin/observe query -q 'filter service_name = "<SERVICE>" and error = true | pick_col start_time, end_time, span_name, attributes, trace_id | limit 10' -i '<SPAN_DATASET>' -r 24h -x -w 0
   

7. Trace a single request end-to-end — follow one trace ID across all services:

   ~/go/bin/observe query -q 'filter trace_id = "<TRACE_ID>" | pick_col start_time, end_time, service_name, span_name, error, duration, attributes' -i '<SPAN_DATASET>' -r 24h -x -w 0
   

8. Cross-reference dimensions — break down errors by attributes (e.g., loyalty level, card type):

   ~/go/bin/observe query -q 'filter service_name = "<SERVICE>" | make_col attr_val: string(attributes["<KEY>"]) | statsby count:count(1), group_by(attr_val, error)' -i '<SPAN_DATASET>' -r 24h
   

Revenue / Business Impact Analysis Playbook

When asked to quantify the business impact of failures, follow this approach using trace span attributes (e.g., app.payment.amount):

1. Get totals by success vs failure — aggregate revenue and transaction counts:

   ~/go/bin/observe query -q 'filter service_name = "<SERVICE>" and span_name = "<SPAN>" | make_col amount: float64(attributes["<AMOUNT_KEY>"]) | statsby total: sum(amount), count: count(1), avg_order: avg(amount), min_order: min(amount), max_order: max(amount), group_by(error)' -i '<SPAN_DATASET>' -r 24h -w 0
   

2. Calculate rates — add hourly burn rate and percentage breakdowns:

   ... | make_col pct_of_total: round(100.0 * count / <TOTAL>.0, 1), hourly_rate: round(total / 24.0, 2)
   

3. Distribution by order size — bucket failed transactions to find where the biggest dollar losses are:

   filter error = true | make_col amount: float64(attributes["<AMOUNT_KEY>"]) | make_col bucket: case(amount < 100, "< $100", amount < 500, "$100-$500", amount < 1000, "$500-$1K", amount < 5000, "$1K-$5K", amount < 10000, "$5K-$10K", true, "$10K+") | statsby lost: sum(amount), count: count(1), group_by(bucket)
   

4. Segment by customer dimension — break down revenue loss by user attributes (loyalty tier, region, etc.):

   filter error = true | make_col segment: string(attributes["<SEGMENT_KEY>"]), amount: float64(attributes["<AMOUNT_KEY>"]) | statsby lost: sum(amount), count: count(1), group_by(segment)
   

5. Break down by product/item — correlate failed traces with product data using the batched trace correlation technique:

   Step 1: Export ALL failed trace IDs and split into batches of ~2,000:

   ~/go/bin/observe query -q 'filter service_name = "<SERVICE>" and span_name = "<SPAN>" and error = true | pick_col start_time, end_time, trace_id' -i '<SPAN_DATASET>' -r 24h --json \
     | python3 -c "
   import sys, json
   traces = list(set(json.loads(l)['trace_id'] for l in sys.stdin if l.strip()))
   print(f'Total unique failed trace IDs: {len(traces)}', file=sys.stderr)
   batch_size = 2000
   for i in range(0, len(traces), batch_size):
       open(f'/tmp/failed_batch_{i//batch_size}.txt','w').write('|'.join(traces[i:i+batch_size]))
   print(f'Created {(len(traces)+batch_size-1)//batch_size} batches')
   "
   

   Step 2: Run all batches in parallel to find correlated spans (e.g., product lookups):

   NUM_BATCHES=<N>  # from step 1
   for i in $(seq 0 $((NUM_BATCHES-1))); do
     TRACES=$(cat /tmp/failed_batch_${i}.txt)
     ~/go/bin/observe query \
       -q "filter service_name = \"<CATALOG_SERVICE>\" and span_name ~ /GetProduct/ and trace_id ~ /${TRACES}/ | make_col product_name: string(attributes[\"<PRODUCT_NAME_KEY>\"]) | statsby count: count(1), group_by(product_name)" \
       -i '<SPAN_DATASET>' -r 24h --json 2>/dev/null > /tmp/product_batch_${i}.json &
   done
   wait
   

   Step 3: Aggregate all batch results and calculate price-weighted revenue:

   import json, glob
   product_counts = {}
   for f in glob.glob('/tmp/product_batch_*.json'):
       for line in open(f):
           obj = json.loads(line.strip())
           name = obj['product_name']
           product_counts[name] = product_counts.get(name, 0) + int(obj['count'])
   
   # Price-weight using reference table data
   total_weighted = sum(prices[name] * count for name, count in product_counts.items())
   for name, count in product_counts.items():
       pct = (prices[name] * count) / total_weighted
       est_lost = pct * total_lost_revenue
   

   This batched approach is necessary because OPAL's leftjoin cannot pre-filter the right-side input dataset, and a single regex with all trace IDs would exceed CLI argument limits. Batches of ~2,000 trace IDs each work well within limits.

   IMPORTANT: When parsing --json output from statsby, numeric values like count are returned as strings, not integers. Always cast with int() in Python.

6. Present the results — always include:

   • Total lost revenue (absolute dollar amount)
   • Failure rate (% of transactions)
   • Hourly/daily burn rate
   • Projected loss if unresolved (extrapolate to 30 days)
   • Distribution by order size (high-value orders often drive most of the dollar loss)
   • Per-product/item breakdown with unit prices when reference data is available
   • Severity recommendation (P0/P1/P2) based on magnitude

OPAL Tips for Numeric Analysis

• Use float64(attributes["key"]) to extract numeric values from JSON attributes for sum(), avg(), min(), max().
• Use round(value, N) to control decimal precision in results.
• Use case() for bucketing continuous values into discrete ranges.
• When statsby on span datasets produces per-span rows instead of aggregates, ensure you are NOT inadvertently including high-cardinality columns (like trace_id, span_id) — these cause implicit grouping. Use only the group_by() columns you intend.

Cross-Trace Correlation Technique (Batched Parallel Queries)

OPAL's leftjoin with two inputs from the same span dataset cannot pre-filter the right side. This means you cannot directly join "failed payment spans" with "product-catalog spans" on trace_id in a single query.

Workaround — Batched Parallel Queries:

1. Export ALL trace IDs from one query (e.g., failed payment spans) using --json
2. Deduplicate and split into batches of ~2,000 (each batch as a regex alternation id1|id2|id3...)
3. Run all batches in parallel using & and wait
4. Aggregate results from all batch output files

# Step 1: Export all trace IDs and split into batches
~/go/bin/observe query -q '... | pick_col start_time, end_time, trace_id' -i '<DATASET>' -r 24h --json \
  | python3 -c "
import sys, json
traces = list(set(json.loads(l)['trace_id'] for l in sys.stdin if l.strip()))
batch_size = 2000
for i in range(0, len(traces), batch_size):
    open(f'/tmp/batch_{i//batch_size}.txt','w').write('|'.join(traces[i:i+batch_size]))
num = (len(traces)+batch_size-1)//batch_size
open('/tmp/num_batches.txt','w').write(str(num))
print(f'{len(traces)} traces -> {num} batches')
"

# Step 2: Run all batches in parallel
NUM=$(cat /tmp/num_batches.txt)
for i in $(seq 0 $((NUM-1))); do
  TRACES=$(cat /tmp/batch_${i}.txt)
  ~/go/bin/observe query \
    -q "filter service_name = \"<SERVICE>\" and trace_id ~ /${TRACES}/ | ..." \
    -i '<DATASET>' -r 24h --json 2>/dev/null > /tmp/result_${i}.json &
done
wait

# Step 3: Aggregate in Python
python3 -c "
import json, glob
totals = {}
for f in glob.glob('/tmp/result_*.json'):
    for line in open(f):
        obj = json.loads(line.strip())
        key = obj['<GROUP_KEY>']
        totals[key] = totals.get(key, 0) + int(obj['count'])
for k, v in sorted(totals.items(), key=lambda x: -x[1]):
    print(f'{k}: {v}')
"

When to use: Correlating data across different services/spans within the same traces (e.g., "which products were in orders that had payment failures"). Use this to cover 100% of traces instead of sampling.

Why batches of ~2,000: Each trace ID is 32 hex chars. A regex alternation of 2,000 IDs is ~66KB, which is within shell argument limits. Batches run in parallel so total wall-clock time is roughly the same as a single query.

Gotcha — --json numeric types: When statsby results are output with --json, numeric aggregation values (like count) are returned as strings, not integers. Always cast with int() or float() when parsing in Python.

Tips for Investigation

• Start broad, then narrow: Begin with high-level counts before drilling into specifics.
• Run queries in parallel: Launch multiple independent queries simultaneously to speed up investigations.
• Use --json -w 0 when you need to see full attribute payloads without truncation.
• Use -x (extended format) for records with many columns or long values.
• Check both logs AND traces: Logs give you error messages and stack traces; traces give you service-to-service call flow, latency, and error propagation paths.
• Steady error rates (consistent per-hour counts) typically indicate a systematic bug; spiky error rates suggest infrastructure or load-related issues.

Listing & Getting Objects

List objects

~/go/bin/observe list <object_type> [substring_filter]

Supported object types: dataset, workspace, user, document, rbacgroup, rbacgroupmember, rbacstatement

Examples:

~/go/bin/observe list dataset
~/go/bin/observe list dataset "Container Logs"
~/go/bin/observe list workspace

Get object details

~/go/bin/observe get <object_type> <object_id>

Returns YAML with config (mutable properties) and state (derived properties) sections.

Example:

~/go/bin/observe get dataset 41042071
~/go/bin/observe get workspace 41042069

Deleting Objects

~/go/bin/observe delete <object_type> <object_id>

Example:

~/go/bin/observe delete document o::1234567890:document:8007654321

Uploading Documents

Upload documents (e.g., prompts for o11y help tool):

~/go/bin/observe upload prompt my-notes.md
~/go/bin/observe upload prompt --as-filename oncall-schedule.md path/to/new/file.md

Prompt documents are indexed for the o11y help tool. Up to 500 prompt documents are supported.

Global Configuration Flags

Flag	Short	Description
--customerid	-C	Numeric tenant ID
--site	-S	Tenant site domain
--authtoken	-A	Bearer token (without "Bearer" prefix)
--profile	-P	Config profile name (default: default)
--config		Config file path (default: ~/.config/observe.yaml)
--workspace		Default workspace for objects
--output	-O	Output file (default: stdout)
--debug	-D	Extra debug logging
--quiet	-Q	Suppress info logs
--show-config		Print resolved config before running

Guidelines

• Always use the full binary path ~/go/bin/observe when running commands.
• When building queries, prefer --json output for programmatic processing and default table format for human-readable output.
• Use --extended (-x) when inspecting records with long field values (JSON objects, log lines).
• For large result sets, always include a limit in the OPAL query to avoid excessive output.
• When the user asks to "query Observe" or "search logs", use the query command with appropriate OPAL.
• When the user asks to "find a dataset" or "what datasets exist", use the list dataset command.
• If authentication fails, suggest re-running login to refresh the token (tokens expire after ~10 days of inactivity).

Gotchas & Common Pitfalls

• Span datasets require start_time and end_time: When using pick_col on span/trace datasets, you MUST include both start_time and end_time or the query will fail. This is because spans are interval-type data.
• Empty results don't mean no data: If a filter returns nothing, try broadening the time window (-r 24h instead of -r 1h) or relaxing the filter pattern.
• statsby output can be very large: When grouping by high-cardinality fields (like body with unique log lines), the result set can be huge. Add a | topk 20, count or similar to limit.
• Regex filters are case-sensitive by default: Use the i flag for case-insensitive matching: filter body ~ /error/i.
• JSON attribute extraction: Use string(attributes["key"]) with make_col to extract values from JSON columns for grouping and filtering.
• Truncated output: Default column width is 64 chars. Use -w 0 to see full values, or --json for complete attribute payloads.
• Large result files: When output exceeds ~30KB, it gets saved to a file. Use --json with limit to keep output manageable.

---

LAUREL INFRASTRUCTURE - ORGANIZATION-SPECIFIC CONTEXT

The following sections provide Laurel-specific datasets, field mappings, authentication, and query patterns for the LaurelAI workspace.

Laurel Authentication

For Laurel infrastructure, use the following customer ID when authenticating:

~/go/bin/observe --customerid 195118325539 --site observeinc.com login <[e-mail]> --sso

Replace <[e-mail]> with your actual Laurel email address.

Laurel Dataset Catalog

The LaurelAI workspace contains the following datasets:

Service & Application Monitoring

• Service Request Logs (LaurelAI.Service Request Logs) - HTTP request logs with status codes, response times, URLs
• Endpoint Logs (LaurelAI.Endpoint Logs) - API endpoint access logs
• Desktop Assistant (LaurelAI.Desktop Assistant) - Desktop application logs
• Frontend (LaurelAI.Frontend) - Browser/mobile frontend observability data

Kubernetes & Container Infrastructure

• Kubernetes Entity (LaurelAI.Kubernetes Explorer/Kubernetes Entity) - K8s resources (pods, services, deployments)
• Kubernetes Logs (LaurelAI.Kubernetes Explorer/Kubernetes Logs) - Container logs
• Parsed Container Logs (LaurelAI.Kubernetes/Parsed Container Logs) - Structured container logs
• Prometheus Metrics (LaurelAI.Kubernetes Explorer/Prometheus Metrics) - K8s cluster metrics

Distributed Tracing

• Trace (LaurelAI.Tracing/Trace) - Distributed traces
• Span (LaurelAI.Tracing/Span) - Individual trace spans
• Service (LaurelAI.Tracing/Service) - Service inventory from traces
• Service Metrics (LaurelAI.Tracing/Service Metrics) - Service-level metrics

Cloud Infrastructure

• AWS (LaurelAI.AWS) - AWS resource inventory and metrics
• AWS Lambda Logs (LaurelAI.AWS/Lambda Logs) - Lambda function logs
• AWS GuardDuty (LaurelAI.AWS GuardDuty) - Security findings
• VPC Flow Logs (LaurelAI.AWS/VPC Flow Logs) - Network traffic logs

Databases & Data Stores

• MongoDB Metrics (LaurelAI.MongoDB/Metrics) - MongoDB performance metrics
• MongoDB Atlas Metrics (LaurelAI.MongoDB Atlas/Metrics) - Atlas-specific metrics
• MongoDB Atlas Cluster (LaurelAI.MongoDB Atlas/Cluster) - Cluster information

Usage & Monitoring

• Monitor Alarms (LaurelAI.usage/Monitor Alarms) - Active alerts and monitors
• Observe Usage Metrics (LaurelAI.usage/Observe Usage Metrics) - Platform usage
• Log Ingest Volume (LaurelAI.usage/Log Ingest Volume) - Log ingestion rates

Authentication & Security

• Auth0 (LaurelAI.Auth0) - Authentication events and logs

Development & CI/CD

• GitHub Actions Run (LaurelAI.github/Actions Run) - CI/CD pipeline runs
• GitHub Pull Requests (LaurelAI.github/Pull Requests) - PR data
• GitHub Issues (LaurelAI.github/Issues) - Issue tracking

Laurel-Specific Field Names

Different datasets use different field names in the LaurelAI workspace. Always verify with limit 1 -x or topk 1 -x first.

Kubernetes Logs (Laurel)

• body - The log message content (NOT message)
• cluster - Kubernetes cluster name (e.g., "iad-prd-eks", "lhr-prd-eks") - top-level field
• namespace - Kubernetes namespace - top-level field
• pod - Pod name - top-level field
• container - Container name
• timestamp - Log timestamp

Note: Kubernetes Logs has cluster, namespace, and pod as top-level fields, making it the most reliable dataset for pod enumeration and namespace-based queries (vs. Kubernetes Entity where these fields are buried in nested/stringified JSON).

Service Request Logs (Laurel)

• service - Service name
• statusCode - HTTP status code
• responseTime - Response time in nanoseconds
• url - Request URL

Prometheus Metrics (Laurel)

• metric - The metric name (e.g., "kube_pod_container_status_restarts_total")
• value - The metric value
• labels - A nested object containing all metric labels
• timestamp - Metric timestamp

Common Prometheus Labels (accessed via labels.<label_name>):

• labels.k8s_cluster_name - Kubernetes cluster name (e.g., "iad-prd-eks", "lhr-prd-eks")
• labels.namespace - Kubernetes namespace
• labels.pod - Pod name
• labels.container - Container name
• labels.k8s_node_name - Node name

Example: To filter by cluster name in Prometheus metrics:

filter metric = "kube_pod_container_status_restarts_total" | make_col cluster:string(labels.k8s_cluster_name)

Tracing/Service (Laurel)

• service_name - Name of the service (NOT name)
• environment - Environment (prd, stg, dev)
• service_namespace - Service namespace
• language - Programming language
• host_type - Host type (k8s, etc.)

Laurel Infrastructure Context

Clusters

Production clusters:

• iad-prd-eks - US East (Virginia) production
• lhr-prd-eks - Europe (London) production

Staging/Dev clusters:

• iad-stg-eks - US East (Virginia) staging

Common Services

Check the Tracing/Service dataset for a full list:

~/go/bin/observe query -q 'topk 100' -i 'LaurelAI.Tracing/Service' -r 24h

Common services include: time, enrichment-api, payment, etc.

Laurel-Specific Query Examples

Investigating Service Errors (Laurel)

# Find recent 5xx errors by service
~/go/bin/observe query -q 'filter statusCode >= 500 | timechart 5m, count(1), group_by(service, statusCode)' -i 'LaurelAI.Service Request Logs' -r 24h

# Get detailed error logs
~/go/bin/observe query -q 'filter statusCode >= 500 | limit 50' -i 'LaurelAI.Service Request Logs' -r 1h -x

Kubernetes Pod Investigation (Laurel)

# Count active pods per namespace across clusters
~/go/bin/observe query -q 'filter namespace = "time" | timechart 1h, count_distinct(pod), group_by(cluster)' -i 'LaurelAI.Kubernetes Explorer/Kubernetes Logs' -r 1h

# List distinct pod names in a namespace
~/go/bin/observe query -q 'filter namespace = "enrichment" and cluster = "iad-prd-eks" | timechart 1h, count_distinct(pod)' -i 'LaurelAI.Kubernetes Explorer/Kubernetes Logs' -r 1h

# List pods via Kubernetes Entity (Resource dataset - use 'topk')
~/go/bin/observe query -q 'filter kind = "Pod" and string(identifiers.clusterName) = "iad-prd-eks" | topk 50' -i 'LaurelAI.Kubernetes Explorer/Kubernetes Entity' -r 1h

Pod Restart Analysis (Laurel)

# Check pod restarts using Prometheus Metrics
~/go/bin/observe query -q 'filter metric = "kube_pod_container_status_restarts_total" and float64(value) > 0 | limit 50' -i 'LaurelAI.Kubernetes Explorer/Prometheus Metrics' -r 24h

# Analyze pod restarts by cluster and namespace
~/go/bin/observe query -q 'filter metric = "kube_pod_container_status_restarts_total" | make_col cluster:string(labels.k8s_cluster_name), namespace:string(labels.namespace), restarts:float64(value) | timechart 48h, max(restarts), group_by(cluster, namespace)' -i 'LaurelAI.Kubernetes Explorer/Prometheus Metrics' -r 48h

Container Log Search (Laurel)

# Search container logs for error patterns
# IMPORTANT: Use 'body' field, NOT 'message'
~/go/bin/observe query -q 'filter body ~ /error|exception|fail/i | limit 50' -i 'LaurelAI.Kubernetes Explorer/Kubernetes Logs' -r 1h -x

# Search specific namespace logs
~/go/bin/observe query -q 'filter namespace = "enrichment" and body ~ /error/i | limit 30' -i 'LaurelAI.Kubernetes Explorer/Kubernetes Logs' -r 6h -x

# Search logs in staging cluster
~/go/bin/observe query -q 'filter cluster = "iad-stg-eks" and body ~ /error/i | limit 30' -i 'LaurelAI.Kubernetes Explorer/Kubernetes Logs' -r 24h -x

Service Tracing (Laurel)

# Find a specific service by name
~/go/bin/observe query -q 'filter service_name = "time" | topk 20' -i 'LaurelAI.Tracing/Service' -r 24h -x

# Find slow traces
~/go/bin/observe query -q 'filter duration > 1000000000 | limit 20' -i 'LaurelAI.Tracing/Trace' -r 1h -x

CI/CD Pipeline Analysis (Laurel)

# Recent GitHub Actions failures (Resource dataset - use 'topk')
~/go/bin/observe query -q 'filter conclusion = "failure" | topk 50' -i 'LaurelAI.github/Actions Run' -r 24h -x

# Find all non-successful runs
~/go/bin/observe query -q 'filter conclusion ~ /(failure|cancelled|timed_out)/ | topk 50' -i 'LaurelAI.github/Actions Run' -r 24h -x

Authentication & Security (Laurel)

# Failed authentication attempts
~/go/bin/observe query -q 'filter type = "f" | limit 50' -i 'LaurelAI.Auth0' -r 24h -x

# AWS GuardDuty findings (last 7 days)
~/go/bin/observe query -q 'limit 20' -i 'LaurelAI.AWS GuardDuty' -r 168h -x

Database Monitoring (Laurel)

# MongoDB metrics
~/go/bin/observe query -q 'timechart 1m, avg(value), group_by(metric_name)' -i 'LaurelAI.MongoDB/Metrics' -r 1h

# MongoDB cluster health
~/go/bin/observe query -q 'topk 10' -i 'LaurelAI.MongoDB Atlas/Cluster' -r 1h -x

Laurel-Specific Troubleshooting

Finding Airflow DAG Errors in Staging

# Search for Airflow pod errors in staging
~/go/bin/observe query -q 'filter cluster = "iad-stg-eks" and pod ~ /airflow/ and body ~ /"level": "ERROR"|"level": "WARNING"/ | limit 30' -i 'LaurelAI.Kubernetes Explorer/Kubernetes Logs' -r 72h -x

# Get full error details in JSON
~/go/bin/observe query -q 'filter cluster = "iad-stg-eks" and pod ~ /airflow/ and body ~ /"level": "ERROR"/ | limit 3' -i 'LaurelAI.Kubernetes Explorer/Kubernetes Logs' -r 72h --json > airflow_errors.json

Filtering by Cluster in Different Datasets (Laurel)

Each dataset stores cluster information differently:

# Kubernetes Logs - use 'cluster' field directly
~/go/bin/observe query -q 'filter cluster = "iad-stg-eks" | limit 50' -i 'LaurelAI.Kubernetes Explorer/Kubernetes Logs' -r 24h

# Kubernetes Entity - use identifiers.clusterName
~/go/bin/observe query -q 'filter string(identifiers.clusterName) = "iad-stg-eks" | topk 50' -i 'LaurelAI.Kubernetes Explorer/Kubernetes Entity' -r 24h

# Prometheus Metrics - use labels.k8s_cluster_name
~/go/bin/observe query -q 'filter string(labels.k8s_cluster_name) = "iad-stg-eks" | limit 50' -i 'LaurelAI.Kubernetes Explorer/Prometheus Metrics' -r 24h

# Service Request Logs - use 'cluster' field directly
~/go/bin/observe query -q 'filter cluster = "iad-stg-eks" | limit 50' -i 'LaurelAI.Service Request Logs' -r 24h

Laurel-Specific Pitfalls

Kubernetes Logs: Use 'body' not 'message'

# WRONG
filter message ~ /error/i

# CORRECT
filter body ~ /error/i

Kubernetes Entity: data field is a JSON string

The data field in Kubernetes Entity is a stringified JSON, NOT a parsed object:

# WRONG - data is a JSON string, not traversable
filter string(data.metadata.namespace) = "time"

# CORRECT - use Kubernetes Logs instead, which has top-level fields
filter namespace = "time"

# OR use regex matching on the stringified data
filter string(data) ~ /"namespace":"time"/

Resource datasets require 'topk' instead of 'limit'

# WRONG for Resource datasets (GitHub Actions, Kubernetes Entity)
limit 50

# CORRECT for Resource datasets
topk 50

# Event datasets (logs, traces, metrics) still use 'limit'

Laurel Quick Reference

# List available Laurel datasets
~/go/bin/observe list dataset | grep -i laurel

# Check Laurel dataset schema
~/go/bin/observe query -q 'limit 1' -i 'LaurelAI.Kubernetes Explorer/Kubernetes Logs' -r 1h -x