dt-obs-logs

name	dt-obs-logs
description	Log querying, filtering, pattern analysis, and error rate calculation. Use when searching application or infrastructure logs, analyzing error patterns, or correlating log data. Trigger: "show error logs", "search logs for keyword", "log error rate", "recent errors", "logs from last hour", "find log entries", "top error messages", "log patterns", "parse JSON logs", "logs by process group", "log trends over time", "log entry counts per minute". Do NOT use for explaining existing queries, product documentation questions, distributed tracing or span analysis (use dt-obs-tracing).
license	Apache-2.0

Log Analysis Skill

Query, filter, and analyze Dynatrace log data using DQL for troubleshooting and monitoring.

What This Skill Covers

• Fetching and filtering logs by severity, content, and entity
• Searching log messages using pattern matching
• Calculating error rates and statistics
• Analyzing log patterns and trends
• Grouping and aggregating log data by dimensions

Cross-source join required: If the query must combine logs with host attributes (OS type, hostname, IP address, cloud provider) → also read dt-dql-essentials/references/smartscape-topology-navigation.md before writing the query.

---

Use Cases

Use this skill when users want to:

• Find specific log entries (e.g., "show me error logs from the last hour")
• Filter logs by severity, process group, or content
• Search logs for specific keywords or phrases
• Calculate error rates or log statistics
• Identify common error messages or patterns
• Analyze log trends over time
• Troubleshoot issues using log data

Key Concepts

Log Data Model

• timestamp: When the log entry was created
• content: The log message text
• status: Log level (ERROR, FATAL, WARN, INFO, etc.)
• dt.process_group.id: Associated process group entity
• dt.process_group.detected_name: Resolves process group IDs to human-readable names

Query Patterns

• fetch logs: Primary command for log data access
• Time ranges: Use from:now() - <duration> for time windows
• Filtering: Apply severity, content, and entity filters
• Aggregation: Group and summarize log data
• Pattern Detection: Use matchesPhrase() and contains() for content search

Common Operations

• Severity filtering (single or multiple levels)
• Content search (simple and full-text)
• Entity-based filtering (process groups)
• Time-series analysis (bucketing, sorting)
• Error rate calculation
• Pattern analysis (exceptions, timeouts, etc.)

Core Workflows

1. Log Searching

Find specific log entries by time, severity, and content.

Typical steps:

1. Define time range
2. Filter by severity (optional)
3. Search content for keywords
4. Select relevant fields
5. Sort and limit results

Example:

fetch logs, from:now() - 1h
| filter status == "ERROR"
| fields timestamp, content, process_group = dt.process_group.detected_name
| sort timestamp desc
| limit 100

2. Log Filtering

Narrow down logs using multiple criteria (severity, entity, content).

Typical steps:

1. Fetch logs with time range
2. Apply severity filters
3. Filter by entity (process_group)
4. Apply content filters
5. Format and sort output

Example:

fetch logs, from:now() - 2h
| filter in(status, {"ERROR", "FATAL", "WARN"})
| summarize count(), by: {dt.process_group.id, dt.process_group.detected_name}
| fieldsAdd process_group = dt.process_group.detected_name
| sort `count()` desc

3. Pattern Analysis

Identify patterns, trends, and anomalies in log data.

Typical steps:

1. Fetch logs with time range
2. Add pattern detection fields
3. Aggregate by entity or time
4. Calculate statistics and ratios
5. Sort by frequency or rate

Example:

fetch logs, from:now() - 2h
| filter status == "ERROR"
| fieldsAdd
    has_exception = if(matchesPhrase(content, "exception"), true, else: false),
    has_timeout = if(matchesPhrase(content, "timeout"), true, else: false)
| summarize
    count(),
    exception_count = countIf(has_exception == true),
    timeout_count = countIf(has_timeout == true),
    by: {process_group = dt.process_group.detected_name}

Key Functions

Filtering

• filter status == "ERROR" - Filter by status level
• in(status, {"ERROR", "FATAL", "WARN"}) - Multi-status filter (use curly braces for literal sets)
• contains(content, "keyword") - Simple substring search
• matchesPhrase(content, "exact phrase") - Full-text phrase search

Entity Operations

• dt.process_group.detected_name - Get human-readable process group name
• filter process_group == "service-name" - Filter by specific entity

Aggregation

• count() - Count all log entries
• countIf(condition) - Conditional count
• by: {dimension} - Group by entity or time bucket
• bin(timestamp, 5m) - Time bucketing for trends

Field Operations

• fields timestamp, content, status - Select specific fields
• fieldsAdd name = expression - Add computed fields
• if(condition, true_value, else: false_value) - Conditional logic

Common Patterns

Content Search

Simple substring search:

fetch logs, from:now() - 1h
| filter contains(content, "database")
| fields timestamp, content, status

Full-text phrase search:

fetch logs, from:now() - 1h
| filter matchesPhrase(content, "connection timeout")
| fields timestamp, content, process_group = dt.process_group.detected_name

Error Rate Calculation

Calculate error rates over time:

fetch logs, from:now() - 2h
| summarize
    total_logs = count(),
    error_logs = countIf(status == "ERROR"),
    by: {time_bucket = bin(timestamp, 5m)}
| fieldsAdd error_rate = (error_logs * 100.0) / total_logs
| sort time_bucket asc

Top Error Messages

Find most common errors:

fetch logs, from:now() - 24h
| filter status == "ERROR"
| summarize error_count = count(), by: {content}
| sort error_count desc
| limit 20

Process Group-Specific Logs

Filter logs by process group:

fetch logs, from:now() - 1h
| fieldsAdd process_group = dt.process_group.detected_name
| filter process_group == "payment-service"
| filter status == "ERROR"
| fields timestamp, content, status
| sort timestamp desc

Structured / JSON Log Parsing

Many applications emit JSON-formatted log lines. Use parse to extract fields instead of dumping raw content:

fetch logs, from:now() - 1h
| filter status == "ERROR"
| parse content, "JSON:log"
| fieldsAdd level = log[level], message = log[msg], error = log[error]
| fields timestamp, level, message, error
| sort timestamp desc
| limit 50

Aggregate by a parsed field:

fetch logs, from:now() - 4h
| filter status == "ERROR"
| parse content, "JSON:log"
| fieldsAdd message = log[msg]
| summarize error_count = count(), by: {message}
| sort error_count desc
| limit 20

Notes:

• parse content, "JSON:log" creates a record field log — access nested values with log[key]
• Filter logs with contains() before parse to reduce parsing overhead
• Works with any JSON-structured field, not just content

Best Practices

1. Always specify time ranges - Use from:now() - <duration> to limit data
2. Apply filters early - Filter by severity and entity before aggregation
3. Use appropriate search methods - contains() for simple, matchesPhrase() for exact
4. Limit results - Add | limit 100 to prevent overwhelming output
5. Sort meaningfully - Sort by timestamp for recent logs, by count for top errors
6. Name entities - Use dt.process_group.detected_name or getNodeName() for human-readable output
7. Use time buckets for trends - bin(timestamp, 5m) for time-series analysis

Integration Points

• Entity model: Uses dt.process_group.id for service correlation
• Time series: Supports temporal analysis with bin() and time ranges
• Content search: Full-text search capabilities via matchesPhrase()
• Aggregation: Statistical analysis using summarize and conditional functions

Limitations & Notes

• Log availability depends on OneAgent configuration and log ingestion
• Full-text search (matchesPhrase) may have performance implications on large datasets
• Entity names require proper OneAgent monitoring for resolution
• Time ranges should be reasonable (avoid unbounded queries)

Troubleshooting

Problem	Cause	Solution
No logs returned	Missing time range or too narrow	Widen from: window; verify log ingestion is active
getNodeName() returns null	OneAgent not monitoring the entity or entity not yet resolved	Verify OneAgent is deployed and entity is discovered; use dt.process_group.detected_name as a reliable alternative
matchesPhrase() slow on large data	Full-text search without pre-filtering	Add filter status == "ERROR" before matchesPhrase()
Wrong field name log.level	Common mistake	Use loglevel (no dot) for severity; see dt-dql-essentials
Empty content field	Log line was empty or not ingested	Check log source configuration in OneAgent

Related Skills

• dt-dql-essentials - Core DQL syntax and query structure for log queries
• dt-obs-tracing - Correlate logs with distributed traces using trace IDs
• dt-obs-problems - Correlate logs with DAVIS-detected problems