Datadog API Key and Application Key from Organization Settings > API Keys
Datadog Agent v7+ installed on hosts/containers that generate security-relevant logs
Log sources configured for ingestion: AWS CloudTrail, VPC Flow Logs, GuardDuty, Azure Activity Logs, GCP Audit Logs, or on-host logs (auth.log, syslog, Windows Security Events)
Python 3.9+ with datadog-api-client library for programmatic rule management
Network access from monitored hosts to Datadog intake endpoints (port 443)
Workflow
Step 1: Deploy and Configure the Datadog Agent for Security
Install the Datadog Agent and enable security-related features in datadog.yaml:
# /etc/datadog-agent/datadog.yamlapi_key:<YOUR_DATADOG_API_KEY>site:datadoghq.com# or datadoghq.eu, us3.datadoghq.com, etc.# Enable log collection for Cloud SIEMlogs_enabled:true# Enable security featuresruntime_security_config:enabled:true# Workload Protection (CSM Threats)activity_dump:enabled:true# Record process activity for investigation
compliance_config:
enabled:
true
# CIS benchmark checks (CSM Misconfigurations)
host_benchmarks:
enabled:
true
Configure log sources for security-relevant files on Linux:
Datadog App > Dashboards > New Dashboard > "Security Operations Overview"
Widgets:
1. Signal Count Over Time (timeseries)
Query: count:security_signal by {signal.rule.name}
Display: Line chart, last 24 hours
2. Top Triggered Rules (top list)
Query: count:security_signal by {signal.rule.name}.as_count()
Display: Top 10
3. Critical Signals (query value)
Query: count:security_signal{severity:critical}
Conditional format: Red if > 0
4. Signals by Source (pie chart)
Query: count:security_signal by {source}
5. Geographic Threat Map (geomap)
Query: count:security_signal by {network.client.geoip.country.name}
6. Top Targeted Users (top list)
Query: count:security_signal by {usr.id}
7. Mean Time to Triage (query value)
Query: avg:security_signal.triage_time
8. Open Signals by Severity (table)
Query: count:security_signal{status:open} by {severity}
Step 6: Configure Notification Workflows
Set up automated notification and response workflows:
Datadog App > Security > Notification Rules
Rule 1: Critical Signal Escalation
Condition: severity:critical
Recipients: @pagerduty-soc-critical @slack-security-incidents
Message: "CRITICAL security signal: {{signal.rule.name}}
Source: {{signal.attributes.network.client.ip}}
Target: {{signal.attributes.usr.id}}
Details: {{signal.message}}"
Rule 2: High Signal SOC Alert
Condition: severity:high
Recipients: @slack-security-alerts
Suppress: After first notification, suppress for 15 minutes
Rule 3: Compliance Violation
Condition: rule_type:compliance
Recipients: @slack-compliance-team @jira-compliance-board
Workflow Automation (Datadog Workflows):
Trigger: Security signal with severity:critical
Steps:
1. Enrich signal with threat intelligence lookup
2. Create Jira incident ticket
3. Send Slack notification with investigation context
4. If source is AWS: Trigger Lambda to isolate resource
Step 7: Validate and Tune Detection Coverage
Test detection rules and tune false positives:
# Generate a test security event (failed SSH login)
ssh -o StrictHostKeyChecking=no invalid_user@localhost 2>/dev/null
# Verify the event appears in Datadog Logs# Datadog App > Logs > source:auth status:error# Check that a security signal was generated# Datadog App > Security > Signals > Filter by rule name# Tune noisy rules by adding suppression queries:# Datadog App > Security > Detection Rules > [Rule] > Edit# Add suppression: Suppress signal when @usr.id:service-account-*
Use the Security Signals API to validate programmatically:
from datadog_api_client import Configuration, ApiClient
from datadog_api_client.v2.api.security_monitoring_api import SecurityMonitoringApi
configuration = Configuration()
# Reads DD_API_KEY and DD_APP_KEY from environmentwith ApiClient(configuration) as api_client:
api = SecurityMonitoringApi(api_client)
signals = api.search_security_monitoring_signals(
body={
"filter": {
"query": "status:open severity:critical",
"from": "now-24h",
"to": "now",
},
"sort": {"field": "timestamp", "order": "desc"},
"page": {"limit": 25},
}
)
for signal in signals.data:
attrs = signal.attributes
print(f"[{attrs.severity}] {attrs.title}")
print(f" Rule: {attrs.custom.get('rule', {}).get('name', 'N/A')}")
print(f" Time: {attrs.timestamp}")
Key Concepts
Term
Definition
Cloud SIEM
Datadog's security information and event management service that analyzes ingested logs in real-time to detect threats using detection rules
Security Signal
An alert generated when a detection rule matches incoming log data; signals have severity, status (open/triage/closed), and investigation context
Detection Rule
A query-based rule that evaluates logs or events against conditions (threshold, anomaly, new value, impossible travel) to generate security signals
CSM (Cloud Security Management)
Datadog platform for infrastructure security including Misconfigurations (compliance benchmarks), Threats (runtime detection), and Vulnerabilities
Workload Protection
CSM Threats component that monitors file, process, and network activity on hosts and containers using eBPF-based Agent rules
Content Pack
Pre-built collection of detection rules, dashboards, and log parsers for a specific integration (AWS, Azure, GCP, Okta, etc.)
Agent Rule
A kernel-level rule evaluated by the Datadog Agent on the host to collect security-relevant events before sending to Datadog for threat detection
Suppression Query
A filter applied to a detection rule to prevent signals from being generated for known-good activity (reduces false positives)
Verification
Datadog Agent is installed and reporting on all target hosts (datadog-agent status shows security agent running)
Security-relevant log sources are ingesting into Datadog (CloudTrail, auth.log, Windows Security Events visible in Log Explorer)
Cloud SIEM Content Packs are enabled for all cloud providers in use (AWS, Azure, GCP)
Out-of-the-box detection rules are active and generating signals for test events
Custom detection rules trigger correctly (test with a simulated failed login burst)
Workload Protection (CSM Threats) is enabled and Agent rules are evaluating on hosts
Security dashboard displays signal counts, top rules, severity breakdown, and geographic data
Notification workflows deliver alerts to Slack, PagerDuty, or Jira for critical and high signals
Suppression queries are configured to reduce false positives on noisy rules
Security Signals API returns results programmatically for automation integration
