["Restore Access","Password Authentication","Biometric Authentication","Strong Password Policy","Restore User Account Access"]
nist_csf
["DE.CM-01","RS.MA-01","GV.OV-01","DE.AE-02"]
Implementing Security Monitoring with Datadog
When to Use
Deploying Cloud SIEM to detect real-time threats across cloud infrastructure (AWS, Azure, GCP)
Creating custom detection rules for attacker techniques, credential abuse, or anomalous behavior
Enabling Workload Protection (CSM Threats) to monitor file, process, and network activity on hosts and containers
Meeting compliance requirements (PCI-DSS, SOC 2, HIPAA) that mandate centralized log monitoring and alerting
Building security dashboards to provide SOC visibility into threat signals, investigation context, and response metrics
Do not use for endpoint-only monitoring without cloud infrastructure; use a dedicated EDR solution for purely on-premises endpoint detection.
Common Misconfigurations & Verification
Logs ingested but not in the SIEM index: Cloud SIEM only evaluates logs flagged for security analytics. If logs_enabled: true works but no signals fire, confirm the source is in a Content Pack / has the security pipeline applied — raw ingestion is not detection.
Workload Protection set in only one file:runtime_security_config.enabled must be true in BOTH datadog.yaml and system-probe.yaml. Verify with sudo datadog-agent status | grep -A5 "Security Agent" showing runtime rules loaded, not just "Agent running".
Detection rule never matches due to attribute mapping: a @evt.outcome:failure query fails if the source isn't parsed into that standard attribute. Confirm in Log Explorer that the facet actually exists before trusting the rule.
Suppression queries over-suppress real alerts: a broad @usr.id:service-account-* suppression also hides an attacker abusing that service account. Scope suppressions narrowly and review the signal-suppressed count.
Confirm end to end, not just config: generate the real event (e.g. a failed-login burst), then verify a Security Signal is created (UI or search_security_monitoring_signals) AND the notification reached Slack/PagerDuty. A green Agent status is not proof a signal would fire.
Datadog App > Dashboards > New Dashboard > "Security Operations Overview"
Widgets:
1. Signal Count Over Time (timeseries)
Query: count:security_signal by {signal.rule.name}
Display: Line chart, last 24 hours
2. Top Triggered Rules (top list)
Query: count:security_signal by {signal.rule.name}.as_count()
Display: Top 10
3. Critical Signals (query value)
Query: count:security_signal{severity:critical}
Conditional format: Red if > 0
4. Signals by Source (pie chart)
Query: count:security_signal by {source}
5. Geographic Threat Map (geomap)
Query: count:security_signal by {network.client.geoip.country.name}
6. Top Targeted Users (top list)
Query: count:security_signal by {usr.id}
7. Mean Time to Triage (query value)
Query: avg:security_signal.triage_time
8. Open Signals by Severity (table)
Query: count:security_signal{status:open} by {severity}
Step 6: Configure Notification Workflows
Set up automated notification and response workflows:
Datadog App > Security > Notification Rules
Rule 1: Critical Signal Escalation
Condition: severity:critical
Recipients: @pagerduty-soc-critical @slack-security-incidents
Message: "CRITICAL security signal: {{signal.rule.name}}
Source: {{signal.attributes.network.client.ip}}
Target: {{signal.attributes.usr.id}}
Details: {{signal.message}}"
Rule 2: High Signal SOC Alert
Condition: severity:high
Recipients: @slack-security-alerts
Suppress: After first notification, suppress for 15 minutes
Rule 3: Compliance Violation
Condition: rule_type:compliance
Recipients: @slack-compliance-team @jira-compliance-board
Workflow Automation (Datadog Workflows):
Trigger: Security signal with severity:critical
Steps:
1. Enrich signal with threat intelligence lookup
2. Create Jira incident ticket
3. Send Slack notification with investigation context
4. If source is AWS: Trigger Lambda to isolate resource
Step 7: Validate and Tune Detection Coverage
Test detection rules and tune false positives:
# Generate a test security event (failed SSH login)
ssh -o StrictHostKeyChecking=no invalid_user@localhost 2>/dev/null
# Verify the event appears in Datadog Logs# Datadog App > Logs > source:auth status:error# Check that a security signal was generated# Datadog App > Security > Signals > Filter by rule name# Tune noisy rules by adding suppression queries:# Datadog App > Security > Detection Rules > [Rule] > Edit# Add suppression: Suppress signal when @usr.id:service-account-*
Use the Security Signals API to validate programmatically:
from datadog_api_client import Configuration, ApiClient
from datadog_api_client.v2.api.security_monitoring_api import SecurityMonitoringApi
configuration = Configuration()
# Reads DD_API_KEY and DD_APP_KEY from environmentwith ApiClient(configuration) as api_client:
api = SecurityMonitoringApi(api_client)
signals = api.search_security_monitoring_signals(
body={
"filter": {
"query": "status:open severity:critical",
"from": "now-24h",
"to": "now",
},
"sort": {"field": "timestamp", "order": "desc"},
"page": {"limit": 25},
}
)
for signal in signals.data:
attrs = signal.attributes
print(f"[{attrs.severity}] {attrs.title}")
print(f" Rule: {attrs.custom.get('rule', {}).get('name', 'N/A')}")
print(f" Time: {attrs.timestamp}")
Key Concepts
Term
Definition
Cloud SIEM
Datadog's security information and event management service that analyzes ingested logs in real-time to detect threats using detection rules
Security Signal
An alert generated when a detection rule matches incoming log data; signals have severity, status (open/triage/closed), and investigation context
Detection Rule
A query-based rule that evaluates logs or events against conditions (threshold, anomaly, new value, impossible travel) to generate security signals
CSM (Cloud Security Management)
Datadog platform for infrastructure security including Misconfigurations (compliance benchmarks), Threats (runtime detection), and Vulnerabilities
Workload Protection
CSM Threats component that monitors file, process, and network activity on hosts and containers using eBPF-based Agent rules
Content Pack
Pre-built collection of detection rules, dashboards, and log parsers for a specific integration (AWS, Azure, GCP, Okta, etc.)
Agent Rule
A kernel-level rule evaluated by the Datadog Agent on the host to collect security-relevant events before sending to Datadog for threat detection
Suppression Query
A filter applied to a detection rule to prevent signals from being generated for known-good activity (reduces false positives)
Verification
Datadog Agent is installed and reporting on all target hosts (datadog-agent status shows security agent running)
Security-relevant log sources are ingesting into Datadog (CloudTrail, auth.log, Windows Security Events visible in Log Explorer)
Cloud SIEM Content Packs are enabled for all cloud providers in use (AWS, Azure, GCP)
Out-of-the-box detection rules are active and generating signals for test events
Custom detection rules trigger correctly (test with a simulated failed login burst)
Workload Protection (CSM Threats) is enabled and Agent rules are evaluating on hosts
Security dashboard displays signal counts, top rules, severity breakdown, and geographic data
Notification workflows deliver alerts to Slack, PagerDuty, or Jira for critical and high signals
Suppression queries are configured to reduce false positives on noisy rules
Security Signals API returns results programmatically for automation integration
