Analyzing Security Logs with Splunk
When to Use
- Investigating a security incident that requires correlation across multiple log sources
- Hunting for adversary activity using known TTPs and IOCs
- Building detection rules for specific attack patterns
- Reconstructing an incident timeline from disparate log sources
- Analyzing authentication anomalies, lateral movement, or data exfiltration patterns
Do not use for real-time packet-level analysis; use Wireshark or Zeek for full packet capture analysis.
Prerequisites
- Splunk Enterprise or Splunk Cloud with Enterprise Security (ES) app installed
- Log sources ingested: Windows Event Logs (via Splunk Universal Forwarder or WEF), firewall, proxy, DNS, EDR, email gateway
- Splunk CIM (Common Information Model) data models configured for normalized field names
- SPL proficiency at intermediate level or higher
- Role-based access with
search and accelerate_search capabilities in Splunk
Workflow
Step 1: Scope the Investigation in Splunk
Define search parameters based on incident triage data:
| Set initial investigation scope
index=windows OR index=firewall OR index=proxy
earliest="2025-11-14T00:00:00" latest="2025-11-16T00:00:00"
(host="WKSTN-042" OR src_ip="10.1.5.42" OR user="jsmith")
| stats count by index, sourcetype, host
| sort -count
This query establishes which log sources contain relevant data for the investigation timeframe and affected assets.
Step 2: Analyze Authentication Events
Investigate suspicious authentication patterns using Windows Security Event Logs:
| Detect brute force and credential stuffing
index=windows sourcetype="WinEventLog:Security" EventCode=4625
earliest=-24h
| stats count as failed_attempts, values(src_ip) as source_ips,
dc(src_ip) as unique_sources by TargetUserName
| where failed_attempts > 10
| sort -failed_attempts
| Detect pass-the-hash (Logon Type 9 - NewCredentials)
index=windows sourcetype="WinEventLog:Security" EventCode=4624
Logon_Type=9
| table _time, host, TargetUserName, src_ip, LogonProcessName
| Detect lateral movement via RDP
index=windows sourcetype="WinEventLog:Security" EventCode=4624
Logon_Type=10
| stats count, values(host) as targets by TargetUserName, src_ip
| where count > 3
| sort -count
Step 3: Trace Process Execution
Use Sysmon logs to reconstruct process execution chains:
| Process creation with parent chain (Sysmon Event ID 1)
index=sysmon EventCode=1 host="WKSTN-042"
earliest="2025-11-15T14:00:00" latest="2025-11-15T15:00:00"
| table _time, ParentImage, ParentCommandLine, Image, CommandLine, User, Hashes
| sort _time
| Detect suspicious PowerShell execution
index=sysmon EventCode=1 Image="*\\powershell.exe"
(CommandLine="*-enc*" OR CommandLine="*-encodedcommand*"
OR CommandLine="*downloadstring*" OR CommandLine="*iex*")
| table _time, host, User, ParentImage, CommandLine
| sort _time
| Detect LSASS credential dumping
index=sysmon EventCode=10 TargetImage="*\\lsass.exe"
GrantedAccess=0x1010
| table _time, host, SourceImage, SourceUser, GrantedAccess
Step 4: Analyze Network Activity
Correlate network logs with endpoint events:
| Detect C2 beaconing pattern
index=proxy OR index=firewall dest_ip="185.220.101.42"
| timechart span=1m count by src_ip
| where count > 0
| Detect DNS tunneling (high query volume to single domain)
index=dns
| rex field=query "(?<subdomain>[^\.]+)\.(?<domain>[^\.]+\.[^\.]+)$"
| stats count, avg(len(query)) as avg_query_len by domain, src_ip
| where count > 500 AND avg_query_len > 40
| sort -count
| Detect large data transfers (potential exfiltration)
index=proxy action=allowed
| stats sum(bytes_out) as total_bytes by src_ip, dest_ip, dest_host
| eval total_MB=round(total_bytes/1024/1024,2)
| where total_MB > 100
| sort -total_MB
Step 5: Build the Incident Timeline
Reconstruct a unified timeline across all log sources:
| Unified incident timeline
index=windows OR index=sysmon OR index=proxy OR index=firewall
(host="WKSTN-042" OR src_ip="10.1.5.42" OR user="jsmith")
earliest="2025-11-15T14:00:00" latest="2025-11-15T16:00:00"
| eval event_summary=case(
sourcetype=="WinEventLog:Security" AND EventCode==4624, "Logon: ".TargetUserName." from ".src_ip,
sourcetype=="WinEventLog:Security" AND EventCode==4625, "Failed logon: ".TargetUserName,
sourcetype=="XmlWinEventLog:Microsoft-Windows-Sysmon/Operational" AND EventCode==1,
"Process: ".Image." by ".User,
sourcetype=="proxy", "Web: ".http_method." ".url,
1==1, sourcetype.": ".EventCode)
| table _time, sourcetype, host, event_summary
| sort _time
Step 6: Create Detection Rules
Convert investigation findings into persistent Splunk correlation searches:
| Correlation search: PowerShell spawned by Office applications
index=sysmon EventCode=1
Image="*\\powershell.exe"
(ParentImage="*\\winword.exe" OR ParentImage="*\\excel.exe"
OR ParentImage="*\\outlook.exe")
| eval severity="high"
| eval mitre_technique="T1059.001"
| collect index=notable_events
Key Concepts
| Term | Definition |
|---|
| SPL (Search Processing Language) | Splunk's query language for searching, filtering, transforming, and visualizing machine data |
| CIM (Common Information Model) | Splunk's field normalization standard that maps vendor-specific field names to common names for cross-source queries |
| Notable Event | An event in Splunk Enterprise Security flagged for analyst review based on a correlation search match |
| Data Model | Structured representation of indexed data in Splunk enabling accelerated searches and pivot-based analysis |
| Sourcetype | Classification label in Splunk that defines the format and parsing rules for a specific log type |
| Correlation Search | Scheduled Splunk search that runs continuously and generates notable events when conditions are met |
| Timechart | SPL command that creates time-series visualizations for identifying patterns, anomalies, and trends |
Tools & Systems
- Splunk Enterprise Security (ES): Premium SIEM application providing correlation searches, risk-based alerting, and investigation workbench
- Splunk SOAR: Orchestration platform integrated with Splunk ES for automated response playbooks
- Sysmon: Microsoft system monitoring tool providing detailed process, network, and file change telemetry ingested into Splunk
- Splunk Attack Analyzer: Automated threat analysis that detonates suspicious files and URLs, feeding results into Splunk
- BOSS of the SOC (BOTS): SANS/Splunk training dataset for practicing incident investigation SPL queries
Common Scenarios
Scenario: Investigating Credential Stuffing Leading to Account Takeover
Context: Security operations receives an alert for multiple successful logins to a single account from geographically dispersed IP addresses within a 30-minute window.
Approach:
- Query Event ID 4624 for the affected account to map all login sources and times
- Correlate login IPs against threat intelligence feeds using a Splunk lookup table
- Check proxy logs for suspicious activity from the authenticated sessions
- Search for lateral movement from the compromised account (Event ID 4624 Type 3 to other hosts)
- Build a timeline showing credential stuffing attempts, successful login, and post-compromise activity
- Create a correlation search to detect similar patterns on other accounts
Pitfalls:
- Searching only the last 24 hours when the credential stuffing may have occurred over weeks
- Not checking for VPN logs that may show the same account authenticating from impossible travel distances
- Failing to normalize timestamps across log sources in different time zones
Output Format
SPLUNK INVESTIGATION REPORT
============================
Incident: INC-2025-1547
Analyst: [Name]
Investigation Period: 2025-11-14 00:00 UTC - 2025-11-16 00:00 UTC
SEARCH SCOPE
Indexes: windows, sysmon, proxy, firewall, dns
Hosts: WKSTN-042, SRV-FILE01
Users: jsmith, svc-backup
Source IPs: 10.1.5.42, 10.1.10.15
KEY FINDINGS
1. [timestamp] - Initial compromise via phishing (Sysmon Event 1)
2. [timestamp] - C2 established (proxy logs, beacon pattern detected)
3. [timestamp] - Credential theft (Sysmon Event 10, LSASS access)
4. [timestamp] - Lateral movement to SRV-FILE01 (Event 4624 Type 3)
5. [timestamp] - Data staging and exfiltration (proxy bytes_out anomaly)
SPL QUERIES USED
[numbered list of key queries with descriptions]
DETECTION GAPS IDENTIFIED
- No Sysmon deployed on SRV-FILE01 (blind spot)
- Proxy logs missing SSL inspection for C2 domain
- PowerShell ScriptBlock logging not enabled
RECOMMENDED DETECTIONS
1. Correlation search for Office-spawned PowerShell
2. Threshold alert for LSASS access patterns
3. Behavioral rule for beacon-interval network traffic
