Examine Linux system artifacts (auth logs, cron/systemd persistence, shell history, SSH keys, and system configuration) to uncover evidence of compromise, detect rootkits or backdoors, and reconstruct user/attacker activity. Use when investigating a compromised Linux server or workstation, hunting for persistence mechanisms, or scoping a Linux-based breach during incident response.
Examine Linux system artifacts (auth logs, cron/systemd persistence, shell history, SSH keys, and system configuration) to uncover evidence of compromise, detect rootkits or backdoors, and reconstruct user/attacker activity. Use when investigating a compromised Linux server or workstation, hunting for persistence mechanisms, or scoping a Linux-based breach during incident response.
# Analyze user accounts for anomalies
python3 << 'PYEOF'print("=== USER ACCOUNT ANALYSIS ===\n")
# Parse /etc/passwd
with open('/cases/case-2024-001/linux/config/passwd') as f:
for line in f:
parts = line.strip().split(':')
if len(parts) >= 7:
username, _, uid, gid, comment, home, shell = parts[0], parts[1], int(parts[2]), int(parts[3]), parts[4], parts[5], parts[6]
# Flag accounts with UID 0 (root equivalent)if uid == 0 and username != 'root':
print(f" ALERT: UID 0 account: {username} (shell: {shell})")
# Flag accounts with login shells that shouldn't have themif shell not in ('/bin/false', '/usr/sbin/nologin', '/bin/sync') and uid >= 1000:
print(f" User: {username} (UID:{uid}, Shell:{shell}, Home:{home})")
# Flag system accounts with login shellsif uid < 1000 and uid > 0 and shell in ('/bin/bash', '/bin/sh', '/bin/zsh'):
print(f" WARNING: System account with shell: {username} (UID:{uid}, Shell:{shell})")
# Parse /etc/shadow for account statusprint("\n=== PASSWORD STATUS ===")
with open('/cases/case-2024-001/linux/config/shadow') as f:
for line in f:
parts = line.strip().split(':')
if len(parts) >= 3:
username = parts[0]
pwd_hash = parts[1]
last_change = parts[2]
if pwd_hash and pwd_hash not in ('*', '!', '!!', ''):
hash_type = 'Unknown'if pwd_hash.startswith('$6$'): hash_type = 'SHA-512'elif pwd_hash.startswith('$5$'): hash_type = 'SHA-256'elif pwd_hash.startswith('$y$'): hash_type = 'yescrypt'elif pwd_hash.startswith('$1$'): hash_type = 'MD5 (WEAK)'print(f" {username}: {hash_type} hash, last changed: day {last_change}")
PYEOF
# Analyze login history
last -f /cases/case-2024-001/linux/logs/wtmp > /cases/case-2024-001/linux/analysis/login_history.txt
lastb -f /cases/case-2024-001/linux/logs/btmp > /cases/case-2024-001/linux/analysis/failed_logins.txt 2>/dev/null
Step 3: Examine Persistence Mechanisms
# Check cron jobs for all usersecho"=== CRON JOBS ===" > /cases/case-2024-001/linux/persistence/cron_analysis.txt
# System cronfor cronfile in /mnt/evidence/etc/crontab /mnt/evidence/etc/cron.d/*; doecho"--- $cronfile ---" >> /cases/case-2024-001/linux/persistence/cron_analysis.txt
cat"$cronfile" 2>/dev/null >> /cases/case-2024-001/linux/persistence/cron_analysis.txt
echo"" >> /cases/case-2024-001/linux/persistence/cron_analysis.txt
done# User cron tabsfor cronfile in /mnt/evidence/var/spool/cron/crontabs/*; doecho"--- User crontab: $(basename $cronfile) ---" >> /cases/case-2024-001/linux/persistence/cron_analysis.txt
cat"$cronfile" 2>/dev/null >> /cases/case-2024-001/linux/persistence/cron_analysis.txt
echo"" >> /cases/case-2024-001/linux/persistence/cron_analysis.txt
done# Check systemd services for persistenceecho"=== SYSTEMD SERVICES ===" > /cases/case-2024-001/linux/persistence/systemd_analysis.txt
find /mnt/evidence/etc/systemd/system/ -name "*.service" -newer /mnt/evidence/etc/os-release \
>> /cases/case-2024-001/linux/persistence/systemd_analysis.txt
for svc in /mnt/evidence/etc/systemd/system/*.service; doecho"--- $(basename $svc) ---" >> /cases/case-2024-001/linux/persistence/systemd_analysis.txt
cat"$svc" >> /cases/case-2024-001/linux/persistence/systemd_analysis.txt
echo"" >> /cases/case-2024-001/linux/persistence/systemd_analysis.txt
done# Check authorized SSH keys (backdoor detection)echo"=== SSH AUTHORIZED KEYS ===" > /cases/case-2024-001/linux/persistence/ssh_keys.txt
find /mnt/evidence/home/ /mnt/evidence/root/ -name "authorized_keys" -exec sh -c \
'echo "--- {} ---"; cat {}; echo ""' \; >> /cases/case-2024-001/linux/persistence/ssh_keys.txt
# Check rc.local and init scriptscat /mnt/evidence/etc/rc.local 2>/dev/null > /cases/case-2024-001/linux/persistence/rc_local.txt
# Check /etc/profile.d/ for login-triggered scriptsls -la /mnt/evidence/etc/profile.d/ > /cases/case-2024-001/linux/persistence/profile_scripts.txt
# Check for LD_PRELOAD hijacking
grep -r "LD_PRELOAD" /mnt/evidence/etc/ 2>/dev/null > /cases/case-2024-001/linux/persistence/ld_preload.txt
cat /mnt/evidence/etc/ld.so.preload 2>/dev/null >> /cases/case-2024-001/linux/persistence/ld_preload.txt
Step 4: Analyze Shell History and Command Execution
# Analyze bash history for each user
python3 << 'PYEOF'
import os, glob
print("=== SHELL HISTORY ANALYSIS ===\n")
suspicious_commands = [
'wget', 'curl', 'nc ', 'ncat', 'netcat', 'python -c', 'python3 -c',
'perl -e', 'base64', 'chmod 777', 'chmod +s', '/dev/tcp', '/dev/udp',
'nmap', 'masscan', 'hydra', 'john', 'hashcat', 'passwd', 'useradd',
'iptables -F', 'ufw disable', 'history -c', 'rm -rf /', 'dd if=',
'crontab', 'at ', 'systemctl enable', 'ssh-keygen', 'scp ', 'rsync',
'tar czf', 'zip -r', 'openssl enc', 'gpg --encrypt', 'shred',
'chattr', 'setfacl', 'awk', '/tmp/', '/dev/shm/'
]
for hist_file in glob.glob('/cases/case-2024-001/linux/users/*/.bash_history'):
username = hist_file.split('/')[-2]
print(f"User: {username}")
with open(hist_file, 'r', errors='ignore') as f:
lines = f.readlines()
print(f" Total commands: {len(lines)}")
flagged = []
for i, line in enumerate(lines):
line = line.strip()
for cmd in suspicious_commands:
if cmd in line.lower():
flagged.append((i+1, line))
breakif flagged:
print(f" Suspicious commands: {len(flagged)}")
for lineno, cmd in flagged:
print(f" Line {lineno}: {cmd[:120]}")
print()
PYEOF
Step 5: Check for Rootkits and Modified Binaries
# Check for known rootkit indicators# Compare system binary hashes against known-good
find /mnt/evidence/usr/bin/ /mnt/evidence/usr/sbin/ /mnt/evidence/bin/ /mnt/evidence/sbin/ \
-type f -executable -execsha256sum {} \; > /cases/case-2024-001/linux/analysis/binary_hashes.txt
# Check for SUID/SGID binaries (potential privilege escalation)
find /mnt/evidence/ -perm -4000 -type f 2>/dev/null > /cases/case-2024-001/linux/analysis/suid_files.txt
find /mnt/evidence/ -perm -2000 -type f 2>/dev/null > /cases/case-2024-001/linux/analysis/sgid_files.txt
# Check for suspicious files in /tmp and /dev/shm
find /mnt/evidence/tmp/ /mnt/evidence/dev/shm/ -type f 2>/dev/null \
-exec file {} \; > /cases/case-2024-001/linux/analysis/tmp_files.txt
# Check for hidden files and directories
find /mnt/evidence/ -name ".*" -not -path "*/\." -type f 2>/dev/null | \
head -100 > /cases/case-2024-001/linux/analysis/hidden_files.txt
# Check kernel modulesls -la /mnt/evidence/lib/modules/$(ls /mnt/evidence/lib/modules/ | head -1)/extra/ 2>/dev/null \
> /cases/case-2024-001/linux/analysis/extra_modules.txt
# Check for modified PAM configuration (authentication backdoors)
diff /mnt/evidence/etc/pam.d/ /cases/baseline/pam.d/ 2>/dev/null \
> /cases/case-2024-001/linux/analysis/pam_changes.txt
Key Concepts
Concept
Description
/var/log/auth.log
Primary authentication log on Debian/Ubuntu systems
/var/log/secure
Primary authentication log on RHEL/CentOS systems
wtmp/btmp
Binary logs recording successful and failed login sessions
.bash_history
User command history file (can be cleared by attackers)
crontab
Scheduled task system commonly used for persistence
authorized_keys
SSH public keys granting passwordless access to an account
SUID bit
File permission allowing execution as the file owner (privilege escalation vector)
LD_PRELOAD
Environment variable that loads a shared library before all others (hooking technique)
Tools & Systems
Tool
Purpose
chkrootkit
Rootkit detection scanner for Linux systems
rkhunter
Rootkit Hunter - checks for rootkits, backdoors, and local exploits
Linux audit framework for system call and file access monitoring
last/lastb
Parse wtmp/btmp for login and failed login history
Plaso/log2timeline
Super-timeline creation including Linux artifacts
osquery
SQL-based system querying for live forensic investigation
Velociraptor
Endpoint agent with Linux artifact collection capabilities
Common Scenarios
Scenario 1: SSH Brute Force Followed by Compromise
Analyze auth.log for failed SSH attempts followed by success, identify the attacking IP, check .bash_history for post-compromise commands, examine authorized_keys for added backdoor keys, check crontab for persistence, review network connections.
Scenario 2: Web Server Compromise via Application Vulnerability
Examine web server access and error logs for exploitation attempts, check /tmp and /dev/shm for webshells, analyze the web server user's activity (www-data), check for privilege escalation via SUID binaries or kernel exploits, review outbound connections.
Scenario 3: Insider Threat on Database Server
Analyze the suspect user's bash_history for database dump commands, check for large tar/zip files in home directory or /tmp, examine scp/rsync commands for data transfer, review cron jobs for automated exfiltration, check USB device logs.
Scenario 4: Crypto-Miner on Cloud Instance
Check for high-CPU processes in /proc (live) or systemd service files, examine crontab entries for miner restart scripts, check /tmp for mining binaries, analyze network connections for mining pool communications, review authorized_keys for attacker access.
