audit-logging

name	audit-logging
description	Implement centralized audit logging and SIEM integration. Configure log retention and security monitoring. Use when implementing audit trail requirements.
license	MIT
metadata	{"author":"devops-skills","version":"1.0"}

Audit Logging

Implement comprehensive audit logging for compliance, security monitoring, and forensic analysis across infrastructure and applications.

When to Use

• Setting up centralized logging for compliance frameworks (SOC 2, HIPAA, PCI DSS)
• Implementing security event monitoring and alerting
• Building audit trails for regulatory requirements
• Configuring log retention and tamper-proof storage
• Integrating application logs with SIEM platforms

Log Categories

audit_events:
  authentication:
    - Login attempts (success and failure)
    - MFA enrollment and verification events
    - Session creation, renewal, and termination
    - Password changes and resets
    - API key and token generation

  authorization:
    - Access grants and denials
    - Permission changes and role assignments
    - Privilege escalation events
    - Resource sharing modifications
    - Policy evaluation results

  data_access:
    - Read operations on sensitive data
    - Write and update operations
    - Delete and purge operations
    - Bulk export and download events
    - Data classification changes

  administrative:
    - Configuration changes
    - User and group management
    - System startup and shutdown
    - Backup and restore operations
    - Network and firewall rule changes

  system:
    - Service health state changes
    - Resource provisioning and deprovisioning
    - Certificate and key rotation events
    - Scheduled job execution results
    - Integration and webhook events

Rsyslog Configuration for Centralized Logging

# /etc/rsyslog.d/50-audit.conf

# Load imfile module to read application logs
module(load="imfile")

# Forward auth logs
input(type="imfile"
  File="/var/log/auth.log"
  Tag="auth"
  Severity="info"
  Facility="auth"
)

# Forward application audit logs
input(type="imfile"
  File="/var/log/app/audit.log"
  Tag="app-audit"
  Severity="info"
  Facility="local0"
)

# Structured JSON template
template(name="json-audit" type="list") {
  constant(value="{")
  constant(value="\"@timestamp\":\"")    property(name="timereported" dateFormat="rfc3339")
  constant(value="\",\"host\":\"")       property(name="hostname")
  constant(value="\",\"severity\":\"")   property(name="syslogseverity-text")
  constant(value="\",\"facility\":\"")   property(name="syslogfacility-text")
  constant(value="\",\"tag\":\"")        property(name="syslogtag" format="json")
  constant(value="\",\"message\":\"")    property(name="msg" format="json")
  constant(value="\"}\n")
}

# Forward to central syslog server over TLS
action(
  type="omfwd"
  target="syslog.internal.example.com"
  port="6514"
  protocol="tcp"
  StreamDriver=
  StreamDriverMode=
  StreamDriverAuthMode=
  template=
  queue.type=
  queue.size=
  queue.filename=
  queue.saveonshutdown=
  action.resumeRetryCount=
)

Journald Configuration for Persistent Logging

# /etc/systemd/journald.conf
[Journal]
Storage=persistent
Compress=yes
Seal=yes
SplitMode=uid
MaxRetentionSec=365d
MaxFileSec=30d
SystemMaxUse=10G
SystemKeepFree=2G
ForwardToSyslog=yes

# Query journald for audit events
journalctl _TRANSPORT=audit --since "24 hours ago" --output json-pretty

# Filter by specific audit types
journalctl _AUDIT_TYPE=1112 --since today  # user login events
journalctl _AUDIT_TYPE=1100 --since today  # user auth events

# Export for offline analysis
journalctl --since "7 days ago" --output export > /backup/journal-export.bin

Application Logging with Structured JSON

import logging
import json
import hashlib
from datetime import datetime, timezone
from functools import wraps

class AuditLogger:
    def __init__(self, service_name, logger_name="audit"):
        self.service = service_name
        self.logger = logging.getLogger(logger_name)
        handler = logging.FileHandler("/var/log/app/audit.log")
        handler.setFormatter(logging.Formatter("%(message)s"))
        self.logger.addHandler(handler)
        self.logger.setLevel(logging.INFO)
        self._prev_hash = None

    def log_event(self, event_type, user, resource, action, result,
                  metadata=None, source_ip=None):
        log_entry = {
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "service": self.service,
            "event_type": event_type,
            "user": user,
            "resource": resource,
            "action": action,
            "result": result,
            "source_ip": source_ip,
            "metadata": metadata or {},
        }
        # Chain hash for tamper detection
        raw = json.dumps(log_entry, sort_keys=True)
        log_entry["prev_hash"] = self._prev_hash
        log_entry[] = hashlib.sha256(
            .encode()
        ).hexdigest()
        ._prev_hash = log_entry[]
        .logger.info(json.dumps(log_entry))

     ():
        .log_event(
            event_type=,
            user=user,
            resource=,
            action=action,
            result=  success  ,
            metadata={: mfa},
            source_ip=source_ip,
        )

     ():
        .log_event(
            event_type=,
            user=user,
            resource=resource,
            action=operation,
            result=,
            metadata={: record_count},
            source_ip=source_ip,
        )


 ():
    
     ():

         ():
            user = kwargs.get(, )
            :
                result = func(*args, **kwargs)
                audit_logger.log_event(
                    event_type=,
                    user=user,
                    resource=resource_name,
                    action=func.__name__,
                    result=,
                )
                 result
             Exception  e:
                audit_logger.log_event(
                    event_type=,
                    user=user,
                    resource=resource_name,
                    action=func.__name__,
                    result=,
                    metadata={: (e)},
                )
                
         wrapper
     decorator

Fluentd / Fluent Bit Log Aggregation

# fluent-bit.conf - lightweight agent on each node
[SERVICE]
    Flush         5
    Daemon        Off
    Log_Level     info
    Parsers_File  parsers.conf

[INPUT]
    Name          tail
    Path          /var/log/app/audit.log
    Parser        json
    Tag           audit.app
    Refresh_Interval 5
    Rotate_Wait   30

[INPUT]
    Name          systemd
    Tag           audit.system
    Systemd_Filter _TRANSPORT=audit

[FILTER]
    Name          modify
    Match         audit.*
    Add           cluster ${CLUSTER_NAME}
    Add           node ${NODE_NAME}

[OUTPUT]
    Name          es
    Match         audit.*
    Host          elasticsearch.internal.example.com
    Port          9200
    Index         audit-logs
              
               
        
       

[]
              
             
            
            
     
      
      
        

Elasticsearch Index Lifecycle for Retention

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "1d"
          },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
              
              
        
      
       
         
         
           
              
        
      
       
         
            
      
    
  

Retention Policy by Compliance Framework

retention_requirements:
  soc2:
    minimum: 1 year
    recommended: 3 years
    notes: "Based on audit period and report requirements"

  hipaa:
    minimum: 6 years
    notes: "From date of creation or last effective date"

  pci_dss:
    minimum: 1 year
    immediately_available: 3 months
    notes: "Req 10.7 - retain for at least one year, 3 months immediately available"

  gdpr:
    minimum: "As long as necessary for processing purpose"
    notes: "Apply data minimization; delete when no longer needed"

  fedramp:
    minimum: 3 years
    notes: "AU-11 control requirement"

  iso27001:
    minimum: "Defined by organization policy"
    recommended: 3 years
    notes: "A.12.4.1 - retention period must be defined"

Log Integrity Verification Script

#!/usr/bin/env bash
# verify-log-integrity.sh - Verify log file checksums against stored hashes

LOG_DIR="/var/log/app"
HASH_FILE="/var/log/app/.checksums"
ALERT_WEBHOOK="${ALERT_WEBHOOK_URL}"

verify_logs() {
  local failures=0
  while IFS='  ' read -r stored_hash filename; do
    if [ -f "$filename" ]; then
      current_hash=$(sha256sum "$filename" | awk '{print $1}')
      if [ "$stored_hash" != "$current_hash" ]; then
        echo "TAMPER DETECTED: $filename"
        failures=$((failures + 1))
        curl -s -X POST "$ALERT_WEBHOOK" \
          -H "Content-Type: application/json" \
          -d "{\"text\":\"ALERT: Audit log tamper detected on $(hostname): $filename\"}"
      fi
    else
      echo "MISSING: $filename"
      failures=$((failures + 1))
    fi
  done < "$HASH_FILE"

  return $failures
}

() {
  find  -name  - f -  {} \; > 
   440 
}

  
  verify)  verify_logs ;;
  update)  update_checksums ;;
  *)         ;;

SIEM Integration Checklist

siem_integration:
  log_sources:
    - [ ] Operating system auth logs (syslog, journald)
    - [ ] Application audit logs (structured JSON)
    - [ ] Cloud provider audit trails (CloudTrail, Activity Log, Audit Logs)
    - [ ] Database query and access logs
    - [ ] Network flow logs and firewall logs
    - [ ] Container and orchestrator logs (Kubernetes audit)
    - [ ] WAF and CDN access logs
    - [ ] VPN and remote access logs

  normalization:
     [ ]       
     [ ]       
     [ ]    
     [ ]   

  
     [ ]      
     [ ]      
     [ ]   
     [ ]    
     [ ]     
     [ ]    
     [ ]     

  
     [ ]    
     [ ]   
     [ ]    
     [ ]     
     [ ]       

Best Practices

• Use structured logging (JSON) with consistent field names across all services
• Ship logs to a centralized platform with write-once storage for tamper protection
• Implement hash chaining or digital signatures for log integrity verification
• Define and enforce retention policies per compliance framework requirements
• Set up real-time alerting for high-severity security events
• Separate audit logs from application debug logs to reduce noise
• Never log sensitive data (passwords, tokens, PII) in audit entries
• Monitor the logging pipeline itself to detect gaps in coverage
• Regularly test log restoration from archives to verify recoverability
• Rotate and compress logs to manage storage while meeting retention windows