Detect unauthorized modifications to running containers by monitoring for binary execution drift, file system changes, and configuration deviations from the original container image. Use when detecting unauthorized modifications to running containers by monitoring for binary.
Detect unauthorized modifications to running containers by monitoring for binary execution drift, file system changes, and configuration deviations from the original container image. Use when detecting unauthorized modifications to running containers by monitoring for binary.
Container drift occurs when running containers deviate from their original image state through unauthorized file modifications, unexpected binary execution, configuration changes, or package installations. Since containers should be treated as immutable infrastructure, any drift is a potential indicator of compromise. Detection techniques leverage the DIE (Detect, Isolate, Evict) model -- an immutable workload should not change during runtime, so any observed change is potentially evidence of malicious activity.
When to Use
Trigger phrases:
"detecting container drift at runtime"
"Detect unauthorized modifications to running containers by monitoring for binary"
When investigating security incidents that require detecting container drift at runtime
When building detection rules or threat hunting queries for this domain
When SOC analysts need structured procedures for this analysis type
When validating security monitoring coverage for related attack techniques
Prerequisites
Kubernetes cluster v1.24+ with runtime security tooling
Falco or Sysdig for runtime drift detection
Container image registry with image manifests available
Familiarity with Linux filesystem layers and OverlayFS
Core Concepts
This section covers core concepts for detecting container drift at runtime.
Ensure all prerequisites are met before proceeding
Follow the documented workflow steps in sequence
Record results and any anomalies encountered during this phase
Types of Container Drift
Binary drift: Execution of binaries not present in the original image (downloaded malware, compiled tools)
File drift: Creation, modification, or deletion of files in the container filesystem
Configuration drift: Changes to environment variables, mounted secrets, or runtime parameters
Package drift: Installation of new packages via apt, yum, pip, or npm at runtime
Network drift: New listening ports or outbound connections not expected for the workload
Detection Methods
Image-Based Comparison: Compare the running container's filesystem against its source image to identify added, modified, or removed files.
Behavioral Monitoring: Use eBPF or kernel-level monitoring to detect process execution, file access, and network activity that deviates from expected behavior.
Digest Verification: Continuously verify that running container image digests match the approved deployment manifests.
Implementation with Falco
This section covers implementation with falco for detecting container drift at runtime.
Ensure all prerequisites are met before proceeding
Follow the documented workflow steps in sequence
Record results and any anomalies encountered during this phase
Detecting New Binary Execution
-rule:DriftDetected(ContainerImageModifiedBinary)desc:Detectexecutionofabinarynotpresentintheoriginalcontainerimagecondition:>
spawned_process and
container and
not proc.pname in (container_entrypoint) and
proc.is_exe_upper_layer = true
output:>
Drift detected: new binary executed in container
(user=%user.name command=%proc.cmdline container=%container.name
image=%container.image.repository:%container.image.tag
exe_path=%proc.exepath)
priority:WARNINGtags: [container, drift]
-rule:ContainerShellSpawneddesc:Detectinteractiveshellinacontainerthatshouldbeimmutablecondition:>
spawned_process and
container and
proc.name in (bash, sh, dash, zsh, csh, ksh) and
not proc.pname in (container_entrypoint)
output:>
Shell spawned in container (user=%user.name shell=%proc.name
container=%container.name image=%container.image.repository)
priority:WARNINGtags: [container, drift, shell]
Detecting Package Manager Usage
-rule:PackageManagerExecutioninContainerdesc:Detectuseofpackagemanagersindicatingdriftcondition:>
spawned_process and
container and
proc.name in (apt, apt-get, yum, dnf, apk, pip, pip3, npm, gem, cargo)
output:>
Package manager executed in container (user=%user.name
command=%proc.cmdline container=%container.name
image=%container.image.repository)
priority:ERRORtags: [container, drift, package-manager]
Detecting File System Modifications
-rule:ContainerFileSystemWritedesc:Detectwritestocontainerupperlayerfilesystemcondition:>
open_write and
container and
fd.typechar = 'f' and
not fd.name startswith /tmp and
not fd.name startswith /var/log and
not fd.name startswith /proc
output:>
File write in container (user=%user.name file=%fd.name
container=%container.name)
priority:NOTICEtags: [container, drift, filesystem]
Implementation with Kubernetes Enforcement
This section covers implementation with kubernetes enforcement for detecting container drift at runtime.
Ensure all prerequisites are met before proceeding
Follow the documented workflow steps in sequence
Record results and any anomalies encountered during this phase
Read-Only Root Filesystem
Prevent drift by making container filesystems immutable:
This section covers image digest verification for detecting container drift at runtime.
Ensure all prerequisites are met before proceeding
Follow the documented workflow steps in sequence
Record results and any anomalies encountered during this phase
Continuous Digest Monitoring
#!/bin/bash# Compare running container digests against approved manifest
NAMESPACE="production"
kubectl get pods -n "$NAMESPACE" -o json | jq -r '
.items[] |
.spec.containers[] |
"\(.image) \(.imageID)"
' | whileread IMAGE IMAGE_ID; do
APPROVED_DIGEST=$(kubectl get deploy -n "$NAMESPACE" -o json | \
jq -r ".items[].spec.template.spec.containers[] | select(.image==\"$IMAGE\") | .image")
if [[ "$IMAGE" != *"@sha256:"* ]]; thenecho"[WARN] Container using mutable tag: $IMAGE"fidone
Microsoft Defender for Containers Integration
For Azure Kubernetes environments, Microsoft Defender provides built-in binary drift detection:
{"alertType":"K8S.NODE_ImageBinaryDrift","severity":"Medium","description":"Binary executed that was not part of the original container image","remediationSteps":["Investigate the binary origin and purpose","Check if the container was compromised","Rebuild the container from a clean image","Enable readOnlyRootFilesystem"]}
Drift Response Playbook
Detect: Alert fires on drift event (Falco, Defender, Sysdig)
Validate: Confirm the drift is not from an approved process (init containers, config reloads)
Isolate: Apply a deny-all NetworkPolicy to the affected pod
Investigate: Capture container filesystem diff and process list
Evict: Delete the drifted pod (ReplicaSet will recreate from clean image)
Remediate: Fix the root cause (patch vulnerability, update image, tighten RBAC)
When NOT to Use
You need to perform the attack to test detection (use performing-* skills)
Task is about analyzing past incidents (use analyzing-* skills)
You need to implement detection rules (use implementing-* skills)
Task is about threat hunting proactively (use hunting-* skills)
You don't have access to logs or monitoring data
Task requires incident response (use IR skills)
Red Flags
Performing actions without explicit written authorization from the asset owner
Testing against production systems without a defined scope and rules of engagement
Modifying cloud IAM policies or security groups without approval
Exposing cloud credentials or secrets in logs or reports
Running scans that generate excessive API calls and trigger billing alerts
Verification
All steps executed successfully against a test environment before production use
Output documented with screenshots or logs demonstrating expected behavior
Cloud resource changes reverted or documented as intentional
IAM policies reviewed for least-privilege compliance after testing
No residual test resources left running (cost and security check)
