Use eBPF for deep kernel-level observability — trace syscalls, network flows, and application behavior without code changes using Cilium, Tetragon, and bpftrace.
Use eBPF for deep kernel-level observability — trace syscalls, network flows, and application behavior without code changes using Cilium, Tetragon, and bpftrace.
license
MIT
metadata
{"author":"devops-skills","version":"1.0"}
eBPF Observability
eBPF (extended Berkeley Packet Filter) allows you to run sandboxed programs in the Linux kernel without modifying kernel source code or loading kernel modules. This skill covers using eBPF for deep observability, network monitoring, and security enforcement across cloud-native infrastructure.
1. When to Use
Use eBPF-based observability when you need:
Deep performance debugging -- trace kernel-level latency, syscall overhead, and scheduling delays that application-level metrics cannot reveal.
Network observability without sidecars -- capture L3/L4/L7 flows, DNS queries, and TCP state transitions directly from the kernel, eliminating the CPU and memory overhead of sidecar proxies.
Security monitoring at the kernel boundary -- detect container escapes, unexpected process execution, sensitive file access, and anomalous syscall patterns in real time.
Continuous profiling in production -- generate CPU flame graphs and memory allocation profiles with negligible overhead (typically under 1% CPU).
Service mesh replacement or augmentation -- Cilium can replace kube-proxy and provide identity-aware network policies enforced at the kernel level.
Avoid eBPF when your kernel version is below 4.19, when you are running on managed platforms that restrict BPF capabilities, or when your debugging needs are fully met by application-level tracing.
2. Prerequisites
Kernel Version Requirements
Feature
Minimum Kernel
Recommended Kernel
Basic BPF maps & probes
4.9
5.10+
BPF CO-RE (BTF support)
5.2
5.10+
BPF ring buffer
5.8
5.10+
BPF LSM hooks
5.7
5.15+
Cilium full features
4.19
5.10+
Tetragon
4.19
5.13+
Verify Kernel Support
# Check kernel versionuname -r
# Verify BTF (BPF Type Format) is enabled -- required for CO-REls /sys/kernel/btf/vmlinux
# Check BPF filesystem is mounted
mount | grep bpf
# If not mounted, mount itsudo mount -t bpf bpf /sys/fs/bpf
# Verify BPF JIT is enabledcat /proc/sys/net/core/bpf_jit_enable
# Should return 1; if not:sudo sysctl net.core.bpf_jit_enable=1
# Port-forward the Hubble Relay
cilium hubble port-forward &
# Observe all flows in real time
hubble observe --follow
# Filter flows by namespace
hubble observe --namespace production --follow
# Filter by verdict (dropped traffic)
hubble observe --verdict DROPPED --follow
# Filter by DNS queries
hubble observe --protocol DNS --follow
# Filter HTTP traffic to a specific service
hubble observe --to-label "app=api-server" --protocol HTTP --follow
# Export flows as JSON for ingestion into SIEM
hubble observe --output json --last 1000 > flows.json
Hubble UI Access
# Port-forward the Hubble UI
kubectl port-forward -n kube-system svc/hubble-ui 12000:80
# Access at http://localhost:12000 -- provides a real-time service dependency map
4. Tetragon for Security
Tetragon is Cilium's runtime security enforcement engine. It uses eBPF to observe and enforce security policies at the kernel level with zero application changes.
# process-monitor.yaml -- TracingPolicy to monitor all process executionsapiVersion:cilium.io/v1alpha1kind:TracingPolicymetadata:name:process-execution-monitorspec:kprobes: []
tracepoints: []
uprobes: []
enforcers: []
# process_exec and process_exit events are always emitted by default# Use tetra CLI to observe them:
# Watch all process executions cluster-wide
kubectl exec -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact --process-exec
# Filter to a specific namespace
kubectl exec -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact \
--namespace production
File Access Tracking
# file-access-policy.yaml -- detect reads/writes to sensitive filesapiVersion:cilium.io/v1alpha1kind:TracingPolicymetadata:name:sensitive-file-accessspec:kprobes:-call:"security_file_open"syscall:falseargs:-index:0type:"file"selectors:-matchArgs:-index:0operator:"Prefix"values:-"/etc/shadow"-"/etc/passwd"-"/etc/kubernetes/pki"-"/var/run/secrets/kubernetes.io"-"/root/.ssh"
Example ebpf_exporter config for tracking OOM kills and run queue latency:
# ebpf-exporter-config.yamlprograms:-name:oom_killsmetrics:counters:-name:oom_kill_totalhelp:"Total number of OOM kills"labels:-name:cgroupsize:128decoders:-name:stringkprobes:oom_kill_process:count_oom-name:runqlatmetrics:histograms:-name:run_queue_latency_secondshelp:"Run queue latency histogram in seconds"bucket_type:exp2bucket_min:0bucket_max:26bucket_multiplier:0.000000001tracepoints:sched:sched_wakeup:trace_wakeupsched:sched_switch:trace_switch
Grafana Dashboard
Import these community dashboards for eBPF metrics:
# Hubble dashboard -- Grafana dashboard ID 16611# Cilium Agent dashboard -- Grafana dashboard ID 16612# Cilium Operator dashboard -- Grafana dashboard ID 16613# Or create a ConfigMap for automatic provisioning
kubectl create configmap grafana-cilium-dashboard \
--from-file=cilium-dashboard.json \
--namespace monitoring \
-o yaml --dry-run=client | \
kubectl label --local -f - grafana_dashboard=1 -o yaml | \
kubectl apply -f -
Key Prometheus queries for eBPF-sourced metrics:
# Dropped packets rate by reason
rate(hubble_drop_total[5m])
# DNS error rate by query type
sum(rate(hubble_dns_responses_total{rcode!="No Error"}[5m])) by (rcode, qtypes)
# HTTP request latency (p99) from Hubble L7 visibility
histogram_quantile(0.99, sum(rate(hubble_http_request_duration_seconds_bucket[5m])) by (le, destination))
# TCP retransmit rate from eBPF exporter
rate(tcp_retransmits_total[5m])
# Run queue latency p99
histogram_quantile(0.99, sum(rate(run_queue_latency_seconds_bucket[5m])) by (le))
7. Network Observability
L3/L4 Flow Logging
# Log all TCP connections with Hubble
hubble observe --type l3/l4 --protocol TCP --follow
# Filter SYN packets only (new connections)
hubble observe --type trace:to-endpoint --tcp-flags SYN --follow
# Export flows to a file for batch analysis
hubble observe --output json --since 1h > network-flows.json
# Count flows by destination service over the last hour
hubble observe --output json --since 1h | \
jq -r '.destination.labels[] | select(startswith("k8s:app="))' | \
sort | uniq -c | sort -rn | head -20
L7 Protocol Visibility
Enable L7 visibility with Cilium annotations on target pods:
# Annotate a namespace for HTTP visibilityapiVersion:v1kind:Namespacemetadata:name:productionannotations:policy.cilium.io/proxy-visibility:"<Egress/53/UDP/DNS>,<Ingress/80/TCP/HTTP>,<Ingress/443/TCP/HTTP>"
# Observe L7 HTTP flows
hubble observe --type l7 --protocol HTTP --follow
# Filter by HTTP status code (5xx errors)
hubble observe --type l7 --http-status "500+" --follow
# Filter by HTTP method and path
hubble observe --type l7 --http-method GET --http-path "/api/v1/.*" --follow
DNS Monitoring
# All DNS queries and responses
hubble observe --type l7 --protocol DNS --follow
# DNS queries that returned NXDOMAIN
hubble observe --type l7 --protocol DNS --dns-rcode NXDOMAIN --follow
# DNS latency analysis with bpftracesudo bpftrace -e 'kprobe:dns_resolve { @start[tid] = nsecs; }
kretprobe:dns_resolve /@start[tid]/ {
@dns_latency_us = hist((nsecs - @start[tid]) / 1000);
delete(@start[tid]);
}'
Service Dependency Map Generation
Hubble UI automatically generates service maps. For programmatic access:
# Get a service map via Hubble Relay API
hubble observe --output json --since 24h | \
jq '{src: .source.labels, dst: .destination.labels, verdict: .verdict}' | \
jq -s 'group_by(.src, .dst) | map({
source: .[0].src,
destination: .[0].dst,
flow_count: length,
verdicts: [.[].verdict] | group_by(.) | map({(.[0]): length}) | add
})' > service-map.json
# Get verbose verifier outputsudo bpftrace -d -e 'your_program_here' 2>&1 | tail -50
# Common causes:# - Unbounded loops (BPF requires bounded loops or unrolled iterations)# - Stack size exceeds 512 bytes# - Accessing memory without null checks# - Back-edges in control flow (pre-5.3 kernels)
BTF not available:
# Check if BTF is compiled into the kernelcat /boot/config-$(uname -r) | grep CONFIG_DEBUG_INFO_BTF
# If not, install BTF data from btfhub# https://github.com/aquasecurity/btfhub
wget "https://github.com/aquasecurity/btfhub-archive/raw/main/ubuntu/22.04/x86_64/$(uname -r).btf.tar.xz"
tar xvf "$(uname -r).btf.tar.xz"
Permission denied:
# BPF requires CAP_BPF (or CAP_SYS_ADMIN on older kernels)# For containers, add to securityContext:# securityContext:# capabilities:# add: ["BPF", "PERFMON", "SYS_RESOURCE"]# Check current capabilitiescat /proc/self/status | grep Cap
capsh --decode=$(cat /proc/self/status | grep CapEff | awk '{print $2}')
Cilium pods not starting:
# Check Cilium agent logs
kubectl logs -n kube-system -l k8s-app=cilium --tail=100
# Verify BPF filesystem
kubectl exec -n kube-system ds/cilium -- mount | grep bpf
# Check for conflicting CNIsls /etc/cni/net.d/
# Run Cilium connectivity test
cilium connectivity test
Tetragon events missing:
# Verify TracingPolicy is loaded
kubectl get tracingpolicies
# Check Tetragon agent logs for verifier errors
kubectl logs -n kube-system ds/tetragon -c tetragon --tail=200 | grep -i error
# Verify the kprobe is attached
kubectl exec -n kube-system ds/tetragon -c tetragon -- \
cat /sys/kernel/debug/kprobes/list | grep your_function
High overhead from eBPF programs:
# List all loaded BPF programs and their run timesudo bpftool prog show
sudo bpftool prog profile id <PROG_ID> duration 5
# Check map memory usagesudo bpftool map show
sudo bpftool map dump id <MAP_ID> | wc -l
# If a program is consuming too much CPU, check its run count and timesudo bpftool prog show id <PROG_ID> --json | jq '{run_cnt, run_time_ns}'# Detach a misbehaving programsudo bpftool prog detach id <PROG_ID> type <ATTACH_TYPE>
Kernel Compatibility Matrix
# Quick check: which eBPF features your kernel supportssudo bpftool feature probe kernel
# Check specific program typessudo bpftool feature probe kernel | grep program_type
# Check available map typessudo bpftool feature probe kernel | grep map_type
# Check available helper functionssudo bpftool feature probe kernel | grep helper