ebpf

name	ebpf
description	eBPF skill for Linux observability and networking. Use when writing eBPF programs with libbpf or bpftrace, attaching kprobes/tracepoints/XDP hooks, debugging verifier errors, working with eBPF maps, or achieving CO-RE portability across kernel versions. Activates on queries about eBPF, bpftool, bpftrace, XDP programs, libbpf, verifier errors, eBPF maps, or kernel tracing with BPF.

eBPF

Purpose

Guide agents through writing, loading, and debugging eBPF programs using libbpf, bpftrace, and bpftool. Covers map types, program types, verifier errors, XDP networking, and CO-RE portability.

Triggers

• "How do I write an eBPF program to trace system calls?"
• "My eBPF program fails with a verifier error"
• "How do I use bpftrace to trace kernel events?"
• "How do I share data between kernel eBPF and userspace?"
• "How do I write an XDP program for packet filtering?"
• "How do I make my eBPF program portable across kernel versions (CO-RE)?"

Workflow

1. Choose the right tool

Goal?
├── One-liner kernel tracing / scripting → bpftrace
├── Production eBPF program with userspace → libbpf (C) or aya (Rust)
├── Inspect loaded programs and maps → bpftool
└── High-performance packet processing → XDP + libbpf

2. bpftrace — quick kernel tracing

# Trace all execve calls with comm and args
bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("%s %s\n", comm, str(args->filename)); }'

# Count syscalls by process
bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'

# Latency histogram for read() syscall
bpftrace -e '
  tracepoint:syscalls:sys_enter_read { @start[tid] = nsecs; }
  tracepoint:syscalls:sys_exit_read  { @us = hist((nsecs - @start[tid]) / 1000); delete(@start[tid]); }'

# List available tracepoints
bpftrace -l 'tracepoint:syscalls:*'
bpftrace -l 'kprobe:tcp_*'

3. libbpf skeleton — minimal C program

// counter.bpf.c — kernel-side
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __type(key, u32);
    __type(value, u64);
    __uint(max_entries, 1024);
} call_count SEC(".maps");

SEC("tracepoint/syscalls/sys_enter_read")
int trace_read(struct trace_event_raw_sys_enter *ctx)
{
    u32 pid = bpf_get_current_pid_tgid() >> 32;
    u64 *cnt = bpf_map_lookup_elem(&call_count, &pid);
    if (cnt)
        (*cnt)++;
    else {
        u64 one = 1;
        bpf_map_update_elem(&call_count, &pid, &one, BPF_ANY);
    }
    return 0;
}

char LICENSE[] SEC("license") = "GPL";

// counter.c — userspace loader
#include "counter.skel.h"

int main(void) {
    struct counter_bpf *skel = counter_bpf__open();
    if (!skel || counter_bpf__load(skel))
        return 1;
    counter_bpf__attach(skel);
    // read map, print results
    counter_bpf__destroy(skel);
}

# Build with libbpf 1.x
clang -g -O2 -target bpf -D__TARGET_ARCH_x86 -I/usr/include/bpf \
      -c counter.bpf.c -o counter.bpf.o
bpftool gen skeleton counter.bpf.o > counter.skel.h
gcc -o counter counter.c -lbpf -lelf -lz

libbpf 1.x API changes:

// Open and load (replaces older bpf_object__open/load split patterns)
struct counter_bpf *skel = counter_bpf__open();
counter_bpf__load(skel);
counter_bpf__attach(skel);

// Or explicit file open
struct bpf_object *obj = bpf_object__open_file("counter.bpf.o", NULL);
bpf_object__load(obj);

// Skeleton generation (always via bpftool)
// bpftool gen skeleton counter.bpf.o name counter > counter.skel.h

4. eBPF map types

Map type	Key→Value	Use case
BPF_MAP_TYPE_HASH	arbitrary→arbitrary	Per-PID counters, state
BPF_MAP_TYPE_ARRAY	u32→fixed	Config, metrics indexed by CPU
BPF_MAP_TYPE_PERCPU_HASH	key→per-CPU val	High-frequency counters without locks
BPF_MAP_TYPE_RINGBUF	—	Efficient kernel→userspace events
BPF_MAP_TYPE_PERF_EVENT_ARRAY	—	Legacy perf event output
BPF_MAP_TYPE_LRU_HASH	key→val	Connection tracking, limited size
BPF_MAP_TYPE_PROG_ARRAY	u32→prog	Tail calls, program chaining
BPF_MAP_TYPE_XSKMAP	—	AF_XDP socket redirection

Use BPF_MAP_TYPE_RINGBUF over PERF_EVENT_ARRAY for new code — lower overhead, variable-size records.

5. Verifier error triage

Error message	Root cause	Fix
invalid mem access 'scalar'	Dereferencing unbounded pointer	Check pointer with null test before use
R0 !read_ok	Return without setting R0	Ensure all paths set a return value
jump out of range	Branch target beyond program end	Restructure conditionals
back-edge detected	Backward jump (loop)	Use bpf_loop() helper (kernel ≥5.17) or bounded loop
unreachable insn	Dead code after return	Remove dead branches
invalid indirect read	Stack read of uninitialised bytes	Zero-init structs: struct foo x = {}
misaligned stack access	Pointer arithmetic off alignment	Align reads to __u64 boundaries

# Get detailed verifier log
bpftool prog load prog.bpf.o /sys/fs/bpf/prog type kprobe \
    2>&1 | head -100

# Check loaded programs
bpftool prog list
bpftool prog dump xlated id 42

6. XDP programs

// xdp_drop_icmp.bpf.c
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_endian.h>

SEC("xdp")
int xdp_filter(struct xdp_md *ctx)
{
    void *data_end = (void *)(long)ctx->data_end;
    void *data     = (void *)(long)ctx->data;
    struct ethhdr *eth = data;

    if ((void *)(eth + 1) > data_end)
        return XDP_PASS;

    if (bpf_ntohs(eth->h_proto) != ETH_P_IP)
        return XDP_PASS;

    struct iphdr *ip = (void *)(eth + 1);
    if ((void *)(ip + 1) > data_end)
        return XDP_PASS;

    if (ip->protocol == IPPROTO_ICMP)
        return XDP_DROP;

    return XDP_PASS;
}
char LICENSE[] SEC("license") = "GPL";

# Attach XDP program to interface
ip link set dev eth0 xdp obj xdp_drop_icmp.bpf.o sec xdp
# Remove
ip link set dev eth0 xdp off
# Use native (driver) mode for best performance
ip link set dev eth0 xdp obj prog.bpf.o sec xdp mode native

XDP return codes: XDP_PASS, XDP_DROP, XDP_TX (hairpin), XDP_REDIRECT.

7. CO-RE — compile once, run everywhere

CO-RE (Compile Once - Run Everywhere) uses BTF type info to relocate field accesses at load time.

// Use BTF-based field access (CO-RE aware)
#include <vmlinux.h>        // generated from running kernel's BTF
#include <bpf/bpf_core_read.h>

SEC("kprobe/tcp_connect")
int trace_connect(struct pt_regs *ctx)
{
    struct sock *sk = (struct sock *)PT_REGS_PARM1(ctx);
    u16 dport = BPF_CORE_READ(sk, __sk_common.skc_dport);
    // BPF_CORE_READ relocates the field offset at load time
    bpf_printk("connect to port %d\n", bpf_ntohs(dport));
    return 0;
}

# Generate vmlinux.h from running kernel
bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

# Verify BTF is enabled
ls /sys/kernel/btf/vmlinux

8. BPF ring buffer vs perf buffer

// Ring buffer (preferred for new programs — better perf, no per-CPU loss)
struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} rb SEC(".maps");

SEC("kprobe/sys_open")
int handle_open(struct pt_regs *ctx)
{
    struct event *e = bpf_ringbuf_reserve(&rb, sizeof(*e), 0);
    if (!e)
        return 0;
    e->pid = bpf_get_current_pid_tgid() >> 32;
    bpf_ringbuf_submit(e, 0);
    // bpf_ringbuf_discard(e, 0) on error paths
    return 0;
}

Feature	Ring buffer	Perf buffer
API	bpf_ringbuf_reserve/submit	bpf_perf_event_output
Backpressure	Reserve fails if full	May drop events
Userspace	ring_buffer__poll (libbpf)	perf_buffer__poll
Multi-producer	Yes	Per-CPU buffers

9. BPF iterators

struct {
    __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
    __uint(max_entries, 1);
} iter_prog SEC(".maps");

SEC("iter/task")
int dump_tasks(struct bpf_iter__task *ctx)
{
    struct task_struct *task = ctx->task;
    if (task)
        bpf_seq_printf(ctx->meta->seq, "%d %s\n", task->tgid, task->comm);
    return 0;
}

# Read iterator output from userspace
bpftool prog tracelog   # or attach iter to seq_file reader
cat /sys/kernel/debug/tracing/trace_pipe

Iterators walk kernel data structures (tasks, maps, TCP sockets) without kprobe overhead per element.

10. BPF atomics

// GCC/Clang atomic builtins in BPF programs (kernel 5.12+)
static __always_inline void inc_counter(__u32 *counter)
{
    __sync_fetch_and_add(counter, 1);
}

// Use for per-CPU or map-backed counters under concurrent probes

Prefer per-CPU array maps for high-frequency counters; use atomics when aggregating into a single map value.

For the full map types reference, see references/ebpf-map-types.md.

Related skills

• Use skills/observability/ebpf-rust for Aya framework Rust eBPF programs
• Use skills/profilers/linux-perf for perf-based tracing without eBPF
• Use skills/runtimes/binary-hardening for seccomp-bpf syscall filtering
• Use skills/low-level-programming/linux-kernel-modules for kernel module development
• Use skills/async-io/af-xdp for XDP_REDIRECT to AF_XDP sockets