eBPF-Based Observability
eBPF (Extended Berkeley Packet Filter) lets you run sandboxed programs directly in the Linux kernel — without modifying kernel source code or loading kernel modules. For observability, this means zero-instrumentation visibility into system calls, network flows, CPU scheduling, and application internals.
+------------------------------------------------------------+
| Traditional Observability vs eBPF |
| |
| Traditional: |
| App Code --> SDK (OTel) --> Agent --> Backend |
| Requires: code change, redeploy, SDK maintenance |
| |
| eBPF: |
| Linux Kernel |
| +--------------------------------------------------+ |
| | syscalls | scheduler | network stack | filesystem | |
| | ^ ^ ^ ^ | |
| | [eBPF] [eBPF] [eBPF] [eBPF] | |
| +--------------------------------------------------+ |
| | | | | |
| profiles scheduling flows file I/O |
| |
| Requires: no code change, no redeploy |
+------------------------------------------------------------+
What eBPF Is
eBPF programs are loaded into the kernel at runtime and verified for safety before execution. The verifier guarantees:
- No infinite loops (bounded loop iterations only)
- No invalid memory access (bounds-checked)
- No crashing the kernel (programs are sandboxed)
- Programs terminate (proven by the verifier)
+------------------------------------------+
| User Space |
| bpftrace / BCC / Cilium / Pixie agent |
| - Write eBPF program (C / BPF bytecode) |
| - Load via bpf() syscall |
+------------------------------------------+
|
v bpf() syscall
+------------------------------------------+
| Linux Kernel |
| eBPF Verifier |
| - Checks safety (no loops, no OOB) |
| - Rejects unsafe programs |
| | |
| v (if safe) |
| JIT Compiler --> native machine code |
| | |
| v |
| Attach to hook point: |
| - kprobe (any kernel function) |
| - tracepoint (stable kernel events) |
| - uprobe (user-space function) |
| - XDP (network packet, before kernel) |
| - tc (traffic control) |
| - LSM (security hooks) |
+------------------------------------------+
|
v BPF Maps (shared memory)
+------------------------------------------+
| User Space reads results |
| from BPF Maps (ring buffer, hash map) |
+------------------------------------------+
eBPF Hook Types
| Hook | Stability | Use Case |
|---|---|---|
| Tracepoint | Stable (kernel API) | syscalls, scheduler, block I/O |
| Kprobe / kretprobe | Unstable (changes across kernel versions) | any kernel function entry/exit |
| Uprobe | Stable per binary | user-space function entry/exit |
| USDT | Stable (defined in app) | Java, Python, Node.js application events |
| XDP | Stable | high-performance packet filtering at NIC level |
| tc (traffic control) | Stable | packet shaping, filtering after NIC |
bpftrace — One-Liner Tracing
bpftrace is an awk-like language for writing eBPF programs. Best for ad-hoc production debugging.
# Requires: Linux 4.9+, bpftrace installed
# Install
apt-get install bpftrace # Ubuntu/Debian
dnf install bpftrace # Fedora/RHEL
# Show all open() syscalls with process name and file
bpftrace -e '
tracepoint:syscalls:sys_enter_openat {
printf("%-16s %s\n", comm, str(args->filename));
}'
# Count syscalls by process name (top-like view)
bpftrace -e '
tracepoint:raw_syscalls:sys_enter {
@[comm] = count();
}
interval:s:5 {
print(@);
clear(@);
}'
# Trace all HTTP GET/POST at write() syscall level
bpftrace -e '
tracepoint:syscalls:sys_enter_write {
if (strncmp("GET ", str(args->buf), 4) == 0 ||
strncmp("POST", str(args->buf), 4) == 0) {
printf("%-16s fd=%-4d: %s\n", comm, args->fd, str(args->buf, 80));
}
}'
# Histogram of read() latency in microseconds
bpftrace -e '
tracepoint:syscalls:sys_enter_read { @start[tid] = nsecs; }
tracepoint:syscalls:sys_exit_read {
if (@start[tid]) {
@latency_us = hist((nsecs - @start[tid]) / 1000);
delete(@start[tid]);
}
}'
# Count TCP connections by destination port
bpftrace -e '
kprobe:tcp_connect {
$sk = (struct sock *)arg0;
@[($sk->__sk_common.skc_dport >> 8) |
(($sk->__sk_common.skc_dport & 0xff) << 8)] = count();
}'
# Trace JVM GC pause time via USDT (Java with -XX:+ExtendedDTraceProbes)
bpftrace -e '
usdt:/usr/lib/jvm/java-21/lib/server/libjvm.so:hotspot:gc__begin {
@gc_start[tid] = nsecs;
}
usdt:/usr/lib/jvm/java-21/lib/server/libjvm.so:hotspot:gc__end {
if (@gc_start[tid]) {
@gc_pause_ms = hist((nsecs - @gc_start[tid]) / 1000000);
delete(@gc_start[tid]);
}
}'
Pre-built bpftrace Scripts
# List all available scripts
ls /usr/share/bpftrace/tools/
# Trace disk I/O latency
bpftrace /usr/share/bpftrace/tools/biolatency.bt
# Trace new process executions
bpftrace /usr/share/bpftrace/tools/execsnoop.bt
# Trace TCP retransmits
bpftrace /usr/share/bpftrace/tools/tcpretrans.bt
# CPU off-CPU time (thread blocking analysis)
bpftrace /usr/share/bpftrace/tools/offcputime.bt
BCC — Python-Based eBPF Tools
BCC (BPF Compiler Collection) provides pre-built production-ready eBPF tools and a Python API for custom programs.
# Install BCC tools
apt-get install bpfcc-tools linux-headers-$(uname -r)
# Core debugging tools
execsnoop-bpfcc # trace new process executions (find hidden child processes)
tcpretrans-bpfcc # TCP retransmits (network reliability issues)
biolatency-bpfcc # block I/O latency histogram
runqlat-bpfcc # CPU run queue latency (scheduler contention)
profile-bpfcc # CPU profiler with flame graph output
offcputime-bpfcc # off-CPU time analysis (blocking, I/O wait)
memleak-bpfcc # detect memory leaks in native code
tcpconnect-bpfcc # trace TCP connection attempts
tcpaccept-bpfcc # trace accepted TCP connections
# Examples
execsnoop-bpfcc # all new processes
tcpconnect-bpfcc -p $(pgrep java) # TCP connects from Java process
biolatency-bpfcc -m # block I/O latency in milliseconds
profile-bpfcc -f 30 > /tmp/cpu.stacks # 30s CPU profile for flame graph
Custom BCC Program (Python)
#!/usr/bin/env python3
"""Trace slow HTTP requests by hooking write() syscall."""
from bcc import BPF
import ctypes
program = """
#include <uapi/linux/ptrace.h>
#include <linux/sched.h>
struct event_t {
u32 pid;
u64 latency_ns;
char comm[TASK_COMM_LEN];
char data[128];
};
BPF_HASH(start_time, u32, u64);
BPF_PERF_OUTPUT(events);
TRACEPOINT_PROBE(syscalls, sys_enter_write) {
u32 tid = bpf_get_current_pid_tgid();
u64 ts = bpf_ktime_get_ns();
start_time.update(&tid, &ts);
return 0;
}
TRACEPOINT_PROBE(syscalls, sys_exit_write) {
u32 tid = bpf_get_current_pid_tgid();
u64 *ts = start_time.lookup(&tid);
if (!ts) return 0;
u64 latency = bpf_ktime_get_ns() - *ts;
start_time.delete(&tid);
// Only report calls slower than 1ms
if (latency < 1000000) return 0;
struct event_t event = {};
event.pid = bpf_get_current_pid_tgid() >> 32;
event.latency_ns = latency;
bpf_get_current_comm(&event.comm, sizeof(event.comm));
events.perf_submit(args, &event, sizeof(event));
return 0;
}
"""
b = BPF(text=program)
def handle_event(cpu, data, size):
event = b["events"].event(data)
print(f"pid={event.pid} comm={event.comm.decode()} "
f"latency={event.latency_ns/1e6:.2f}ms")
b["events"].open_perf_buffer(handle_event)
print("Tracing slow write() calls... Ctrl-C to stop")
while True:
b.perf_buffer_poll()
Pixie — Auto-Instrumented Kubernetes Observability
Pixie uses eBPF to automatically instrument every service in your Kubernetes cluster — no code changes, no sidecars.
+-----------------------------------------------+
| Kubernetes Cluster |
| |
| +------------------+ +------------------+ |
| | Service A | | Service B | |
| | (no OTel SDK) | | (no OTel SDK) | |
| +------------------+ +------------------+ |
| | HTTP/gRPC | |
| v v |
| +-------------------------------------+ |
| | Pixie Vizier (DaemonSet per node) | |
| | eBPF: captures all traffic | |
| | Stores in-cluster (no egress) | |
| +-------------------------------------+ |
| |
| Pixie UI / PxL queries |
| px/http_data -- HTTP request traces |
| px/mysql_data -- MySQL query traces |
| px/jvm -- JVM heap + GC metrics |
+-----------------------------------------------+
Install Pixie
# Install px CLI
bash -c "$(curl -fsSL https://withpixie.ai/install.sh)"
# Deploy Pixie into your cluster
px deploy
# Run a PxL script
px run px/http_data # all HTTP traffic for 30s
px run px/mysql_data # all MySQL queries
px run px/node_memory_usage # memory per node
px run px/jvm_stats # JVM heap, GC, threads
# Live UI
px live px/service_stats # per-service latency/error rate/RPS
PxL Query (SQL-like)
# PxL: find slowest HTTP endpoints in the last 5 minutes
import px
df = px.DataFrame(table='http_events', start_time='-5m')
df = df[df.resp_status >= 400] # only errors
df.latency_ms = df.latency / 1e6
df = df.groupby(['service', 'req_path']).agg(
p99_latency_ms=('latency_ms', px.quantiles(0.99)),
error_count=('resp_status', px.count),
)
df = df.sort(by='p99_latency_ms', ascending=False)
px.display(df, 'Slow/Erroring Endpoints')
What Pixie Auto-Detects (No Code Changes)
- HTTP/1.x, HTTP/2, gRPC (encrypted HTTPS via uprobes on OpenSSL)
- MySQL, PostgreSQL, Redis, Cassandra, MongoDB
- DNS queries and responses
- JVM metrics (heap, GC, class loading) via JVM USDT probes
- CPU profiles per service (flame graph)
Cilium and Hubble — eBPF Networking + Observability
Cilium replaces iptables with eBPF for Kubernetes networking. Hubble is the built-in observability layer.
+--------------------------------------------------+
| Cilium Architecture |
| |
| Pod A ---[eBPF datapath]---> Pod B |
| | |
| | Flow recorded |
| v |
| Hubble Agent (per node) |
| - HTTP method, path, status |
| - DNS query/response |
| - TCP connection open/close |
| - Policy verdict (allowed/denied) |
| | |
| v |
| Hubble Relay (cluster-wide aggregation) |
| | |
| v |
| Hubble UI / Grafana / Prometheus |
+--------------------------------------------------+
Install Cilium + Hubble
# Install Cilium CLI
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
curl -L --fail-with-body \
"https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz" \
| sudo tar xz -C /usr/local/bin
# Install Cilium (replace kube-proxy)
cilium install --version 1.15.0
# Enable Hubble
cilium hubble enable --ui
# Verify
cilium status --wait
# Port-forward Hubble UI
cilium hubble ui # opens browser at localhost:12000
Hubble CLI — Flow Inspection
# Install hubble CLI
export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
curl -L --fail-with-body \
"https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz" \
| sudo tar xz -C /usr/local/bin
# Port-forward Hubble relay
cilium hubble port-forward &
# Watch all flows
hubble observe
# Filter to a specific namespace
hubble observe --namespace production
# Filter HTTP flows only
hubble observe --protocol http
# Filter by source pod
hubble observe --from-pod production/frontend
# Watch for dropped packets (policy violations)
hubble observe --verdict DROPPED
# JSON output for further processing
hubble observe --output json | jq '.flow | {src: .source.pod_name, dst: .destination.pod_name, type: .l7.type}'
Hubble Metrics in Grafana
# Cilium values for Prometheus metrics
# cilium-values.yaml
hubble:
enabled: true
metrics:
enabled:
- dns:query;ignoreAAAA
- drop
- tcp
- flow
- port-distribution
- icmp
- http
serviceMonitor:
enabled: true # requires Prometheus Operator
mTLS Without Sidecars (Cilium 1.14+)
# CiliumNetworkPolicy with mutual auth (no Istio/Envoy needed)
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: require-mtls
namespace: production
spec:
endpointSelector:
matchLabels:
app: payment-service
ingress:
- fromEndpoints:
- matchLabels:
app: order-service
authentication:
mode: required # enforce mTLS between these services
Continuous Profiling with Pyroscope
Pyroscope collects CPU and memory profiles continuously from all processes using eBPF — no code change needed.
+--------------------------------------------------+
| Pyroscope Architecture |
| |
| Pyroscope Agent (eBPF, DaemonSet) |
| - Sample stack traces every 10ms (perf_events) |
| - Attribute to PID / container / K8s pod |
| - No app instrumentation needed |
| | |
| v |
| Pyroscope Server |
| - Aggregate flame graphs per service |
| - Store time-series profiles |
| - Label: service, pod, namespace |
| | |
| v |
| Grafana (Pyroscope datasource) |
| - Correlate profiles with traces (Tempo) |
| - "Why is p99 high?" --> drill to flame graph |
+--------------------------------------------------+
Deploy Pyroscope in Kubernetes
# Add Helm repo
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
# Install Pyroscope with eBPF profiling
helm install pyroscope grafana/pyroscope \
--namespace monitoring \
--set ebpf.enabled=true \
--set ebpf.spy=true
# pyroscope-values.yaml
pyroscope:
extraArgs:
- -scrape-configs.reload-period=30s
ebpf:
enabled: true
# Profile all processes including JVM, Python, Rust, Go
# No per-language agent needed for CPU profiles
# Grafana integration
grafana:
enabled: true
sidecar:
datasources:
enabled: true
Pyroscope SDK — Push Profiles (When You Need App Labels)
// Java: push profiles with custom labels (service name, version)
// build.gradle: implementation 'io.pyroscope:agent:0.13.0'
PyroscopeAgent.start(
new Config.Builder()
.setApplicationName("order-service")
.setProfilingEvent(EventType.ITIMER) // CPU
.setProfilingAlloc("512k") // allocation profiling
.setServerAddress("http://pyroscope:4040")
.setLabels(Map.of(
"region", System.getenv("AWS_REGION"),
"version", System.getenv("APP_VERSION")
))
.build()
);
// Go: SDK for labeled profiles
import "github.com/grafana/pyroscope-go"
profiler, _ := pyroscope.Start(pyroscope.Config{
ApplicationName: "order-service",
ServerAddress: "http://pyroscope:4040",
ProfileTypes: []pyroscope.ProfileType{
pyroscope.ProfileCPU,
pyroscope.ProfileAllocObjects,
pyroscope.ProfileAllocSpace,
pyroscope.ProfileInuseObjects,
pyroscope.ProfileInuseSpace,
},
Tags: map[string]string{
"region": os.Getenv("AWS_REGION"),
"version": os.Getenv("APP_VERSION"),
},
})
defer profiler.Stop()
Correlate Profiles With Traces (Grafana)
# grafana datasource: link Tempo traces to Pyroscope profiles
apiVersion: 1
datasources:
- name: Tempo
type: tempo
url: http://tempo:3200
jsonData:
tracesToProfiles:
datasourceUid: pyroscope
tags: [{ key: "service.name", value: "service_name" }]
profileTypeId: "process_cpu:cpu:nanoseconds:cpu:nanoseconds"
- name: Pyroscope
type: grafana-pyroscope-datasource
url: http://pyroscope:4040
eBPF Kernel Requirements and Portability
+------------------------------------------+
| Kernel Version Requirements |
| |
| 4.9 - kprobes, tracepoints, BPF maps |
| 4.18 - BTF (type info for CO-RE) |
| 5.2 - BPF_PROG_TYPE_TRACING |
| 5.5 - BPF ring buffer |
| 5.8 - BPF iterator, fentry/fexit |
| 5.13 - BPF CO-RE stable |
| |
| Check your kernel: |
| uname -r |
| Check BTF support: |
| ls /sys/kernel/btf/vmlinux |
+------------------------------------------+
BPF CO-RE (Compile Once, Run Everywhere)
CO-RE allows a single eBPF binary to run across different kernel versions without recompilation.
# Check if BTF is available (needed for CO-RE)
ls /sys/kernel/btf/vmlinux # exists on kernels compiled with CONFIG_DEBUG_INFO_BTF=y
# Amazon Linux 2023, Ubuntu 22.04+, RHEL 9+: BTF enabled by default
# GKE, EKS (Amazon Linux 2023 node groups): BTF supported
# EKS Bottlerocket: BTF supported
# bpftool: inspect running eBPF programs
bpftool prog list # list loaded eBPF programs
bpftool map list # list BPF maps
bpftool map dump id <map_id> # dump map contents
bpftool net list # show eBPF attached to network interfaces
bpftool perf list # show eBPF attached to perf events
When to Use eBPF Observability
Decision tree:
Can you modify the application?
YES --> Use OTel SDK (richer business context, distributed tracing)
NO --> eBPF is your only option for visibility
Need kernel-level visibility?
(syscalls, network bytes, scheduler, file I/O)
YES --> eBPF only; OTel SDK can't see this layer
NO --> OTel SDK sufficient
Performance-critical path? (< 1% overhead budget)
YES --> eBPF (< 0.5% overhead); OTel can be 1-5%
NO --> OTel SDK easier to maintain
Kubernetes service mesh (zero-config mTLS + L7 visibility)?
YES --> Cilium + Hubble replaces Istio/Envoy; uses eBPF
NO --> N/A
Continuous CPU/memory profiling?
YES --> Pyroscope with eBPF spy (no SDK needed)
OR Pyroscope SDK for labeled profiles
Ad-hoc production debugging (legacy app, no source)?
YES --> bpftrace one-liners or BCC tools
Limitations
| Limitation | Detail | Workaround |
|---|---|---|
| Kernel version | Many features need 5.8+ | Check uname -r; upgrade if needed |
| BTF required for CO-RE | Without BTF, must compile eBPF per kernel version | Use libbpf-bootstrap + CO-RE; modern distros have BTF |
| Container security restrictions | eBPF needs CAP_BPF or CAP_SYS_ADMIN |
DaemonSet with privileged=true; restrict to observability namespace |
| Not for high-cardinality business metrics | BPF maps have size limits; no string interpolation | Use OTel SDK for business events; eBPF for infrastructure |
| Encrypted user-space traffic | HTTPS encrypted before kernel TCP layer | Uprobe on SSL library (libssl.so) or use Pixie's TLS interception |
| JVM JIT-compiled code | JIT-compiled methods have no stable symbols | Use async-profiler or JVM USDT probes (-XX:+ExtendedDTraceProbes) |
Checklist
Before Deploying eBPF Tools
-
uname -r— kernel 5.8+ for full feature set -
ls /sys/kernel/btf/vmlinux— BTF available for CO-RE - Node security group / PSA allows
CAP_BPFfor DaemonSet - Confirm eBPF tool overhead acceptable (< 1% CPU on idle node)
Pixie Deployment
- Pixie Vizier deployed as DaemonSet
- Test:
px run px/http_datareturns data within 30s - Verify TLS interception works:
px run px/http_datashows HTTPS traffic
Cilium + Hubble
-
cilium statusshows all components healthy -
hubble observereturns flows - Hubble metrics exported to Prometheus
- Grafana dashboard showing per-service HTTP error rate
Pyroscope Profiling
- eBPF spy enabled in Helm values
- Flame graph visible in Grafana for at least one service
- Profile-to-trace correlation configured (Tempo + Pyroscope datasources linked)
bpftrace / BCC (Ad-Hoc)
-
bpftrace --versioninstalled; verify version 0.16+ - Run
bpftrace -l 'tracepoint:syscalls:*'to verify tracepoints available - Test:
bpftrace -e 'BEGIN { printf("eBPF works\n"); exit(); }'