Cilium Ebpf

Cilium and eBPF

Purpose

Implement Cilium-based Kubernetes networking with eBPF for high-performance networking, security policies, observability, and multi-cluster connectivity.

Architecture Decision Trees

Cilium vs Other CNIs

Cilium Agent Mode: kube-proxy replacement vs standard

Network Policy Enforcement

Hubble vs Prometheus for Observability

CiliumInstallation Decision

Kernel version >= 5.10?
├── Yes → kube-proxy replacement mode (better performance, more features)
└── No → Standard iptables mode (fallback, fewer features)
    └── Consider upgrading kernel to 5.10+ for full eBPF benefits

Need encryption between nodes?
├── Yes → WireGuard (simpler, better perf) or IPsec (legacy compat)
└── No → Skip encryption config

Need multi-cluster communication?
├── Yes → Cluster Mesh with KV store mesh
└── No → Single cluster only

Encryption Mode Decision

Kernel supports WireGuard?
├── Yes → WireGuard (better performance, simpler config)
│   ├── Kernel module available → encryption.type=wireguard
│   └── No kernel module → encryption.wireguard.userspaceFallback=true
└── No → IPsec (broader kernel compatibility, higher overhead)

Quick Start

Helm install Cilium → Verify with cilium status → kube-proxy replacement → Network policies (L3/L4, L7) → Hubble for observability → Cluster mesh for multi-cluster → Service mesh for L7.

Core Workflow

Step 1: Cilium Installation (Helm)

# Install Cilium with kube-proxy replacement
helm repo add cilium https://helm.cilium.io/
helm upgrade --install cilium cilium/cilium --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true \
  --set encryption.type=wireguard \
  --set ipam.mode=kubernetes \
  --set devices='{eth0}' \
  --set routingMode=native \
  --set autoDirectNodeRoutes=true \
  --set bpf.masquerade=true \
  --set l7Proxy=true

# Verify installation
cilium status --wait
cilium connectivity test

# Check eBPF is being used
cilium status | grep BPF

Step 2: L3/L4 Network Policies

# policies/l3-l4-policy.yaml
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: payment-service-policy
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: payment-service
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: api-gateway
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
  egress:
    - toEndpoints:
        - matchLabels:
            app: notification-service
      toPorts:
        - ports:
            - port: "9090"
              protocol: TCP
    - toEntities:
        - kube-apiserver
      toPorts:
        - ports:
            - port: "6443"
              protocol: TCP
    - toFQDNs:
        - matchPattern: "*.internal.example.com"
        - matchName: "api.stripe.com"

Step 3: L7 Network Policies (HTTP/gRPC)

# policies/l7-http-policy.yaml
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: l7-payment-policy
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: payment-service
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: api-gateway
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: "POST"
                path: "/api/payments"
                headers:
                  - "X-Idempotency-Key"
              - method: "GET"
                path: "/api/payments/[0-9]+"
              - method: "GET"
                path: "/health"
            # Deny all other HTTP methods/paths

Step 4: L7 Kafka Policy

# policies/l7-kafka-policy.yaml
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: kafka-producer-policy
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: payment-producer
  egress:
    - toEndpoints:
        - matchLabels:
            app: kafka-broker
      toPorts:
        - ports:
            - port: "9092"
              protocol: TCP
          rules:
            kafka:
              - apiKey: "produce"
                topic: "payments"
                clientID: "payment-service"
              - apiKey: "metadata"
                topic: "payments"

Step 5: Hubble Observability

# Enable Hubble Relay and UI (already enabled in install)
kubectl port-forward -n kube-system svc/hubble-ui 12000:80
# Open http://localhost:12000

# Hubble CLI
hubble observe --from-pod production/payment-service
hubble observe --to-pod production/notification-service
hubble observe --verdict DROPPED
hubble observe --protocol http --last 100

# Hubble metrics (export to Prometheus)
# --set hubble.metrics.enabled="{dns,drop,tcp,flow,icmp,http}"

Step 6: Cluster Mesh (Multi-Cluster)

# cluster-mesh/cluster-mesh-values.yaml
# Install on cluster 1
clustermesh:
  useAPIServer: true
  config:
    clusters:
      - name: cluster-us-east
        ips: ["10.0.0.1"]
        port: 32379
      - name: cluster-us-west
        ips: ["10.0.1.1"]
        port: 32379

# Enable cluster mesh
cilium clustermesh enable --service-type ClusterIP --enable-kvstoremesh
# Generate connection secret for cluster 2
cilium clustermesh status
cilium clustermesh create --context cluster-us-east --hostname cluster-us-east.example.com
cilium clustermesh connect --context cluster-us-west \
  --destination-context cluster-us-east

# cluster-mesh/service-export.yaml
apiVersion: cilium.io/v2alpha1
kind: CiliumClusterwideServiceExport
metadata:
  name: payment-service
spec:
  type: ClusterSetIP
  ports:
    - port: 8080
      protocol: TCP
---
apiVersion: cilium.io/v2alpha1
kind: CiliumClusterwideService
metadata:
  name: payment-service
spec:
  type: ClusterSetIP
  ports:
    - port: 8080

Step 7: Bandwidth Management

# bandwidth/pod-annotations.yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/ingress-bandwidth: "100M"
    kubernetes.io/egress-bandwidth: "100M"
  name: bandwidth-limited-pod
spec:
  containers:
    - name: app
      image: nginx

Step 8: Encryption with WireGuard

# Set during install:
# --set encryption.type=wireguard
# --set encryption.wireguard.userspaceFallback=true

# Verify encryption
cilium encrypt status
cilium bpf ipcache list | grep encrypt

# Node-to-node encryption is automatic
# Can verify with tcpdump on any node:
# tcpdump -i any -nn "udp and port 51871"

Step 9: Cilium Service Mesh — L7 Ingress

# service-mesh/cilium-ingress.yaml
apiVersion: cilium.io/v2alpha1
kind: CiliumIngress
metadata:
  name: payment-ingress
  namespace: production
spec:
  ingressClassName: cilium
  rules:
    - host: api.payments.example.com
      http:
        paths:
          - path: /api/payments
            pathType: Prefix
            backend:
              service:
                name: payment-service
                port:
                  number: 8080
  tls:
    - hosts: [api.payments.example.com]
      secretName: payment-tls

Step 10: CiliumClusterWideNetworkPolicy — Default Deny

# policies/default-deny.yaml
apiVersion: cilium.io/v2
kind: CiliumClusterWideNetworkPolicy
metadata:
  name: default-deny-ingress
spec:
  endpointSelector: {}
  ingress:
    - fromEndpoints:
        - matchLabels:
            "k8s:io.kubernetes.pod.namespace": kube-system
  # All other ingress is denied by default

Step 11: eBPF-native Ingress Load Balancing

# lb/cilium-lb.yaml
apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
  name: production-pool
spec:
  blocks:
    - start: 192.168.10.100
      stop: 192.168.10.200
---
apiVersion: cilium.io/v2alpha1
kind: CiliumL2AnnouncementPolicy
metadata:
  name: production-l2-announce
spec:
  loadBalancerIPs: true
  interfaces:
    - eth0
  externalIPs: true

Step 12: Hubble Metrics for Prometheus

# Enable Hubble metrics export
helm upgrade cilium cilium/cilium --namespace kube-system --reuse-values \
  --set hubble.metrics.enabled="{dns,drop,tcp,flow,icmp,http}" \
  --set hubble.metrics.destination=prometheus

# Grafana dashboards available at:
# https://github.com/cilium/cilium/tree/main/install/kubernetes/cilium/environment/hubble/grafana

Step 13: Cilium Monitor — Real-time Debugging

# Real-time packet flow monitoring
cilium monitor --verbose

# Monitor specific service
cilium monitor --related-to payment-service

# Capture dropped packets only
cilium monitor --type drop

# Monitor HTTP traffic at L7
cilium monitor --type l7

# Capture and decode with tcpdump-like output
cilium monitor --hex

Step 14: Cilium Endpoint and Identity Inspection

# List all Cilium endpoints
kubectl get ciliumendpoints -A

# Get endpoint identity details
cilium endpoint list
cilium endpoint get <endpoint-id>

# Identity-based troubleshooting
cilium identity list
cilium identity get <identity-id>

# Check security identity labels
kubectl get ciliumendpoints --all-namespaces -o json \
  | jq '.items[] | {pod: .metadata.name, ns: .metadata.namespace, identity: .status.identity.id}'

Tool Comparison: eBPF-based Security Tools

Cilium Helm Values — Production Profile

# cilium-production-values.yaml
kubeProxyReplacement: true
routingMode: native
autoDirectNodeRoutes: true
bpf:
  masquerade: true
  tproxy: true
ipam:
  mode: kubernetes
encryption:
  type: wireguard
  wireguard:
    userspaceFallback: true
l7Proxy: true
policyEnforcementMode: always
hubble:
  relay:
    enabled: true
  ui:
    enabled: true
  metrics:
    enabled:
      - dns
      - drop
      - tcp
      - flow
      - icmp
      - http
    destination: prometheus
rollOutCiliumPods: true
resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    cpu: 2
    memory: 2Gi

Security Considerations

• CiliumNetworkPolicy with policyEnforcementMode: always prevents all traffic by default
• Enable Hubble audit logging for all dropped packets — store in SIEM for compliance
• Use toFQDN policies instead of allowing all egress — prevents data exfiltration via DNS
• WireGuard encryption ensures node-to-node traffic is secure even on untrusted networks
• Cilium identities are tied to Kubernetes service accounts — never run containers as root
• Hubble Relay API should be internal-only (not exposed externally)
• Enable TLS for Hubble gRPC communication between relay and UI
• Use CiliumClusterWideNetworkPolicy for baseline security that spans all namespaces
• Monitor Cilium agent logs for policy enforcement errors and identity allocation failures

Production Considerations

Security

• Enable encryption (WireGuard) for node-to-node traffic.
• Use CiliumNetworkPolicy with L7 enforcement for critical services.
• Apply CiliumClusterWideNetworkPolicy for baseline security (deny all by default).
• Enable Hubble for network audit trail.
• Use toFQDN policies instead of allowing all egress to APNs.
• Enable policy enforcement mode: always.

Performance

• Enable kube-proxy replacement for better service routing performance.
• Use native routing mode with autoDirectNodeRoutes.
• Enable BPF masquerading instead of iptables MASQUERADE.
• Use bandwidth annotations for noisy-neighbor prevention.
• Monitor Hubble dropped packet metrics for network issues.
• Use XDP acceleration for DDoS mitigation at the NIC level.

Troubleshooting

• cilium connectivity test for end-to-end validation.
• hubble observe --verdict DROPPED to see dropped packets.
• cilium monitor for real-time packet flow.
• cilium bpf commands for low-level BPF map inspection.
• cilium endpoint list to verify identity labels match expected policies.

Operations

• Upgrade Cilium using helm upgrade with rollOutCiliumPods=true for gradual rollout
• Test policy changes with cilium connectivity test before applying to production
• Monitor Cilium agent resource usage — enable auto-scaling for large clusters
• Enable etcd or CRD-backed KV store for Cluster Mesh reliability
• Set bpf.lbBurstSize proportionally to expected concurrent connections

Anti-Patterns

Anti-Pattern 1: Running on Old Kernels

Cilium's eBPF features require kernel >= 5.10 (ideally 6.x). On older kernels, Cilium falls back to iptables, losing most performance and feature benefits.

Anti-Pattern 2: No Network Policies

Installing Cilium but using default allow-all policy. Cilium's primary value is its policy engine — enable with --set policyEnforcementMode=always.

Anti-Pattern 3: Ignoring Hubble

Not deploying Hubble alongside Cilium. Hubble provides per-packet visibility that no other tool offers — deploy Hubble Relay and Prometheus metrics.

Anti-Pattern 4: Direct Routing Without Native CIDR

Using tunneling (VXLAN/Geneve) when direct routing works. Native routing with autoDirectNodeRoutes provides better performance.

Anti-Pattern 5: Too Permissive L7 Policies

Allowing all HTTP methods/paths without restrictions. L7 policies should be as specific as possible — only allow required paths and methods.

Anti-Pattern 6: Default Allow Egress

Not restricting pod egress traffic. Pods should only be allowed to talk to required services (API server, DNS, specific endpoints).

Anti-Pattern 7: Hubble Disabled for Performance Reasons

Disabling Hubble thinking it adds overhead. Hubble's eBPF-based observability has negligible overhead (< 5% CPU) and provides invaluable debugging.

Rules & Constraints

• Kernel >= 5.10 required for full eBPF features.
• Always deploy Hubble alongside Cilium.
• Enable kube-proxy replacement on new clusters.
• Use CiliumNetworkPolicy (not K8s NetworkPolicy) for L7 features.
• Enable encryption for multi-node clusters.
• Test connectivity with cilium connectivity test after install.
• Set policyEnforcementMode=always in production.
• Native routing requires proper podCIDR configuration on all nodes.
• Cilium agent log level: info in production, debug during troubleshooting only.

References

• references/cilium-architecture.md
• references/cilium-ebpf-advanced.md
• references/cilium-ebpf-fundamentals.md
• references/cluster-mesh.md
• references/ebpf-deep-dive.md
• references/network-policies.md
• references/observability-hubble.md
• references/cilium-service-mesh-guide.md

Handoff

Next: service-mesh — Istio/Linkerd service mesh integration with Cilium.