What Is eBPF and Why Does It Matter for Observability?

Working within the Linux kernel is ideal when implementing security, networking, and observability features. However, it’s not without its challenges. Whether modifying kernel source code or adding modules, developers have traditionally found themselves contending with complicated infrastructure and abstracted layers that are difficult to debug. Extended BPF (eBPF) solves both of these problems.

Extended Berkeley Packet Filter (eBPF) is a kernel technology (starting in Linux 4.x) that allows programs to run without having to change the kernel source code or adding additional modules. You can think of it as a lightweight, sandbox virtual machine (VM) inside the Linux kernel, where programmers can run BPF bytecode that takes advantage of specific kernel resources.

Using eBPF eliminates the need to change kernel source code and streamlines the ability of software to leverage existing layers. As a result, it’s a powerful technology with the potential to fundamentally alter how services like networking, observability, and security are delivered.

Here’s a closer look at what it is, how it works, and when to consider implementing it.

How eBPF works

eBPF programs are event-driven and attached to a code path. The code path contains specific triggers—called hooks—which execute any attached eBPF programs when they’re passed. Some examples of hooks include network events, system calls, function entries, and kernel tracepoints.

When triggered, the code is compiled first to the BPF bytecode. In turn, the bytecode is verified before it runs, to ensure it doesn’t create a loop. This step prevents the program from compromising the Linux kernel either accidentally or on purpose.

After a program is triggered at a hook, it then makes helper calls. These helper calls are functions that equip eBPF with many features for accessing memory. Helper calls need to be pre-defined by the kernel, but the list of what functions exist continues to grow.

eBPF was initially used as a way to increase observability and security when filtering network packets. However, over time, it became a way to make the implementation of user-supplied code safer, more convenient, and better-performing.

The advantages of eBPF

eBPF is typically used to trace user-space processes, and its advantages shine here. It’s a safe and useful method to ensure:

• Speed and performance. eBPF can move packet processing from the kernel-space and into the user-space. Likewise, eBPF is a just-in-time (JIT) compiler. After the bytecode is compiled, eBPF is invoked rather than a new interpretation of the bytecode for every method.
• Low intrusiveness. When leveraged as a debugger, eBPF doesn’t need to stop a program to observe its state.
• Security. Programs are effectively sandboxed, meaning kernel source code remains protected and unchanged. The verification step ensures that resources don’t get choked up with programs that run infinite loops.
• Convenience. It’s less work to create code that hooks kernel functions than it is to build and maintain kernel modules.
• Unified tracing. eBPF gives you a single, powerful, and accessible framework for tracing processes. This increases visibility and security.
• Programmability. Using eBPF helps increase the feature-richness of an environment without adding additional layers. Likewise, since code is run directly in the kernel, it’s possible to store data between eBPF events instead of dumping it like other tracers do.
• Expressiveness. eBPF is expressive, capable of performing functions usually only found in high-level languages.

eBPF best practices

Since eBPF is such a new technology, many things remain unexplored. Best practices around eBPF are still evolving as the technology gains prominence. While no defined set of best practices exist, there are a few things that you can do to ensure effective, efficient programs.

If you’re using eBPF for your ecosystem, we recommend that you:

• Use LLVM Clang to compile C into bytecode. When eBPF first hit the scene, it was necessary to code and assemble the program by hand. Then, developers used the kernel’s assembler to generate bytecode. Fortunately, it’s no longer necessary to do this. Clang provides infrastructure for frontend and tooling in C languages. 
• Use the BCC toolkit when writing BPF programs. The BPF Compiler Collection (BCC) is a toolkit that can help you create efficient kernel tracing and manipulation programs. It’s especially suited for tasks related to performance analysis and controlling network traffic.

The pitfalls of eBPF

Although it’s powerful, eBPF is not a silver bullet that suits every project or ecosystem. eBPF does have some notable disadvantages, which can make it frustrating to work within certain instances. Some developers might find eBPF inadequate to use because:

• It’s restricted to Linux and a recent kernel. eBPF was developed in the Linux kernel and is completely oriented around it. That makes it less portable than other tracers. Additionally, you need a fairly recent kernel. If you’re running anything older than v 4.13, you won’t be able to use it.
• Sandboxed programs are limited. eBPF derives increased security by limiting what resources programs can access. However, by limiting what parts of the OS a program can access, functionality is also potentially limited.

When eBPF typically works well

eBPF is rapidly gaining traction in cloud native applications. As a result, eBPF is used most commonly in two situations:

• There’s a need for observability using kernel tracing. In this situation, eBPF is faster and more accurate. There are no context switches involved, and eBPF programs are event-based so nothing runs without a specific trigger—and you won’t miss any occurrences.
• Traditional security monitoring doesn’t work. eBPF is finding lots of uses in distributed and container-based environments, including Kubernetes. In these environments, eBPF can close the visibility gap because it can provide visibility into HTTP traffic.

You might also find eBPF deployed for other security measures, including:

• Firewalls
• Device drivers
• Network performance monitoring

New Relic and eBPF

Pixie (acquired by New Relic), is an open source, kubernetes-native-in-cluster observability platform that provides instant visibility into Kubernetes workloads with no manual instrumentation. eBPF provides most of the magic behind the Pixie platform. As described earlier, eBPF allows you to run restricted code when an event is triggered. This event could be a function call either in kernel space(kprobes) or userspace(uprobes). Pixie uses both uprobes and kprobes to enable observability across services and applications. 

Pixie automatically harvests telemetry data by leveraging eBPF, and its edge-machine intelligence connects this data with Kubernetes metadata to provide visibility while maintaining data locality. This visibility complements New Relic’s powerful Kubernetes observability solution. And starting in late May, you'll be able to send Pixie-generated telemetry data to New Relic One, gaining scalable retention, powerful visualizations, advanced correlation, and intelligent alerting capabilities.

eBPF is observability made efficient

eBPF is a new technology that improves observability, networking, and security in the Linux kernel. It eliminates the need to change kernel source code or add modules, so you can create a richer infrastructure to support your system without overcomplicating it.

We took a look at what eBPF is, how it works, and why it’s so useful in distributed environments. By monitoring from the kernel layer, many of the challenges associated with observability in the cloud are solved. You can enjoy deeper visibility, more context, and more accuracy in your data.