I like Rust. I like the ecosystem, I like the performance, it's the perfect blend for me. Coupled with the safety features, I feel like I have a wise old friend guiding me through areas that could otherwise cause me some grand issues.
So why Rust? C is fine
As you saw in the last article, BPF programs are typically programmed in C. But why couldn't we use Rust? It can be just as low level as C.
The biggest win in my mind is the ergonomics that could be achieved (which I will hopefully be able to improve across these next few posts) along with the crates ecosystem that has enormous potential when compared to the C BPF ecosystem. It could be argued that because of the verifier the borrow checker is moot, but I don't see it in quite the same light. Sure the verifier will enforce specific checks and be arguably far more strict than the borrow checker but that
doesn't mean we can't get any use out of leaning on Rust's borrowing system.
Of course there is the subjective benefit that I'm more productive in Rust, and far more proficient with Rust than with C. Plus if I'm writing the supporting applications/libraries in Rust it'd be nice to stay in a single language when possible.
RedBPF
As it turns out there are a number of crates dedicated to writing BPF code in Rust! Many are in varying states of completeness and activity. The set of crates I've found most complete and promising is the redbpf
collection. It's not perfect, but it shows great promise and the developers are super friendly and helpful. In this post I hope to dive into the design decisions and some new ideas for improving the networking portions of the redbpf
crates.
If you remember all the way back to part 1, the problem we were originally trying to solve was a networking one, where we needed to validate a packet and multiplex a single ingress port to multiple ingress ports depending on parts of the payload.
Whats in the Box
The redbpf
repository is divided into several crates:
-
bpf-sys
: provides bindings tolibbpf
and parts of BCC by usingbindgen
-
cargo-bpf
: Acargo
subcommand that handles the boilerplate of setting up and building a BPF project in Rust- also contains a library for generating additional bindings, and a development loader
-
redbpf
: A userspace library for loading and interacting with BPF programs -
rebpf-probes
: A library to writing kprobes, uprobes, and XDP or Socket BPF programs in Rust- also provides a
bindings
module where it usesbindgen
to generate bindings forlibbpf
- also provides a
-
rebpf-macros
: A procedural macro crate which contains the proc-macros used to wrap BPF functions and code inredbpf-probes
-
rebpf-tools
: Sample projects generated bycargo-bpf
to demonstrate how to structure code
Interestingly, bpf-sys
is only used by cargo-bpf
and redbpf
, but not by redbpf-probes
or redbpf-macros
. I asked about this on Github and was told the reasons for both were historical and due to stability, but now that things have improved they plan to merge everything down to bpf-sys
with libbpf
. This will be great, because it caused a little confusion early on while investigating the code.
We can somewhat ignore redbpf-tools
as it's pretty much just example code. I would prefer if it was in the examples/
dir, but that's just subjective preference I don't plan on trying to move it. Finally, redbpf-macros
is used heavily by redbpf-probes
, and since redbpf-probes
will be our main focus for this post we will also touch the proc-macros to some extent.
There is another tool, bpf-linker
which is not part of the RedBPF collection, but was written recently by the primary author of the RedBPF crates. Although I haven't started using it heavily yet as it's so new, I have no doubt it will become key to this space.
Example Current Solution to Part 1 Problem
We'll first create a solution the problem we spoke about in Part 1 using RedBPF as it stands (kind of). This will allow us to contrast the solution with future iterations, and discover/discuss design changes and improvements.
To write BPF code in Rust, it's easiest to use
cargo-bpf
(part of the redbpf
suite) which handles setting up the project and can even function as a development loader.
I recommend installing it via cargo install
from the git
repository, but first you must make sure you have all the required development files such as LLVM 11, kernel headers, etc. (see the repository for details as it's quite explicit and outside the scope of this post)
Once you have all the required packages, install cargo-bpf
with:
$ cargo install cargo-bpf --git https://github.com/redsift/redbpf
We'll create our example project called mplex
:
$ cargo new mplex
$ cd mplex/
Our project will most likely contain a custom loader in the future, and at least one BPF object. mplex
will be the place holder for our "outer" userspace application and loader. We'll use cargo-bpf
to add BPF objects to this
project, which will be in the form of stand alone binaries.
The way cargo-bpf
does this is by creating additional binaries using cargo's [[bin]]
tables, and requiring specific cargo features to be passed in order to compile the BPF code. When you run cargo bpf build
it will search through the project and find the binaries listed, and build them with the appropriate cargo features enabled. In this way we can split out the dependencies required by our BPF program(s) and our userspace application/loader.
At first though we will be using cargo-bpf
as the loader, so we can focus on the important parts of this post.
XDP, TC, or Socket?
Ok, we're at our first decision point; where to hook in our BPF program? We know we want a networking hook, and the socket layer is too high as we want to affect the incoming port. We could use TC, but we only need the ingress side, and since we'll be re-writing part of the packet it'd be nice to have direct access to the packet memory.
XDP
XDP checks all of our boxes, and is the earliest possible point to observe or mutate a packet.
The argument against XDP is that if we'll most likely be passing the packet up the networking stack anyways (after mutating) we don't really save anything by not allocating a socket buffer in the TC layer.
XDP is normally best when you're just trying to drop/redirect packets out (firewalling or routing), however, we are validating packets and potentially dropping a fair percentage so using XDP is fine. Perhaps in a later post I will come back and show a TC variant as well so we can contrast the two solutions. In fact if I do my job well, the two solutions should not be too different by the time we're done improving these crates.
We can then tell cargo bpf
to create a BPF executable for us which we'll call mplex_xdp
:
$ cargo bpf add mplex_xdp
Our project structure now has two binaries, mplex
(at src/main.rs
) will be the eventual userspace loader, and a new src/mplex_xdp/main.rs
has been added which will be the BPF object.
It's going to feel like we just jumped from level 0 to 100 skipping all the steps in-between. But the purpose of this post is to discuss the current state of RedBPF networking, vs future improvements we can make. Not the specifics of how
we can implement a program for the problem in part 1.
Current RedBPF XDP Example
Removing the generated example from src/mplex_xdp/main.rs
we accomplish our set out task with the following simplified code (see caveat below):
#![no_std]
#![no_main]
use redbpf_probes::{
bindings::tcphdr,
net::Transport,
xdp::prelude::*,
};
program!(0xFFFFFFFE, "GPL");
#[xdp]
pub fn mplex(ctx: XdpContext) -> XdpResult {
// only match TCP
if let Ok(Transport::TCP(tcp)) = ctx.transport() {
unsafe {
// Only match destination port 5000
if u16::from_be((*tcp).dest) == 5000 {
let d = ctx.data()?; // get the payload
let ds = d.slice(d.len())?; // turn the payload into a byte slice
// "Validation" ensure payload length is within the narrow window
let payload_len = ds.len();
if payload_len < 290 || payload_len > 294 {
// Drop packet if not
return Ok(XdpAction::Drop);
}
// Multiplex based on entity tag which is 3 bytes, at an offset of
// 20 bytes into the payload
//
// Double slice means "skip 20 bytes, then return the next 3"
match &ds[20..][..3] {
b"600" => {
// re-write destination port
(*(tcp as *mut tcphdr)).dest = u16::to_be(5001);
}
b"601" => {
(*(tcp as *mut tcphdr)).dest = u16::to_be(5002);
}
b"602" => {
(*(tcp as *mut tcphdr)).dest = u16::to_be(5003);
}
_ => return Ok(XdpAction::Drop);, // no matching tag means invalid; drop
}
}
}
}
// Allow to pass up the stack
Ok(XdpAction::Pass)
}
CAVEAT: The caveat is that the above code will fail the BPF verifier incorrectly because it thinks we're not doing proper bounds checking. This is actually one of the things that lead me to look at improving the RedBPF networking. So this example is roughly what a solution would look like sans a few changes that aren't super important right now.
But generally, we can already see that this is leaps and bounds better in terms of ergonomics and readability (IMO) than the equivalent C.
If we were to run this on our incoming interface, (after calming the verifier) we'd see that in-fact all the problems described in part 1 are magically gone. The server application now believes multiple ports are being utilized, so it's happy, and the external source system is only sending data across a single port so they're happy too. Meanwhile this tiny XDP shim is churning along, with no performance hit.
Can we do Better?
But we're not done! We can make the verifier happy with standard Rust idioms, we shouldn't have to jump through hoops just to make the verifier happy when it appear like we're already doing those things by telling Rust to do it. There
is also a little too much unsafe
for me, and the abstractions fall a little short since it's a little too coupled to the 4 protocols currently supported.
After speaking with the authors of redbpf-probes
, they stated networking/XDP wasn't their main focus, as they instead they utilize the tracing aspects far more frequently so the networking API is not as solid or had as much thought put into it. It was also some of the first code written as merely a proof of concept.
It's open source, so let's see if we can help with that!
I reached out to the original author, and we've been going back and forth about potential changes. He's on board, and we're both super excited to see where we can take this!
Wrap Up
In the part of this series we'll do that deep dive into the current implementation and note areas that we want to mark for improvement or change.
Top comments (0)