Since the beginning of 2022, we have been working on the move to cgroups v2. It requires changes to the Linux kernel as well as userspace integration. That is why the devcgprog utility came about.
As I wrote in my previous post, we are gradually working on the planned move to cgroups v2. Kernel changes are still in progress, alongside userspace integration: controller delegation, different ways of configuring and reading parameters, and finally integration with vpsAdmin. For me, the devices controller has probably been the toughest part so far. I will describe what it is for and how it works, for anyone interested.
The devices controller is essential to us. We cannot operate without it because it controls access to every device: block devices such as disks at /dev/sda and character devices such as /dev/tty, /dev/null and /dev/zero. Since you can use mknod inside a VPS to create any device node, the devices cgroup is necessary to control access.
By default, we allow access to /dev/{console, full, kmsg, null, ptmx, random, urandom, tty*}, and depending on the VPS features, also /dev/{net/tun, kvm, ppp}. Our most important requirement is to prevent access to the disk devices underlying ZFS.
The devices controller in cgroups v1 is beautifully simple. Allowed devices appear in devices.list:
# cat /sys/fs/cgroup/devices/devices.list c 1:3 rwm c 1:5 rwm c 1:7 rwm c 1:8 rwm c 1:9 rwm c 1:11 rwm c 5:0 rwm c 5:1 rwm c 5:2 rwm c 136:* rwm b *:* m c *:* m
That means read-write-mknod (rwm) for selected devices and mknod (m) for everything else. Normally, Linux does not allow mknod in a user namespace, but vpsAdminOS does. This is useful when unpacking an archive containing device files, for example.
To get this devices.list configuration, we first deny access to everything:
# echo a > devices.deny
Then allow selected devices:
echo c 1:3 rwm > devices.allow echo c 1:5 rwm > devices.allow [...]
It is wonderfully simple, both to inspect and to configure. The only catch is that echo a > devices.deny works only if the cgroup has no children. Since cgroups are used from various places, I addressed this by configuring the devices cgroup first when starting the system or creating a VPS.
For a long time, cgroups v2 had no devices controller at all... and when it arrived, it used BPF. You write a BPF program, compile it, load it into the kernel and attach it to the cgroup. When a device is accessed, that BPF program runs and either allows or denies access. An example program is included in the kernel.
My biggest problem was that I could not get it to compile, no matter what I tried. Of course, I no longer remember the details, but I was utterly desperate. Compiling programs this way would not be ideal, since it requires LLVM and increases the rootfs size... but at least it would have been a start.
Projects such as systemd and LXC also use the devices cgroup. They do not compile programs with LLVM; they construct them directly from BPF instructions. You can see this nicely in LXC.
I liked this approach much more, but still got nowhere. We cannot use this LXC feature directly because we run LXC as an unprivileged user, and only root can configure the devices cgroup. Copying the code out is not simple either: it needs a lot of surrounding code, and apparently I did not have the patience for it. I tried libbpf too, without success.
I gave up and returned to it about a year later... the situation was exactly the same. But I found a Go library that lets me assemble a program from instructions, load it into the kernel and even attach it to a cgroup! Best of all, it needs no LLVM, kernel source files, libbpf or anything else: Go is enough.
Another necessary piece is the bpffs virtual filesystem. A BPF program only stays in the kernel while the process that loaded it is running. We effectively have no such permanent process; everything can restart during an update... which would suddenly remove all the programs guarding device access.
But if we create a reference to the BPF program in bpffs, the program stays in the kernel even after the process that created it exits. We then release the program by deleting the file (its reference) in bpffs. Similarly, we can create references to a program's attachment (attach/link) to a cgroup.
Our whole stack is in Ruby, so I wrote a separate Go program to configure the devices cgroup: devcgprog. We call it from Ruby with the cgroup path and a list of devices to allow. devcgprog creates a BPF program, loads it into the kernel, stores it in bpffs and attaches it to the cgroup. If anyone else happens to need this, devcgprog is on GitHub.
Usage is simple:
devcgprog set /sys/fs/bpf/my-program \
/sys/fs/cgroup/my-cgroup \
/sys/fs/bpf/my-program-on-my-cgroup \
allow \
c:1:3:rwm c:1:5:rwm [...]
The parameters are: the reference to create in bpffs, the cgroup path, the reference to the cgroup link, the list type (allow for an allowlist and deny for a denylist), and the device list.
You can load a BPF program once and use it with multiple cgroups through devcgprog new/attach. For us, the possible combinations are limited by VPS features. We name programs using a hash of all the devices, giving programs for the different combinations of TUN/TAP, KVM and PPP. If a program with that hash already exists, we simply attach it to another VPS cgroup.
Compared with cgroups v1, this is still harder to inspect. A BPF program is opaque next to devices.list. A cgroup can also be deleted and recreated with the same name, leaving the link file in bpffs even though the program is no longer attached. So a reference existing in bpffs does not mean everything is correct. We therefore call bpftool regularly to check that programs are configured properly. This is just a precaution: only root on the node controls the cgroups we configure, so the situation should not arise.
I am not saying BPF is not great. For tracing, flame graphs and so on, it is excellent. It is just not always easy to use.