We've changed how selected files in proc are virtualized, replacing LXCFS with namespace-based virtualization.
LXCFS is a FUSE (Filesystem in Userspace) filesystem we've used since vpsAdminOS began to virtualize selected files in /proc and /sys inside VPS containers.
Linux supports namespace isolation and cgroup limits, but without LXCFS, top inside a VPS would still show all the node's CPUs, free would show the whole node's memory usage, load average would be higher than expected, and so on. We gradually reduced LXCFS's role, and from vpsAdminOS kernel 6.6.10, we've removed it entirely. It's no longer needed!
Our kernel now includes patches virtualizing selected /proc files based on namespaces and cgroup limits. LXCFS had to obtain these values from userspace by reading /sys/fs/cgroup parameters. To calculate load average, it even had to scan all the VPS's processes through /proc. Reading these files acquires shared locks, delaying other accesses.
LXCFS also only worked where it was mounted. If you ran containers inside your VPS, you either had to bind-mount LXCFS into them manually or their /proc values weren't virtualized. Some utilities obtain system status through the sysinfo syscall, bypassing LXCFS entirely. sysinfo is used by Busybox, for example, and even glibc itself. On Alpine Linux, which uses Busybox by default, uptime therefore showed the entire node's load average.
On nodes with an older kernel, we can see which files LXCFS virtualizes in its final stage of use:
$ grep lxcfs /proc/mounts lxcfs /proc/cpuinfo fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0 lxcfs /proc/diskstats fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0 lxcfs /proc/loadavg fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0 lxcfs /proc/uptime fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
In the past, this also included /proc/stat, /proc/meminfo, /proc/swaps and /sys/devices/system/cpu/online. Back then, we added VPS CPU usage virtualization based on CFS quotas to LXCFS ourselves.
We started with a single LXCFS instance serving all VPSes on a node, which worked fine on nodes with up to 100 VPSes. Moving to nodes with 300/600 VPSes, one LXCFS process could no longer keep up. We briefly ran a separate LXCFS instance for each VPS, but that was a very bad idea: 300/600 processes competed to read /proc and cgroup parameters. We finally moved to LXCFS worker processes, with one instance serving at most 50 VPSes and more instances starting as needed:
[root@node24.prg.vpsfree.cz] ~ # osctl lxcfs worker ls NAME ENABLED SIZE MAX_SIZE CPU_PACKAGE LOADAVG CFS worker.0.cpu1 true 49 50 1 true true worker.1.cpu1 true 50 50 1 true true worker.2.cpu1 true 50 50 1 true true worker.3.cpu1 true 50 50 1 true true worker.4.cpu1 true 50 50 1 true true worker.5.cpu1 true 48 50 1 true true worker.6.cpu1 true 18 50 1 true true worker.7.cpu0 true 50 50 0 true true worker.8.cpu0 true 50 50 0 true true worker.9.cpu0 true 50 50 0 true true worker.10.cpu0 true 49 50 0 true true worker.11.cpu0 true 50 50 0 true true worker.12.cpu0 true 36 50 0 true true
node23/24.prg have two CPU sockets, so we also make sure a VPS uses an LXCFS instance on the same socket. Bandwidth between sockets is limited, so we try to keep each VPS within one socket.
None of this is needed now, and we've removed LXCFS entirely from vpsAdminOS. You'll always see virtualized values inside your VPS. If you run LXCFS inside your VPS yourself, nothing changes; you can keep using it. Our changes simply hide the fact that your VPS is itself a container.