After a few weeks, here is another update on vpsAdminOS, what we are working on and what we are struggling with.
We are still working on LXCFS on node1.stg because the upstream version and kernel cannot do what we need. We need to set the number of available CPUs without pinning specific CPUs with cpuset, as OpenVZ can. Each VPS should be able to use any 8 CPUs, depending on which are currently free. It should not see more than that, even if the node has more: node1.stg has 32, for example. This is confusing, especially for users. If you see 32 available CPUs but can only fully use 8, you conclude that it does not work.
On vpsAdminOS, we use CPU quotas (cfs_quota/period_us) to set CPU limits, but there is no mechanism available to limit the number of visible CPUs. So I am currently modifying LXCFS to show the correct CPU count in /proc/stat and /proc/cpuinfo based on the quota. These are the files read by top, htop and other monitoring tools.
LXCFS only solves part of the problem, though. If you ask the kernel how many CPUs you have through a syscall, you still get the "wrong" answer. The kernel does not care about modified values in LXCFS. For example, nproc on node1.stg still returned 32 instead of 8. Various build systems may use nproc, so I tried to address that. The result is a short kernel patch that changes the return value of sched_getaffinity() to contain only a limited number of CPUs. With this change, nproc returns the correct number, but the solution is not ideal and may break some applications. We will see.
These changes have been deployed on node1.stg since last week. Please report anything unusual or not working as it should.
We also updated the kernel to 4.17. A NixOS bug that made the kernel incompatible with ZFS held us up for a long time. Now we are eagerly awaiting 4.18, which will support FUSE in a user namespace. That should allow sshfs and similar tools to work.
node1.stg will stay available for at least a few more months, and there is plenty of space. Join in. We have not had much feedback yet. You can run anything that can tolerate frequent outages. We do not delete the VPSes you create as long as there is space.