We've reconsidered and greatly expanded our original plan to buy more powerful virtualization machines. This means major energy savings without increasing membership fees.
The end of the year is fast approaching, along with the end of our old energy prices. We expect a substantial increase in the new year. Our original plan to cut consumption by a few kilowatts would probably have left us around 100 thousand crowns short every month.
We had to rethink our approach considerably and put much more into it than originally planned. The only sensible way to reduce consumption is to consolidate smaller machines into far fewer, much larger ones.
Even more virtual servers per machine
In the previous post, I mentioned a server with two 64-core EPYC CPUs and 1 TB RAM. Like node22, it was originally meant to run 300 VPSes and provide reserve capacity for CPU-intensive workloads. It turned out that even one such processor is more than enough, leaving nothing to move over.
With the hockey-stick graph of electricity prices, the next step more or less suggested itself: we fitted node23 with two terabytes of RAM and bought node24 with the same configuration.
We're therefore adding two machines, each comfortably fitting 600 VPSes. Together, node21 (1TB/64 core Naples, 300 VPSes), node22 (1TB/64 core Rome, 300 VPSes) and now node23+node24 (2TB/128 core Rome, 600 VPSes) can hold all production VPSes in Prague, with a little room to spare.
To put power consumption in perspective: a machine for 80 VPSes needs about 300 W. A new 1T/64c machine for 300 VPSes uses roughly 400 W (single socket Rome, node22) to 500 W (dual socket Naples, node21). We don't yet have data from 600 VPSes on node23/24, but expect something below 800 W. They do have HPC models of those EPYCs, after all.
It isn't cheap
This is probably the point to say that none of this is cheap. We're very lucky that second-hand prices have fallen to more usable levels over the past two years; otherwise the numbers wouldn't really work. Even so, finishing node23 and buying node24 together cost 1.5 million crowns.
For budgeting, borrowing the money isn't especially significant. We saved up and paid cash for node22 from the same surplus we'll now use for repayments. After all, borrowing from members is nothing new; we were used to it for a long time. That tradition only stopped during the odd period in the last pre-crisis years, when upgrades were overdue but the numbers didn't work either way. Energy savings didn't justify the high investment needed for brand-new large EPYC machines then.
If we'd upgraded earlier to modern servers still sized for 80 VPSes, consumption would only have fallen slightly. We'd have newer hardware, but not better resource sharing. We'd now have a big problem, still paying off hardware whose electricity we couldn't afford. That would probably have forced a membership fee increase.
By waiting, we've achieved a double benefit: we won't have to raise membership fees, because we can move to hardware that genuinely saves money. It has taken a lot of work, though. It's very hard to simulate enough load to tune a shared kernel on such a large configuration in advance.
The hardest part is behind us
I'd like to apologize to all members for creating extra work too. We still have a few weeks of fairly frequent restarts on the large machines as we tune them for the intended load. We're much further along and more confident than at the start; I'd say the worst is behind us. Thanks to everyone with VPSes on node22 for bearing with us and helping tune production so far.
I've left the least pleasant part until last, since it also means a lot of work for everyone involved: we need to shut down the OpenVZ machines by the end of the year.
The OpenVZ Legacy kernel is based on RHEL 6, meaning version 2.6.32. It wasn't designed for machines this large. There's no reasonable way to move those VPSes to larger machines while retaining every feature. For members who really can't migrate and aren't running KVM inside, we could create one QEMU instance the size of an original machine. Honestly, we'd rather avoid it.
Move to vpsAdminOS
I'm asking all members still running VPSes on OpenVZ: please start taking the steps you've been planning and postponing as soon as possible, so you can migrate your VPS to vpsAdminOS. Time is running out, and we need to enter the new year with consumption as low as possible.
In general, it's very simple. If an OpenVZ VPS doesn't use the NAS and runs a distribution still supported by its developers, we can migrate it without problems or any intervention from the member.
If it uses the NAS, you need to reconfigure the NAS mount from vpsAdmin-managed mounting to ordinary NFS mounting inside the VPS, using your distribution's usual method.
The remaining cases involve distribution versions no longer supported even by their own developers. Around the end of November, we should know whether we'll need a virtualized OpenVZ Legacy node or, hopefully, not.
Half the power consumption
Current consumption is 11.5 kW, and our aim is to cut it to less than half. Besides replacing servers, we need other changes, such as adjusting infrastructure and switching off surplus support boxes. We can do it. If you have questions or ideas for improvements, contact support.