Our first server with SSDs and 512 GB of RAM

We bought a new server full of SSDs at a very good price and moved our most demanding VPSes onto it.

Some time ago, our server supplier contacted us to say that Intel was selling off its DCS3700 SSDs because it had a new 3710 series. We could buy the remaining stock at half price: an 800GB drive reduced from 28 to 14 thousand crowns. We know these drives, have had excellent experience with them, and they deliver a constant IOPS figure of around 10K. You cannot overwhelm a drive like that.

We also connected the drives through three powerful SAS3 controllers, which are considerably faster than the onboard controller. That removes any bottleneck and lets us make full use of the drives under a high, sustained load.

So we bought a new server with eight of these drives in a single RAIDZ2 (something like RAID 6). The rest of the server is up to the same standard: two of the latest 14nm Broadwell-EP processors with ten physical cores each. With HT, the system sees 40 cores. Unless your terminal is big enough, you cannot press one in top to display them all.

We also had 512GB of DDR4 memory installed, because the price has finally fallen to where it should have been all along. A 32GB module costs only about 20% more than a 16GB DDR3 module. So a small extra payment gets us twice as much memory.

A machine this powerful suits members who need high disk performance. Even with our aggressive caching, plenty of node memory and SSD caches, writes can still put a regular node under pressure. We have quite a lot of them, and ZFS starts syncing its write cache after every 64 MB of unwritten data. We reach that very quickly, so we are writing practically all the time. At that point, disks are busy writing and have less time for reads. Demanding applications can struggle.

That is why we moved the most demanding VPSes to the new node 14, taking some load off the other servers. We have not hit the new server's limit yet, and we will see how much it can handle. There are certainly more and more services capable of putting a heavy load on ordinary disks.

Eventually, we would like everything to be on SSDs, but prices still need to fall by at least a third. Replacing the drives in one node with SSDs currently costs about 90 thousand crowns. We need another price drop before we can start making those replacements.