Have you ever replaced a server's motherboard? Yes, sometimes we have to head to the datacentre to do that too.
The SAS controller connecting some of the disks in our Node1 started behaving strangely. Disks connected through this external chip (which isn't part of the chipset) had unusually high latency. We ordered a newer revision of the motherboard and went to the Prague datacentre overnight at the weekend to replace it.
The first surprise was the different arrangement of SATA ports on the board. We had to move some cables, which changed the order in which the system detected the disks. Luckily, ZFS doesn't mind at all, and the array came up without trouble. With ZFS version 0.6.2 or earlier, though, we'd have had a problem. Those versions needed some wild symlinking.
When we first booted the server with the new board, the BIOS status code "B7" stayed lit in the bottom-right corner. Google revealed that this AMI BIOS message means something is wrong with the RAM. We were sure the problem involved CPU0, and had to rearrange the memory twice before everything started without errors. Our fingers paid the price: we both ended up with plenty of cuts.
Replacing the board did help. Previously, disk svctime in iostat was very high; afterwards, only atime (access time) remained high. svctime measures how long a disk takes to service a request once the kernel sends it. By contrast, atime covers the request's entire wait, including time in the kernel's disk queue.
So the problem is only partly resolved. The results show that ZFS overloads two disks on the external controller, giving them more work than the others. The array is unbalanced and will need a clean reinstall. We'll therefore have to migrate to Node12, reinstall, and move back. At the same time, we'll replace the current 1TB disks with 2TB ones, which are already waiting impatiently in the rack. Unfortunately, we can't mix them in the zpool, because the new ones use 4k sectors.
To see how the whole night's work went, look in our gallery, where there are plenty more pictures.