Migrating our infrastructure to 20 gigabits

We spent several years working to increase the capacity of the links between our servers. Why did we need it, and how did we do it?

Historically, our servers were connected through two bonded gigabit links. This gradually became a limit, not so much for user traffic as for migrating virtual servers between nodes. With at least 256 GB of RAM in our servers today, a move took tens of minutes, during which the server was unusable. In recent months, the NAS has also become a major network user, reaching 3 Gbps at peaks.

So we began considering a substantial speed increase, which naturally required new network cards and switches. For various reasons, we had to postpone the change several times and only managed it at the start of this year. Unfortunately, we are still a relatively small organization, and a project this demanding sometimes takes us quite a while.

We chose Dell S4048-ON switches for their support for the ONIE operating system. We wanted to use that and, as always, take the most open-source route possible. Unfortunately, as we later found, nothing is flawless.

Of the original five units ordered for Prague (3×) and Brno (2×), two broke in strange ways during configuration and testing. The first failed during an attempt to update the system. The second lost its management card when connected to the network after a long period of inactivity. A closer look showed that part of the board had died, and repairing it meant replacing the unit. So we were back to the price of a whole switch, if not more.

The problem is that upgrading from a very old firmware version requires installing every intermediate update released along the way. Skip one, and the switch stops working. In the end, someone who handles updates like this every day had to help us.

It was a difficult road, but we finally started the migration on Sunday night into Monday, 20 January. We began at midnight with our NASbox, which needed reconfiguring to share ZFS datasets through sharenfs.

Then we moved on to the nodes themselves. Each had to be shut down, have its network cards replaced, get a Linux kernel update while we were there, and start again. This happened in two phases. Network cards were installed in the servers last year, and we kept expecting to begin the migration at any moment. In the end, firmware problems pushed the migration itself back to January, although everything had been connected and ready since autumn.

A lesson for next time: it is better to leave the money in the bank and buy the equipment for a change like this mostly at once, rather than gradually. The price of 10GE networking fell dramatically during the process. Today, we could at least have newer network cards. Lesson number two: never say that changes like this are easy and nearly done.