99.986% availability

Prospective members often ask about the uptime and availability of our servers. Here are the statistics.

We have been measuring availability at several points in our infrastructure for a long time, so we have precise data on our availability and the length of individual outages. The results are quite good.

The simplest measurement is a ping to some of our servers. Its long-term figure is 99.986%. That means our servers are unavailable for a total of about 6 minutes per month (calculator). This does not account for the overall functioning of every component, though. It only confirms that the network works and the server responds to ping.

Regular HTTP requests provide a deeper test. They can establish that the whole application works, the database responds and the filesystem serves data. Our long-term availability for this measurement is 99.875%. The measured servers are then unavailable for 54 minutes per month, or 12 minutes per week. This shows that every hundredth of a percent makes a big difference in absolute numbers.

In practice, however, most outages last less than ten minutes. Together, these short outages account for almost 80% of cases. Only five outages in the past year lasted longer than 30 minutes. So it is not too dramatic, although there is certainly room for improvement.

Reaching higher figures is very difficult. The Pareto principle applies: achieving 80% success takes 20% of the effort. But as you approach 100%, the effort required rises ever more steeply. We are not letting up, though, and continue to make changes that help us improve availability a little at a time.

At present, the combination of Linux and ZFS causes us the most trouble. The Linux kernel is not designed for heavy memory pressure and sometimes deadlocks. Solaris, for example, does not have this problem. But by actively reporting bugs and debugging with the ZFS on Linux developers, we have resolved most of the major problems. In practice, hangs are now rare, but we are continuing to debug.

Eventually, we would like to set up our own QA test cluster at the Base48 hackerspace in Brno, so we can catch problems before deploying a new kernel in production. We would then maintain our own set of tested components, knowing that they work and that no new bug has been introduced during recent development.