Maintaining a Mirror Network

Here are some notes about maintaining a mirror network. I'm talking about the Sage software, which is a nice open source mathematics program. Besides the source, there are many prebuilt binaries for various platforms. In total, for each release once a month, there have to move about 14GB in 27 files to 18 mirror websites on all continents of the world. It's important to have nearby mirrors, because the network connectivity might be weak in some areas. In Nov. 2009, the virtual box image for windows alone generated about 1.5 TB of traffic - all other binaries and source combined probably also 1.5 TB. Therefore, the mirror networks net efficiency is approximately a tenfold increases of the outgoing data volume.

To ensure that only those mirrors which are online and up-to-date are listed on the website, a small Python script checks a time-stamp file with a content-specific checksum. Only if it is correct and retrievable, the mirror is included. This check happens every 10 minutes, using Linux's cron mechanism.

Recently I enhanced this to visualize how the mirrors perform over time. This is especially interesting when a new release is mirrored out into the world. To see how this looks like, here is the graphic for the Sage 4.2.1 rollout:

Time starts at the bottom, you see it's around 14:00 GMT on Nov. 17th. The first working mirror is "UW", which is the master. The master mirror is excluded, once there is at least another one in North America online - as you can see that's SFU and Boston (Harvard). You can also see that it takes more than one day to sync with the mirrors. Besides that the transfer takes its time, they are probably scheduled to start sync only once a day, the master mirror rejects too many simultaneous connections, too many users download directly from the main server or there is simply some other timeout! Why are there two breaks? Well, not all binaries are ready at the same time and the timestamp is for the whole mirror, so that you do not access an outdated mirror. The broken line of Yandex is also a bit odd, but that's because they have multiple servers and seem to switch between. 

How does it look right now? Here is the website for the mirror log visualization. There is also a link to how it looked last week.

Beyond just retrieving a file through http or ftp you can use a metalink file. Through a client it enables downloading from all available mirrors simultaneously and the client selects a few but fastest mirrors automatically. It also checks consistency and downloads are resumable. That's the ideal solution if you have a weaker connection and need to stop a huge transfer for some time or it is just flaky. Hint: use aria2 or DownThemAll. 

Ok, at last some words about hosting a mirror. Two things are important: sync often to avoid wasting time when you could transfer data, but only sync once at a time ;) I've written how i did set up a mirror (the Boston one) in a MirrorNetwork wiki page.


In case the main website is down, you can find a mirror here.


Thank's to all our mirror hosters! Without them Sage would probably never reach their users and new mirrors are always welcome, especially in southern asia!