In this text, B means byte and b means bit, in acronyms as MB/s and Gb/s (megabyte per second and gigabit per second).
For the debian release, we also have the resources of cdimage.debian.org, which is a donation by HP and a modern PC server with an external disk array (HP DL360 + MSA30).
In addition to this we borrowed two HP DL145 (dual opteron, 8 GB of RAM) from the local supercomputing center at the university (HPC2N) as temporary machines for just the release. These have only a slow IDE drive internally, so delivering data from disk is not an option. But delivering data from RAM, they can easily saturate a gigabit ethernet uplink. These would serve as http offload hosts.
And of course, the network. Thanks to the university, the computer club has several gigabit ethernet ports available, from a switch that has dual gigabit uplink to the campus network and in turn to the swedish university network.
Software, we run AIX on ftp.acc.umu.se and Linux (Debian/Ubuntu) on the other hosts, with apache2 patched for LFS and the worker MPM. We typically tune apache to allow 2000-5000 concurrent downloads.
We tune the TCP settings according to my network performance guide to get resonable performance.
Before the release, we put in http redirect rules on cdimage.debian.org, pointing all downloaders to ftp.acc.umu.se, to leave cdimage.debian.org free to act as master node for all the mirrors that needed to get their copy.
After we had built all the images, we triggered mirrors and copied out the data to the temporary hosts, then we started distributing the load by means of http redirects to the different hosts. This worked pretty well, but as we had promised not to use more than 2Gbit/s we pushed more and more traffic off-campus during the day.
Soon there was also a rebuild of a fixed version, due to a bug in the first set. This also made the distribution hard to coordinate and our bandwidth graphs were not as impressive as they could have been.
Network statistics from stats.sunet.se for the entire university, "in" means in towards the core network, that is, downloads from our archive:
After talking to the network people, they were fine with us actually pushing the limit on the 2Gbit/s uplink to the computer club. So we did not need to push the traffic off-campus as we approached it, saturation was fine.
We also learnt the importance of verifiying that the isos you redirect to are actually there, in current versions, before adding redirects. Mistakes were made, late that night.
One thing that made the Ubuntu release easier than Debian is that they only released two to four (counting Kubuntu) CD images per architecture and only two architectures (i386 and amd64) had high demand. This meant that the data in high demand easily fit into the 16 GB of ram we had available.
Because I slept late, first ftp.acc.umu.se had to take all the load itself, which it did resonably fine to start with, but after a while the load on some of the cluster nodes became high enough for it to be rather unusable. Once we introduced the http redirects, the load quickly fell away to managable levels on the cluster.
The load balancing was ad-hoc by looking on how many concurrent downloads of the different CD images there were on ftp.acc.umu.se and the two HP servers, adding http redirects as needed. This ramped the network load up rapidly, and we were soon hitting the 2Gbit/s limit on our network uplink.
The day was spent with some small tunings to keep the load even, and not to put too much load on ftp.acc.umu.se, while making sure that the offload hosts had a small enough working set to keep it all in RAM. We kept the network load pinning our uplink all day, until it dropped off in the evening. While the load picked up in the days after, we didn't saturate our capacity again.
Network statistics from stats.sunet.se for the entire university, "in" means in towards the core network, that is, downloads from our archive:
Our network statistics for the offload hosts and cdimage.debian.org (Farbror in the graph, not actually serving Ubuntu CD images, but Debian CD images):
The 42TB total network traffic over the week around the Breezy release shown in this last graph is equivalent to about 70 thousand cd-images. We estimate that about 10-15 thousand cd-images were downloaded during the first day and about 100 thousand cd-images (60TB) during the week following the release.
As we found out later, our activities had both filled an uplink two "hops" up from our university, at the Nordic University Network level to the point of warnings going off and made some DoS-alarms go off at the national level. All part of a fun release.
To HP and HPC2N and PDC for the equipment used.
To Debian and Ubuntu for eager users, wanting downloads.