Slow downloads from download.microsoft.com

A customer complained about abysmally slow downloads of applications from Microsoft.

Upon our requests for specific information, they said that everything under download.microsoft.com seems to be concerned. Downloads from this site often run at only 4-5 kilobytes per second, although the site has a 100 Mb/s connection to the SWITCH backbone. The customer didn't tell us their IP address, but explained that they work behind the site's firewall (possibly behind a NAT).

A first test downloading one of the applications (.NET Framework Version 1.1 Redistributable Package) from a well-connected "reference" host near the complaining site resulted almost 1000 times higher transfer rates:

: leinen@lsmp2[leinen]; wget http://download.microsoft.com/download/a/a/c/aac39226-8825-44ce-90e3-bf8203e74006/dotnetfx.exe
--13:44:16--  http://download.microsoft.com/download/a/a/c/aac39226-8825-44ce-90e3-bf8203e74006/dotnetfx.exe
           => `dotnetfx.exe'
Resolving download.microsoft.com... 195.176.255.136, 195.176.255.135
Connecting to download.microsoft.com[195.176.255.136]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 24,265,736 [application/octet-stream]

100%[====================================>] 24,265,736     4.49M/s    ETA 00:00

13:44:22 (3.54 MB/s) - `dotnetfx.exe' saved [24265736/24265736]

In this test, the hostname download.microsoft.com was mapped to an address (195.176.255.136) within SWITCH's own network. This address belongs to one of several servers of the Akamai ContentDistributionNetwork, which is apparently used by Microsoft to host downloadable applications. A traceroute from the reference host shows that the route is very short:

: leinen@lsmp2[leinen]; traceroute download.microsoft.com
traceroute: Warning: download.microsoft.com has multiple addresses; using 195.176.255.135
traceroute to a767.ms.akamai.net (195.176.255.135), 30 hops max, 38
byte packets 1  swiLST76-G2-8 (130.59.35.45)  0.223 ms  0.163 ms 0.156 ms
 2  swiEZT76-G2-2 (130.59.35.57)  3.624 ms  3.568 ms  3.558 ms
 3  swiCS3-G3-14 (130.59.15.93)  3.646 ms  3.584 ms  3.570 ms
 4  a195-176-255-135.deploy.akamaitechnologies.com (195.176.255.135) 3.589 ms 3.522 ms  3.535 ms

The first thought on seeing this speed discrepancy is that the complaining site might be experiencing a local problem, such as bad cabling, overloaded firewalls, or protocol problems induced by middleboxes.

However, another test from the reference host resulted in vastly slower performance:

: leinen@lsmp2[leinen]; wget http://download.microsoft.com/download/a/a/c/aac39226-8825-44ce-90e3-bf8203e74006/dotnetfx.exe
--13:53:29--  http://download.microsoft.com/download/a/a/c/aac39226-8825-44ce-90e3-bf8203e74006/dotnetfx.exe
           => `dotnetfx.exe.1'
Resolving download.microsoft.com... 212.162.0.30
Connecting to download.microsoft.com[212.162.0.30]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 24,265,736 [application/x-msdownload]

12% [===>                                 ] 2,968,483      7.37K/s    ETA 51:32

What happened here is that the DNS mapping of download.microsoft.com suddently changed to 212.162.0.30, an alternative download server outside SWITCH's network, which seems to be overloaded. Traceroute shows a decent-looking path with stable delays, so we assume that the overload is in fact located on the file server (too many parallel downloads?).

: leinen@lsmp2[leinen]; traceroute download.microsoft.com
traceroute to download.microsoft.com.c.footprint.net (212.162.0.30), 30 hops max, 38 byte packets
 1  swiLST76-G2-8 (130.59.35.45)  0.233 ms  0.162 ms  0.157 ms
 2  swiLS2-10GE-1-4 (130.59.35.49)  0.199 ms  0.179 ms  0.182 ms
 3  swiCE2-G2-3 (130.59.36.49)  1.201 ms  1.105 ms  1.103 ms
 4  swiCE3-10GE-1-4 (130.59.36.210)  1.225 ms  1.161 ms  1.150 ms
 5  so-1-1-0.ar2.CDG2.gblx.net (64.213.33.221)  26.738 ms  26.792 ms  26.969 ms
 6  so1-0-0-2488M.ar1.CDG3.gblx.net (67.17.75.206)  26.911 ms so0-0-0-2488M.ar1.CDG3.gblx.net (67.17.75.202)  26.874 ms so1-0-0-2488M.ar1.CDG3.gblx.net (67.17.75.206)  26.939 ms
 7  ge-3-2.core1.Paris1.Level3.net (4.68.127.97)  22.975 ms  23.042 ms  23.028 ms
 8  ae-0-18.mp2.Paris1.Level3.net (212.73.240.114)  23.572 ms  23.462 ms  23.450 ms
 9  ae-1-0.bbr2.Frankfurt1.Level3.net (212.187.128.29)  25.255 ms  25.295 ms so-1-0-0.bbr1.Frankfurt1.Level3.net (212.187.128.33)  25.325 ms
10  ge-11-2.ipcolo1.Frankfurt1.Level3.net (195.122.136.115)  25.277 ms ge-10-2.ipcolo1.Frankfurt1.Level3.net (195.122.136.99)  22.344 ms ge-10-0.ipcolo1.Frankfurt1.Level3.net (195.122.136.7)  25.325 ms
11  212.162.0.2 (212.162.0.2)  25.556 ms  25.605 ms  25.519 ms

Further experiments uncovered many different servers to which download.microsoft.com can be mapped, including some in the US (which worked reasonably well at around 25 KB/s).

We told the customer that Microsoft's global server load sharing sometimes yields very badly performing servers even when good servers (such as the Akamai installation at SWITCH) are available. This is something that we cannot really improve, but the customer - who is also a Microsoft customer - will notify Microsoft Switzerland about this.

As an ugly short-time workaround, we suggested that the customer could map download.microsoft.com to an address of a767.g.akamai.net locally, which should force it to one of the Akamai servers at SWITCH. But besides breaking the globally-unique tree of the DNS, this could result in all kinds of bad side-effects once the hosting arrangement for download.microsoft.com changes somehow.

-- SimonLeinen - 19 Apr 2005