Slow Wiki Uploads

Note: The official documentation for this issue is on the PERT Ticket System (PTS). This page has been created to provide convenient storage for composite documents, such as text referring to multiple graphs.

Two-end packet traces

We have performed some tests with packet traces (Ethereal or tcpdump to a file) running on both endpoints. The idea is that the traces can be correlated to see where packets are lost or delayed.

Unfortunately the packet traces don't have the same time base. The receiving end (cemp1.switch.ch) is NTP-synchronized, but the sending end (TobyRodwell's laptop) isn't. The time difference is about 2:30 minutes.

Example 1: Slow upload

• conn-1107-dante.pcap: slow connection, seen at sender
• conn-1107-switch.pcap: slow connection, seen at receiver

Example 2: Fast upload

• conn-1102-dante.pcap: fast connection, seen at sender
• conn-1102-switch.pcap: fast connection, seen at receiver

Jabber Transcripts

These transcripts contain conversation between the testers at both ends: TobyRodwell and SimonLeinen. The timestamps can be used to relate the different parts of the traces.

• 2005-09-15-jabber-log.html: Thu, 15-Sep-2005 Jabber log
• 2005-09-16-jabber-log.html: Fri, 16-Sep-2005 Jabber log

Processing packet traces

The following sections contain some information in the hope that it is useful for people who want to analyze packet traces like this. Eventually this kind of information should be moved to the PERT Knowledge Base, but I first have to find a nice place for it there.

Splitting a trace into connections

The traces often contain multiple TCP connections (from several HTTP accesses), but we're only interested in the ones that correspond to uploads. So we should first split the traces into TCP connections. Here's one way to find the local port numbers of the outgoing TCP port 80 connections that contain more than 100 packets:

tcpdump -r toby-fri.pcap dst port 80 \
  | perl -e 'while (<>) { ++$pc{$port} if ($port) = /\.(\d+) > /; }
             foreach $port (sort keys %pc) {
               print $pc{$port},"\t",$port,"\n" if $pc{$port} > 100; }' | sort -n

The result looks like this:

219     1152
230     1094
436     1102
436     1145
447     1099
477     1088
819     1123

Now you can use the right-hand column to construct another command that extracts one .pcap file for each connection:

for eph_port in 1152 1094 1102 1145 1099 1088 1123
do
  tcpdump -r toby-fri.pcap -w conn-$eph_port-switch.pcap tcp port $eph_port
done

Bash one-liner to quickly compare to traces

If you use the bash shell under Unix, you can use the following to find whether two traces of the same connection (taken at different endpoints) contain identical packet sets:

diff -u <(tcpdump -r fri-09-58-dante.pcap | cut -c 17- | sort) \
        <(tcpdump -r fri-09-58-switch.pcap | cut -c 17- | sort)

If the traces contain the same packets, then the output will be empty. The packets can be ordered differently though (and typically will be), and the timestamps will certainly be different - not just because of clock differences, but also because of packet travel times. Therefore we have to cut off the timestamps and sort the packets before doing the comparison.

-- SimonLeinen - 16 Sep 2005