Bus architecture and performance

Network (and other) interfaces attach to the motherboard using some kind of interconnect. Typically a motherboard has a couple of bus media, e.g., a number of PCI, AGP, PCI-X or similar slots. How different adapters attach to these interconnects has significant performance implications.

Maximum performance of some buses

• ISA 8bit /8Mhz: 8 MB/sec
• ISA 16bit/8Mhz: 16 MB/sec

• PCI 32bit/33Mhz: 133 MB/sec
• PCI 2.x 64bit/66Mhz: 532 MB/sec

• PCI-X 1.0 64bit/66Mhz: 532 MB/sec
• PCI-X 1.0 64bit/133Mhz: 1.07 GB/sec
• PCI-X 2.0 64bit/266Mhz: 2.13 GB/sec
• PCI-X 2.0 64bit/533Mhz: 4.26 GB/sec

All PCI and PCI-X buses are backward compatible, except latest models no longer support 5V adapters. Note that the slowest adapter in the bus determines the maximum speed of the bus. If multiple adapters are plugged into the same bus, the speed also decreases. Regardless of the numbers above, an adapter in 32bit/33Mhz PCI slot can typically only achieve ~300-400 Mbit/s TCP/IP performance.

Newer systems often include the PCI Express bus, which scales to higher speeds than classical PCI and also features smaller physical adapters. The first wide use of PCI Express in adapters was for graphics adapters, where high throughput is critical. More recently, vendors of network adapters have also started to use PCI Express, in particular for Gigabit Ethernet and 10 Gigabit Ethernet interfaces.

• PCI Express x1: 400 MB/sec
• PCI Express x2: 800 MB/sec
• PCI Express x4: 1.6 GB/sec
• PCI Express x8: 3.2 GB/sec
• PCI Express x16: 6.4 GB/sec

Measuring BUS performance

There are tools to measure I/O performance of PCI buses. For example, at least on Linux, "pcitop" utility may be of great help.

Common Issues: Bus support and adapter placement

Check what bus types and speeds a network adapter supports. Verify how many distinct channels the motherboard supports. Verify that only the network adapter is plugged into a bus so that the performance is not degraded by slower or multiple cards.

Common Issues: Memory read registers

The default operating system/driver's or BIOS's Maximum Memory Read Byte Count (MMRBC) might be set to e.g., 512 bytes. A higher value (e.g., 4096 bytes) could even double the performance in 10 Gbit/s grade tests (in one case, from 2Gbit/s to 4 Gbit/s). The value may in some cases be set from BIOS, but in others must be adjusted in OS's PCI-X settings (e.g., 'setpci' on Linux).

It's not clear whether this is also the same thing, in some cases adjusting "PCI payload size" in BIOS made a huge difference, especially when using small packets. This is because the overhead in PCIe was significant. See more on the netdev post referred below.

References

• http://en.wikipedia.org/wiki/Industry_Standard_Architecture
• http://en.wikipedia.org/wiki/PCI-X
• http://en.wikipedia.org/wiki/Pci_express
• http://www.intel.com/support/network/adapter/pro100/sb/cs-008400.htm
• http://pcitop.berlios.de/
• Thread: "what is the expeted performance from a dual port 10G card ?" (sic) on netdev@vger.kernel.org, Nov 2008

-- PekkaSavola - 15 Nov 2006
-- SimonLeinen - 06 May 2007
-- PekkaSavola - 15 Aug 2008, 01 Dec 2008