Networking
Various kernel parameters affect network settings. These kernel parameters can be inspected and modified using the sysctl tool or the files under /proc/sys/. Below, they are referred to using the sysctl name.
Socket Parameters
The following are some of the settings that affect socket networking parameters for all protocols.
net.core.rmem_default
net.core.wmem_default
net.core.rmem_max
net.core.wmem_max
The rmem parameters refer to the socket receive buffer size and the wmem refer to the send buffer size. Under TCP socket parameters, a sending host will typically need to buffer data to support the bandwidth delay product (BDP) of the link, i.e. enough memory for all bits in flight; the higher the RTT of the link, the more memory that is required (see the TCP Throughput Calculator [TTC] resource for an example of a BDP and buffer size calculator). As the names imply, those ending in default set the default value used, and those ending in max set the maximum value that can be requested by a program. A privileged program can set a buffer size beyond the maximum value [socket].
TCP Socket Parameters
The following parameters affect socket sizes for the Transmission Control Protocol (TCP) protocol in particular.
net.ipv4.tcp_mem
net.ipv4.tcp_rmem
net.ipv4.tcp_wmem
The tcp_rmem and tcp_wmem parameters are similar to the socket parameters in that they influence the socket buffer size but they are set differently. Each takes three values: a minimum, a default and a maximum. The default value overrides the value set by rmem_default and wmem_default. The maximum values do not override the settings of rmem_max and wmem_max. The minimum and maximum values set the range in which TCP can dynamically adjust the buffer size. This has no effect on the buffer size that a program can request. The setting tcp_mem affects how TCP manages its memory usage. It takes three values: low, pressure and high. Below the value of low, TCP will stop regulating memory usage (if it was previously). Above the value of pressure, TCP will start regulating its memory usage. The high value is the maximum amount of memory that TCP will use. These are not set in bytes but in the number of pages of memory. They refer to global memory usage not individual sockets [tcp].
Typically, larger buffer sizes are better for higher throughput or data transfers with a high round-trip-time. However, unnecessarily high values will waste RAM.
Another TCP parameter which can have a significant effect on performance is the congestion control algorithm used. The following parameters influence which algorithm is used.
net.ipv4.tcp_available_congestion_control
net.ipv4.tcp_allowed_congestion_control
net.ipv4.tcp_congestion_control
The tcp_available_congestion_control parameter is read-only and shows what algorithms are available on the system. The tcp_congestion_control parameter sets the default algorithm used; the Fasterdata site recommends htcp, but the newer TCP BBR developed by Google and being standardised in the IETF is certainly worth considering due to its greater resilience to packet loss). A program may instead use any of the algorithms set in tcp_allowed_congestion_control and a privileged program can use any of the available algorithms [tcp].
In order for a particular algorithm to be available it may require that a module is loaded into the kernel. While choosing the right congestion control algorithm can improve performance it also affects fairness. On shared links, some algorithms may negatively impact the network performance of other systems. For example, TCP BBR is more “aggressive” and thus may dominate some traditional TCP algorithms.
The ss command can be used to get detailed information about current TCP connections including what congestion control algorithm they are using:
ss --tcp --info
NIC Driver Settings
In addition to kernel parameters, network performance can be tuned by adjusting settings that affect the actual network hardware. NIC manufacturers may supply proprietary tools to adjust settings. However, some network driver settings can be modified using the ethtool command. Two important settings that should be looked at are the size of the send and receive rings and the offload features.
ethtool --show-ring INTERFACE
ethtool --show-features INTERFACE
Larger rings typically improve throughput but can have a negative impact on latency or jitter. Offloading features allow some work that would need to be done by the CPU to be done on the NIC instead. This can lead to lower CPU utilization and higher throughput.
MTU Settings
The Maximum Transmission Unit (MTU) size can have a significant effect on throughput. For most typical Ethernet networks, the default MTU is 1500 bytes. This means each Ethernet frame sent will have a maximum size of 1500 bytes. However, some networks may support larger MTUs, often up to 9000 bytes. Where such MTUs are available end to end, greater throughput can be achieved; see the TCP Throughput Calculator [TTC] to explore the effect. A higher MTU also puts less pressure on devices on the path in terms of their requirements to support high packet per second processing.
Many NREN backbone networks support 9000 MTU, some support slightly higher to allow for some framing overhead. Where a Science DMZ network is connected directly on to an NREN backbone through its site router, it may be possible to configure some or all of the Science DMZ to a higher MTU.
In Appendix A, the example tuning for the GTS DTN setup uses an MTU of 8986 bytes, using a simple ifconfig to set the MTU value.
