TCP Tuning

Introduction This page serves as a basic introduction into the parameters that can be tweaked to improve TCP performance, the tools that can be used to perform those tweaks, and basic settings that should be an improvement over linux defaults. This is based primarily upon the work of Brian Tierney. You can visit his TCP-Tuning page at the Lawrence Berkeley National Laboratory (http://dsd.lbl.gov/TCP-tuning/background.html). This HOWTO is divided in order of experience. If you know about how TCP works, skip that section as it is very basic. Similarly, the /proc/sys and sysctl sections are basic introductions to those facilities.  Background  Brief Introduction to TCP TCP stands for Transmission Control Protocol. It is the most visible networking protocol for end users, because it is the transport protocol for HTTP, FTP, GnuTella, SSH, and most application-layer protocols. It provides a "stream-like" abstraction over the complicated links in a computer network or the Internet. Some of the services provided by TCP are:  Reliable Data Delivery In-Order Data Delivery Error-free Data Delivery Congestion Control  Why TCP needs to be tweaked The last feature of TCP listed above, congestion control, can limit the performance of a TCP connection due to innappropriately configured protocol parameters. The main issue is the TCP "congestion window," a number representing the maximum amount of data that can be in-transit over the network at any one time. If this number is too low, the kernel will not transmit data, even when the link is uncongested.  How to solve this problem This can occur when the sender or receiver have unreasonably small buffer sizes. These buffers are the amount of memory on the computer that is reserved to hold data just received or data that has yet to be transmitted.  Receiver Buffer If the receiver's receive buffer is too small, the sender will stop sending information even when the link has unused bandwith. The biggest improvement that can be made is to increase the default and maximum size of your receive window. This can greatly increase your TCP performance, often enough that you can visibly notice the difference. However, you need to be careful not to make the receive buffer too large, or the receiver can become flooded with packets if the sender can transmit faster than the receiver can receive.  Transmit Buffer There is little harm in increasing the size of your transmission buffer, as long as you have sufficient RAM. This may also increase performance slightly, as it reduces overhead of transferring data from the Application Layer to the Transport Layer of the network protocol stack. <span class="mw-headline"> Minimum, Default, and Maximum values The current linux kernels support dynamic sizing of these buffers. This means that the buffers change in size to attempt to fit the current TCP connection. This also means that each of these buffers has a minimum, default, and maximum value. As can be expected, the kernel will never decrease the buffer size below the minum, or above the maximum, and the buffer starts at the default size. <span class="mw-headline"> Practical Stuff <span class="mw-headline"> /proc/sys/ The proc filesystem is required by the Gentoo Handbook, so you should already have it on your Gentoo system. If you've never paid any attention to it, it is mounted at <font color="green" style="font-family: monospace;">/proc and contains all sorts of information about the current state of the kernel and drivers. The subdirectory <font color="green" style="font-family: monospace;">/proc/sys contains files and directories related to kernel variables. These can be written to by the root user to alter the variables, even while the kernel is running! The relevant directories for tuning TCP are <font color="green" style="font-family: monospace;">/proc/sys/net/ipv4 and <font color="green" style="font-family: monospace;"> /proc/sys/net/core. The IPv4 directory contains settings for TCP/IP while the core directory contains settings for all networking protocols. I highly recommend that you look at the proc(5), ip(7), and tcp(7) man pages for information about the variables that can be changed. <span class="mw-headline"> sysctl "sysctl" is a program in sys-process/procps that allows you to view and alter kernel parameters on a running kernel. Any user can view the parameters, but you need to be root to set parameters. This is the tool you need to use to improve the kernel's TCP settings. A /proc/sys variable can be set using a command like: <div style="padding: 1px 2px 1px 2px; background-color: #32CD32; border: solid 1px #a0ffa0; font-size: -1">Tip: Don't use slashes with sysctl! Replace the '/' in the variable path with a period: '.'. Besides this, sysctl exactly follows the structure of the /proc/sys directory structure. This sets the variable "variable" to the new value "value." Note that this change is only for the running kernel. The variable will revert to its default setting upon a reboot. Obviously, it would be quite tedious to set many settings manually every time you boot your computer, so you can edit <font color="green" style="font-family: monospace;">/etc/sysctl.conf to provide settings to automatically be applied at every boot. The lines in this file use identical syntax to the sysctl parameters. <span class="mw-headline"> Variables to Change, and suggested Values <dl> net.core.rmem_max="16777216" This setting changes the maximum network receive buffer size to 16777216 bytes.</dl><dl> net.core.wmem_max="16777216"  The same thing for the send buffer</dl><dl> net.ipv4.tcp_rmem="4096 87380 16777216"  This sets the kernel's minimum, default, and maximum TCP receive buffer sizes. You might be surprised, seeing the maximum of 16M, that many Unix-like operating systems still have a maximum of 256K!</dl><dl> net.ipv4.tcp_wmem="4096 65536 16777216" A similar setting for the TCP send buffer. Note that the default value is a little lower. Don't worry about this, the send buffer size is less important than the receive buffer.</dl><dl> net.ipv4.tcp_no_metrics_save=1 This removes an odd behavior in the 2.6 kernels, whereby the kernel stores the slow start threshold for a client between TCP sessions. This can cause undesired results, as a single period of congestion can affect many subsequent connections. I recommend that you disable it.<div style="padding: .2em .3em; margin: .2em .2em; background-color: #ffc1c1; border: solid 2px #ff7070; font-size: 92%;">Warning: DO NOT USE THE SETTING ABOVE WITH 2.4 KERNELS. Instead, use net.ipv4.route.flush = 1. </dl><dl> net.ipv4.ip_default_ttl = &lt;&lt;something random between 60 and 100 &gt;&gt; I recommend putting a random number between 60 and 100 here because many attackers use the TTL (time to live) as a parameter for profiling your operating system. It is a simple way to very slightly increase your networking security.</dl> <span class="mw-headline"> Advanced <span class="mw-headline"> Congestion Control Algorithms The current default congestion algorithm on Gentoo seems to be BIC, and it should satisfy almost everyone's needs. However, the 2.6.13 kernels and above now support pluggable congestion control algorithms. These can be changed using the net.ipv4.tcp_congestion_control variable using sysctl. The available algorithms are:  BIC - The default on Gentoo</li> Reno - The classic TCP protocol. Most OSes use this.</li> highspeed - HighSpeed TCP: Sally Floyd's suggested algorithm</li> htcp - Hamilton TCP</li> hybla - For Satellite Links</li> scalable - Scalable TCP</li> vegas - Vegas TCP</li> westwood - Optimized for lossy networks</li></ul> There's a LWN Article (http://lwn.net/Articles/128681/) about TCP Congestion Algorithms. Many thanks to Brian Tierney for the list above. Note that you will need to add these algorithms to your kernel in order to use them. They can be found at: <div style="padding: .2em .3em; margin: .2em .2em; background-color: #c1ffc1; border: solid 1px #a0ffa0; font-size: 92%;">Note: It seems that in my kernel, 2.6.15, Reno is no longer even an option. This is because Reno is the default algorithm. If you unselect all others the kernel will use Reno. <div class="printfooter"> Retrieved from http://www.gentoo-wiki.info/Tuning_TCP