Network Performance Document
Linux Configure Jumbo Frames to Boost Network Performance / Throughput
Most modern Linux distros (read as Linux Kernel 2.6.17+) does support frames larger than 1500 bytes. This can improve the performance. First, make sure your network driver supports custom MTU. Second you need to have a compatible gigabit NIC

Jumbo frames can reduce server overhead such as a big decrease in CPU usage when transferring larger file. Also you should see some increase in network throughput.

# ifconfig eth0 mtu 9000

Make changes permanent

Edit the network configuration file for eth0 interface – for example, /etc/sysconfig/network-script/ifcfg-eth0 (CentOS / RHEL / Fedora Linux):

# vi /etc/sysconfig/network-script/ifcfg-eth0

Append the following configuration directive, which specifies the size of the frame in bytes:

MTU=9000

# /etc/init.d/networking restart

To confirm the MTU used between two specific devices. use ip command as follows:

# ip route get {IP-address}
# ip route get 192.168.1.1

Output:
192.168.1.1 dev eth0 src 192.168.1.100 cache mtu 9000 advmss 1460 hoplimit 64

Tuning sysctl.conf

The sysctl.conf of a server is something that is seldom optimized for performance. You can get a tremendous boost in throughput by adjusting these settings. This configuration has been written by Steve from Rack911. I have applied this configuration to servers ranging from Celeron 1.7Ghz to Dual Xeon 2.8Ghz servers, and on the whole, the load on each lowered after making the changes.

First make a backup of your old /etc/sysctl.conf file by running the following command, logged in as root:
cp /etc/sysctl.conf /etc/sysctl.conf.bak
Now enter:
vi /etc/sysctl.conf
and replace the contents of the file with the following:

# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and
# sysctl.conf(5) for more details.
# Disables packet forwarding
net.ipv4.ip_forward=0
# Disables IP source routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.lo.accept_source_route = 0
net.ipv4.conf.eth0.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
# Enable IP spoofing protection, turn on source route verification
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.lo.rp_filter = 1
net.ipv4.conf.eth0.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
# Disable ICMP Redirect Acceptance
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.lo.accept_redirects = 0
net.ipv4.conf.eth0.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
# Enable Log Spoofed Packets, Source Routed Packets, Redirect Packets
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.lo.log_martians = 0
net.ipv4.conf.eth0.log_martians = 0
# Disables IP source routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.lo.accept_source_route = 0
net.ipv4.conf.eth0.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
# Enable IP spoofing protection, turn on source route verification
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.lo.rp_filter = 1
net.ipv4.conf.eth0.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
# Disable ICMP Redirect Acceptance
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.lo.accept_redirects = 0
net.ipv4.conf.eth0.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
# Disables the magic-sysrq key
kernel.sysrq = 0
# Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout = 15
# Decrease the time default value for tcp_keepalive_time connection
net.ipv4.tcp_keepalive_time = 1800
# Turn off the tcp_window_scaling
net.ipv4.tcp_window_scaling = 0
# Turn off the tcp_sack
net.ipv4.tcp_sack = 0
# Turn off the tcp_timestamps
net.ipv4.tcp_timestamps = 0
# Enable TCP SYN Cookie Protection
net.ipv4.tcp_syncookies = 1
# Enable ignoring broadcasts request
net.ipv4.icmp_echo_ignore_broadcasts = 1
# Enable bad error message Protection
net.ipv4.icmp_ignore_bogus_error_responses = 1
# Log Spoofed Packets, Source Routed Packets, Redirect Packets
net.ipv4.conf.all.log_martians = 1
# Increases the size of the socket queue (effectively, q0).
net.ipv4.tcp_max_syn_backlog = 1024
# Increase the tcp-time-wait buckets pool size
net.ipv4.tcp_max_tw_buckets = 1440000
# Allowed local port range
net.ipv4.ip_local_port_range = 16384 65536
CTRL + X to exit and save the file
To make your changes take effect immediately, type this command:
/sbin/sysctl -p
You can watch your server load by entering “uptime” command via SSH.There you have it, a quick few things you can do to your server to boost performance and lower CPU load.
TCP Tuning
For servers that are serving up huge numbers of concurrent sessions, there are some TCP options that should probably be enabled. In order to optimize TCP performance for this situation, I would suggest tuning the following parameters:
#echo 1024 65000 > /proc/sys/net/ipv4/ip_local_port_range
Tuning Syn Backlog
Set the Syn backlog to a high number so Squid will never be starved due to a kernel limiting network connection. The default value is 1024.
#echo 16384 > /proc/sys/net/ipv4/tcp_max_syn_backlog
Set up Network Bonding
Linux allows binding multiple network interfaces into a single channel/NIC using special kernel module called bonding. The Linux bonding driver provides a method for aggregating multiple network interfaces into a single logical “bonded” interface.

TCP Tuning Guide
Linux TCP Tuning
There are a lot of differences between Linux version 2.4 and 2.6, so first we’ll cover the tuning issues that are the same in both 2.4 and 2.6. To change TCP settings in, you add the entries below to the file /etc/sysctl.conf, and then run “sysctl -p”.
Like all operating systems, the default maximum Linux TCP buffer sizes are way too small. I suggest changing them to the following settings:
# increase TCP max buffer size setable using setsockopt()
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
# increase Linux autotuning TCP buffer limits
# min, default, and max number of bytes to use
# set max to at least 4MB, or higher if you use very high BDP paths
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
Note: you should leave tcp_mem alone. The defaults are fine.
Another thing you can try that may help increase TCP throughput is to increase the size of the interface queue. To do this, do the following:
ifconfig eth0 txqueuelen 1000
I’ve seen increases in bandwidth of up to 8x by doing this on some long, fast paths. This is only a good idea for Gigabit Ethernet connected hosts, and may have other side effects such as uneven sharing between multiple streams.
Also, I’ve been told that for some network paths, using the Linux ‘tc’ (traffic control) system to pace traffic out of the host can help improve total throughput.
________________________________________
Linux 2.4
Starting with Linux 2.4, Linux has implemented a sender-side autotuning mechanism, so that setting the opitimal buffer size on the sender is not needed. This assumes you have set large buffers on the recieve side, as the sending buffer will not grow beyond the size of the recieve buffer.
However, Linux 2.4 has some other strange behavior that one needs to be aware of. For example: The value for ssthresh for a given path is cached in the routing table. This means that if a connection has has a retransmition and reduces its window, then all connections to that host for the next 10 minutes will use a reduced window size, and not even try to increase its window. The only way to disable this behavior is to do the following before all new connections (you must be root):
sysctl -w net.ipv4.route.flush=1
More information on various tuning parameters for Linux 2.4 are available in the Ipsysctl tutorial .
________________________________________
Linux 2.6
Starting in Linux 2.6.7 (and back-ported to 2.4.27), linux includes alternative congestion control algoritms beside the traditional ‘reno’ algorithm. These are designed to recover quickly from packet loss on high-speed WANs.
Linux 2.6 also includes and both send and receiver-side automatic buffer tuning (up to the maximum sizes specified above). There is also a setting to fix the ssthresh caching weirdness described above.
There are a couple additional sysctl settings for 2.6:
# don’t cache ssthresh from previous connection
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_moderate_rcvbuf = 1
# recommended to increase this for 1000 BT or higher
net.core.netdev_max_backlog = 2500
# for 10 GigE, use this
# net.core.netdev_max_backlog = 30000
Starting with version 2.6.13, Linux supports pluggable congestion control algorithms . The congestion control algorithm used is set using the sysctl variable net.ipv4.tcp_congestion_control, which is set to cubic or reno by default, depending on which version of the 2.6 kernel you are using.
To get a list of congestion control algorithms that are available in your kernel, run:
sysctl net.ipv4.tcp_available_congestion_control
The choice of congestion control options is selected when you build the kernel. The following are some of the options are available in the 2.6.23 kernel:
• reno: Traditional TCP used by almost all other OSes. (default)
• cubic: CUBIC-TCP
• bic: BIC-TCP
• htcp: Hamilton TCP
• vegas: TCP Vegas
• westwood: optimized for lossy networks
For very long fast paths, I suggest trying cubic or htcp if reno is not is not performing as desired. To set this, do the following:

sysctl -w net.ipv4.tcp_congestion_control=htcp
More information on each of these algorithms and some results can be found here .
More information on tuning parameters and defaults for Linux 2.6 are available in the file ip-sysctl.txt, which is part of the 2.6 source distribution.
And finally a warning for both 2.4 and 2.6: for very large BDP paths where the TCP window is > 20 MB, you are likely to hit the Linux SACK implementation problem. If Linux has too many packets in flight when it gets a SACK event, it takes too long to located the SACKed packet, and you get a TCP timeout and CWND goes back to 1 packet. Restricting the TCP buffer size to about 12 MB seems to avoid this problem, but clearly limits your total throughput. Another solution is to disable SACK.
________________________________________
Linux 2.2
If you are still running Linux 2.2, upgrade! If this is not possible, add the following to /etc/rc.d/rc.local
echo 8388608 > /proc/sys/net/core/wmem_max
echo 8388608 > /proc/sys/net/core/rmem_max
echo 65536 > /proc/sys/net/core/rmem_default
echo 65536 > /proc/sys/net/core/wmem_default

Network Monitoring Tools

http://acs.lbl.gov/NCS/download/