Pushing TCP a little farther on Linux

I’ve been seeing an interesting error on our servers over the past week.  Here’s the meat of what shows up in dmesg:

[4262686.822339] swapper: page allocation failure. order:1, mode:0×4020
[4262686.822394] Pid: 0, comm: swapper Not tainted 2.6.24-gentoo-r8_64-08.19.08 #1
[4262686.822474]
[4262686.822475] Call Trace:
[4262686.822548]  <IRQ>  [<ffffffff802555f9>] __alloc_pages+0x2e2/0x2fb
[4262686.822605]  [<ffffffff804f2ebb>] __netdev_alloc_skb+0×29/0×43
[4262686.822657]  [<ffffffff8021732c>] smp_apic_timer_interrupt+0×40/0×54
[4262686.822711]  [<ffffffff802725e4>] new_slab+0×63/0x22a
[4262686.822759]  [<ffffffff8027295d>] __slab_alloc+0x1b2/0x3ac
[4262686.822809]  [<ffffffff804f2ebb>] __netdev_alloc_skb+0×29/0×43
[4262686.822860]  [<ffffffff804f2ebb>] __netdev_alloc_skb+0×29/0×43
[4262686.822911]  [<ffffffff8027399b>] __kmalloc_node_track_caller+0x8c/0xc8
[4262686.822965]  [<ffffffff804f22e7>] __alloc_skb+0×69/0×132
[4262686.823014]  [<ffffffff804f2ebb>] __netdev_alloc_skb+0×29/0×43
[4262686.823070]  [<ffffffff8044b1b7>] tg3_alloc_rx_skb+0xc4/0×150
[4262686.823124]  [<ffffffff80452bf3>] tg3_poll+0×393/0x91c
[4262686.823175]  [<ffffffff80270d40>] get_partial_node+0×15/0x7e
[4262686.823228]  [<ffffffff804f8c1f>] net_rx_action+0x6a/0x18a
[4262686.823279]  [<ffffffff80230762>] __do_softirq+0×55/0xc4
[4262686.823329]  [<ffffffff8020c72c>] call_softirq+0x1c/0×28
[4262686.823378]  [<ffffffff8020e336>] do_softirq+0x2c/0x7d
[4262686.823427]  [<ffffffff802306c8>] irq_exit+0x3f/0×84
[4262686.823475]  [<ffffffff8020e43d>] do_IRQ+0xb6/0xd5
[4262686.823522]  [<ffffffff8020a947>] mwait_idle+0×0/0×45
[4262686.823570]  [<ffffffff8020bab1>] ret_from_intr+0×0/0xa
[4262686.823618]  <EOI>  [<ffffffff8020a989>] mwait_idle+0×42/0×45
[4262686.823670]  [<ffffffff8020a8db>] cpu_idle+0x6f/0×96

To summarize, the network driver is allocating socket buffers in the kernel so it can move data around.  For some reason, it is failing.  A brief look on the system uncovers 2 possibly interesting paramters:

# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: off

-and-

# sysctl vm.min_free_kbytes
vm.min_free_kbytes = 11495

TCP Segmentation Offload, or TSO is a clever way of reducing CPU usage.  Usually, a NIC operates with a standard MTU of 1500 bytes, and so the kernel sends data to the NIC in MTU-sized chunks.  With TSO enabled (assuming the NIC and driver support it), the kernel will send a much larger packet to the NIC, and let the network processor take care of splitting that large packet down into proper MTU-sized chunks for the network.  I believe this in combination with a rather low VM reclaimation value could result in an out-of-memory situation under certain conditions.  I’ll see if I can put this theory to the test and report back on it later…

06.20.10 – Update
Just thought I’d post a little update. After making these changes, the error no longer occurs. If you’re seeing the same problems, hopefully this may help you.