Setting Socket Buffer Sizes

NOTE: Do NOT use kernel.DIRT when trying to modify send or receive buffers larger than 16KB. Instead, use one of the ALTQ kernels, which includes a bug fix for window sizes (see below).

Modifying Send/Receive Buffer

There are two ways modify the send and/or receive buffer on a host:

If the socket options aren't set, then the values in net.inet.tcp.recvspace and net.inet.tcp.sendspace will be used.

The maximum size of a socket send/recv buffer is limited by two more variables, kern.ipc.maxsockbuf and kern.ipc.nmbclusters. If an error message like "no buffer space available" is received when trying to create socket with very large send or receive buffers, most likely you'll need to increase the maximum size of the socket buffer. Setting kern.ipc.maxsockbuf=1048576 will give a maximum socket send/recv buffer of 1M. The total amount of memory given to *all* sockets is controlled by kern.ipc.nmbclusters. If you plan on creating many sockets with large buffers, you'll want to increase kern.ipc.nmbclusters. Unfortunately, this cannot be resized while the machine is running. There are two options for changing kern.ipc.nmbclusters, the first (and more drastic) is adding the line options NMBCLUSTERS=<size> to the kernel config file and recompiling the kernel. The second involves adding the line kern.ipc.nmbcluster="<size>" to the file /boot/loader.conf and rebooting the system. This increase the total number of Mbufs (the data structure underlying socket send and receive buffers) allowing for larger window sizes and more sockets in general.

ALTQ Window Size Fix

In [my kernel]/netinet/in_rmx.c, there's a section of code where tcp_sendspace and tcp_recvspace are set to values provided by the routing mechanism. Commenting out this code allows TCP windows to grow past 16KB. This fix is present in all ALTQ kernels (i.e., it's included in the patch).
// in_rmx.c
	 * We also specify a send and receive pipe size for every
	 * route added, to help TCP a bit.  TCP doesn't actually
	 * want a true pipe size, which would be prohibitive in memory
	 * costs and is hard to compute anyway; it simply uses these
	 * values to size its buffers.  So, we fill them in with the
	 * same values that TCP would have used anyway, and allow the
	 * installing program or the link layer to override these values
	 * as it sees fit.  This will hopefully allow TCP more
	 * opportunities to save its ssthresh value.
#if !defined(ALTQ)  /* commented out for the tcp window problem  --kjc */
	if (!rt->rt_rmx.rmx_sendpipe && !(rt->rt_rmx.rmx_locks & RTV_SPIPE))
		rt->rt_rmx.rmx_sendpipe = tcp_sendspace;

	if (!rt->rt_rmx.rmx_recvpipe && !(rt->rt_rmx.rmx_locks & RTV_RPIPE))
		rt->rt_rmx.rmx_recvpipe = tcp_recvspace;

Other DiRT documents
Author: Michele Clark / Andy Jones