Setting Socket Buffer Sizes

Modifying Send/Receive Buffer

There are two ways modify the send and/or receive buffer on a host:

• For every connection on a host -- use sysctl to modify the net.inet.tcp.sendspace and/or net.inet.tcp.recvspace variables.
• For a single connection -- use setsocktopt() in the application that creates the socket. The option must be set on the listening socket before the listen() call is made. This makes sense, because the window scale option is agreed upon in the SYN packets which are exchanged in the listen() call.

If the socket options aren't set, then the values in net.inet.tcp.recvspace and net.inet.tcp.sendspace will be used.

The maximum size of a socket send/recv buffer is limited by two more variables, kern.ipc.maxsockbuf and kern.ipc.nmbclusters. If an error message like "no buffer space available" is received when trying to create socket with very large send or receive buffers, most likely you'll need to increase the maximum size of the socket buffer. Setting kern.ipc.maxsockbuf=1048576 will give a maximum socket send/recv buffer of 1M. The total amount of memory given to *all* sockets is controlled by kern.ipc.nmbclusters. If you plan on creating many sockets with large buffers, you'll want to increase kern.ipc.nmbclusters. Unfortunately, this cannot be resized while the machine is running. There are two options for changing kern.ipc.nmbclusters, the first (and more drastic) is adding the line options NMBCLUSTERS=<size> to the kernel config file and recompiling the kernel. The second involves adding the line kern.ipc.nmbcluster="<size>" to the file /boot/loader.conf and rebooting the system. This increase the total number of Mbufs (the data structure underlying socket send and receive buffers) allowing for larger window sizes and more sockets in general.

ALTQ Window Size Fix

// in_rmx.c
	/*
	 * We also specify a send and receive pipe size for every
	 * route added, to help TCP a bit.  TCP doesn't actually
	 * want a true pipe size, which would be prohibitive in memory
	 * costs and is hard to compute anyway; it simply uses these
	 * values to size its buffers.  So, we fill them in with the
	 * same values that TCP would have used anyway, and allow the
	 * installing program or the link layer to override these values
	 * as it sees fit.  This will hopefully allow TCP more
	 * opportunities to save its ssthresh value.
	 */
#if !defined(ALTQ)  /* commented out for the tcp window problem  --kjc */
	if (!rt->rt_rmx.rmx_sendpipe && !(rt->rt_rmx.rmx_locks & RTV_SPIPE))
		rt->rt_rmx.rmx_sendpipe = tcp_sendspace;

	if (!rt->rt_rmx.rmx_recvpipe && !(rt->rt_rmx.rmx_locks & RTV_RPIPE))
		rt->rt_rmx.rmx_recvpipe = tcp_recvspace;
#endif