Effect of TCP Buffer Size and Window Size for One Rapid Flow
Machines running rapid flows: flow from smithfd-linux1 to smithfd-linux2
Rapid Version: rapid_v0.9
Network Paths: the routed path in netlab
TCP Configuration: For the sender smithfd-linux1: net.core.wmem_max=16777216, net.ipv4.tcp_wmem=4096 16386 16777216; for the receiver smithfd-linux2, net.core.rmem_max=16777216, net.ipv4.tcp_rmem=4096 16386 16777216
List of Experiments:
The experiments contain three steps. In step one, run the single rapid flow with default MAX_IN_FLIGHT value and let linux autotune the send and receive buffers. Try cubic for comparison.
In step two, with the default window size MAX_IN_FLIGHT=2000, increase tcp buffer size:
single rapid flow with 0ms delay (BDP=12.2K), with -w 128K/256K/512K/1M/2M/4M/6M/8M, MAX_IN_FLIGHT=2000
single rapid flow with 10ms delay (BDP=1.2M), with -w 128K/256K/512K/1M/2M/4M/6M/8M, MAX_IN_FLIGHT=2000
single rapid flow with 30ms delay (BDP=3.6M), with -w 128K/256K/512K/1M/2M/4M/6M/8M, MAX_IN_FLIGHT=2000
single rapid flow with 50ms delay (BDP=6M), with -w 128K/256K/512K/1M/2M/4M/6M/8M, MAX_IN_FLIGHT=2000
In step three, for those experiments with long delays where the throughput failed to achieve its best, increase congestion window size by increase MAX_IN_FLIGHT and repeat the above experiments. Try MAX_IN_FLIGHT=4000 and 8000.
2. Experimental Results:
Throughputput of the single rapid/cubic flow with linux autotuning buffer size:
Single rapid flow:
0ms 2000entry 889
10ms 2000entry 580
30ms 2000entry 71.8
30ms 4000entry 62
50ms 2000entry 16.6
50ms 4000entry 104
50ms 8000entry 65.1
Single cubic flow:
Increase send/receiver buffers by different "-w" options in iperf
128K 256K 512K 1M 2M 4M 6M 8M
0ms 413 691 904 911 899 900 892
10ms 69.3 127 249 464 885 882 876
30ms 29.2 56.1 108 205 364 543 563 554
50ms 18.4 37.5 72.3 140 276 373 361 364
If tcp send/receive buffer is large enough for Rapid flow, it manages to achieve the best throughput. For flows with delays 30ms or 50ms, the throughput was limited by congestion window size which was 2000 segments.
Increase congestion window size: MAX_IN_FLIGHT
Throughput of one rapid flow with 30ms delay with MAX_IN_FLIGHT=2000 and MAX_IN_FLIGHT=4000
Throughput of one rapid flow with 50ms delay with MAX_IN_FLIGHT=2000, MAX_IN_FLIGHT=4000 and MAX_IN_FLIGHT=8000
In rapid_v0.9, MAX_IN_FLIGHT indicates congestion window size. So it cwin is large enough for rapid, it successfully achieve the best throughput.
Linux autotuning fails work for rapid
Rapid requires enough send/receive buffers. It should be at least BDP.
In rapid code, congestion window size should be large enought. It should be at least BDP.
Probe array size MAX_IN_FLIGHT should be independent of BDP because large MAX_IN_FLIGHT takes too much memory space if it corresponds to cwin size. Constant MAX_IN_FLIGHT may lead to overwriting of early probe streams, which is reasonable indicating that latest probes provide the most useful information.