Effect of TCP Buffer Size and Window Size for One Rapid Flow

1. Description:

Machines running rapid flows: flow from smithfd-linux1 to smithfd-linux2
Rapid Version: rapid_v0.9
Network Paths: the routed path in netlab
TCP Configuration: For the sender smithfd-linux1: net.core.wmem_max=16777216, net.ipv4.tcp_wmem=4096 16386 16777216; for the receiver smithfd-linux2, net.core.rmem_max=16777216, net.ipv4.tcp_rmem=4096 16386 16777216
List of Experiments:
  • The experiments contain three steps. In step one, run the single rapid flow with default MAX_IN_FLIGHT value and let linux autotune the send and receive buffers. Try cubic for comparison.
  • In step two, with the default window size MAX_IN_FLIGHT=2000, increase tcp buffer size:
  • single rapid flow with 0ms delay (BDP=12.2K), with -w 128K/256K/512K/1M/2M/4M/6M/8M, MAX_IN_FLIGHT=2000
  • single rapid flow with 10ms delay (BDP=1.2M), with -w 128K/256K/512K/1M/2M/4M/6M/8M, MAX_IN_FLIGHT=2000
  • single rapid flow with 30ms delay (BDP=3.6M), with -w 128K/256K/512K/1M/2M/4M/6M/8M, MAX_IN_FLIGHT=2000
  • single rapid flow with 50ms delay (BDP=6M), with -w 128K/256K/512K/1M/2M/4M/6M/8M, MAX_IN_FLIGHT=2000
  • In step three, for those experiments with long delays where the throughput failed to achieve its best, increase congestion window size by increase MAX_IN_FLIGHT and repeat the above experiments. Try MAX_IN_FLIGHT=4000 and 8000.

    2. Experimental Results:

  • Throughputput of the single rapid/cubic flow with linux autotuning buffer size:
    Single rapid flow:
    0ms 2000entry		889		
    10ms 2000entry		580					
    30ms 2000entry		71.8						
    30ms 4000entry		62							
    50ms 2000entry		16.6
    50ms 4000entry		104
    50ms 8000entry		65.1
    
    Single cubic flow:
    0ms	941
    10ms	933
    30ms	915
    50ms	899
    
  • Increase send/receiver buffers by different "-w" options in iperf
    	128K	256K	512K	1M	2M	4M	6M	8M
    0ms	413	691	904	911	899	900	892
    10ms	69.3	127	249	464	885	882	876
    30ms	29.2	56.1	108	205	364	543	563	554
    50ms	18.4	37.5	72.3	140	276	373	361	364
    

    If tcp send/receive buffer is large enough for Rapid flow, it manages to achieve the best throughput. For flows with delays 30ms or 50ms, the throughput was limited by congestion window size which was 2000 segments.

  • Increase congestion window size: MAX_IN_FLIGHT
  • Throughput of one rapid flow with 30ms delay with MAX_IN_FLIGHT=2000 and MAX_IN_FLIGHT=4000
  • Throughput of one rapid flow with 50ms delay with MAX_IN_FLIGHT=2000, MAX_IN_FLIGHT=4000 and MAX_IN_FLIGHT=8000
    In rapid_v0.9, MAX_IN_FLIGHT indicates congestion window size. So it cwin is large enough for rapid, it successfully achieve the best throughput.

    3. Conclusion:

  • Linux autotuning fails work for rapid
  • Rapid requires enough send/receive buffers. It should be at least BDP.
  • In rapid code, congestion window size should be large enought. It should be at least BDP.
  • Probe array size MAX_IN_FLIGHT should be independent of BDP because large MAX_IN_FLIGHT takes too much memory space if it corresponds to cwin size. Constant MAX_IN_FLIGHT may lead to overwriting of early probe streams, which is reasonable indicating that latest probes provide the most useful information.