Rapid_v0.9.2: Turn Off Interrupt Coalescence

Turn Off Interrupt Coalescence

According to the experiments results in previous experiments, two suggestions are made:

  • 1. Make scheduled send gap no smaller than the gap the link can create
  • 2. The big issue is interrupt latencies on the receiver side. This might be caused by interrupt coalescence on the NIC card or CPU scheduling latency. So turn off interrupt coalescence on the NIC card.

    Experiments Description:

    one rapid flow from smithfd-rapid22 to smithfd-rapid1 via the private network lasts for 5s. During 3-5s UDP cross traffic starts from smithfd-rapid6 to smithfd-rapid3 to share the link with the rapid flow.

    Configuration:

    Default sysctl parameters. Use commands "ethtool -C eth2 rx-usecs 0 rx-frames 1 rx-usecs-irq 0 rx-frames-irq 1" "ethtool -C eth2 tx-usecs 0 tx-frames 1 tx-usecs-irq 0 tx-frames-irq 1" to turn off the interrupt coalescence on the NIC cards on both end hosts.

    Experimental Results:

    1. Throughput:

    [  4]  0.0- 0.5 sec  22.2 MBytes   372 Mbits/sec
    [  4]  0.5- 1.0 sec  46.7 MBytes   784 Mbits/sec
    [  4]  1.0- 1.5 sec  47.9 MBytes   804 Mbits/sec
    [  4]  1.5- 2.0 sec  48.0 MBytes   806 Mbits/sec
    [  4]  2.0- 2.5 sec  47.9 MBytes   804 Mbits/sec
    [  4]  2.5- 3.0 sec  39.3 MBytes   659 Mbits/sec
    [  4]  3.0- 3.5 sec  20.7 MBytes   347 Mbits/sec
    [  4]  3.5- 4.0 sec  12.9 MBytes   216 Mbits/sec
    [  4]  4.0- 4.5 sec  16.3 MBytes   274 Mbits/sec
    [  4]  4.5- 5.0 sec  22.2 MBytes   372 Mbits/sec
    [  4]  5.0- 5.5 sec  16.1 MBytes   270 Mbits/sec
    [  4]  5.5- 6.0 sec  16.6 MBytes   279 Mbits/sec
    [  4]  6.0- 6.5 sec  47.0 MBytes   789 Mbits/sec
    [  4]  6.5- 7.0 sec  47.9 MBytes   804 Mbits/sec
    [  4]  7.0- 7.5 sec  48.0 MBytes   806 Mbits/sec
    [  4]  7.5- 8.0 sec  47.9 MBytes   803 Mbits/sec
    [  4]  8.0- 8.5 sec  47.0 MBytes   789 Mbits/sec
    [  4]  0.0- 8.7 sec   609 MBytes   588 Mbits/sec
    

    2. AB_estimation for each probe stream




    3. Gaps from one probe stream when cross traffic is on:

    id= 4035 send_gap= 92532 dag_gap= 71000 recv_gap= 70000 seqno= 882331416 cum_size= 2896
    id= 4035 send_gap= 92532 dag_gap= 86000 recv_gap= 86000 seqno= 882334312 cum_size= 2896
    id= 4035 send_gap= 92532 dag_gap= 93000 recv_gap= 93000 seqno= 882337208 cum_size= 2896
    id= 4035 send_gap= 86568 dag_gap= 86000 recv_gap= 87000 seqno= 882340104 cum_size= 2896
    id= 4035 send_gap= 80604 dag_gap= 81000 recv_gap= 81000 seqno= 882343000 cum_size= 2896
    id= 4035 send_gap= 80604 dag_gap= 80000 recv_gap= 80000 seqno= 882345896 cum_size= 2896
    id= 4035 send_gap= 80604 dag_gap= 81000 recv_gap= 80000 seqno= 882348792 cum_size= 2896
    id= 4035 send_gap= 75408 dag_gap= 75000 recv_gap= 76000 seqno= 882351688 cum_size= 2896
    id= 4035 send_gap= 70212 dag_gap= 71000 recv_gap= 70000 seqno= 882354584 cum_size= 2896
    id= 4035 send_gap= 70212 dag_gap= 70000 recv_gap= 70000 seqno= 882357480 cum_size= 2896
    id= 4035 send_gap= 70212 dag_gap= 70000 recv_gap= 70000 seqno= 882360376 cum_size= 2896
    id= 4035 send_gap= 65686 dag_gap= 72000 recv_gap= 73000 seqno= 882363272 cum_size= 2896
    id= 4035 send_gap= 61160 dag_gap= 56000 recv_gap= 55000 seqno= 882366168 cum_size= 2896
    id= 4035 send_gap= 61160 dag_gap= 60000 recv_gap= 60000 seqno= 882369064 cum_size= 2896
    id= 4035 send_gap= 61160 dag_gap= 61000 recv_gap= 61000 seqno= 882371960 cum_size= 2896
    id= 4035 send_gap= 57218 dag_gap= 59000 recv_gap= 61000 seqno= 882374856 cum_size= 2896
    id= 4035 send_gap= 53276 dag_gap= 64000 recv_gap= 62000 seqno= 882377752 cum_size= 2896
    id= 4035 send_gap= 53276 dag_gap= 41000 recv_gap= 44000 seqno= 882380648 cum_size= 2896
    id= 4035 send_gap= 53276 dag_gap= 54000 recv_gap= 54000 seqno= 882383544 cum_size= 2896
    id= 4035 send_gap= 49842 dag_gap= 63000 recv_gap= 60000 seqno= 882386440 cum_size= 2896
    id= 4035 send_gap= 46408 dag_gap= 33000 recv_gap= 32000 seqno= 882389336 cum_size= 2896
    id= 4035 send_gap= 46408 dag_gap= 59000 recv_gap= 63000 seqno= 882392232 cum_size= 2896
    id= 4035 send_gap= 46408 dag_gap= 33000 recv_gap= 30000 seqno= 882395128 cum_size= 2896
    id= 4035 send_gap= 43416 dag_gap= 50000 recv_gap= 46000 seqno= 882398024 cum_size= 2896
    id= 4035 send_gap= 40424 dag_gap= 43000 recv_gap= 48000 seqno= 882400920 cum_size= 2896
    id= 4035 send_gap= 40424 dag_gap= 37000 recv_gap= 35000 seqno= 882403816 cum_size= 2896
    id= 4035 send_gap= 40424 dag_gap= 44000 recv_gap= 49000 seqno= 882406712 cum_size= 2896
    id= 4035 send_gap= 37818 dag_gap= 37000 recv_gap= 29000 seqno= 882409608 cum_size= 2896
    id= 4035 send_gap= 35212 dag_gap= 27000 recv_gap= 32000 seqno= 882412504 cum_size= 2896
    id= 4035 send_gap= 35212 dag_gap= 49000 recv_gap= 49000 seqno= 882415400 cum_size= 2896
    id= 4035 send_gap= 35212 dag_gap= 25000 recv_gap= 30000 seqno= 882418296 cum_size= 2896
    id= 4035 send_gap= 32942 dag_gap= 49000 recv_gap= 43000 seqno= 882421192 cum_size= 2896
    id= 4035 send_gap= 30672 dag_gap= 25000 recv_gap= 24000 seqno= 882424088 cum_size= 2896
    id= 4035 send_gap= 30672 dag_gap= 37000 recv_gap= 39000 seqno= 882426984 cum_size= 2896
    id= 4035 send_gap= 30672 dag_gap= 37000 recv_gap= 34000 seqno= 882429880 cum_size= 2896
    id= 4035 send_gap= 28695 dag_gap= 24000 recv_gap= 25000 seqno= 882432776 cum_size= 2896
    id= 4035 send_gap= 26718 dag_gap= 37000 recv_gap= 37000 seqno= 882435672 cum_size= 2896
    

    4. netstat statistics

    279 segments dropped at the router.
    7011 segments got retransmited by the sender.
    0 packets rejects at the receiver.
    

    5. Summary of Results:

  • When interrupt coalescence is turned off, the interrupt latency problem on the receiver side is alleviated.
  • The AB_est algorithm still prevents us to detect the right available bandwidth. The reason is that the cross traffic on the router not only increase send gaps, but also make some segments back to back. Thus the send gap becomes the gap the switch interface can creates. So when the forwarding speed at the switch is faster than actual ab, the send gap decreases. Otherwise, the send gap increases to the value the switch interface can create.

    How to calculate AB_est? -- My heuristics

    Description of the heuristic:

    All the previous AB_est methods assume that the cross traffic follows a fliud model, which means that they have infinite small packet size. But it's not true in real world. The cross traffic might both increase or decrease the packet-pair gaps. So when the receive gap is much larger than send gap, this pair of gaps is useful to calculate AB_est. The pseudocode is:
    
    for i-th element in pruned stream
      
      if(recv_gap[i] > send_gap[i] + GAP_EPSILON_NS)
          excursion_start = i
           j = i
           for j-th element in pruned stream
              if( recv_gap[j] > send_gap[j] )
                 avg_bw_per_probe[j] = C* (1 - (recv_gap[j] - send_gap[j])/send_gap[j])
              if(  recv_gap[j] > send_gap[j] + threshold)
                 counter++
           j++
      i=j
    i++
    
    if counter >= 4
       excursion is valid
       compute abest using only elements in pruned stream from index= excursion-start to the end
    
    

    1. Throughput:

  • No cross traffic:
    [  4]  0.0- 0.5 sec  24.2 MBytes   406 Mbits/sec
    [  4]  0.5- 1.0 sec  44.9 MBytes   753 Mbits/sec
    [  4]  1.0- 1.5 sec  45.5 MBytes   763 Mbits/sec
    [  4]  1.5- 2.0 sec  34.4 MBytes   577 Mbits/sec
    [  4]  2.0- 2.5 sec  41.3 MBytes   692 Mbits/sec
    [  4]  2.5- 3.0 sec  45.5 MBytes   763 Mbits/sec
    [  4]  3.0- 3.5 sec  47.8 MBytes   803 Mbits/sec
    [  4]  3.5- 4.0 sec  47.4 MBytes   795 Mbits/sec
    [  4]  4.0- 4.5 sec  47.5 MBytes   797 Mbits/sec
    [  4]  4.5- 5.0 sec  47.6 MBytes   798 Mbits/sec
    [  4]  5.0- 5.5 sec  48.0 MBytes   804 Mbits/sec
    [  4]  5.5- 6.0 sec  47.2 MBytes   791 Mbits/sec
    [  4]  6.0- 6.5 sec  47.4 MBytes   795 Mbits/sec
    [  4]  6.5- 7.0 sec  47.4 MBytes   795 Mbits/sec
    [  4]  7.0- 7.5 sec  47.9 MBytes   804 Mbits/sec
    [  4]  7.5- 8.0 sec  47.6 MBytes   798 Mbits/sec
    [  4]  8.0- 8.5 sec  47.6 MBytes   799 Mbits/sec
    [  4]  8.5- 9.0 sec  48.0 MBytes   806 Mbits/sec
    [  4]  9.0- 9.5 sec  47.9 MBytes   804 Mbits/sec
    [  4]  9.5-10.0 sec  39.3 MBytes   660 Mbits/sec
    [  4]  0.0-10.0 sec   896 MBytes   750 Mbits/sec
    
  • 300Mbps cross traffic: 318Mbps throughput
    [  4]  0.0- 0.5 sec  29.4 MBytes   493 Mbits/sec
    [  4]  0.5- 1.0 sec  34.4 MBytes   578 Mbits/sec
    [  4]  1.0- 1.5 sec  34.1 MBytes   572 Mbits/sec
    [  4]  1.5- 2.0 sec  34.1 MBytes   573 Mbits/sec
    [  4]  2.0- 2.5 sec  34.2 MBytes   575 Mbits/sec
    [  4]  2.5- 3.0 sec  33.8 MBytes   567 Mbits/sec
    [  4]  3.0- 3.5 sec  34.1 MBytes   573 Mbits/sec
    [  4]  3.5- 4.0 sec  34.4 MBytes   578 Mbits/sec
    [  4]  4.0- 4.5 sec  34.3 MBytes   575 Mbits/sec
    [  4]  4.5- 5.0 sec  34.7 MBytes   582 Mbits/sec
    [  4]  5.0- 5.5 sec  33.4 MBytes   561 Mbits/sec
    [  4]  5.5- 6.0 sec  34.1 MBytes   572 Mbits/sec
    [  4]  0.0- 6.0 sec   408 MBytes   567 Mbits/sec
    
  • 500Mbps cross traffic: 535Mbps throughput
    [  4]  0.0- 0.5 sec  9.77 MBytes   164 Mbits/sec
    [  4]  0.5- 1.0 sec  13.5 MBytes   226 Mbits/sec
    [  4]  1.0- 1.5 sec  6.63 MBytes   111 Mbits/sec
    [  4]  1.5- 2.0 sec  15.6 MBytes   261 Mbits/sec
    [  4]  2.0- 2.5 sec  12.6 MBytes   212 Mbits/sec
    [  4]  2.5- 3.0 sec  5.74 MBytes  96.3 Mbits/sec
    [  4]  3.0- 3.5 sec  7.07 MBytes   119 Mbits/sec
    [  4]  3.5- 4.0 sec  6.88 MBytes   116 Mbits/sec
    [  4]  4.0- 4.5 sec  6.35 MBytes   107 Mbits/sec
    [  4]  4.5- 5.0 sec  5.43 MBytes  91.1 Mbits/sec
    [  4]  5.0- 5.5 sec  7.93 MBytes   133 Mbits/sec
    [  4]  5.5- 6.0 sec  16.3 MBytes   273 Mbits/sec
    [  4]  0.0- 6.1 sec   116 MBytes   160 Mbits/sec
    

    2. AB_est for each probe stream

  • No Cross traffic:

  • 300Mbps UDP:

    3. Summery of Results: