Gaps Smaller Than Link Capacity <h2>Do we need to send gaps smaller than link capacity?</h2>
  • One conjecture is that if we set send gaps much smaller than 12us, the gaps will be disturbed by OS scheduling issue on the sender, which prevent us from estimating the approporitate available bandwidth.
  • So here we tried to keep the scheduled send gaps no less than 12302us, which corresponds to link capacity 1G.

    Comparison of experimental results before and after the modification:

    Throughputs:

  • With 300Mbps cross traffic: With send gaps smaller than 12us:
    [  4]  0.0- 0.5 sec  7.81 MBytes   131 Mbits/sec
    [  4]  0.5- 1.0 sec  42.6 MBytes   715 Mbits/sec
    [  4]  1.0- 1.5 sec  54.8 MBytes   920 Mbits/sec
    [  4]  1.5- 2.0 sec  55.2 MBytes   926 Mbits/sec
    [  4]  2.0- 2.5 sec  55.9 MBytes   938 Mbits/sec
    [  4]  2.5- 3.0 sec  52.5 MBytes   880 Mbits/sec
    [  4]  3.0- 3.5 sec  16.9 MBytes   284 Mbits/sec
    [  4]  3.5- 4.0 sec  14.9 MBytes   250 Mbits/sec
    [  4]  4.0- 4.5 sec  16.4 MBytes   275 Mbits/sec
    [  4]  4.5- 5.0 sec  18.3 MBytes   308 Mbits/sec
    [  4]  5.0- 5.5 sec  24.6 MBytes   413 Mbits/sec
    [  4]  5.5- 6.0 sec  14.7 MBytes   246 Mbits/sec
    [  4]  6.0- 6.5 sec  14.7 MBytes   246 Mbits/sec
    [  4]  6.5- 7.0 sec  55.9 MBytes   938 Mbits/sec
    [  4]  7.0- 7.5 sec  55.1 MBytes   924 Mbits/sec
    [  4]  7.5- 8.0 sec  46.5 MBytes   780 Mbits/sec
    [  4]  0.0- 8.4 sec   582 MBytes   582 Mbits/sec
    
  • With 300Mbps cross traffic: With send gaps no smaller than 12us:
    [  4]  0.0- 0.5 sec  6.75 MBytes   113 Mbits/sec
    [  4]  0.5- 1.0 sec  35.9 MBytes   601 Mbits/sec
    [  4]  1.0- 1.5 sec  47.1 MBytes   790 Mbits/sec
    [  4]  1.5- 2.0 sec  47.9 MBytes   804 Mbits/sec
    [  4]  2.0- 2.5 sec  44.8 MBytes   752 Mbits/sec
    [  4]  2.5- 3.0 sec  42.7 MBytes   717 Mbits/sec
    [  4]  3.0- 3.5 sec  13.4 MBytes   225 Mbits/sec
    [  4]  3.5- 4.0 sec  20.7 MBytes   346 Mbits/sec
    [  4]  4.0- 4.5 sec  13.1 MBytes   219 Mbits/sec
    [  4]  4.5- 5.0 sec  13.7 MBytes   230 Mbits/sec
    [  4]  5.0- 5.5 sec  14.5 MBytes   243 Mbits/sec
    [  4]  5.5- 6.0 sec  14.6 MBytes   246 Mbits/sec
    [  4]  6.0- 6.5 sec  47.2 MBytes   792 Mbits/sec
    [  4]  6.5- 7.0 sec  47.1 MBytes   791 Mbits/sec
    [  4]  7.0- 7.5 sec  47.8 MBytes   803 Mbits/sec
    [  4]  7.5- 8.0 sec  48.0 MBytes   805 Mbits/sec
    [  4]  0.0- 8.4 sec   539 MBytes   537 Mbits/sec
    
  • With 700Mbps cross traffic: With send gaps no smaller than 12us:
    [  4]  0.0- 0.5 sec  19.9 MBytes   334 Mbits/sec
    [  4]  0.5- 1.0 sec  46.7 MBytes   784 Mbits/sec
    [  4]  1.0- 1.5 sec  48.0 MBytes   805 Mbits/sec
    [  4]  1.5- 2.0 sec  47.0 MBytes   789 Mbits/sec
    [  4]  2.0- 2.5 sec  47.3 MBytes   793 Mbits/sec
    [  4]  2.5- 3.0 sec  44.0 MBytes   738 Mbits/sec
    [  4]  3.0- 3.5 sec  4.07 MBytes  68.3 Mbits/sec
    [  4]  3.5- 4.0 sec  7.49 MBytes   126 Mbits/sec
    [  4]  4.0- 4.5 sec  9.59 MBytes   161 Mbits/sec
    [  4]  4.5- 5.0 sec  8.59 MBytes   144 Mbits/sec
    [  4]  5.0- 5.5 sec  8.97 MBytes   151 Mbits/sec
    [  4]  5.5- 6.0 sec  9.46 MBytes   159 Mbits/sec
    [  4]  6.0- 6.5 sec  35.8 MBytes   601 Mbits/sec
    [  4]  6.5- 7.0 sec  48.0 MBytes   805 Mbits/sec
    [  4]  7.0- 7.5 sec  48.0 MBytes   805 Mbits/sec
    [  4]  7.5- 8.0 sec  47.8 MBytes   801 Mbits/sec
    [  4]  0.0- 8.4 sec   510 MBytes   507 Mbits/sec
    
  • So the throughputs failed to improve after the modification.

    AB_estimation:

    When UDP cross traffic is 300Mbps, after loss recovery, the AB_estimation keeps increasing till 1g although there's 300Mbps cross traffic. But for 700Mbps UDP cross traffic, the Ab_ests are very close to available bandwidth 300Mbps.

    Scheduled send gaps and actual send gaps when there's no cross traffic:

  • With send gaps smaller than 12us:
    id= 250 send_gap= 44012 dag_gap= 24000 recv_gap= 53000 seqno= 1453993333 cum_size= 2896
    id= 250 send_gap= 44012 dag_gap= 25000 recv_gap= 16000 seqno= 1453996229 cum_size= 2896
    id= 250 send_gap= 44012 dag_gap= 25000 recv_gap= 12000 seqno= 1453999125 cum_size= 2896
    id= 250 send_gap= 41175 dag_gap= 24000 recv_gap= 43000 seqno= 1454002021 cum_size= 2896
    id= 250 send_gap= 38338 dag_gap= 25000 recv_gap= 12000 seqno= 1454004917 cum_size= 2896
    id= 250 send_gap= 38338 dag_gap= 24000 recv_gap= 13000 seqno= 1454007813 cum_size= 2896
    id= 250 send_gap= 38338 dag_gap= 25000 recv_gap= 52000 seqno= 1454010709 cum_size= 2896
    id= 250 send_gap= 35866 dag_gap= 25000 recv_gap= 15000 seqno= 1454013605 cum_size= 2896
    id= 250 send_gap= 33394 dag_gap= 24000 recv_gap= 12000 seqno= 1454016501 cum_size= 2896
    id= 250 send_gap= 33394 dag_gap= 30000 recv_gap= 63000 seqno= 1454019397 cum_size= 2896
    id= 250 send_gap= 33394 dag_gap= 34000 recv_gap= 13000 seqno= 1454022293 cum_size= 2896
    id= 250 send_gap= 31241 dag_gap= 31000 recv_gap= 48000 seqno= 1454025189 cum_size= 2896
    id= 250 send_gap= 29088 dag_gap= 29000 recv_gap= 14000 seqno= 1454028085 cum_size= 2896
    id= 250 send_gap= 29088 dag_gap= 29000 recv_gap= 12000 seqno= 1454030981 cum_size= 2896
    id= 250 send_gap= 29088 dag_gap= 29000 recv_gap= 57000 seqno= 1454033877 cum_size= 2896
    id= 250 send_gap= 27213 dag_gap= 27000 recv_gap= 16000 seqno= 1454036773 cum_size= 2896
    id= 250 send_gap= 25338 dag_gap= 25000 recv_gap= 14000 seqno= 1454039669 cum_size= 2896
    id= 250 send_gap= 25338 dag_gap= 26000 recv_gap= 41000 seqno= 1454042565 cum_size= 2896
    id= 250 send_gap= 25338 dag_gap= 25000 recv_gap= 12000 seqno= 1454045461 cum_size= 2896
    id= 250 send_gap= 23704 dag_gap= 25000 recv_gap= 13000 seqno= 1454048357 cum_size= 2896
    id= 250 send_gap= 22070 dag_gap= 25000 recv_gap= 54000 seqno= 1454051253 cum_size= 2896
    id= 250 send_gap= 22070 dag_gap= 25000 recv_gap= 15000 seqno= 1454054149 cum_size= 2896
    id= 250 send_gap= 22070 dag_gap= 24000 recv_gap= 14000 seqno= 1454057045 cum_size= 2896
    id= 250 send_gap= 20647 dag_gap= 25000 recv_gap= 42000 seqno= 1454059941 cum_size= 2896
    id= 250 send_gap= 19224 dag_gap= 24000 recv_gap= 13000 seqno= 1454062837 cum_size= 2896
    id= 250 send_gap= 19224 dag_gap= 25000 recv_gap= 12000 seqno= 1454065733 cum_size= 2896
    id= 250 send_gap= 19224 dag_gap= 25000 recv_gap= 51000 seqno= 1454068629 cum_size= 2896
    id= 250 send_gap= 17984 dag_gap= 24000 recv_gap= 16000 seqno= 1454071525 cum_size= 2896
    id= 250 send_gap= 16744 dag_gap= 25000 recv_gap= 12000 seqno= 1454074421 cum_size= 2896
    id= 250 send_gap= 16744 dag_gap= 24000 recv_gap= 42000 seqno= 1454077317 cum_size= 2896
    id= 250 send_gap= 16744 dag_gap= 25000 recv_gap= 12000 seqno= 1454080213 cum_size= 2896
    id= 250 send_gap= 15664 dag_gap= 25000 recv_gap= 13000 seqno= 1454083109 cum_size= 2896
    id= 250 send_gap= 14584 dag_gap= 24000 recv_gap= 53000 seqno= 1454086005 cum_size= 2896
    id= 250 send_gap= 14584 dag_gap= 25000 recv_gap= 15000 seqno= 1454088901 cum_size= 2896
    id= 250 send_gap= 14584 dag_gap= 25000 recv_gap= 12000 seqno= 1454091797 cum_size= 2896
    id= 250 send_gap= 13644 dag_gap= 24000 recv_gap= 42000 seqno= 1454094693 cum_size= 2896
    id= 250 send_gap= 12704 dag_gap= 25000 recv_gap= 12000 seqno= 1454097589 cum_size= 2896
    id= 250 send_gap= 12704 dag_gap= 24000 recv_gap= 12000 seqno= 1454100485 cum_size= 2896
    id= 250 send_gap= 12704 dag_gap= 25000 recv_gap= 55000 seqno= 1454103381 cum_size= 2896
    
  • With send gaps no smaller than 12us:
    id= 137 send_gap= 42094 dag_gap= 42000 recv_gap= 30000 seqno= 2466595906 cum_size= 1448
    id= 137 send_gap= 42094 dag_gap= 42000 recv_gap= 47000 seqno= 2466597354 cum_size= 1448
    id= 137 send_gap= 42094 dag_gap= 42000 recv_gap= 36000 seqno= 2466598802 cum_size= 1448
    id= 137 send_gap= 42094 dag_gap= 42000 recv_gap= 45000 seqno= 2466600250 cum_size= 1448
    id= 137 send_gap= 42094 dag_gap= 42000 recv_gap= 41000 seqno= 2466601698 cum_size= 1448
    id= 137 send_gap= 42094 dag_gap= 42000 recv_gap= 41000 seqno= 2466603146 cum_size= 1448
    id= 137 send_gap= 42094 dag_gap= 42000 recv_gap= 42000 seqno= 2466604594 cum_size= 1448
    id= 137 send_gap= 36667 dag_gap= 37000 recv_gap= 37000 seqno= 2466606042 cum_size= 1448
    id= 137 send_gap= 36667 dag_gap= 37000 recv_gap= 36000 seqno= 2466607490 cum_size= 1448
    id= 137 send_gap= 36667 dag_gap= 36000 recv_gap= 37000 seqno= 2466608938 cum_size= 1448
    id= 137 send_gap= 36667 dag_gap= 37000 recv_gap= 38000 seqno= 2466610386 cum_size= 1448
    id= 137 send_gap= 36667 dag_gap= 37000 recv_gap= 36000 seqno= 2466611834 cum_size= 1448
    id= 137 send_gap= 36667 dag_gap= 36000 recv_gap= 37000 seqno= 2466613282 cum_size= 1448
    id= 137 send_gap= 36667 dag_gap= 36000 recv_gap= 36000 seqno= 2466614730 cum_size= 1448
    id= 137 send_gap= 36667 dag_gap= 38000 recv_gap= 37000 seqno= 2466616178 cum_size= 1448
    id= 137 send_gap= 31940 dag_gap= 32000 recv_gap= 33000 seqno= 2466617626 cum_size= 1448
    id= 137 send_gap= 31940 dag_gap= 32000 recv_gap= 30000 seqno= 2466619074 cum_size= 1448
    id= 137 send_gap= 31940 dag_gap= 32000 recv_gap= 33000 seqno= 2466620522 cum_size= 1448
    id= 137 send_gap= 31940 dag_gap= 32000 recv_gap= 32000 seqno= 2466621970 cum_size= 1448
    id= 137 send_gap= 31940 dag_gap= 31000 recv_gap= 32000 seqno= 2466623418 cum_size= 1448
    id= 137 send_gap= 31940 dag_gap= 32000 recv_gap= 32000 seqno= 2466624866 cum_size= 1448
    id= 137 send_gap= 31940 dag_gap= 32000 recv_gap= 32000 seqno= 2466626314 cum_size= 1448
    id= 137 send_gap= 31940 dag_gap= 32000 recv_gap= 34000 seqno= 2466627762 cum_size= 1448
    id= 137 send_gap= 27822 dag_gap= 28000 recv_gap= 26000 seqno= 2466629210 cum_size= 1448
    id= 137 send_gap= 27822 dag_gap= 28000 recv_gap= 36000 seqno= 2466630658 cum_size= 1448
    id= 137 send_gap= 27822 dag_gap= 28000 recv_gap= 24000 seqno= 2466632106 cum_size= 1448
    id= 137 send_gap= 27822 dag_gap= 28000 recv_gap= 21000 seqno= 2466633554 cum_size= 1448
    id= 137 send_gap= 27822 dag_gap= 27000 recv_gap= 30000 seqno= 2466635002 cum_size= 1448
    id= 137 send_gap= 27822 dag_gap= 28000 recv_gap= 28000 seqno= 2466636450 cum_size= 1448
    id= 137 send_gap= 27822 dag_gap= 28000 recv_gap= 29000 seqno= 2466637898 cum_size= 1448
    id= 137 send_gap= 27822 dag_gap= 28000 recv_gap= 29000 seqno= 2466639346 cum_size= 1448
    id= 137 send_gap= 24235 dag_gap= 24000 recv_gap= 25000 seqno= 2466640794 cum_size= 1448
    id= 137 send_gap= 24235 dag_gap= 24000 recv_gap= 24000 seqno= 2466642242 cum_size= 1448
    id= 137 send_gap= 24235 dag_gap= 24000 recv_gap= 25000 seqno= 2466643690 cum_size= 1448
    id= 137 send_gap= 24235 dag_gap= 25000 recv_gap= 25000 seqno= 2466645138 cum_size= 1448
    id= 137 send_gap= 24235 dag_gap= 24000 recv_gap= 23000 seqno= 2466646586 cum_size= 1448
    id= 137 send_gap= 24235 dag_gap= 24000 recv_gap= 24000 seqno= 2466648034 cum_size= 1448
    id= 137 send_gap= 24235 dag_gap= 24000 recv_gap= 24000 seqno= 2466649482 cum_size= 1448
    id= 137 send_gap= 24235 dag_gap= 25000 recv_gap= 21000 seqno= 2466650930 cum_size= 1448
    id= 137 send_gap= 21110 dag_gap= 21000 recv_gap= 21000 seqno= 2466652378 cum_size= 1448
    id= 137 send_gap= 21110 dag_gap= 21000 recv_gap= 21000 seqno= 2466653826 cum_size= 1448
    id= 137 send_gap= 21110 dag_gap= 21000 recv_gap= 22000 seqno= 2466655274 cum_size= 1448
    id= 137 send_gap= 21110 dag_gap= 21000 recv_gap= 21000 seqno= 2466656722 cum_size= 1448
    id= 137 send_gap= 21110 dag_gap= 21000 recv_gap= 21000 seqno= 2466658170 cum_size= 1448
    id= 137 send_gap= 21110 dag_gap= 21000 recv_gap= 21000 seqno= 2466659618 cum_size= 1448
    id= 137 send_gap= 21110 dag_gap= 22000 recv_gap= 21000 seqno= 2466661066 cum_size= 1448
    id= 137 send_gap= 21110 dag_gap= 21000 recv_gap= 76000 seqno= 2466662514 cum_size= 1448
    id= 137 send_gap= 18388 dag_gap= 18000 recv_gap= 10000 seqno= 2466663962 cum_size= 1448
    id= 137 send_gap= 18388 dag_gap= 18000 recv_gap= 7000 seqno= 2466665410 cum_size= 1448
    id= 137 send_gap= 18388 dag_gap= 19000 recv_gap= 7000 seqno= 2466666858 cum_size= 1448
    id= 137 send_gap= 18388 dag_gap= 18000 recv_gap= 6000 seqno= 2466668306 cum_size= 1448
    id= 137 send_gap= 18388 dag_gap= 18000 recv_gap= 60000 seqno= 2466669754 cum_size= 1448
    id= 137 send_gap= 18388 dag_gap= 19000 recv_gap= 10000 seqno= 2466671202 cum_size= 1448
    id= 137 send_gap= 18388 dag_gap= 18000 recv_gap= 7000 seqno= 2466672650 cum_size= 1448
    id= 137 send_gap= 18388 dag_gap= 19000 recv_gap= 6000 seqno= 2466674098 cum_size= 1448
    id= 137 send_gap= 16017 dag_gap= 16000 recv_gap= 7000 seqno= 2466675546 cum_size= 1448
    id= 137 send_gap= 16017 dag_gap= 16000 recv_gap= 53000 seqno= 2466676994 cum_size= 1448
    id= 137 send_gap= 16017 dag_gap= 16000 recv_gap= 10000 seqno= 2466678442 cum_size= 1448
    id= 137 send_gap= 16017 dag_gap= 16000 recv_gap= 7000 seqno= 2466679890 cum_size= 1448
    id= 137 send_gap= 16017 dag_gap= 15000 recv_gap= 5000 seqno= 2466681338 cum_size= 1448
    id= 137 send_gap= 16017 dag_gap= 17000 recv_gap= 7000 seqno= 2466682786 cum_size= 1448
    id= 137 send_gap= 16017 dag_gap= 16000 recv_gap= 6000 seqno= 2466684234 cum_size= 1448
    id= 137 send_gap= 29969 dag_gap= 30000 recv_gap= 58000 seqno= 2466687130 cum_size= 2896
    id= 137 send_gap= 27904 dag_gap= 28000 recv_gap= 20000 seqno= 2466690026 cum_size= 2896
    id= 137 send_gap= 27904 dag_gap= 28000 recv_gap= 18000 seqno= 2466692922 cum_size= 2896
    id= 137 send_gap= 27904 dag_gap= 28000 recv_gap= 20000 seqno= 2466695818 cum_size= 2896
    id= 137 send_gap= 26256 dag_gap= 26000 recv_gap= 49000 seqno= 2466698714 cum_size= 2896
    id= 137 send_gap= 24608 dag_gap= 24000 recv_gap= 13000 seqno= 2466701610 cum_size= 2896
    id= 137 send_gap= 24608 dag_gap= 25000 recv_gap= 44000 seqno= 2466704506 cum_size= 2896
    id= 137 send_gap= 24608 dag_gap= 25000 recv_gap= 16000 seqno= 2466707402 cum_size= 2896
    
  • When scheduled send gap is smaller than 12304 us between two segments, the modified code presented perfect match between scheduled send gaps and actual send gaps shown in the dag trace. However, for the unmodified version which allows scheduled send gap smaller than 12us, the actual send gaps are larger than scheduled for those segments, which leads to the fact that actual send gaps are smaller than scheduled gaps for the segments in the beginning of one probe stream. But this will no affect AB_estimation because the probing rate in the beginning of one probe stream is much smaller than available bandwidth.

    Scheduled gaps and actual gaps when there's 300M udp cross traffic

  • With 300Mbps UDP Cross traffic: with send gaps smaller than 12us:
    id= 4220 send_gap= 71630 dag_gap= 37000 recv_gap= 22000 seqno= 1721680925 cum_size= 2896
    id= 4220 send_gap= 71630 dag_gap= 37000 recv_gap= 44000 seqno= 1721683821 cum_size= 2896
    id= 4220 send_gap= 71630 dag_gap= 49000 recv_gap= 49000 seqno= 1721686717 cum_size= 2896
    id= 4220 send_gap= 67013 dag_gap= 49000 recv_gap= 49000 seqno= 1721689613 cum_size= 2896
    id= 4220 send_gap= 62396 dag_gap= 57000 recv_gap= 48000 seqno= 1721692509 cum_size= 2896
    id= 4220 send_gap= 62396 dag_gap= 69000 recv_gap= 74000 seqno= 1721695405 cum_size= 2896
    id= 4220 send_gap= 62396 dag_gap= 59000 recv_gap= 53000 seqno= 1721698301 cum_size= 2896
    id= 4220 send_gap= 58374 dag_gap= 56000 recv_gap= 51000 seqno= 1721701197 cum_size= 2896
    id= 4220 send_gap= 54352 dag_gap= 55000 recv_gap= 70000 seqno= 1721704093 cum_size= 2896
    id= 4220 send_gap= 54352 dag_gap= 64000 recv_gap= 72000 seqno= 1721706989 cum_size= 2896
    id= 4220 send_gap= 54352 dag_gap= 56000 recv_gap= 43000 seqno= 1721709885 cum_size= 2896
    id= 4220 send_gap= 50848 dag_gap= 39000 recv_gap= 48000 seqno= 1721712781 cum_size= 2896
    id= 4220 send_gap= 47344 dag_gap= 61000 recv_gap= 61000 seqno= 1721715677 cum_size= 2896
    id= 4220 send_gap= 47344 dag_gap= 34000 recv_gap= 20000 seqno= 1721718573 cum_size= 2896
    id= 4220 send_gap= 47344 dag_gap= 49000 recv_gap= 65000 seqno= 1721721469 cum_size= 2896
    id= 4220 send_gap= 32386 dag_gap= 34000 recv_gap= 23000 seqno= 1721723477 cum_size= 2008
    id= 4220 send_gap= 41240 dag_gap= 42000 recv_gap= 55000 seqno= 1721726373 cum_size= 2896
    id= 4220 send_gap= 41240 dag_gap= 36000 recv_gap= 22000 seqno= 1721729269 cum_size= 2896
    id= 4220 send_gap= 41240 dag_gap= 50000 recv_gap= 79000 seqno= 1721732165 cum_size= 2896
    id= 4220 send_gap= 38581 dag_gap= 31000 recv_gap= 15000 seqno= 1721735061 cum_size= 2896
    id= 4220 send_gap= 35922 dag_gap= 49000 recv_gap= 73000 seqno= 1721737957 cum_size= 2896
    id= 4220 send_gap= 35922 dag_gap= 25000 recv_gap= 12000 seqno= 1721740853 cum_size= 2896
    id= 4220 send_gap= 35922 dag_gap= 49000 recv_gap= 52000 seqno= 1721743749 cum_size= 2896
    id= 4220 send_gap= 33606 dag_gap= 25000 recv_gap= 15000 seqno= 1721746645 cum_size= 2896
    id= 4220 send_gap= 31290 dag_gap= 49000 recv_gap= 70000 seqno= 1721749541 cum_size= 2896
    id= 4220 send_gap= 31290 dag_gap= 24000 recv_gap= 15000 seqno= 1721752437 cum_size= 2896
    id= 4220 send_gap= 31290 dag_gap= 37000 recv_gap= 13000 seqno= 1721755333 cum_size= 2896
    id= 4220 send_gap= 29273 dag_gap= 37000 recv_gap= 57000 seqno= 1721758229 cum_size= 2896
    id= 4220 send_gap= 27256 dag_gap= 25000 recv_gap= 13000 seqno= 1721761125 cum_size= 2896
    id= 4220 send_gap= 27256 dag_gap= 37000 recv_gap= 27000 seqno= 1721764021 cum_size= 2896
    id= 4220 send_gap= 27256 dag_gap= 37000 recv_gap= 59000 seqno= 1721766917 cum_size= 2896
    id= 4220 send_gap= 25499 dag_gap= 24000 recv_gap= 14000 seqno= 1721769813 cum_size= 2896
    id= 4220 send_gap= 23742 dag_gap= 37000 recv_gap= 27000 seqno= 1721772709 cum_size= 2896
    id= 4220 send_gap= 23742 dag_gap= 37000 recv_gap= 58000 seqno= 1721775605 cum_size= 2896
    id= 4220 send_gap= 23742 dag_gap= 25000 recv_gap= 12000 seqno= 1721778501 cum_size= 2896
    id= 4220 send_gap= 22211 dag_gap= 24000 recv_gap= 13000 seqno= 1721781397 cum_size= 2896
    id= 4220 send_gap= 20680 dag_gap= 49000 recv_gap= 75000 seqno= 1721784293 cum_size= 2896
    id= 4220 send_gap= 20680 dag_gap= 25000 recv_gap= 15000 seqno= 1721787189 cum_size= 2896
    id= 4220 send_gap= 20680 dag_gap= 25000 recv_gap= 12000 seqno= 1721790085 cum_size= 2896
    
  • With 300Mbps UDP Cross Traffic: with send gaps no smaller than 12us:
    id= 6611 send_gap= 83872 dag_gap= 49000 recv_gap= 37000 seqno= 2734423466 cum_size= 2896
    id= 6611 send_gap= 83872 dag_gap= 74000 recv_gap= 73000 seqno= 2734426362 cum_size= 2896
    id= 6611 send_gap= 83872 dag_gap= 84000 recv_gap= 84000 seqno= 2734429258 cum_size= 2896
    id= 6611 send_gap= 73060 dag_gap= 73000 recv_gap= 73000 seqno= 2734432154 cum_size= 2896
    id= 6611 send_gap= 73060 dag_gap= 73000 recv_gap= 73000 seqno= 2734435050 cum_size= 2896
    id= 6611 send_gap= 73060 dag_gap= 73000 recv_gap= 73000 seqno= 2734437946 cum_size= 2896
    id= 6611 send_gap= 73060 dag_gap= 74000 recv_gap= 74000 seqno= 2734440842 cum_size= 2896
    id= 6611 send_gap= 63642 dag_gap= 62000 recv_gap= 63000 seqno= 2734443738 cum_size= 2896
    id= 6611 send_gap= 63642 dag_gap= 65000 recv_gap= 64000 seqno= 2734446634 cum_size= 2896
    id= 6611 send_gap= 63642 dag_gap= 63000 recv_gap= 63000 seqno= 2734449530 cum_size= 2896
    id= 6611 send_gap= 63642 dag_gap= 64000 recv_gap= 64000 seqno= 2734452426 cum_size= 2896
    id= 6611 send_gap= 55438 dag_gap= 65000 recv_gap= 83000 seqno= 2734455322 cum_size= 2896
    id= 6611 send_gap= 55438 dag_gap= 54000 recv_gap= 40000 seqno= 2734458218 cum_size= 2896
    id= 6611 send_gap= 55438 dag_gap= 46000 recv_gap= 43000 seqno= 2734461114 cum_size= 2896
    id= 6611 send_gap= 55438 dag_gap= 57000 recv_gap= 60000 seqno= 2734464010 cum_size= 2896
    id= 6611 send_gap= 48290 dag_gap= 53000 recv_gap= 54000 seqno= 2734466906 cum_size= 2896
    id= 6611 send_gap= 48290 dag_gap= 43000 recv_gap= 49000 seqno= 2734469802 cum_size= 2896
    id= 6611 send_gap= 48290 dag_gap= 57000 recv_gap= 66000 seqno= 2734472698 cum_size= 2896
    id= 6611 send_gap= 48290 dag_gap= 40000 recv_gap= 20000 seqno= 2734475594 cum_size= 2896
    id= 6611 send_gap= 42064 dag_gap= 49000 recv_gap= 74000 seqno= 2734478490 cum_size= 2896
    id= 6611 send_gap= 42064 dag_gap= 43000 recv_gap= 22000 seqno= 2734481386 cum_size= 2896
    id= 6611 send_gap= 42064 dag_gap= 36000 recv_gap= 43000 seqno= 2734484282 cum_size= 2896
    id= 6611 send_gap= 42064 dag_gap= 45000 recv_gap= 40000 seqno= 2734487178 cum_size= 2896
    id= 6611 send_gap= 36640 dag_gap= 36000 recv_gap= 60000 seqno= 2734490074 cum_size= 2896
    id= 6611 send_gap= 36640 dag_gap= 39000 recv_gap= 12000 seqno= 2734492970 cum_size= 2896
    id= 6611 send_gap= 36640 dag_gap= 37000 recv_gap= 57000 seqno= 2734495866 cum_size= 2896
    id= 6611 send_gap= 36640 dag_gap= 41000 recv_gap= 27000 seqno= 2734498762 cum_size= 2896
    id= 6611 send_gap= 31916 dag_gap= 37000 recv_gap= 69000 seqno= 2734501658 cum_size= 2896
    id= 6611 send_gap= 31916 dag_gap= 25000 recv_gap= 12000 seqno= 2734504554 cum_size= 2896
    id= 6611 send_gap= 31916 dag_gap= 49000 recv_gap= 64000 seqno= 2734507450 cum_size= 2896
    id= 6611 send_gap= 31916 dag_gap= 24000 recv_gap= 15000 seqno= 2734510346 cum_size= 2896
    id= 6611 send_gap= 27800 dag_gap= 37000 recv_gap= 12000 seqno= 2734513242 cum_size= 2896
    id= 6611 send_gap= 27800 dag_gap= 37000 recv_gap= 58000 seqno= 2734516138 cum_size= 2896
    id= 6611 send_gap= 27800 dag_gap= 25000 recv_gap= 12000 seqno= 2734519034 cum_size= 2896
    id= 6611 send_gap= 27800 dag_gap= 37000 recv_gap= 28000 seqno= 2734521930 cum_size= 2896
    id= 6611 send_gap= 24608 dag_gap= 37000 recv_gap= 58000 seqno= 2734524826 cum_size= 2896
    id= 6611 send_gap= 24608 dag_gap= 24000 recv_gap= 13000 seqno= 2734527722 cum_size= 2896
    id= 6611 send_gap= 24608 dag_gap= 25000 recv_gap= 13000 seqno= 2734530618 cum_size= 2896
    id= 6611 send_gap= 24608 dag_gap= 49000 recv_gap= 56000 seqno= 2734533514 cum_size= 2896
    
  • For both cases when there's 300Mbps UDP cross traffic, the actual send gaps at the tail of the probe stream fails to match after the send gap is smaller than 33526(gaps for 700Mbps available bandwidth for two segments) for cum_size= 2896. If newmexico machine is between two routers, then these dag_gaps can reflect the queuing facts on the router. What should be more concerned about is the interrupt-coalesence effects on the receiver.
  • With 700Mbps UDP Cross Traffic:
    id= 4640 send_gap= 185314 dag_gap= 172000 recv_gap= 173000 seqno= 202586501 cum_size= 2896
    id= 4640 send_gap= 161424 dag_gap= 136000 recv_gap= 135000 seqno= 202589397 cum_size= 2896
    id= 4640 send_gap= 161424 dag_gap= 159000 recv_gap= 159000 seqno= 202592293 cum_size= 2896
    id= 4640 send_gap= 161424 dag_gap= 148000 recv_gap= 149000 seqno= 202595189 cum_size= 2896
    id= 4640 send_gap= 161424 dag_gap= 147000 recv_gap= 147000 seqno= 202598085 cum_size= 2896
    id= 4640 send_gap= 140614 dag_gap= 136000 recv_gap= 135000 seqno= 202600981 cum_size= 2896
    id= 4640 send_gap= 140614 dag_gap= 123000 recv_gap= 123000 seqno= 202603877 cum_size= 2896
    id= 4640 send_gap= 140614 dag_gap= 135000 recv_gap= 134000 seqno= 202606773 cum_size= 2896
    id= 4640 send_gap= 140614 dag_gap= 135000 recv_gap= 136000 seqno= 202609669 cum_size= 2896
    id= 4640 send_gap= 122486 dag_gap= 123000 recv_gap= 123000 seqno= 202612565 cum_size= 2896
    id= 4640 send_gap= 122486 dag_gap= 110000 recv_gap= 111000 seqno= 202615461 cum_size= 2896
    id= 4640 send_gap= 122486 dag_gap= 123000 recv_gap= 122000 seqno= 202618357 cum_size= 2896
    id= 4640 send_gap= 122486 dag_gap= 111000 recv_gap= 111000 seqno= 202621253 cum_size= 2896
    id= 4640 send_gap= 106696 dag_gap= 111000 recv_gap= 111000 seqno= 202624149 cum_size= 2896
    id= 4640 send_gap= 106696 dag_gap= 110000 recv_gap= 111000 seqno= 202627045 cum_size= 2896
    id= 4640 send_gap= 106696 dag_gap= 111000 recv_gap= 111000 seqno= 202629941 cum_size= 2896
    id= 4640 send_gap= 106696 dag_gap= 110000 recv_gap= 110000 seqno= 202632837 cum_size= 2896
    id= 4640 send_gap= 92942 dag_gap= 86000 recv_gap= 85000 seqno= 202635733 cum_size= 2896
    id= 4640 send_gap= 92942 dag_gap= 99000 recv_gap= 99000 seqno= 202638629 cum_size= 2896
    id= 4640 send_gap= 92942 dag_gap= 98000 recv_gap= 99000 seqno= 202641525 cum_size= 2896
    id= 4640 send_gap= 92942 dag_gap= 86000 recv_gap= 86000 seqno= 202644421 cum_size= 2896
    id= 4640 send_gap= 80960 dag_gap= 99000 recv_gap= 98000 seqno= 202647317 cum_size= 2896
    id= 4640 send_gap= 80960 dag_gap= 73000 recv_gap= 74000 seqno= 202650213 cum_size= 2896
    id= 4640 send_gap= 80960 dag_gap= 99000 recv_gap= 99000 seqno= 202653109 cum_size= 2896
    id= 4640 send_gap= 80960 dag_gap= 86000 recv_gap= 85000 seqno= 202656005 cum_size= 2896
    id= 4640 send_gap= 70522 dag_gap= 73000 recv_gap= 73000 seqno= 202658901 cum_size= 2896
    id= 4640 send_gap= 70522 dag_gap= 74000 recv_gap= 76000 seqno= 202661797 cum_size= 2896
    id= 4640 send_gap= 70522 dag_gap= 86000 recv_gap= 84000 seqno= 202664693 cum_size= 2896
    id= 4640 send_gap= 70522 dag_gap= 74000 recv_gap= 74000 seqno= 202667589 cum_size= 2896
    id= 4640 send_gap= 61430 dag_gap= 74000 recv_gap= 74000 seqno= 202670485 cum_size= 2896
    id= 4640 send_gap= 61430 dag_gap= 74000 recv_gap= 74000 seqno= 202673381 cum_size= 2896
    id= 4640 send_gap= 61430 dag_gap= 73000 recv_gap= 73000 seqno= 202676277 cum_size= 2896
    id= 4640 send_gap= 61430 dag_gap= 74000 recv_gap= 74000 seqno= 202679173 cum_size= 2896
    id= 4640 send_gap= 53510 dag_gap= 62000 recv_gap= 61000 seqno= 202682069 cum_size= 2896
    id= 4640 send_gap= 53510 dag_gap= 61000 recv_gap= 62000 seqno= 202684965 cum_size= 2896
    id= 4640 send_gap= 53510 dag_gap= 74000 recv_gap= 79000 seqno= 202687861 cum_size= 2896
    id= 4640 send_gap= 53510 dag_gap= 61000 recv_gap= 58000 seqno= 202690757 cum_size= 2896
    
    When cross traffic is large enought, queue starts to appear when send_gap is smaller than available bandwidth. And the buffering issue fails to occur because the send_gap is large enough. So the interrupt coalesence at the receiver side is a bigger problem???

    Some Throughputs:

  • 1. To create the right send gap in the beginning of each probe stream, we need to limit the scheduled send gap no smaller than the gap the full link capacity can create. But the negative effect is that rapid will fail to reach full link capacity when available bandwidth is full link capacity.
  • 2. If the send gaps from the dag trace reflect the queuing issues on the router due to cross traffic, we need to change the excursion-searching algorithm. The reason is that the cross traffic not only increases the send gap, but decreases some gaps because two segments might be queued back to back on the router.
  • 3. When throughput is high enough, interrupt coalesence is a big issue on the receiver because it destroys the orchestrated gaps.