Don Smith created a nice dummynet modification for FreeBSD 4.5 during Summer 2002. The idea is to create per-flow delays sampled from a cdf, rather than using the same delay value for all flows associated with a dummynet pipe.
There are three sysctl variables involved:
Long Le added an additional sysctl variable several months after Don's work that allows you to set the seed used by the random number generator. This variable is:
Note that the assignment of delay values using the given distribution is done when two pipe configuration conditions are true:
If you statically configure either of these to non-zero values, then this per-flow assignment scheme no longer functions.
If you need to disable Don's feature (e.g., you need to use dummynet to generate loss or queue size limitations only), then your only option is to set either delay or bandwidth to some non-zero value. Setting net.inet.ip.dummynet.delay_cdf_p1 and net.inet.ip.dummynet.delay_cdf_p2 both to zero results in random values that are not range-constrained. (I tried it just now.)
Don's source code for FreeBSD 4.5 is at ~smithfd/src/delaybox/dummynet/src/4.5. Long's revision, also for FreeBSD 4.5, is on puck at /usr/src/sys-altq/netinet. Here's a source listing for Long's code:
Below is Don's original email message announcing the new feature, and Long's email mentioning his additional modification.
-------- Original Message --------
Subject: FYI
Date: Thu, 20 Jun 2002 10:36:57 -0400
From: Don Smith
I have been able to get some new tools working that some of you may
find useful. First, I modified dummynet to add a new mode in which it
delays packets on a per-flow basis. In this mode, each time dummynet
sees a new flow (address 5-tuple) it picks a random delay for that
flow and creates a new queue for packets belonging to that flow. Each
arriving packet of the flow is delayed by the delay value chosen for
that flow. The delay for a flow is chosen from a uniform
distribution, but that can be changed easily if we want. Jay and Long
are now using per-flow delays (on the clients) in the gigabit
experimental networks and it seems to work fine on the highest
aggregate throughput we tried (>250 Mbps). The source for FreeBSD-4.5
is at ~smithfd/src/delaybox/dummynet/src/4.5 (search for "FDS" to find
the changes). This turned out to be really trivial since most of the
mechanisms are already built into dummynet to do per-flow bandwidth
allocation.
In order to test the per-flow delays, I also put together some
programs to extract RTT values for TCP connections from tcpdump
traces. For this you need to trace both directions of a link (as we
do in the gigabit experimental network). The process is fairly
complicated and consists of a series of steps to first prepare the
tcpdump files using tcpdump and sort, extracting data and ACK segments
from the traces, a merge-sort to order each connection's data segments
from one trace with its ACK segments from the other trace, and finally
using the merged data to compute RTT values. The final step produces
two files: (a) one with the RTT for each data segment for which there
is a corresponding ACK for the highest sequence # in the segment (+1),
and (b) one with the min, max, and mean of these RTT values for each
connection in the traces. The programs are in
~smithfd/src/delaybox/dummynet/{2way_prep.c, get_data_seq.c,
get_ack_seq.c, data_rtt.c}. The whole process is done by the scripts,
prepRTT and doRTT, run in that order (also found in the same
directory).
-------- Original Message --------
Subject: Re: Modified dummynet
Date: Mon, 26 Aug 2002 00:09:04 -0400 (EDT)
From: Long Le
To: Don Smith
Hi Don,
I made the change that you suggested. The new file is in
/usr/src/sys-altq/netinet/ip_dummynet.c on puck. The seed for
the random number generator can now be read or set via the sysctl
variable net.inet.ip.dummynet.dn_randseed.
-- long