This documents how to run experiments using the system as per February 16th, 2004. You can find the following section in this document:
The trace replay system has two main pieces of software:
Before running an experiment, a suitable environment in the testbed administration host must be created following these steps:
# NTP
${fwcmd} add pass udp from clock1.unc.edu to any
${fwcmd} add pass udp from clock.psu.edu to any
${fwcmd} add pass udp from clock-1.cs.cmu.edu to any
${fwcmd} add pass udp from timex.cs.columbia.edu to any
mkdir bin exp tmix_conf tcvecs treplay_confSee the next section for a description of a description of the purpose of these subdirectories.
cp -r $RELEASE/tcvecs /yourmaindir/tcvecs
cp $RELEASE/treplay/treplay.pl yourmaindir/bin
cp $RELEASE/treplay/bin/* yourmaindir/binThis includes run_mon, run_mon_tcpdump, run_tmix, set_sysparams, get_sysparams, change_kernel, and set_delays. You also need a binary version of the traffic generation program tmix (i.e., compile the sources under $RELEASE/tmix with make tmix and copy tmix to yourmaindir/bin.
cp $RELEASE/bin-treplay/* yourmaindir/binThis is include tcpdump-hiperf, bpfmon and ifmon.
Run an experiment with:
/yourmaindir/bin/treplay.pl treplay_conf/treplay_example.conf
Note that each run requires a fresh results directory, that will be created under yourmaindir/exp with the name specified by the Exp_Prefix configuration variable. I recommend you create a new .conf file for each experiment (the idea is that this file fully documents the experiment, so it should be kept for future reference). The files in the result directory include the output of each traffic generator instance and the traffic monitors. The extensions of each output file explain their content:
end_ts_in_ms pkt_count byte_count
Start_t_in_millis Duration a_i b_i ta_i i tcvec_num cvecid
Start_t_in_millis Duration Tot_Bytes Tot_Quiet_Time tcvec_num cvecid Expected_Start_t
Start_t_in_millis Duration a_i b_i ta_i i tcvec_num cvecidStart_t_in_millis Duration Tot_Bytes Tot_Quiet_Time tcvec_num
cvecid Start_t_in_millis Duration Interrupted_Epoch Inter_State Replayed_Data Pending_Data
Time_in_secs Cvecs_started Cvecs_done Min_conns_used Max_conns_used conn_tmos read_reset write_epipe CPU_perc Mem_bytes select_calls select_time select_ready_fds select_tmos select_busy_conns
Some analysis programs are described in the last section of the page.
This is the usage message of the program:
treplay version 1.7
Felix Hernandez-Campos, 2003.
Description:
This program configures the infrastructure and runs a trace replay
experiment in a network testbed. It requires the following directory
structure under the current directory (whose absolute path must be
specified using the configuration variable Abs_Path):
bin/ programs and scripts used during the experiments
(run_tmix set_inbound_delay set_outbound_delay set_sysparams get_sysparams change_kernel).
exp/ results directory (results are copied into a subdirectory of
this directory of the form exp/Exp_Prefix).
tmix_conf/ stores temporary files for configuring tmix.
tcvecs/ stores sets of connection vector subtraces to be replayed
during the experiments (each set is stored in a
subdirectory).
treplay_conf/ experiment configuration files.
Abs_Path must accessible from all of the hosts participating in the
experiment (through NFS).
Configuration:
See a fully-documented example in
~fhernand/src/tmix/treplay/treplay_conf/treplay_example.conf.
Usage:
treplay.pl [options] config_fname
Options:
-c Check configuration variables only (no setup, no experiment).
-h Show this help message.
-r Do not reboot hosts (it also disables kernel changing).
-s Set up the network only (do not run experiment).
-v Show version message.
This is an example configuration file (treplay_conf/treplay_example.conf):
# This is a configuration file for a trace replay experiment
# See treplay.pl for further information
#
# Felix Hernandez-Campos
# July 2003 -- February 2004
# Prefix added to all output files, and name of directory under exp/
# in which the files generated by each traffic generator are copied to
# at the end of the simulation
# Each experiment MUST have a unique name
Exp_Prefix = treplay_example
# An email is sent to the following address when the experiment is done
Email_Addr = fhernand@cs.unc.edu
### DIRECTORIES ############################################
# This path points to the location of the scripts and configuration files
# It *must* be available in all hosts involved in the experiment, including
# the administration host
Abs_Path = /net/goldberg/playpen2/fhernand
Abs_Path_Mount_Point = /net/goldberg/playpen2
# Local storage in end hosts
# All files are remove after experiment
End_Host_Local_Dir = /usr/home2
# Local storage in router hosts
# All files are remove after experiment
# Router_Local_Dir = /usr/home2
# Local storage in monitors
# Some file may be left here (see bin/run_mon[_tcpdump])
Monitor_Local_Dir = /playpen2/fhernand
### TRACE ##################################################
# Experiment duration in seconds (it should be longer than the duration of the
# set of connection vectors, e.g., 3600 + 300 seconds)
# 120 extra-seconds are always added to take into account the copy and
# clean-up operation that take place after the traffic generation is done
Duration = 3900
# This parameter changes the rate at which connection are started
# E.g., 2.0 makes the start times two times slower
# Duration should be change accordingly
# E.g., if 2.0 for 1h tcvec then Duration should be 2.0 * 3600 + 300
tcvec_Rate_Scaling = 1.0
# File name prefix for subtraces (before tr_num)
tcvec_Prefix = 2003_Apr_30_Wed_1900.em1.stnc
# Directory in which subtraces are stored
tcvec_Dir = unc03-bidir-wed-1900-stnc-21
# Replay cinit, crecv or both sides of the trace
tcvec_Side = both
# Minimum thresholds for time replay (default is 1000 milliseconds)
# Cvec_Min_ta = 300
# Cvec_Min_tb = 300
### END-POINT PARAMETERS ###################################
# End Host Script (for running tmix):
# e.g., run_tmix
# End_Host_Script = run_tmix_ping
# Traffic generator executable (must be located under bin/)
tmix_Executable = tmix
# Directory in which tmix configuration files will be stored
tmix_Config_Dir = tmix_conf
# Set system limits script (e.g., set_sysparams)
# Note that arguments can follow the name of the script
# (e.g., set_sysparams_args 35040 1 0 1)
# System_Parameters_Script = set_sysparams_args 70080 1 0 1
# TCP_Window_Size = 70080
# This function is executed for every host, so it can be used
# to change the default system parameters for specific hosts
# The subtrace number corresponding to each host is $trnum
# Sysparams_Function =
# if ($trnum <= 5) {
# $System_Parameters_Script = "set_sysparams_args 5840 1 0 1";
# }
# print " $trnum. $System_Parameters_Script\n";
# END
# Size of IO buffer for read/write calls on sockets
# A multiple of 1460, the maximum payload, is recommended to avoid artifacts
# in the distribution packet sizes (default value is 17520 = 12 * 1460)
# IO_Buffer_Size = 1460
# This function is executed for every host, so it can be used
# to change the default tmix parameters for specific hosts
# tmix_Conf_Function =
# if ($trnum <= 5) {
# $IO_Buffer_Size = 5840; $TCP_Window_Size = 5840;
# }
# print " $trnum. IO_Buffer_Size=$IO_Buffer_Size TCP_Window_Size=$TCP_Window_Size\n";
# END
### PATH PARAMETERS ########################################
# RTT, Loss and Bandwidth Settings
# Specify the direction in which the delays are applied
# Possible values are none, inbound, outbound and both
RTT_LHS_Mode = outbound
RTT_RHS_Mode = none
# Random Number Seed for Artificial Round-Trip-Time Delays
RTT_Seed = 1
# These delays are applied in each direction, so total
# RTTs are distributed between 2*min and 2*max
RTT_Min_Value = 10
RTT_Max_Value = 150
# This function is executed for every host, so it can be used
# to change the default round-trip time settings for specific hosts
# RTT parameters are available as $RTT_*
# The subtrace number corresponding to each host is $trnum
# The side of the replay is available as $trside (cinit or crecv)
# RTT_Function =
# if ($trnum >= 5) {
# $RTT_Min_Value = 50;
# $RTT_Max_Value = 500;
# $RTT_LHS_Mode = "both";
# $RTT_RHS_Mode = "both";
# $RTT_Inbound_Setup = "set_inbound_flow_bw 28800bit/s";
# }
# END
### TOPOLOGY ##############################################
# END HOSTS
# Limit the number of traffic generating hosts (and therefore the number
# of replayed subtraces)
Number_of_End_Hosts = 21
# List of traffic generating hosts on
# the left hand side of the experimental network
LHS_Hosts = howard135, lovey135, gilligan135, chen135, borg135, minnow135, shylock135, bianca135, cordelia135, falstaff135, talley135, marcus135, wako135, pedro135, hero135, auntbea135, granny135, marvin135, kate135, laertes135, elmer135
# ginger135
# and their network mask
LHS_Hosts_Netmask = 152.2.135.0/24
# List of traffic generating hosts on
# the right hand side of the experimental network
RHS_Hosts = floyd136, goober136, cornelius136, cromwell136, lear136, caesar136, cicero136, iago136, claudio136, cleopatra136, gertrude136, beatrice136, macbeth136, yosemite136, sam136, henry136, wilee136, thurston136, pericles136, juliet136, krishnan136
# desdemona136
# and their network mask
RHS_Hosts_Netmask = 152.2.136.0/24
End_Host_Kernel = DIRT_ECN
# ROUTERS
Router_Hosts = priam135, portia136
Router_Host_Kernel = DIRT_AQM
LHS_Router = priam135
LHS_Setup_Routes = setup_gig_routes_priam135
RHS_Router = portia136
RHS_Setup_Routes = setup_gig_routes_portia136
# Router monitoring scripts
# LHS_Router_Script = run_fifo_priam135
# RHS_Router_Script = run_fifo_portia136
# MONITORS
Monitor_Hosts = sylvester, richard
Monitor_Host_Kernel = DIRT
# Monitoring script:
# run_mon monitors packet arrivals using bpfmon only (this is the default)
# run_mon_tcpdump collects a full tcpdump traces (stored in the monitors)
Monitor_Script = run_gig_mon
Parsing configuration file treplay_conf/treplay_revised.conf... Exp_Prefix: treplay_revised Email_Addr: fhernand@cs.unc.edu tmix_Executable: tmix tmix_Config_Dir: tmix_conf Monitor_Script: run_mon_tcpdump Abs_Path: /net/goldberg/playpen2/fhernand Abs_Path_Mount_Point: /net/goldberg/playpen2 End_Host_Local_Dir: /usr/home2/exp Monitor_Local_Dir: /playpen3/fhernand/exp Duration: 3900 tcvec_Rate_Scaling: 1.0 tcvec_Prefix: 2003_Apr_30_Wed_1900.em1 tcvec_Dir: oct3 tcvec_Side: both RTT_Seed: 1 RTT_Min_Value: 5 RTT_Max_Value: 250 Number_of_End_Hosts: 22 LHS_Hosts: howard135 lovey135 gilligan135 chen135 borg135 minnow135 shylock135 bianca135 cordelia135 falstaff135 ginger135 talley135 marcus135 wako135 pedro135 hero135 auntbea135 granny135 marvin135 kate135 laertes135 elmer135 LHS_Hosts_Netmask: 152.2.135.0/24 RHS_Hosts: floyd136 goober136 cornelius136 cromwell136 lear136 caesar136 cicero136 iago136 claudio136 cleopatra136 desdemona136 gertrude136 beatrice136 macbeth136 yosemite136 sam136 henry136 wilee136 thurston136 pericles136 juliet136 krishnan136 RHS_Hosts_Netmask: 152.2.136.0/24 End_Host_Kernel: DIRT_ECN Router_Hosts: priam135 portia136 Router_Host_Kernel: DIRT_AQM Monitor_Hosts: sylvester richard Monitor_Host_Kernel: DIRT Synchronizing local clock using ntpdate... Checking subtraces and directories... Number of traffic generation hosts on each side: 22 Checking lhs hosts... Hosts are reachable. Hosts respond to rsh. Checking rhs hosts... Hosts are reachable. Hosts respond to rsh. Checking router hosts... Hosts are reachable. Hosts respond to rsh. Checking monitor hosts... Hosts are reachable. Hosts respond to rsh. Changing end host kernels... All hosts running kernel DIRT_ECN. Changing router kernels... All hosts running kernel DIRT_AQM. Changing monitor kernels... All hosts running kernel DIRT. Rebooting routers... (waiting for reboot to complete) Rebooting non-routers... (waiting for reboot to complete) Checking end host kernels... Checking router kernels... Checking monitor kernels... Checking NFS mount... Required NFS mount found in all hosts. Flushing ipfw rules... All firewall rules flushed. Setting up extra delays in LHS hosts (U[5,75] seed 1)... All delays set. Setting up extra delays in RHS hosts (U[5,75] seed 1)... All delays set. Setting up gigabit ethernet routes... Routing configured. Writing tmix configuration files in /net/goldberg/playpen2/fhernand/tmix_conf... Start time: 9:28:37 Minimum intra- and inter-epoch times: 1 second. Starting traffic generators... All traffic generators started. Starting traffic monitors... All monitors started. Running experiment for 4020 seconds... 5 minutes. 10 minutes. 15 minutes. 20 minutes. 25 minutes. 30 minutes. 35 minutes. 40 minutes. 45 minutes. 50 minutes. 55 minutes. 60 minutes. 65 minutes. 70 minutes. Done.
The easiest way to understand the use of tmix is to follow an example run in debugging mode. The files under
Note that tmix instances generates traffic working in pairs
.(run the programs in the results directory)