There are the steps required to reproduce all the experiments in our paper:
J. Bakita, S. Ahmed, S. H. Osborne, S. Tang, J. Chen, F. D. Smith, J. H. Anderson, “Simultaneous Multithreading in Real-Time Mixed-Criticality Systems”, Proceedings of the 27th IEEE Real-Time and Embedded Technology and Applications Symposium, May 2021, to appear. (PDF)
Each step relies on completion of all the previous ones:
For full reproduction, each of these steps relies on all of the previous ones to have completed. If any steps are not feasible due to errors or hardware limitations, please double-check that you have the latest code of our code and instructions before skipping ahead. Where possible, we prepopulate each step’s configuration so that you can still continue with evaluation even if you are unable to directly derive every configuration parameter.
These steps were last updated
.
These benchmarks measure the slowdown incurred by each task when paired onto the SMT threads of a core. We measure this slowdown under three different system configurations:
xi)i3)i)These benchmarks reproduce Tbl. 1 and the “SMT Effect” entries of
Tbl. 2, plus
,
and
for all
and all
.
We measure all
and
under all system configurations (xi, i3, and
i), but only measure
values without isolation (xi) as keeping with our system
design.
Platform Compatibility An AMD Ryzen 9 3950X-based
system is needed for exact reproduction, but any AMD
Zen-2-based processor should yield similar results. Cache coloring
requires the processor’s cache mapping function, and is thus not
available on other systems. However, the xi and
i3 configurations do not require cache coloring and can be
run on any Linux kernel/processor configuration which supports Intel
CAT/AMD QoS Extensions via resctrl.
Our scripts automatically check for the MC^2 kernel however, and these
checks will have to be commented out to run on other systems.
Running the Experiments
CONFIG_DEFAULT_CMDLINE, add
isolcpus=15,31 and nohz_full=15,31 if you
intend to benchmark on core 16 (CPU 15) in a 16-core system. The second
number is the CPU ID that Linux uses for the thread paired with your
core. You can obtain this second number by taking your benchmarking core
and adding the total number of cores in your system.irqaffinity=0 to the kernel cmdline.sudo service irqbalance stopcd mc2-scripts-and-benchmarks; make benchmarkscd wbinvd; make; sudo insmod wbinvd.kocd thrasher; makeae.ae:sudo ./run_all_tacle.sh 100 ae 15 31
sudo ./run_all_dis.sh 100 ae 15 31
sudo ./run_all_sd-vbs.sh 100 ae 15 31
Output Format Each output file uses the following format:
Date Hour Isolation type Run ID
v v v v
Mar09-13-c-xi-async-my_1k_run_id.txt
^ ^ ^
Contention type └ Pairing type └ Pair ID (not shown)
xc (no contending tasks) or
c (contending tasks). The scripts automatically run all
experiments with contention.xi, i3, or i
as introduced earlier.async or non-present. This field
indicates if we are measuring
values, which do not require the pairs to start synchronously.A, B, or non-present. This is
used to indicate which half of the pair generated the output file for
paired experiments, and is not needed when we compute the unpaired base
times.Analyzing the Results for Level-A and Level-B We
provide scripts in smt_analysis which automatically parse
the output files and generate the
and
values for each experiment and isolation type. To get
values, run computeSMTslowdown.py with the baseline
()
and paired
()
timing results. For example, to generate the values for DIS from a run
on March 9th at 2pm with ID ae, run:
user@machine:/mc2-scripts-and-benchmarks$ ./smt_analysis/computeSMTslowdown.py dis/Mar09-16-c-i-ae-A.txt dis/Mar09-16-c-i-ae-B.txt dis/Mar09-14-c-i-ae.txt
Analyzing results using Level-A/B methodology...
Average offset is: 116.90ns
Bench field matrix neighborho pointer transitive update
field : 0.397 0.617 0.47 -0.0136 -0.357 0.0846
matrix : N/A 0.599 0.789 0.142 0.248 0.341
neighborhood : N/A N/A 0.523 N/A N/A -0.293
pointer : N/A N/A N/A 0.114 0.218 0.0994
transitive : N/A N/A N/A N/A 0.0815 0.129
update : N/A N/A N/A N/A N/A 0.136
Overall average is 0.228 with standard deviation 0.287 using `max`
3 of 19 M_i:j values are at most zero - 15.789473684210526 percent
0 of 19 M_i:j values are greater than one - 0.0 percent
For 16 of 19 M_i:j values in (0, 1], average: 0.312 with std. dev. 0.22 (coeff. var. 0.706) using `max`
bucket | ________________________________________________________________________________ Total Counts
[0.081493, 0.152198) | ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀ 7
[0.152198, 0.222903) | ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 1
[0.222903, 0.293608) | ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 1
[0.293608, 0.364313) | ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 1
[0.364313, 0.435018) | ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 1
[0.435018, 0.505723) | ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 1
[0.505723, 0.576428) | ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 1
[0.576428, 0.647132) | ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡷⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 2
[0.647132, 0.717837) | ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 0
[0.717837, 0.788542) | ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 1
‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
The first line of results details the average offset between the start of each member of a pair from the other. This should be fairly small. The following is a table of values. Next several summary statistics are presented alongside a simple histogram of the values. How does this correspond to the values in Tbl. 1?
M_i:j values in (0, 1]M_i:j values greater than one /
100coeff. var. on the
M_i:j values in (0, 1] lineThe Tbl. 1 values also appear in bottom right section of Tbl. 2.
Analyzing the Results for Level-C The process for
this is very similar to that used for Level-A and Level-B, except you
use computeLCslowdown.py and need to specify fewer inputs.
For example, to generate the values for DIS from a run on March 9th at
2pm with ID ae, run:
user@machine:/mc2-scripts-and-benchmarks$ ./smt_analysis/computeLCslowdown.py dis/Mar09-14-c-xi-async-ae.txt dis/Mar09-14-c-xi-ae.txt
Reading file using asynchronous pair format...
Bench M_i
field : 1.89
matrix : 2.17
neighborhood : 2.26
pointer : 1.51
transitive : 1.09
update : 1.54
Overall average is 1.75 with standard deviation 0.408 using `mean`
0 of 6 M_i values are at most one - 0.0 percent
2 of 6 M_i values are greater than two - 33.333333333333336 percent
bucket | ________________________________________________________________________________ Total Counts
[1.092161, 1.209424) | ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 1
[1.209424, 1.326688) | ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 0
[1.326688, 1.443951) | ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 0
[1.443951, 1.561215) | ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀ 2
[1.561215, 1.678478) | ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 0
[1.678478, 1.795741) | ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 0
[1.795741, 1.913005) | ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 1
[1.913005, 2.030268) | ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 0
[2.030268, 2.147531) | ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 0
[2.147531, 2.264795) | ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀ 2
‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
As values are a representation of the worst slowdown possible against any other task, the table is not all-to-all as with the values. The overall average and standard deviation correspond to the numbers in the bottom-right of Tbl. 2.
These benchmarks are needed to reproduce Fig. 7.
Note: The 3-D plot inset to Fig. 7, the 3-D plots in the supplemental appendix, and the derived “Cache Sens.” entries in Tbl. 2 can also be generated via a variant of these benchmarks, but the experiments suffer a combinatorial explosion and are extremely lengthy. Instructions are not included here, but can be made available upon request via the AE PC chair.
Platform Compatibility Exact reproduction requires the AMD Ryzen 9 3950X and the MC^2 kernel, but similar results should be evident on any Linux system with a Intel CAT/AMD QoS Extension-supporting processor. Our scripts automatically check for the MC^2 kernel, and these checks will have to be commented out to run on other kernels.
Running the Experiments
cd mc2-scripts-and-benchamrks; make benchmarks.cd thrasher; make; cd ...cache-ae on core 16 (CPU
ID 15):
cd dis; sudo ../run_bench.sh -m dis -p 15 -l 100 -b dis2MbInNames.txt -B -W inputs/4mb_WSS -T in0 -C inputs/caches_all cache-ae
(see the help text of run_bench.sh for additional parameter
detail).Plotting the Results
matplotlib, and
numpy. To install these on Ubuntu:
sudo apt install python3 python3-matplotlib python3-numpy.postproc.sh.
For an experiment run at 2pm on March 9th:
./postproc.sh Mar9-14-c-xi-cache-ae../plot_cache_sensitivity_at_4mb.py Mar9-14-c-xi-cache-ae-pp.txt.cleanThese experiments are needed to reproduce the plots in Sec. 6 and our online appendix. If you are unable to run SMT Interference Experiments and Cache Sensitivity Benchmarks, the data from Tbl. 2 in our paper has been prepopulated.
After completing Schedulabilty Framework
Setup, see mc2-sched-study/README.md for detailed
instructions on running, plotting, and summarizing the output.
These experiments use the schedulability study code to generate tasksets for our case study in Sec. 7.
After completing Schedulabilty Framework
Setup, see mc2-sched-study/README.md for detailed
instructions.
These experiments test the holistic viability of our system by running and checking for the tardiness of some schedulable task sets from our schedulability study. If you were unable to complete Taskset Generation, the tasksets for our paper are included and can be used instead.
This requires an AMD Zen-2-based processor with at least 8 cores and SMT.
Running the Case Study
export LIBLITMUS=$(pwd)/liblitmusmake case-study from
mc2-scripts-and-benchmarkssudo setsched MC2 from liblitmussudo -E ./rtas21-ae-case-study.sh to use the task
sets from our paper (these are stored in
ABmod_light_TACle-SDVBS_5.5)sudo -E ./rtas21-ae-case-study.sh <directory with your task sets> <number of task sets>
to use your own task set(<program name>/<pid>:<job number>) <log message>.
If a job is tardy, the log message includes the absolute and relative
tardiness. Otherwise, regular liveness messages are printed by the
Level-C tasks. (You can review _rt_start_loop() in
mc2-scripts-and-benchmarks/extra.h to see how this
works.)Note: We have found our scheduler to be highly robust to non-real-time interference, so it should be safe to continue running other work on the system while the case study executes.
Obtain, build, and install the kernel sources by running:
git clone --branch rtas21-ae https://github.com/JoshuaJB/litmus-rt.git
cd litmus-rt
sudo ./SETUP_MC2.sh # This installs kernel build dependencies
make bzImage modules -j8
sudo make INSTALL_MOD_STRIP=1 modules_install install # This copies the kernel to /boot/
To boot with the kernel, simply reboot and select the mc2-v2 kernel from your bootloader.
Note: We do not support graphics drivers when page coloring is enabled, so plan to only interact with your system over SSH, serial, or some other remote access protocol after rebooting.
Compatibility Note: If you are not running on an AMD
Zen-2-based system, SETUP_MC2.sh should print a warning and
automatically disable page coloring.
To download and make this library, run the following:
git clone https://github.com/JoshuaJB/liblitmus.git
cd liblitmus
make
Note: You must complete the Kernel Setup first and have python
on your PATH.
Download our code and documentation by running:
git clone --recurse-submodules --branch rtas21-ae http://rtsrv.cs.unc.edu/public/mc2-scripts-and-benchmarks.git
cd mc2-scripts-and-benchmarks
This framework is needed for the Schedulability Study and Taskset Generation. It requires Python
3.6 or newer, NumPy, Matplotlib and Gurobi with Python bindings. Gurobi
is commercial software, however free academic licences are available.
Please follow these
instructions to obtain Gurobi. Please see your package maintainers
on how to obtain a recent version of Python if your current version is
too old. If running on Ubuntu,
sudo apt install python3-matplotlib python3-numpy is
sufficient for the other dependencies. We do not support running on
Windows.
After installing the dependencies, download our code by running:
git clone --branch rtas21-ae https://github.com/shosborn/MC2-2020.git mc2-sched-study
cd mc2-sched-study