There are the steps required to reproduce all the experiments in our paper:
J. Bakita, S. Ahmed, S. H. Osborne, S. Tang, J. Chen, F. D. Smith, J. H. Anderson, "Simultaneous Multithreading in Real-Time Mixed-Criticality Systems", Proceedings of the 27th IEEE Real-Time and Embedded Technology and Applications Symposium, May 2021, to appear. (PDF)
Each step relies on completion of all the previous ones:
For full reproduction, each of these steps relies on all of the previous ones to have completed. If any steps are not feasible due to errors or hardware limitations, please double-check that you have the latest code of our code and instructions before skipping ahead. Where possible, we prepopulate each step's configuration so that you can still continue with evaluation even if you are unable to directly derive every configuration parameter.
These steps were last updated .
These benchmarks measure the slowdown incurred by each task when paired onto the SMT threads of a core. We measure this slowdown under three different system configurations:
xi)i3)i)These benchmarks reproduce Tbl. 1 and the "SMT Effect" entries of Tbl. 2, plus PETi, PETi* and PETi : j for all i and all j. We measure all PETi and PETi : j under all system configurations (xi, i3, and i), but only measure PETi : j values without isolation (xi) as keeping with our system design.
Platform Compatibility An AMD Ryzen 9 3950X-based system is needed for exact reproduction, but any AMD Zen-2-based processor should yield similar results. Cache coloring requires the processor's cache mapping function, and is thus not available on other systems. However, the xi and i3 configurations do not require cache coloring and can be run on any Linux kernel/processor configuration which supports Intel CAT/AMD QoS Extensions via resctrl. Our scripts automatically check for the MC^2 kernel however, and these checks will have to be commented out to run on other systems.
Running the Experiments
CONFIG_DEFAULT_CMDLINE, add isolcpus=15,31 and nohz_full=15,31 if you intend to benchmark on core 16 (CPU 15) in a 16-core system. The second number is the CPU ID that Linux uses for the thread paired with your core. You can obtain this second number by taking your benchmarking core and adding the total number of cores in your system.irqaffinity=0 to the kernel cmdline.sudo service irqbalance stopcd mc2-scripts-and-benchmarks; make benchmarkscd wbinvd; make; sudo insmod wbinvd.kocd thrasher; makeae.Run all the benchmarks. For an experiment on core 16 (Linux CPU IDs of 15 and 31) with 100 samples and a run identifier of ae:
sudo ./run_all_tacle.sh 100 ae 15 31
sudo ./run_all_dis.sh 100 ae 15 31
sudo ./run_all_sd-vbs.sh 100 ae 15 31Wait until the scripts finish. This process is very sensitive to interference, so please refrain from running other work on the system as much as possible. When each script completes, it prints the names and locations of each output file. Record these for later analysis.
Output Format Each output file uses the following format:
Date Hour Isolation type Run ID
v v v v
Mar09-13-c-xi-async-my_1k_run_id.txt
^ ^ ^
Contention type └ Pairing type └ Pair ID (not shown)xc (no contending tasks) or c (contending tasks). The scripts automatically run all experiments with contention.xi, i3, or i as introduced earlier.async or non-present. This field indicates if we are measuring PETi* values, which do not require the pairs to start synchronously.A, B, or non-present. This is used to indicate which half of the pair generated the output file for paired experiments, and is not needed when we compute the unpaired base times.Analyzing the Results for Level-A and Level-B We provide scripts in smt_analysis which automatically parse the output files and generate the Mi : j and Mi values for each experiment and isolation type. To get Mi : j values, run computeSMTslowdown.py with the baseline (PETi) and paired (PETi : j) timing results. For example, to generate the values for DIS from a run on March 9th at 2pm with ID ae, run:
user@machine:/mc2-scripts-and-benchmarks$ ./smt_analysis/computeSMTslowdown.py dis/Mar09-16-c-i-ae-A.txt dis/Mar09-16-c-i-ae-B.txt dis/Mar09-14-c-i-ae.txt
Analyzing results using Level-A/B methodology...
Average offset is: 116.90ns
Bench field matrix neighborho pointer transitive update
field : 0.397 0.617 0.47 -0.0136 -0.357 0.0846
matrix : N/A 0.599 0.789 0.142 0.248 0.341
neighborhood : N/A N/A 0.523 N/A N/A -0.293
pointer : N/A N/A N/A 0.114 0.218 0.0994
transitive : N/A N/A N/A N/A 0.0815 0.129
update : N/A N/A N/A N/A N/A 0.136
Overall average is 0.228 with standard deviation 0.287 using `max`
3 of 19 M_i:j values are at most zero - 15.789473684210526 percent
0 of 19 M_i:j values are greater than one - 0.0 percent
For 16 of 19 M_i:j values in (0, 1], average: 0.312 with std. dev. 0.22 (coeff. var. 0.706) using `max`
bucket | ________________________________________________________________________________ Total Counts
[0.081493, 0.152198) | ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀ 7
[0.152198, 0.222903) | ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 1
[0.222903, 0.293608) | ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 1
[0.293608, 0.364313) | ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 1
[0.364313, 0.435018) | ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 1
[0.435018, 0.505723) | ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 1
[0.505723, 0.576428) | ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 1
[0.576428, 0.647132) | ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡷⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 2
[0.647132, 0.717837) | ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 0
[0.717837, 0.788542) | ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 1
‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾The first line of results details the average offset between the start of each member of a pair from the other. This should be fairly small. The following is a table of Mi : j values. Next several summary statistics are presented alongside a simple histogram of the Mi : j values. How does this correspond to the values in Tbl. 1?
M_i:j values in (0, 1]M_i:j values greater than one / 100coeff. var. on the M_i:j values in (0, 1] lineThe Tbl. 1 values also appear in bottom right section of Tbl. 2.
Analyzing the Results for Level-C The process for this is very similar to that used for Level-A and Level-B, except you use computeLCslowdown.py and need to specify fewer inputs. For example, to generate the values for DIS from a run on March 9th at 2pm with ID ae, run:
user@machine:/mc2-scripts-and-benchmarks$ ./smt_analysis/computeLCslowdown.py dis/Mar09-14-c-xi-async-ae.txt dis/Mar09-14-c-xi-ae.txt
Reading file using asynchronous pair format...
Bench M_i
field : 1.89
matrix : 2.17
neighborhood : 2.26
pointer : 1.51
transitive : 1.09
update : 1.54
Overall average is 1.75 with standard deviation 0.408 using `mean`
0 of 6 M_i values are at most one - 0.0 percent
2 of 6 M_i values are greater than two - 33.333333333333336 percent
bucket | ________________________________________________________________________________ Total Counts
[1.092161, 1.209424) | ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 1
[1.209424, 1.326688) | ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 0
[1.326688, 1.443951) | ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 0
[1.443951, 1.561215) | ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀ 2
[1.561215, 1.678478) | ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 0
[1.678478, 1.795741) | ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 0
[1.795741, 1.913005) | ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 1
[1.913005, 2.030268) | ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 0
[2.030268, 2.147531) | ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ 0
[2.147531, 2.264795) | ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀ 2
‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾As Mi values are a representation of the worst slowdown possible against any other task, the table is not all-to-all as with the Mi : j values. The overall average and standard deviation correspond to the numbers in the bottom-right of Tbl. 2.
These benchmarks are needed to reproduce Fig. 7.
Note: The 3-D plot inset to Fig. 7, the 3-D plots in the supplemental appendix, and the derived "Cache Sens." entries in Tbl. 2 can also be generated via a variant of these benchmarks, but the experiments suffer a combinatorial explosion and are extremely lengthy. Instructions are not included here, but can be made available upon request via the AE PC chair.
Platform Compatibility Exact reproduction requires the AMD Ryzen 9 3950X and the MC^2 kernel, but similar results should be evident on any Linux system with a Intel CAT/AMD QoS Extension-supporting processor. Our scripts automatically check for the MC^2 kernel, and these checks will have to be commented out to run on other kernels.
Running the Experiments
cd mc2-scripts-and-benchamrks; make benchmarks.cd thrasher; make; cd ...cache-ae on core 16 (CPU ID 15): cd dis; sudo ../run_bench.sh -m dis -p 15 -l 100 -b dis2MbInNames.txt -B -W inputs/4mb_WSS -T in0 -C inputs/caches_all cache-ae (see the help text of run_bench.sh for additional parameter detail).Plotting the Results
matplotlib, and numpy. To install these on Ubuntu: sudo apt install python3 python3-matplotlib python3-numpy.postproc.sh. For an experiment run at 2pm on March 9th: ./postproc.sh Mar9-14-c-xi-cache-ae../plot_cache_sensitivity_at_4mb.py Mar9-14-c-xi-cache-ae-pp.txt.cleanThese experiments are needed to reproduce the plots in Sec. 6 and our online appendix. If you are unable to run SMT Interference Experiments and Cache Sensitivity Benchmarks, the data from Tbl. 2 in our paper has been prepopulated.
After completing Schedulabilty Framework Setup, see mc2-sched-study/README.md for detailed instructions on running, plotting, and summarizing the output.
These experiments use the schedulability study code to generate tasksets for our case study in Sec. 7.
After completing Schedulabilty Framework Setup, see mc2-sched-study/README.md for detailed instructions.
These experiments test the holistic viability of our system by running and checking for the tardiness of some schedulable task sets from our schedulability study. If you were unable to complete Taskset Generation, the tasksets for our paper are included and can be used instead.
This requires an AMD Zen-2-based processor with at least 8 cores and SMT.
Running the Case Study
export LIBLITMUS=$(pwd)/liblitmusmake case-study from mc2-scripts-and-benchmarkssudo setsched MC2 from liblitmussudo -E ./rtas21-ae-case-study.sh to use the task sets from our paper (these are stored in ABmod_light_TACle-SDVBS_5.5)sudo -E ./rtas21-ae-case-study.sh <directory with your task sets> <number of task sets> to use your own task set(<program name>/<pid>:<job number>) <log message>. If a job is tardy, the log message includes the absolute and relative tardiness. Otherwise, regular liveness messages are printed by the Level-C tasks. (You can review _rt_start_loop() in mc2-scripts-and-benchmarks/extra.h to see how this works.)Note: We have found our scheduler to be highly robust to non-real-time interference, so it should be safe to continue running other work on the system while the case study executes.
Obtain, build, and install the kernel sources by running:
git clone --branch rtas21-ae https://github.com/JoshuaJB/litmus-rt.git
cd litmus-rt
sudo ./SETUP_MC2.sh # This installs kernel build dependencies
make bzImage modules -j8
sudo make INSTALL_MOD_STRIP=1 modules_install install # This copies the kernel to /boot/To boot with the kernel, simply reboot and select the mc2-v2 kernel from your bootloader.
Note: We do not support graphics drivers when page coloring is enabled, so plan to only interact with your system over SSH, serial, or some other remote access protocol after rebooting.
Compatibility Note: If you are not running on an AMD Zen-2-based system, SETUP_MC2.sh should print a warning and automatically disable page coloring.
To download and make this library, run the following:
git clone https://github.com/JoshuaJB/liblitmus.git
cd liblitmus
makeNote: You must complete the Kernel Setup first and have python on your PATH.
Download our code and documentation by running:
git clone --recurse-submodules --branch rtas21-ae http://rtsrv.cs.unc.edu/cgit/cgit.cgi/mc2-scripts-and-benchmarks.git
cd mc2-scripts-and-benchmarksThis framework is needed for the Schedulability Study and Taskset Generation. It requires Python 3.6 or newer, NumPy, Matplotlib and Gurobi with Python bindings. Gurobi is commercial software, however free academic licences are available. Please follow these instructions to obtain Gurobi. Please see your package maintainers on how to obtain a recent version of Python if your current version is too old. If running on Ubuntu, sudo apt install python3-matplotlib python3-numpy is sufficient for the other dependencies. We do not support running on Windows.
After installing the dependencies, download our code by running:
git clone --branch rtas21-ae https://github.com/shosborn/MC2-2020.git mc2-sched-study
cd mc2-sched-study