COMP 633 Parallel Computing http://www.cs.unc.edu/~prins/Classes/633/ Fall 2020 (Tue Aug 11 - Tue Nov 17) TTh 3:00 PM - 4:15 PM online instruction Instructor: Jan Prins, FB 334, Tel: 919-590-6213, prins@cs.unc.edu Office hours: Mon 11AM - 12.30PM, Wed 9AM - 10.30AM. Use the class zoom id. If you plan to attend, send an an email in advance and I'll acknowledge. (email me with questions anytime)

Overview and Syllabus

• be able to design and analyze parallel algorithms for a variety of problems and computational models,
• be familiar with the hardware and software organization of high-performance parallel computing systems, and
• have experience with the implementation of parallel applications on high-performance computing systems, and be able to measure, tune, and report on their performance.

Course details can be found in the course information and syllabus.

Announcements

• Online Lectures
  • Lectures will use Zoom. The link for this class is https://unc.zoom.us/j/95771378990. Zoom requires you to authenticate with a unc.edu or *.unc.edu account. The meetings are recorded, so don't worry if there is a problem.
  • Zoom enables you to ask questions during the lecture, either by "hand raising" so I can call on you, or simply unmuting your microphone and asking your question. We will figure out what works best.
  • Lectures will be recorded, so you can review them later or watch them at a more suitable time. Of course, you will not be able to ask questions in this setting.
• Questions, answers, and discussions outside of lectures
  • We will use Piazza for asynchronous discussions.
  • Please join our discussion group using this link: http://piazza.com/unc/fall2020/comp633. Be sure to sign up using your unc email so I can restrict participation to students in this class.
  • Please access the discussion group using this link: http://piazza.com/unc/fall2020/comp633/home

Course Materials

• Lecture slides
• Lecture recordings

Reading Assignments

• (for Tue Nov 19) Look over Kumar et al., Basic Communication Operations.
• (For Thu Oct 22) Skim MPI tutorial by Blaise Barney, LLNL.
• (For Thu Oct 8) Skim the Questions and Answers about BSP pp 1-25. We will not use BSPLib directly, rather we use the BSP model together with communication operations from the MPI library.
• (For Tue Sep 29) Look over the Cuda programming guide (9.2) and the Cuda best practices guide (9.2).
  Read Nyland et. al, Fast N-Body Simulation with CUDA.
• (For Tue Sep 22) Hennessy & Patterson Ch. 8, sections 8.5 - 8.6
• (For Thu Sep 17) Look through Memory consistency models tutorial (sections 1-6, pp 1 -17).
• (For Tue Sep 8) Look through Open MP Tutorial sections 7-9 and read about TASK directives.
  Cilk is simple extension of C that includes tasking. For a modern look at Cilk, see this short tutorial on Intel Cilk Plus which also includes expression of data parallelism using matlib-like array notations (that use the vector operations on Intel Xeon processors).
• (For Thu Sep 3) Look through OpenMP Tutorial sections 3-5 and section 6 only up to the first exercise. Most examples are shown in C/C++ and in Fortran, so just ignore the Fortran.
• (For Tue Sep 1) Read the overview of Memory Hierarchy in Cache-based Systems
• (For Thu Aug 27) Read PRAM Handout skim section 5 (pp 20 - 23)
• (Tue Aug 25) Classes canceled by College of Arts and Sciences.
• (For Thu Aug 20) Read PRAM Handout secns 3.6, 4.1 (pp 15 - 19)
• (For Tue Aug 18) Read PRAM Handout secns 3.2, 3.3, 3.5 (pp 9 - 15)
• (For Thu Aug 13) Read PRAM Handout secns 1, 2, 3.1 (pp 1 - 8)

Written and Programming Assignments

1. Written Assignments
   • (Aug 18) Written assignment WA1. Due date is Thu Sep 3.
     WA1 sample solutions
   • (Sep 26) Written Assignment WA2. Due date is Mon Oct 5 anytime. You may work with another classmate or on your own.
     WA2 sample solutions
   • (Nov 5) Written Assignment WA3. Due date is Tue Nov 17 before the start of class. You may work with another classmate or on your own.
     (Nov 18) WA3 sample solutions
2. Programming Assignments
   • (Sep 8) Programming Assignment PA1(a) is available. You can work together with a partner on this project or you can choose to work on your own. Due date is Tue Sep 15 (submit via email to me before the start of class!).
     Here are some sample implementations shown in class
     • A three time-step evolution of sample 3-body system was posted on piazza.
     • v2.c is an all-pairs implementation in C.
       Compile with icc or gcc: icc -Ofast -fopenmp -o v2 v2.c
     • v3n.c is a half-pairs sample implementation in C.
       compile with icc or gcc: icc -Ofast -fopenmp -o v3n v3n.c
   • (Sep 17) Programming Assignment PA1(b) is available. Due date is Oct 1 (start of class).
   • (Oct 20) Programming Assignment PA2 is available. Due date is Nov 17.
     LLoyds K-means clustering algorithm.

Available Platforms

• phaedra is an Intel Xeon E5-2650v4 compute server dedicated to this class. It has with 20 cores and an attached Nvidia Titan V100 accelerator. OpenMP, Cilk, and Cuda programming models are supported. Login using your onyen (instead of a CS login).
• longleaf is a research computing cluster with ~350 Intel Xeon E5-2643 nodes providing 24 cores per node. OpenMP and Cilk programming models are supported on individual nodes. Compute jobs are submitted using slurm.
• dogwood is a research computing cluster with ~240 Intel Xeon E5-2699A nodes providing 44 cores per node. The MPI programming model is supported to coordinate and communicate among nodes. A subset of the nodes have Intel Xeon Phi (KNL) accelerators. The individual nodes support MPI, OpenMP, OpenACC, and Intel offload (Intel Xeon Phi) programming models. Compute jobs are submitted using slurm.

Available Tools

Compilers

• Intel C/C++ compiler (icc/icpc 2020)
  • supports OpenMP 4.5 with tasking and accelerator offload.
  • supports Cilk extensions to C including Cilk array notation (until Sep 10 2020)
  • On phaedra, source /opt/intel/bin/compilervars.sh intel64 (bash) or source /opt/intel/bin/compilervars.csh intel64 (csh) to access the Intel compilers and tools.
  • On research computing clusters use "module add icc" to access Intel compilers.

OpenMP

• Shared memory parallel programming. Specification of the OpenMP 4.5 API for C/C++ (supported by the Intel compilers). For a more accessible introduction see the tutorial for OpenMP 3.1 in the Bibliography below.

Accelerators

• Nvidia GPUs: programmed using Cuda C (Compute Capability 9.2 for V100 on phaedra).
  • CUDA C Programming Guide (v9.2 Aug 2018)
  • Be sure to include /usr/local/cuda-9.2/bin in your path on phaedra to access the nvcc compiler and other Nvidia tools.

MPI

• MPI reference material
  • Comprehensive site organizing all reference and tutorial information on MPI.
  • A short overview of MPI.
• MPI programs can be submitted to dogwood

Bibliography

1. PRAM Algorithms, S. Chatterjee, J. Prins, COMP 633 course notes, 2015.
2. Memory Hierarchy in Cache-Based Systems, R. v.d. Pas, Sun Microsystems, 2003.
3. OpenMP 3.1 tutorial, Blaise Barney, LLNL, 2015.
4. Cilk Plus Tutorial (online tutorial).
5. Shared Memory Consistency Models: A Tutorial, S. V. Adve, K. Gharachorloo, DEC Western Research Labs Report 95/7, 1995.
6. Computer Architecture: A Quantitative Approach 2nd ed, D. Patterson, J. Hennessy, Morgan-Kaufmann 1996.
7. Fast N-Body Simulation with CUDA, L. Nyland, M. Harris, J. Prins, GPUGems 3, 2008.
8. Questions and Answers about BSP, D. Skillicorn, J. Hill, and W. McColl, Scientific Programming 6, 1997.
9. Message Passing Interface, Blaise Barney, LLNL 2015
10. Introduction to Parallel Computing: Design and Analysis of Algorithms - Chapter 3, V. Kumar, A. Grama, A. Gupta, G. Karypis, Benjamin-Cummings, 1994.