COMP 633 Parallel Computing http://www.cs.unc.edu/~prins/Classes/633/ Fall 2017 (Tue Aug 22 - Tue Dec 5) TTh 2:00 - 3:15 PM FB 009 Instructor: Jan Prins, FB 334, Tel: 919-590-6213, prins@cs.unc.edu Office hours: Mon 2 - 3 pm and by appt. (email me with questions anytime)

Overview

• be able to design and analyze parallel algorithms for a variety of problems and computational models,
• be familiar with the hardware and software organization of high-performance parallel computing systems, and
• have experience with the implementation of parallel applications on high-performance computing systems, and be able to measure, tune, and report on their performance.

Additional information including the course syllabus can be found in the course overview.

All parallel programming models discussed in this class are supported on BASS or phaedra which are available for use in this class.

Announcements

• This course will use Piazza to manage class questions and discussions online.
  • Please join our discussion group using this link: http://piazza.com/unc/fall2017/comp633
  • Please access the discussion group using this link: http://piazza.com/unc/fall2017/comp633/home

On-line Handouts

• Course Lecture Notes
• Some sample midterm questions are available.

Reading Assignments

• (for Thu Oct 12) Read material on Intel Xeon Phi
• (for Tue Oct 10) Read Nyland et. al, Fast N-Body Simulation with CUDA. Check supplementary materials for Cuda in the Software section below.
• (for Thu Oct 5) Skim through introductory material on CUDA introduction
• (for Thu Sep 28) Hennessy & Patterson Ch. 8, sections 8.5 - 8.6 (synchronization primitives in shared memory, and memory consistency models).
• (for Tue Sep 26) Look through Memory consistency models tutorial (sections 1-6, pp 1 -17).
• (For Thu Sep 21) Glance through The Implementation of the Cilk-5 Multithreaded Language, sections 1 - 3.
  For a modern look at Cilk, see this short tutorial on Intel Cilk Plus which also includes expression of data parallelism using matlib-like array notations (which generally are executed using the advanced vector extensions in Intel processors).
• (For Tue Sep 19) Look through Open MP Tutorial sections 7-9 and read about TASK directives.
• (For Tue Sep 12) Look through OpenMP Tutorial sections 1-6. Most examples are shown in C/C++ and in Fortran. We will be using C/C++. Ignore WORKSHARE and TASK directives, and discussion of nested parallel constructs.
• (for Thu Sep 7) Read the overview of Memory Hierarchy in Cache-based Systems.
• (For Tue Sep 5) PRAM handout, skim section 5
• (For Thu Aug 31) PRAM handout, sections 3.6, 4.1
• (For Tue Aug 29) PRAM handout, sections 3.2, 3.3, 3.5
• (For Thu Aug 24) Read PRAM Handout secns 1, 2, 3.1 (pp 1 - 8)
• (For Tue Aug 22) Read through the course overview.

Written and Programming Assignments

• (Aug 29) Written assignment WA1 is available. Due date is Sep 19.
  WA1 sample solutions
• (Sep 12) Programming assignment PA1a is available. Due date is Tue Sep 26.
  • A sample sequential implementation pa1-ref.c of the all-pairs method that yields reasonable performance on a modern machine.
  • Here is a simulation of 3 bodies to which you can compare your implementation. Set the initial positions, velocities, masses, and value of G as shown for Time = 0 below. Run for nts (k) time steps. You don't need exact agreement, but I would expect 5 or 6 digits of accuracy. Note that the 3 bodies are following a figure 8 orbit, and the final position should be close to the initial postion. Also note that this sort of simulation (n small, #time steps large) is not what we're trying to optimize. We are interested in large n performance (and only perform a few timesteps to measure it).

    	nbref sequential all-pairs n-body, n = 3, nts = 6325914, G = 1, DeltaT = 1e-06
    	Time = 0
    	P0 = (-0.97000436,0.24308753)        V0 = (-0.46620368,-0.43236573)  m = 1
    	P1 = (0.97000436,-0.24308753)        V1 = (-0.46620368,-0.43236573)  m = 1
    	P2 = (0,0)                           V2 = (0.93240737,0.86473146)    m = 1
    
    	Time = 1e-06
            P0 = (-0.97000483,0.2430871)         V0 = (-0.46620247,-0.43236603)  m = 1
            P1 = (0.97000389,-0.24308796)        V1 = (-0.4662049,-0.43236543)   m = 1
            P2 = (9.3240737e-07,8.6473146e-07)   V2 = (0.93240737,0.86473146)    m = 1
    
    	Time = 3.162957
    	P0 = (0.97000754,0.24311667)         V0 = (0.46624203,-0.43234625)   m = 1
    	P1 = (-0.97003188,-0.24309202)       V1 = (0.46616351,-0.43236621)   m = 1
    	P2 = (2.4340532e-05,-2.4652558e-05)  V2 = (-0.93240554,0.86471246)   m = 1
    
    	Time = 6.325914
    	P0 = (-0.96997182,0.24313017)        V0 = (-0.46636412,-0.43232847)  m = 1
    	P1 = (0.9701019,-0.24301235)         V1 = (-0.46601452,-0.432412)    m = 1
    	P2 = (-0.00013007996,-0.00011781508) V2 = (0.93237864,0.86474047)    m = 1
    
    	Initial momentum = (0,       0)
    	Final   momentum = (-1.726397e-13, -6.71685e-14)
    	Relative error = 1.85246e-13
    

• (Sep 28) Programming assignment PA1b is available. Due date is Oct. 24.
• (Oct 7) Written Assignment WA2 is available. Due date is Sun Oct 15.
  WA2 Sample Solutions are available

Platforms and Programming Models

Platforms

• The Bass system supports the OpenMP and MPI programming models. The general instructions for getting started on bass are supplemented below with specific instructions for each programming model. When you login to bass.cs.unc.edu you are connected to a specific node on bass dedicated to interactive program development. You can compile programs on this node. Shared-memory programs run within an individual node on bass. Distributed-memory programs run across multiple nodes in Bass. The login node should not be used to run your programs, although a short debug test for a few seconds and no more than 4 cores should be OK. In general programs that need multiple nodes or dedicated nodes or GPUs should be submitted to queues that are managed by the Grid Engine job scheduler.
• The Phaedra system is a compute server supporting OpenMP, Cilk, and Xeon Phi accelerator programming models.
• gamma-x51-1 is a server supporting the CUDA accelerator programming model.

OpenMP

• OpenMP reference: Specification of OpenMP 3.0 API for C/C++. You may be more interested in OpenMP support in gcc 4.4.7 (the compiler on Bass).
• Bass-specific material
  • Getting started on Bass.
  • To get accurate performance information run your programs on a dedicated node as a batch job with a shell script myjob using
    qsub -pe smp 16 myjob
    or interactively via
    qlogin -pe smp 16
    Do not park yourself on this node as everyone else in the class will be held up.
  • A directory with the sample diffusion program discussed in class.
  • Command lines for the compilation and execution of programs on bass.unc.edu.
    1. C compilation to create a sequential program (compiler ignores OpenMP directives and does not link with the OpenMP runtime library):
       gcc -O3 -o prog prog.c (Gnu C compiler 4.4.7)
    2. C compilation to create a parallel program (OpenMP 3.0 directives honored and program linked with the OpenMP runtime library)
       gcc -fopenmp -O3 -o prog prog.c (Gnu C compiler 4.4.7)
• Phaedra-specific material
  • Phaedra is a 20-core Xeon E5-2650 server with eight attached Intel Xeon Phi 5110P accelerators. The server hosts the Intel Parallel Studio XE 2017 compilers and performance analysis tools to access the accelerators. Students in COMP 633 have a login on phaedra, and OpenMP programs run directly on the server.
  • The Intel C compiler icc directly supports OpenMP 4.0 with tasking and accelerator offload, and Cilk extensions to C including Cilk array notation. (C arrays permit aliasing which can inhibit vectorization).
  • Be sure to source /opt/intel/bin/compilervars.sh (bash) or source /opt/intel/bin/compilervars.csh (csh) to access the Intel compilers and tools.

• GPU: Cuda 7.0 on gamma-x51-1.
  • CUDA C Programming Guide (v8.0 Sep 2016)
  • Be sure to include /usr/local/cuda-7.0/bin in your path to access to the nvcc compiler and other Nvidia tools.
• Intel Xeon Phi: Parallel Studio XE 2017 on phaedra
  • For an introduction read An Overview of Programming for Intel Xeon processors and Intel Xeon Phi coprocessors by James Reinders.
  • For programming details see our class slides and Programming, Compiling and Optimizing for the Intel Xeon Phi Coprocessor by Martyn Corden
  • Access to the accelerators using OpenMP acc or Intel offload directives is supported by the Intel compiler icc.
  • Be sure to source /opt/intel/bin/compilervars.sh (bash) or source /opt/intel/bin/compilervars.csh (csh) to access the Intel compilers and tools.

MPI

• MPI reference material
  • Comprehensive site organizing all reference and tutorial information on MPI.
  • A short overview of MPI.
• Running MPI programs on Bass
  • Set up your environment on Bass as directed here. Select the openmpi-x86_64 MPI implementation (this is not in the list shown in the instructions). Note that the instructions presuppose you are running the bash shell. If your default shell is not bash, make sure you get into a bash shell by executing "bash -l". If you do not do this, the next step will fail.
  • You can compile your program on the Bass front end as follows (use mpiCC for C++ programs and remember to #include mpi.h in your programs)):
    mpicc -o myprog myprog.c
  • To execute your program you need to submit it to the grid engine for scheduling. You need to prepare a shell script that will be used to start your program with the appropriate command line arguments, and includes instructions to the scheduler for resources required.

  • Here is an example job submission script runprog.sh. See the instructions in lecture 18 for additional details. Note that if myprog above requires an argument you need to include the argument in the script below after $WDIR/myprog.

    #================== runprog.sh =================================
    #$ -S /bin/bash
    #$ -j y
    #$ -cwd
    #$ -l h_rt=0:5:0 
    #
    source /usr/local/bin/pathmunge.sh
    pathmunge usempi openmpi-x86_64
    WDIR=/home/yourname/path-to-your-project-dir
    cd $WDIR
    `which mpirun` -np $NSLOTS $WDIR/myprog  
    #=============================================================
    

  • The shell script specifies (through directives in the comments) that the script should be run in a bash shell, that all output should be placed in the current working directory, and that the job has a run time limit of 5 minutes. The subsequent shell commands specify the invocation of myprog on each processor assigned to the job.
  • To submit job to run using 8 processors use:
    qsub -pe MPI 8 runprog.sh
    
  • To make a larger scaling run use:
    qsub -pe MPI 32 runprog.sh (successive MPI processes will fill one node before being placed on the next)
    or
    qsub -pe rrMPI 32 runprog (successive MPI processes will be placed on different nodes according to availability)
    Note that your job and your MPI processes may end up sharing nodes with other requests, although this is not very likely given the small number of uses of Bass at the moment.
  • Once the job is submitted you can check its status using qstat
  • It can take several minutes longer than the expected runtime of your job for it to complete and for the files to show up in your directory.

Bibliography

1. PRAM Algorithms, S. Chatterjee, J. Prins, COMP 633 course notes, 2015.
2. Memory Hierarchy in Cache-Based Systems, R. v.d. Pas, Sun Microsystems, 2003.
3. OpenMP 3.1 tutorial, Blaise Barney, LLNL, 2015.
4. The Implementation of the Cilk-5 Multithreaded Language, M. Frigo, C. Leiserson, K. Randall, in Proceedings of ACM Conf. on Programming Language Design and Implementation, 1998.
5. Shared Memory Consistency Models: A Tutorial, S. V. Adve, K. Gharachorloo, DEC Western Research Labs Report 95/7, 1995.
6. Computer Architecture: A Quantitative Approach 2nd ed, D. Patterson, J. Hennessy, Morgan-Kaufmann 1996.
7. Fast N-Body Simulation with CUDA, L. Nyland, M. Harris, J. Prins, GPUGems 3, 2008.
8. An Overview of Programming for Intel Xeon processors an Intel Xeon Phi coprocessors, James Reinders, Intel Corp, 2012.
9. "Questions and Answers about BSP", D. Skillicorn, J. Hill, and W. McColl, Scientific Programming 6, 1997.
10. Message Passing Interface, Blaise Barney, LLNL 2015
11. Introduction to Parallel Computing: Design and Analysis of Algorithms, V. Kumar, A. Grama, A. Gupta, G. Karypis, Benjamin-Cummings, 1994.