COMP 633 Parallel Computing http://www.cs.unc.edu/~prins/Classes/633/ Fall 2016 (Tue Aug 23 - Tue Dec 6) TTh 12:30 - 1:45 PM FB 009 Instructor: Jan Prins, FB 334, Tel: 919-590-6213, prins@cs.unc.edu Office hours: Mon 2 - 3 pm and by appt. (email me with questions anytime)

Overview

• be able to design and analyze parallel algorithms for a variety of problems and computational models,
• be familiar with the hardware and software organization of high-performance parallel computing systems, and
• have experience with the implementation of parallel applications on high-performance computing systems, and be able to measure, tune, and report on their performance.

Additional information including the course syllabus can be found in the course overview.

All parallel programming models discussed in this class are supported on BASS or phaedra which are available for use in this class.

Announcements

• This course will use Piazza to manage class questions and discussions online.
  • Please join our discussion group using this link: http://piazza.com/unc/fall2016/comp633
  • Please access the discussion group using this link: http://piazza.com/unc/fall2016/comp633/home

On-line Handouts

• Course Lecture Notes

Reading Assignments

• (For Thu Sep 22) Glance through The Implementation of the Cilk-5 Multithreaded Language, sections 1 - 3.
  For a modern look at Cilk, see this short tutorial on Intel Cilk Plus which also includes expression of data parallelism using matlib-like array notations (which generally are executed using the advanced vector extensions in Intel processors).
• (For Tue Sep 20) Look through Open MP Tutorial sections 7-9 and read about TASK directives.
• (For Tue Sep 13) Look through OpenMP Tutorial sections 1-6. Most examples are shown in C/C++ and in Fortran. We will be using C/C++. Ignore WORKSHARE and TASK directives, and discussion of nested parallel constructs.
• (for Thu Sep 8) Read the overview of Memory Hierarchy in Cache-based Systems.
• (For Tue Sep 6) PRAM Handout, sections 3.6, 4.1, skim section 5
• (For Tue Aug 30) PRAM handout, sections 3.2, 3.3, 3.5
• (For Thu Aug 25) Read PRAM Handout secns 1, 2, 3.1 (pp 1 - 8)
• (For Tue Aug 23) Look over the course overview.

Written and Programming Assignments

• (Sep 1) Written assignment WA1 is available. Due date is Sep 20.
  
• (Sep 13) Programming assignment PA1a is available. Due date is Tue Sep 27.

Software

We will be using the Bass system for programming assignments. The Bass system supports all the programming models studied in this class except those used with Intel Xeon Phi accelerators and Cilk -- a separate compute server phaedra provides access to these capabilities.

The general instructions for getting started on bass are supplemented below with specific instructions for each programming model. When you login to bass.cs.unc.edu you are connected to a specific node on bass dedicated to interactive program development. You can compile programs on this node. Shared-memory programs run within an individual node on bass. Distributed-memory programs run across multiple nodes in Bass. The login node should not be used to run your programs, although a short debug test for a few seconds and no more than 4 cores should be OK. In general programs that need multiple nodes or dedicated nodes or GPUs should be submitted to queues that are managed by the Grid Engine job scheduler.

OpenMP

• OpenMP reference: Specification of OpenMP 3.0 API for C/C++. You may be more interested in OpenMP support in gcc 4.4.7 (the compiler on Bass).
• Bass-specific material
  • Getting started on Bass.
  • To get accurate performance information run your programs on a dedicated node as a batch job with a shell script myjob using
    qsub -pe smp 16 myjob
    or interactively via
    qlogin -pe smp 16
    Do not park yourself on this node as everyone else in the class will be held up.
  • A directory with the sample diffusion program discussed in class.
  • Command lines for the compilation and execution of programs on bass.unc.edu.
    1. C compilation to create a sequential program (compiler ignores OpenMP directives and does not link with the OpenMP runtime library):
       gcc -O3 -o prog prog.c (Gnu C compiler 4.4.7)
    2. C compilation to create a parallel program (OpenMP 3.0 directives honored and program linked with the OpenMP runtime library)
       gcc -fopenmp -O3 -o prog prog.c (Gnu C compiler 4.4.7)

CUDA

• Cuda 4.2 on Bass
  • CUDA C Programming Guide (v4.2 Apr 2012)
  • Compile your code on the login node. See the instructions on the Bass web page to set the environment correctly.
  • Get a login on a GPU node: qlogin -l gpu_host,gpus=1 and run your program.
• Cuda 5.0 on killdevil and cuda 5.5 on other GPU-equipped machines
  • CUDA C Programming Guide (v5.5 July 2013)

MPI

• MPI reference material
  • Comprehensive site organizing all reference and tutorial information on MPI.
  • A short overview of MPI.
• Running MPI programs on Bass
  • Set up your environment on Bass as directed here. Select the openmpi-x86_64 MPI implementation (this is not in the list shown in the instructions). Note that the instructions presuppose you are running the bash shell. If your default shell is not bash, make sure you get into a bash shell by executing "bash -l". If you do not do this, the next step will fail.
  • You can compile your program on the Bass front end as follows (use mpiCC for C++ programs and remember to #include mpi.h in your programs)):
    mpicc -o myprog myprog.c
  • To execute your program you need to submit it to the grid engine for scheduling. You need to prepare a shell script that will be used to start your program with the appropriate command line arguments, and includes instructions to the scheduler for resources required.

  • Here is an example job submission script runprog.sh. See the instructions in lecture 18 for additional details. Note that if myprog above requires an argument you need to include the argument in the script below after $WDIR/myprog.

    #================== runprog.sh =================================
    #$ -S /bin/bash
    #$ -j y
    #$ -cwd
    #$ -l h_rt=0:5:0 
    #
    source /usr/local/bin/pathmunge.sh
    pathmunge usempi openmpi-x86_64
    WDIR=/home/yourname/path-to-your-project-dir
    cd $WDIR
    `which mpirun` -np $NSLOTS $WDIR/myprog  
    #=============================================================
    

  • The shell script specifies (through directives in the comments) that the script should be run in a bash shell, that all output should be placed in the current working directory, and that the job has a run time limit of 5 minutes. The subsequent shell commands specify the invocation of myprog on each processor assigned to the job.
  • To submit job to run using 8 processors use:
    qsub -pe MPI 8 runprog.sh
    
  • To make a larger scaling run use:
    qsub -pe MPI 32 runprog.sh (successive MPI processes will fill one node before being placed on the next)
    or
    qsub -pe rrMPI 32 runprog (successive MPI processes will be placed on different nodes according to availability)
    Note that your job and your MPI processes may end up sharing nodes with other requests, although this is not very likely given the small number of uses of Bass at the moment.
  • Once the job is submitted you can check its status using qstat
  • It can take several minutes longer than the expected runtime of your job for it to complete and for the files to show up in your directory.

• For an introduction to the Xeon Phi accelerators, start by reading An Overview of Programming for Intel Xeon processors and Intel Xeon Phi coprocessors by James Reinders.
• Programming, Compiling and Optimizing for the Intel Xeon Phi Coprocessor by Martyn Corden provides helpful programming details.
• The Intel C compiler icc directly supports the Cilk extensions to C without need of any compilation flags. Cilk also includes an array notation can be vectorized directly (C arrays permit aliasing which can inhibit vectorization).

Bibliography

1. PRAM Algorithms, S. Chatterjee, J. Prins, COMP 633 course notes, 2015.
2. Memory Hierarchy in Cache-Based Systems, R. v.d. Pas, Sun Microsystems, 2003.
3. OpenMP tutorial, Blaise Barney, LLNL, 2013.
4. The Implementation of the Cilk-5 Multithreaded Language, M. Frigo, C. Leiserson, K. Randall, in Proceedings of ACM Conf. on Programming Language Design and Implementation, 1998.
5. Shared Memory Consistency Models: A Tutorial, S. V. Adve, K. Gharachorloo, DEC Western Research Labs Report 95/7, 1995.
6. Computer Architecture: A Quantitative Approach 2nd ed, D. Patterson, J. Hennessy, Morgan-Kaufmann 1996.
7. Fast N-Body Simulation with CUDA, L. Nyland, M. Harris, J. Prins, GPUGems 3, 2008.
8. An Overview of Programming for Intel Xeon processors an Intel Xeon Phi coprocessors, James Reinders, Intel Corp, 2012.
9. "Questions and Answers about BSP", D. Skillicorn, J. Hill, and W. McColl, Scientific Programming 6, 1997.
10. Message Passing Interface, Blaise Barney, LLNL 2015
11. Introduction to Parallel Computing: Design and Analysis of Algorithms, V. Kumar, A. Grama, A. Gupta, G. Karypis, Benjamin-Cummings, 1994.