COMP 633 Parallel Computing http://www.cs.unc.edu/~prins/Classes/633/ Fall 2013 (Aug 20 - Dec 3) TTh 12:30 - 1:45 PM FB 009 Instructor: Jan Prins, FB 334, Tel: 919-590-6213, prins@cs.unc.edu Office hours: Mon 3:00 -4:00 PM and by appt. (email me with questions anytime) TA: Duo Zhao SN 370, Tel: TBD, duozhao@cs.unc.edu Office hours: Tue 3:00 - 5:00 PM

Overview

• be able to design and analyze parallel algorithms for a variety of problems and computational models,
• be familiar with the hardware and software organization of high-performance parallel computing systems, and
• have experience with the implementation of parallel applications on high-performance computing systems, and be able to measure, tune, and report on their performance.

Additional information including the course syllabus can be found in the course overview.

All parallel programming models discussed in this class are supported on BASS which will be available for use in this class.

Announcements

• Next offering of this class is planned for fall 2015.

On-line Handouts

• Course Lecture Notes

Reading Assignments

• (for Thu Nov 21) Read Kumar et al., Basic Communication Operations.
• (for Thu Nov 10) Read Foster, Chapter 8, Overview of MPI.
• (for Tue Nov 5) Read Skillicorn et al., Questions and Answers about BSP.
• Read Nyland et. al, Fast N-Body Simulation with CUDA. Check supplementary materials for Cuda in the Software section below.
• (for Thu Oct 10) Read Hennessy & Patterson Ch. 8, sections 8.5 - 8.6 (synchronization primitives in shared memory, and memory consistency models).
• (for Tue Oct 8) Read Memory consistency models tutorial (sections 1-6, pp 1 -17).
• (For Thu Sep 26) Read The Implementation of the Cilk-5 Multithreaded Language, sections 1 - 3.
• (For Thu Sep 19) Look through Open MP Tutorial sections 7-9. Details specific to OpenMP support on Bass (using gcc 4.4.7) can be found here .
• (For Tue Sep 17) Look through Open MP Tutorial sections 1-6. Most examples are in Fortran, but don't require any knowledge of Fortran beyond the obvious. Ignore WORKSHARE and TASK directives, and discussion of nested parallel constructs.
• (for Thu Sep 12) Read the overview of Memory Hierarchy in Cache-based Systems (pg 1-9).
• (For Thu Sep 5) PRAM Handout, (review section 4.1), section 5.
• (For Tue Sep 3) PRAM Handout, sections 3.4, 3.6.
• (For Thu Aug 29) PRAM handout, sections 3.2, 3.3, 3.5.
• (For Tue Aug 27) Read PRAM Handout secns 1, 2, 3.1 (pp 1 - 8)
• (For Thu Aug 22) Look over the course overview.

Written and Programming Assignments

Software

We will be using the Bass system for programming assignments. The Bass system supports all the programming models studied in this class. The general instructions for getting started on bass are supplemented below with specific instructions for each programming model. When you login to bass.cs.unc.edu you are connected to a specific node on bass dedicated to interactive program development. You can compile programs on this node. Shared-memory programs run within an individual node on bass. Distributed-memory programs run across multiple nodes in Bass. The login node should not be used to run your programs, although a short debug test for a few seconds and no more than 4 cores should be OK. In general programs that need multiple nodes or dedicated nodes or GPUs should be submitted to queues that are managed by the Grid Engine job scheduler.

OpenMP

• OpenMP reference: Specification of OpenMP 3.0 API for C/C++. You may be more interested in OpenMP support in gcc 4.4.7 (the compiler on Bass).
• Bass-specific material
  • Getting started on Bass.
  • To get accurate performance information run your programs on a dedicated node as a batch job with a shell script myjob using
    qsub -pe smp 16 myjob
    or interactively via
    qlogin -pe smp 16
    Do not park yourself on this node as everyone else in the class will be held up.
  • A directory with the sample diffusion program discussed in class.
  • Command lines for the compilation and execution of programs on bass.unc.edu.
    1. C compilation to create a sequential program (compiler ignores OpenMP directives and does not link with the OpenMP runtime library):
       gcc -O3 -o prog prog.c (Gnu C compiler 4.4.7)
    2. C compilation to create a parallel program (OpenMP 3.0 directives honored and program linked with the OpenMP runtime library)
       gcc -fopenmp -O3 -o prog prog.c (Gnu C compiler 4.4.7)

Cilk and Cilk++

• This Cilk reference manual refers to a slightly older revision of the Cilk system, but is accurate with respect to the language definition.
• Cilk runs on Bass, but there is no public installation. We recommend you install and use Cilk++ instead of Cilk.
• Cilk++ can be downloaded from Intel. For Bass select the 64-bit linux version. You can install it in your home directory. You can run it via qlogin or batch submission just like OpenMP programs.

CUDA

• Cuda 4.2 on Bass
  • CUDA C Programming Guide (v4.2 Apr 2012)
  • Compile your code on the login node. See the instructions on the Bass web page to set the environment correctly.
  • Get a login on a GPU node: qlogin -l gpu_host,gpus=1 and run your program.
• Cuda 5.0 on killdevil and cuda 5.5 on other GPU-equipped machines
  • CUDA C Programming Guide (v5.5 July 2013)

Java

• Java threads reference material
  • Java 6 Documentation and SDK
  • Class Thread API description.
  • Java threads and locks specification (not easy going, unfortunately).
  • Some background on recent efforts to update the Java memory model to yield a better specification for Java concurrency.
• Java threads execute in parallel on any Bass node.

UPC

TBD

MPI

• MPI reference material
  • Comprehensive site organizing all reference and tutorial information on MPI.
  • A short overview of MPI.
• Running MPI programs on Bass
  • Set up your environment on Bass as directed here. Select the openmpi-1.4.3 environment. Note that the instructions presuppose you are running the bash shell. If your default shell is not bash, make sure you get into a bash shell by executing "bash -l". If you do not do this, the next step will fail.
  • You can compile your program on the Bass front end as follows (use mpiCC for C++ programs and remember to #include mpi.h in your programs)):
    mpicc -o prog prog.c
  • To execute your program you need to submit it to the grid engine for scheduling. You need to prepare a shell script that will be used to start your program with the appropriate command line arguments, and includes instructions to the scheduler for resources required.

  • Here is an example script called runprog. See the instructions in lecture 17 for additional details.

    #================== runprog =================================
    #$ -S /bin/bash
    #$ -j y
    #$ -cwd
    #$ -l h_rt=0:5:0 
    #
    source /usr/local/bin/pathmunge.sh
    pathmunge usempi openmpi-1.4.3
    WDIR=/home/yourname/projectname
    cd $WDIR
    `which mpirun` -np $NSLOTS -mca btl_openib_warn_default_gid_prefix 0  $WDIR/myprog  
    #=============================================================
    

  • The shell script specifies (through directives in the comments) that the script should be run in a bash shell, that all output should be placed in the current working directory, and that the job has a run time limit of 5 minutes. The subsequent shell commands specify the invocation of myprog on each processor assigned to the job.
  • To submit job to run using 8 processors use:
    qsub -pe MPI 8 runprog
    
  • To make a larger scaling run use:
    qsub -pe MPI 32 -l nonexc runprog (successive MPI processes will fill one node before being placed on the next)
    or
    qsub -pe rrMPI 32 -l nonexc runprog (successive MPI processes will be placed on different nodes according to availability)
    Note that your job may be queued for quite some time if you request a large processor count, and that your processes may end up sharing nodes with other processes.

Bibliography

1. PRAM Algorithms, S. Chatterjee, J. Prins, COMP 633 course notes, 2013.
2. Memory Hierarchy in Cache-Based Systems, R. v.d. Pas, Sun Microsystems, 2003.
3. OpenMP tutorial, Blaise Barney, LLNL, 2013.
4. The Implementation of the Cilk-5 Multithreaded Language, M. Frigo, C. Leiserson, K. Randall, in Proceedings of ACM Conf. on Programming Language Design and Implementation, 1998.
5. Shared Memory Consistency Models: A Tutorial, S. V. Adve, K. Gharachorloo, DEC Western Research Labs Report 95/7, 1995.
6. Computer Architecture: A Quantitative Approach 2nd ed, D. Patterson, J. Hennessy, Morgan-Kaufmann 1996.
7. "Questions and Answers about BSP", D. Skillicorn, J. Hill, and W. McColl, Scientific Programming 6, 1997.
8. Designing and Building Parallel Programs, I. Foster, Addison-Wesley, 1995. Online text.
9. Introduction to Parallel Computing: Design and Analysis of Algorithms, V. Kumar, A. Grama, A. Gupta, G. Karypis, Benjamin-Cummings, 1994.