COMP 633 Parallel Computing http://www.cs.unc.edu/~prins/Classes/633/ Fall 2017 (Tue Aug 22 - Tue Dec 5) TTh 2:00 - 3:15 PM FB 009 Instructor: Jan Prins, FB 334, Tel: 919-590-6213, prins@cs.unc.edu Office hours: Mon 2 - 3 pm and by appt. (email me with questions anytime)

Overview

• be able to design and analyze parallel algorithms for a variety of problems and computational models,
• be familiar with the hardware and software organization of high-performance parallel computing systems, and
• have experience with the implementation of parallel applications on high-performance computing systems, and be able to measure, tune, and report on their performance.

Additional information including the course syllabus can be found in the course overview.

All parallel programming models discussed in this class are supported on BASS or phaedra which are available for use in this class.

Announcements

On-line Handouts

• Course Lecture Notes

Reading Assignments

Written and Programming Assignments

Platforms and Programming Models

Platforms

• The Bass system supports the OpenMP and MPI programming models. The general instructions for getting started on bass are supplemented below with specific instructions for each programming model. When you login to bass.cs.unc.edu you are connected to a specific node on bass dedicated to interactive program development. You can compile programs on this node. Shared-memory programs run within an individual node on bass. Distributed-memory programs run across multiple nodes in Bass. The login node should not be used to run your programs, although a short debug test for a few seconds and no more than 4 cores should be OK. In general programs that need multiple nodes or dedicated nodes or GPUs should be submitted to queues that are managed by the Grid Engine job scheduler.
• The Phaedra system is an Intel Xeon E5-2650v4 compute server supporting OpenMP, Cilk, and Xeon Phi accelerator programming models.

OpenMP

• OpenMP reference: Specification of OpenMP 3.0 API for C/C++. You may be more interested in OpenMP support in gcc 4.4.7 (the compiler on Bass).
• Bass-specific material
  • Getting started on Bass.
  • To get accurate performance information run your programs on a dedicated node as a batch job with a shell script myjob using
    qsub -pe smp 16 myjob
    or interactively via
    qlogin -pe smp 16
    Do not park yourself on this node as everyone else in the class will be held up.
  • A directory with the sample diffusion program discussed in class.
  • Command lines for the compilation and execution of programs on bass.unc.edu.
    1. C compilation to create a sequential program (compiler ignores OpenMP directives and does not link with the OpenMP runtime library):
       gcc -O3 -o prog prog.c (Gnu C compiler 4.4.7)
    2. C compilation to create a parallel program (OpenMP 3.0 directives honored and program linked with the OpenMP runtime library)
       gcc -fopenmp -O3 -o prog prog.c (Gnu C compiler 4.4.7)
• Phaedra-specific material
  • Phaedra is a 20-core Xeon E5-2650 server with eight attached Intel Xeon Phi 5110P accelerators. The server hosts the Intel Parallel Studio XE 2017 compilers and performance analysis tools to access the accelerators. Students in COMP 633 have a login on phaedra, and OpenMP programs run directly on the server.
  • The Intel C compiler icc directly supports OpenMP 4.0 with tasking and accelerator offload, and Cilk extensions to C including Cilk array notation. (C arrays permit aliasing which can inhibit vectorization).
  • Be sure to source /opt/intel/bin/compilervars.sh (bash) or source /opt/intel/bin/compilervars.csh (csh) to access the Intel compilers and tools.

• GPU: Cuda 7.0 on gamma-x51-1.
  • CUDA C Programming Guide (v8.0 Sep 2016)
  • Be sure to include /usr/local/cuda-7.0/bin in your path to access to the nvcc compiler and other Nvidia tools.
• Intel Xeon Phi: Parallel Studio XE 2017 on phaedra
  • For an introduction read An Overview of Programming for Intel Xeon processors and Intel Xeon Phi coprocessors by James Reinders.
  • For programming details see our class slides and Programming, Compiling and Optimizing for the Intel Xeon Phi Coprocessor by Martyn Corden
  • Access to the accelerators using OpenMP acc or Intel offload directives is supported by the Intel compiler icc.
  • Be sure to source /opt/intel/bin/compilervars.sh (bash) or source /opt/intel/bin/compilervars.csh (csh) to access the Intel compilers and tools.

MPI

• MPI reference material
  • Comprehensive site organizing all reference and tutorial information on MPI.
  • A short overview of MPI.
• Running MPI programs on Bass
  • Set up your environment on Bass as directed here. Select the openmpi-x86_64 MPI implementation (this is not in the list shown in the instructions). Note that the instructions presuppose you are running the bash shell. If your default shell is not bash, make sure you get into a bash shell by executing "bash -l". If you do not do this, the next step will fail.
  • You can compile your program on the Bass front end as follows (use mpiCC for C++ programs and remember to #include mpi.h in your programs)):
    mpicc -o myprog myprog.c
  • To execute your program you need to submit it to the grid engine for scheduling. You need to prepare a shell script that will be used to start your program with the appropriate command line arguments, and includes instructions to the scheduler for resources required.

  • Here is an example job submission script runprog.sh. See the instructions in lecture 18 for additional details. Note that if myprog above requires an argument you need to include the argument in the script below after $WDIR/myprog.

    #================== runprog.sh =================================
    #$ -S /bin/bash
    #$ -j y
    #$ -cwd
    #$ -l h_rt=0:5:0 
    #
    source /usr/local/bin/pathmunge.sh
    pathmunge usempi openmpi-x86_64
    WDIR=/home/yourname/path-to-your-project-dir
    cd $WDIR
    `which mpirun` -np $NSLOTS $WDIR/myprog  
    #=============================================================
    

  • The shell script specifies (through directives in the comments) that the script should be run in a bash shell, that all output should be placed in the current working directory, and that the job has a run time limit of 5 minutes. The subsequent shell commands specify the invocation of myprog on each processor assigned to the job.
  • To submit job to run using 8 processors use:
    qsub -pe MPI 8 runprog.sh
    
  • To make a larger scaling run use:
    qsub -pe MPI 32 runprog.sh (successive MPI processes will fill one node before being placed on the next)
    or
    qsub -pe rrMPI 32 runprog (successive MPI processes will be placed on different nodes according to availability)
    Note that your job and your MPI processes may end up sharing nodes with other requests, although this is not very likely given the small number of uses of Bass at the moment.
  • Once the job is submitted you can check its status using qstat
  • It can take several minutes longer than the expected runtime of your job for it to complete and for the files to show up in your directory.

Bibliography

1. PRAM Algorithms, S. Chatterjee, J. Prins, COMP 633 course notes, 2015.
2. Memory Hierarchy in Cache-Based Systems, R. v.d. Pas, Sun Microsystems, 2003.
3. OpenMP 3.1 tutorial, Blaise Barney, LLNL, 2015.
4. The Implementation of the Cilk-5 Multithreaded Language, M. Frigo, C. Leiserson, K. Randall, in Proceedings of ACM Conf. on Programming Language Design and Implementation, 1998.
5. Shared Memory Consistency Models: A Tutorial, S. V. Adve, K. Gharachorloo, DEC Western Research Labs Report 95/7, 1995.
6. Computer Architecture: A Quantitative Approach 2nd ed, D. Patterson, J. Hennessy, Morgan-Kaufmann 1996.
7. Fast N-Body Simulation with CUDA, L. Nyland, M. Harris, J. Prins, GPUGems 3, 2008.
8. An Overview of Programming for Intel Xeon processors an Intel Xeon Phi coprocessors, James Reinders, Intel Corp, 2012.
9. "Questions and Answers about BSP", D. Skillicorn, J. Hill, and W. McColl, Scientific Programming 6, 1997.
10. Message Passing Interface, Blaise Barney, LLNL 2015
11. Introduction to Parallel Computing: Design and Analysis of Algorithms, V. Kumar, A. Grama, A. Gupta, G. Karypis, Benjamin-Cummings, 1994.