CUDA Programming (targetted towards use with the CLASSE Farm)

CUDA Documentation

Introduction

CUDA is an Nvidia developed parallel compute environment and API. CUDA once stood for Compute Unified Device Architecture but it's use as an acronym has been dropped. (CUDA wikipedia)

• https://developer.nvidia.com/about-cuda

Programming Guide

• http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#axzz30Cfvm4QM
• Programming Guide PDF

CUDA 8 development

A CLASSE Scientific Linux 6 maintained distribution of the CUDA 8.0 development suite, that will work with the nVidia drivers we support, is available in /nfs/opt/cuda-8-0.sl6 .

A CLASSE Scientific Linux 7 maintained distribution of the CUDA 9.1 development suite, that will work with the nVidia drivers we support, is available in /nfs/opt/cuda-9-1.sl7 .

As an introduction to CUDA development, nVidia provided sample code is in /nfs/opt/cuda-8-0.sl6/samples/ and in /nfs/opt/cuda-9-1.sl7/samples/, along with the locally built binaries in /nfs/opt/cuda-8-0.sl6/samples/bin/x86_64/linux/release/ and /nfs/opt/cuda-9-1.sl7/samples/bin/x86_64/linux/release/. For more documentation, please see the CUDA Samples and page.

To do any test build of the samples, you first must set the correct environment, by typing:

# Intialize newer gcc
source /opt/rh/devtoolset-4/enable

# Set actual package locations
export CUDA_PATH=/nfs/opt/cuda-9-1.sl7
export NVIDIALIBDIR=/usr/lib64/nvidia
export MPIDIR=/usr/lib64/openmpi

# Set Bin and Lib directories
export CUDABINDIR=${CUDA_PATH}/bin
export CUDALIBDIR=${CUDA_PATH}/lib64
export MPIBINDIR=${MPIDIR}/bin
export MPILIBDIR=${MPIDIR}/lib

# Set build paths
export PATH=${CUDABINDIR}:${MPIBINDIR}:$PATH
export LD_LIBRARY_PATH=${CUDALIBDIR}:${NVIDIALIBDIR}:${MPILIBDIR}:$LD_LIBRARY_PATH

# Set the RPATH lib paths for Binaries
export EXTRA_LDFLAGS="-rpath-link=${CUDALIBDIR} -rpath=${CUDALIBDIR} -L${CUDALIBDIR}"
export EXTRA_CCFLAGS="-I ${CUDA_PATH}/include -I ${CUDA_PATH}/samples/common/inc/"
export SAMPLE_ENABLED=1

In general, the above environmental variables would need to be set for any CUDA project.

For more details about CUDA programming, please see the CUDA Programing Guide.

CUDA resources

We currently have 1 CUDA processor in the CLASSE Farm. We hope to add more as use and resources permit.

In addition, most Linux workstations and desktops use nVidia graphics cards that support CUDA, to check type: nvidia-smi.

For more information about GPU card Compute Capability, please see:

• https://developer.nvidia.com/cuda-gpus

Submitting a CUDA job on the Farm

To submit a job that uses one cuda resource, add -l cuda_free=1 to your qsub or qrsh command (where "l" is a lowercase L). For example:

• qsub -l cuda_free=1 myjob.sh

For more information about running jobs on the CLASSE compute farm, please see GridEngine.

Running an interactive CUDA session on the Farm

Use the following command to start an interactive CUDA session:

qrsh -q interactive.q -l cuda_free=1

For more information please see the Interactive Job section of GridEngine page.

CUDA 8 SL6 Benchmark

nbody is a useful tool to benchmark CUDA performance of nVidia graphics card:

For information about command line arguments, type:

/nfs/opt/cuda-8-0.sl6/samples/bin/x86_64/linux/release/nbody --help

This results is:

Command line options
   -fullscreen       (run n-body simulation in fullscreen mode)
   -fp64             (use double precision floating point values for simulation)
   -hostmem          (stores simulation data in host memory)
   -benchmark        (run benchmark to measure performance) 
   -numbodies=<N>    (number of bodies (>= 1) to run in simulation) 
   -device=<d>       (where d=0,1,2.... for the CUDA device to use)
   -numdevices=<i>   (where i=(number of CUDA devices > 0) to use for simulation)
   -compare          (compares simulation results running once on the default GPU and once on the CPU)
   -cpu              (run n-body simulation on the CPU)
   -tipsy=<file.bin> (load a tipsy model file for simulation)

An example SL6 command could be:

/nfs/opt/cuda-8-0.sl6/samples/bin/x86_64/linux/release/nbody -fp64 --benchmark -numbodies=1024 -numdevices=1

For CUDA 9 SL7 Benchmark, please substitute "cuda-9-1.sl7" for "cuda-8-0.sl6".