This page gives a number of Intel Math Kernel Library (MKL) examples, in particular calls to routines that are part of the SCALAPACK group of routines. There is a tar file that can be downloaded to RA that contains the source for the various programs, most in both C and Fortran, a Makefile, and a pbs script. The source for the examples, the Makefile and an run script can extracted from the tar file. You then do a make and submit the script to RA to run the example.
The first example is psgesv.f90. The original source was downloaded from http://acts.nersc.gov/scalapack/hands-on/example4.html but it was modified to be free format compliant. This example is designed to run using 6 processing cores.
The second example is ex1.f90 and ex1.c This example was discussed in the RA introductory workshops see: http://geco.mines.edu/workshop/class2. Ex1 is designed to run using 4 processing cores and again calls psgesv as its main routine.
The third example pdpttr_2.f90 and pdpttr_2.c shows the use of the ScaLAPACK routines PDPTTRF and PPPTTRS to factor and solve a symmetric positive definite tridiagonal system of linear equations Tx = b, in two distinct contexts, with two processors each and distinct right-hand sides. These examples were obtained from http://acts.nersc.gov/scalapack/hands-on. This web page and its links are an excelent resource on SCALAPACK.
The final example, linsolve.f90 and linsolve.c goes back to PSGESV, but it is a more complete example. It also calls PSGEMV, PSAXPY, and PSNRM2 and does so in the context of a MPI program. This example is from chapter 15 of Pacheco, Peter S. Parallel Programming with MPI. San Francisco: Morgan Kaufmann, 1997. It has variable matrix sizes and processor layouts.
What you need to type (Copy/Paste) is in red.
(1) On "ra.mines.edu", create a new empty directory.
(2) Go to the directory.
(3) Download the files
wget -o wget.out http://geco.mines.edu/software/mkl/example.tgz
(3b) You might want to do an "ls" to make sure the file "example.tgz" actually downloaded.
(4) Untar the files
tar -xzf example.tgz
(4b) You might want to do an "ls" to make sure you have the files: ex1.c, ex1.f90, example.tgz, in.dat, linsolve.f90, linsolve.c, linsolve.h, Makefile, pbsfile, pdpttr_2.c, pdpttr_2.f90, and psgesv.f90.
(5) Make the executables
(6) Submit your job
After a few moments your job will run and put the output into the file "out*." You will have something like outx8.114418.ra.local
In the Makefile you will see the libraries required for static linking are:
LIBS= \ $(LIBPATH)/libmkl_scalapack_lp64.a \ $(LIBPATH)/libmkl_intel_lp64.a \ $(LIBPATH)/libmkl_blacs_openmpi_lp64.a \ $(LIBPATH)/libmkl_sequential.a \ $(LIBPATH)/libmkl_core.a \ $(LIBPATH)/libmkl_sequential.a \ $(LIBPATH)/libmkl_core.a \ -lpthread
Where LIBPATH is the path to the MKL library directory. The Makefile gives the library path for MKL versions 10, 11, and 11.1.
If you link dynamically then this is replaced with:
LIBS= \ -L$(LIBPATH) \ -lmkl_scalapack_lp64 \ -lmkl_intel_lp64 \ -lmkl_blacs_openmpi_lp64 \ -lmkl_sequential \ -lmkl_core \ -lmkl_sequential \ -lmkl_core \ -lpthread
The listing of "core" and "sequential" twice in the list of libraries is not a mistake. This is to over come a problem in the Intel linker. You will also see a note in the Makefile that there is a bug in version 10 of the Intel compiler that prevents psgesv.f90 from compiling at optimization above -O1.
We have some examples of calls to the serial version of MKL (LAPACK routines) and the Portland Group version of LAPACK and the Cuda CULA package on the page http://geco.mines.edu/tesla/culaexample/index.shtml.