Click here for the old version of this page that has more obscure examples.
This page shows you how to build and run a simple example on AuN, or Mio. To run the example, copy/paste the text shown below in red:
If you would like a copy of the example files and you don't have an account on one of Mines' machines the files discussed here can also be obtained from:
While these examples were completed on Mio, the procedure is the same on Mc2 and Aun with a minor exception noted below.
Note that the "makefile" and run scripts discussed here can be used as templates for other applications.
To run the quick start example, create a directory for your example and go to it.
[joeuser@mio001 bins]$ mkdir guide [joeuser@mio001 bins]$ cd guide
Copy the file that contains our example code to your directory and unpack it.
[joeuser@mio001 guide]$ wget http://hpc.mines.edu/examples/examples/slurm/slurm.tgz [joeuser@mio001 guide]$ tar -xzf *
If you like, do an ls to see what you have.
[joeuser@mio001 guide]$ ls doit index.html my_time README.html slurm_record st.in helloc.c invertf.f90 nodes slurm_array slurm_simple hellof.f90 makedat p_array slurm_basic stc_01.c heterogen makefile phostone.c slurm_openmp stf_01.f90
Mio has two nodes, ppc001 and ppc002, that are based on IBM Power processors instead of the more common Intel x86 processor family. There are minor changes to the build and run procedures for these nodes. Click here.
Next we want to ensure that your environment is set up to run parallel applications. The following two commands will give you a clean, tested environment:
[joeuser@mio001 guide]$ module purge [joeuser@mio001 guide]$ module load StdEnv
Make the example programs:
[joeuser@mio001 guide]$ make echo mio001 mpicc -O3 stc_01.c -lm -o stc_01 mpif90 -O3 stf_01.f90 -o stf_01 ifort: command line remark #10010: option '-pthread' is deprecated and will be removed in a future release. See '-help deprecated' rm *mod mpicc -fopenmp phostone.c -o phostone mpicc helloc.c -o helloc mpif90 hellof.f90 -o hellof ifort: command line remark #10010: option '-pthread' is deprecated and will be removed in a future release. See '-help deprecated' cp helloc hello gfortran -O3 -fopenmp invertf.f90 -o invertf rm *mod
This directory contains a number of more advanced programs and scripts. Have a look at the README.html file for details.
On AuN and Mc2 you need to supply an account number to run parallel applications. Mio does not require account numbers. So, next find out which accounts you are authorized to use on each machine:
[joeuser@aun002 auto]$ /opt/utility/accounts Account -------------------- science test
If you run this command on Mio you will get:
[joeuser@mio001 guide]$ /opt/utility/accounts Accounts strings are not required on Mio
So, to run a parallel application on Mio you would do the following:
[joeuser@mio001 guide]$ sbatch slurm_simple Submitted batch job 1993
On AuN and Mc2 you add a -A option to the command line followed by the account string from the command given above.
[joeuser@aun001 guide]$ sbatch -A test slurm_simple Submitted batch job 1993
If you receive the message shown below that means that the account you have specified has run out of time. Try another.
sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)
If you quickly enter the command below you may/will see your job waiting to run or running. A "USER ST" of "PD" implies that it is waiting; "R" means it is running.
[joeuser@mio001 guide]$ squeue -u $USER JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 1993 compute hybrid joeuser PD 0:00 2 (Priority)
If this command returns no jobs listed then your job is finished. If the machine is very busy then it could take some time to run.
When the job is complete there will be an output file in your directory that starts with the word "slurm" then contains the jobid from the sbatch command followed by the word out.
[joeuser@mio001 guide]$ ls slurm* slurm-722122.out
This simple test program is a glorified parallel "hello world" program. You will see 8 lines that start with the name of the nodes on which you are running, followed by the MPI task id which should be in the range 0-8 and the the number 0 which is the number of tasks you are running.
The command cat slurm*.out will show you the output of the job. To see your output in a nice order you can use the sort command:
[joeuser@mio001 guide]$ sort slurm*.out Hello from compute065 0 8 Hello from compute065 1 8 Hello from compute065 2 8 Hello from compute065 3 8 Hello from compute066 4 8 Hello from compute066 5 8 Hello from compute066 6 8 Hello from compute066 7 8
On Mio, individual research groups own nodes. They have priority access to their nodes. You request priority access to your nodes by specifying a partition. Please ask your PI or instructor which partition you should be using on Mio.
On AuN there is a debug partition which allows for short small jobs, no more than 15 minutes and up to 4 nodes.
Add the string -p PARTITION_NAME to your sbatch command line. For example:
[joeuser@aun001 guide]$ sbatch -A test -p debug slurm_simple Submitted batch job 1993
Mio has two nodes, ppc001 and ppc002, that are based on IBM Power processors instead of the more common Intel x86 processor family. It is not possible to build applications for these nodes on the Mio headnode. You must launch an interactive session on one of these two nodes to build applications for them. An interactive session can be launched by running the command:
[joeuser@mio001 guide]$ srun -N 1 --tasks-per-node=1 -p ppc-build --share --time=1:00:00 --pty bash [joeuser@ppc002 guide]$
Note that the prompt has changed to ppc002 or ppc001 to show that you are now on the Power nodes.
Running this command is a little different from doing an ssh. In particular you are placed in the directory from which you launched the command instead of your home directory. Also some of the setup that is done in a normal session is not done. Finally, your environment might have references specific to Intel specific commands. You can clean your environment by running the commands:
[joeuser@mio001 guide]$ source /etc/profile [joeuser@mio001 guide]$ source /software/apps/utility/rm_intel [joeuser@mio001 guide]$ module load XL [joeuser@mio001 guide]$ module load OpenMPI
Also, if the nodes are busy running batch jobs you may not get the interactive session immediately.
You can then do a make as described above.
You will need to edit your slrum* files to include the following lines after the last SBATCH line.
source /etc/profile source /software/apps/utility/rm_intel module load XL module load OpenMPI
At this point you should exit your interactive session by entering exit.
[joeuser@ppc002 guide]$ exit exit
Finally, to run a parallel application on Mio Power nodes you would do the following:
[joeuser@mio001 guide]$ sbatch -p ppc slurm_simple Submitted batch job 1299071
The option -p ppc forces your job to run on the Power nodes. This can also be specified in the script.