How do I do a simple build and run?

Click here for the old version of this page that has more obscure examples.

This page shows you how to build and run a simple example on AuN, or Mio. To run the example, copy/paste the text shown below in red:

If you would like a copy of the example files and you don't have an account on one of Mines' machines the files discussed here can also be obtained from:
http://hpc.mines.edu/bluem/quickfiles/example.tgz

While these examples were completed on Mio, the procedure is the same on Mc2 and Aun with a minor exception noted below.

Note that the "makefile" and run scripts discussed here can be used as templates for other applications.

To run the quick start example, create a directory for your example and go to it.

[joeuser@mio001 bins]$  mkdir guide
[joeuser@mio001 bins]$  cd guide

Copy the file that contains our example code to your directory and unpack it.

[joeuser@mio001 guide]$  wget http://hpc.mines.edu/examples/examples/slurm/slurm.tgz
[joeuser@mio001 guide]$  tar -xzf *

If you like, do an ls to see what you have.

[joeuser@mio001 guide]$  ls
doit        index.html   my_time     README.html   slurm_record  st.in
helloc.c    invertf.f90  nodes       slurm_array   slurm_simple  
hellof.f90  makedat      p_array     slurm_basic   stc_01.c
heterogen   makefile     phostone.c  slurm_openmp  stf_01.f90

Special instructions for building and running on the ppc001 and ppc002 (Power) nodes of Mio.

Mio has two nodes, ppc001 and ppc002, that are based on IBM Power processors instead of the more common Intel x86 processor family. There are minor changes to the build and run procedures for these nodes. Click here.


Next we want to ensure that your environment is set up to run parallel applications. The following two commands will give you a clean, tested environment:

[joeuser@mio001 guide]$  module purge
[joeuser@mio001 guide]$  module load StdEnv

Make the example programs:

[joeuser@mio001 guide]$ make
echo mio001
mpicc -O3 stc_01.c -lm -o stc_01
mpif90 -O3 stf_01.f90 -o stf_01
ifort: command line remark #10010: option '-pthread' is deprecated and will be removed in a future release. See '-help deprecated'
rm *mod
mpicc -fopenmp phostone.c -o phostone
mpicc helloc.c -o helloc
mpif90 hellof.f90 -o hellof
ifort: command line remark #10010: option '-pthread' is deprecated and will be removed in a future release. See '-help deprecated'
cp helloc hello
gfortran -O3 -fopenmp invertf.f90 -o invertf
rm *mod

This directory contains a number of more advanced programs and scripts. Have a look at the README.html file for details.

On AuN and Mc2 you need to supply an account number to run parallel applications. Mio does not require account numbers. So, next find out which accounts you are authorized to use on each machine:

[joeuser@aun002 auto]$  /opt/utility/accounts
Account
--------------------
science
test

If you run this command on Mio you will get:

[joeuser@mio001 guide]$  /opt/utility/accounts 
Accounts strings are not required on Mio

So, to run a parallel application on Mio you would do the following:

[joeuser@mio001 guide]$  sbatch  slurm_simple
Submitted batch job 1993

On AuN and Mc2 you add a -A option to the command line followed by the account string from the command given above.

[joeuser@aun001 guide]$  sbatch  -A test  slurm_simple
Submitted batch job 1993

If you receive the message shown below that means that the account you have specified has run out of time. Try another.

sbatch: error: Batch job submission failed: Job violates accounting/QOS policy 
(job submit limit, user's size and/or time limits)

If you quickly enter the command below you may/will see your job waiting to run or running. A "USER ST" of "PD" implies that it is waiting; "R" means it is running.

[joeuser@mio001 guide]$  squeue -u $USER
  JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
   1993   compute   hybrid  joeuser PD       0:00      2 (Priority)

If this command returns no jobs listed then your job is finished. If the machine is very busy then it could take some time to run.

When the job is complete there will be an output file in your directory that starts with the word "slurm" then contains the jobid from the sbatch command followed by the word out.

For example:

[joeuser@mio001 guide]$  ls slurm*
slurm-722122.out

This simple test program is a glorified parallel "hello world" program. You will see 8 lines that start with the name of the nodes on which you are running, followed by the MPI task id which should be in the range 0-8 and the the number 0 which is the number of tasks you are running.

The command cat slurm*.out will show you the output of the job. To see your output in a nice order you can use the sort command:

[joeuser@mio001 guide]$  sort slurm*.out

Hello from compute065 0 8
Hello from compute065 1 8
Hello from compute065 2 8
Hello from compute065 3 8
Hello from compute066 4 8
Hello from compute066 5 8
Hello from compute066 6 8
Hello from compute066 7 8

Queue and Partition Information

On Mio, individual research groups own nodes. They have priority access to their nodes. You request priority access to your nodes by specifying a partition. Please ask your PI or instructor which partition you should be using on Mio.

On AuN there is a debug partition which allows for short small jobs, no more than 15 minutes and up to 4 nodes.

Add the string -p PARTITION_NAME to your sbatch command line. For example:

[joeuser@aun001 guide]$  sbatch  -A test  -p debug slurm_simple
Submitted batch job 1993

Special instructions for building and running on the ppc001 and ppc002 (Power) nodes of Mio

Mio has two nodes, ppc001 and ppc002, that are based on IBM Power processors instead of the more common Intel x86 processor family. It is not possible to build applications for these nodes on the Mio headnode. You must launch an interactive session on one of these two nodes to build applications for them. An interactive session can be launched by running the command:

[joeuser@mio001 guide]$ srun -N 1 --tasks-per-node=1 -p ppc-build --share --time=1:00:00  --pty bash
[joeuser@ppc002 guide]$ 

Note that the prompt has changed to ppc002 or ppc001 to show that you are now on the Power nodes.

Running this command is a little different from doing an ssh. In particular you are placed in the directory from which you launched the command instead of your home directory. Also some of the setup that is done in a normal session is not done. Finally, your environment might have references specific to Intel specific commands. You can clean your environment by running the commands:

[joeuser@mio001 guide]$  source /etc/profile
[joeuser@mio001 guide]$  source /software/apps/utility/rm_intel
[joeuser@mio001 guide]$  module load XL
[joeuser@mio001 guide]$  module load OpenMPI

Also, if the nodes are busy running batch jobs you may not get the interactive session immediately.

You can then do a make as described above.

You will need to edit your slrum* files to include the following lines after the last SBATCH line.

 source /etc/profile
 source /software/apps/utility/rm_intel
 module load XL
 module load OpenMPI

At this point you should exit your interactive session by entering exit.

[joeuser@ppc002 guide]$ exit
exit

Finally, to run a parallel application on Mio Power nodes you would do the following:

[joeuser@mio001 guide]$  sbatch -p ppc slurm_simple
Submitted batch job 1299071

The option -p ppc forces your job to run on the Power nodes. This can also be specified in the script.

Files:

slurm_basic 661

stc_01.c 9719

stf_01.f90 13671

st.in 54

p_array 1024

slurm_openmp 617

helloc.c 643

phostone.c 8637

invertf.f90 6246

hellof.f90 693

slurm_record 1835

slurm_simple 429

makefile 1296

slurm_array 1405

index.html 2058

README.html 4758

makedat 216

heterogen 927

slurm.tgz 15588