How do I do a simple build and run?

This page shows you how to build and run a simple example on AuN, Mc2, or Mio. To run the example, copy/paste the text shown below in red:

If you would like a copy of the example files and you don't have an account on one of Mines' machines the files discussed here can also be obtained from:
http://hpc.mines.edu/bluem/quickfiles/example.tgz

While these examples were completed on Mio, the procedure is the same on Mc2 and Aun with a minor exception noted below.

Note that the "makefile" and run scripts discussed here can be used as templates for other applications.

To run the quick start example, create a directory for your example and go to it.

[joeuser@mio001 bins]$  mkdir guide
[joeuser@mio001 bins]$  cd guide

Copy the file that contains our example code to your directory and unpack it.

[joeuser@mio001 guide]$  wget http://geco.mines.edu/prototype/How_do_I_do_a_simple_build_and_run/example.tgz
[joeuser@mio001 guide]$  tar -xzf *

If you like, do an ls to see what you have.

[joeuser@mio001 guide]$  ls
aun_script     docol.f90    helloc.c      makefile    mio1_script  phostname.c  
power_script   simple       slurm_script  color.f90   example.tgz  info.html  
mc2_script     out.dat      phostone.c    set_alias   simple_slurm

Special instructions for building and running on the ppc001 and ppc002 (Power) nodes of Mio.

Mio has two nodes, ppc001 and ppc002, that are based on IBM Power processors instead of the more common Intel x86 processor family. There are minor changes to the build and run procedures for these nodes. Check below.


Next we want to ensure that your environment is set up to run parallel applications. The following two commands will give you a clean, tested environment:

[joeuser@mio001 guide]$  module purge
[joeuser@mio001 guide]$  module load StdEnv

Make the program:

[joeuser@mio001 guide]$ make
echo mio001
mio001
mpif90 -c color.f90
mpicc -DNODE_COLOR=node_color_  helloc.c color.o -lifcore -o helloc
rm -rf *.o

On AuN and Mc2 you need to supply an account number to run parallel applications. Mio does not require account numbers. So, next find out which accounts you are authorized to use on each machine:

[joeuser@aun002 auto]$  /opt/utility/accounts
Account
--------------------
science
test

If you run this command on Mio you will get:

[joeuser@mio001 guide]$  /opt/utility/accounts 
Accounts strings are not required on Mio

So, to run a parallel application on Mio you would do the following:

[joeuser@mio001 guide]$  sbatch  simple_slurm
Submitted batch job 1993

On AuN and Mc2 you add a -A option to the command line followed by the account string from the command given above.

[joeuser@aun001 guide]$  sbatch  -A test  simple_slurm
Submitted batch job 1993

If you receive the message shown below that means that the account you have specified has run out of time. Try another.

sbatch: error: Batch job submission failed: Job violates accounting/QOS policy 
(job submit limit, user's size and/or time limits)

If you quickly enter the command below you may/will see your job waiting to run or running. A "USER ST" of "PD" implies that it is waiting; "R" means it is running.

[joeuser@mio001 guide]$  squeue -u $USER
  JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
   1993   compute   hybrid  joeuser PD       0:00      2 (Priority)

If this command returns no jobs listed then your job is finished. If the machine is very busy then it could take some time to run.

When the job is complete there will be an output file in your directory that starts with the word "slurm" then contains the jobid from the sbatch command followed by the word out.

For example:

[joeuser@mio001 guide]$  ls slurm*
slurm-722122.out

This simple test program is a glorified parallel "hello world" program. You will see 16 lines that start with the name of the nodes on which you are running, followed by the MPI task id which should be in the range 0-15 and the the number 16 which is the number of tasks you are running. Next we have a number which will be either 0 or 8. This is the MPI task number of the lowest task running on a node.

You will also see two additional lines that are basically the same output described above but for the words "First task". There is one line output per node.

The command cat slurm*.out will show you the output of the job. To see your output in a nice order you can use the sort command:

[joeuser@mio001 guide]$  sort slurm*.out  -k1,1 -k2,2n | grep 16
compute028 0 16 0
compute028 1 16 0
compute028 2 16 0
compute028 3 16 0
compute028 4 16 0
compute028 5 16 0
compute028 6 16 0
compute028 7 16 0
compute029 8 16 8
compute029 9 16 8
compute029 10 16 8
compute029 11 16 8
compute029 12 16 8
compute029 13 16 8
compute029 14 16 8
compute029 15 16 8
First task on node compute028 is 0 16 0
First task on node compute029 is 8 16 8

Just to note, the sort options -k1,1 sorts on the first word in the output. The next option -k2,2n sorts on the second column numerically. The grep command filters out every line that does not contain "16", giving us only those lines of interest.

Congratulations, you have run your first supercomputing program.

The script complex_slurm runs the same program but it adds a number of features to the run. It first creates a new directory for your run, then goes to it and runs your program there.

The script threads_slurm shows how to run a hybrid MPI/OpenMP program. The program it runs is /opt/utility/phostname. This is again a glorified "hello world" program that also prints thread ID. Note the source for this program is included in the directory and it can be made using the command make phostname.

Queue and Partition Information

On Mio, individual research groups own nodes. They have priority access to their nodes. You request priority access to your nodes by specifying a partition. Please ask your PI or instructor which partition you should be using on Mio.

On AuN there is a debug partition which allows for short small jobs, no more than 15 minutes and up to 4 nodes.

Add the string -p PARTITION_NAME to your sbatch command line. For example:

[joeuser@aun001 guide]$  sbatch  -A test  -p debug simple_slurm
Submitted batch job 1993

Special instructions for building and running on the ppc001 and ppc002 (Power) nodes of Mio

Mio has two nodes, ppc001 and ppc002, that are based on IBM Power processors instead of the more common Intel x86 processor family. It is not possible to build applications for these nodes on the Mio headnode. You must launch an interactive session on one of these two nodes to build applications for them. An interactive session can be launched by running the command:

[joeuser@mio001 guide]$ srun -N 1 --tasks-per-node=1 -p ppc-build --share --time=1:00:00  --pty bash
[joeuser@ppc002 guide]$ 

Note that the prompt has changed to ppc002 or ppc001 to show that you are now on the Power nodes.

Alternatively, the file set_alias creates an alias p8for this command. You could do a

source set_alias
p8

You may want to add this alias to your .bashrc file so it is available every time you login.

Running this command is a little different from doing an ssh. In particular you are placed in the directory from which you launched the command instead of your home directory.

Also, if the nodes are busy running batch jobs you may not get the interactive session immediately.

After you have obtained the interactive session you proceed as shown above.

We want to ensure that your environment is set up to run parallel applications. The following two commands will give you a clean, tested environment:

[joeuser@mio001 guide]$  module purge
[joeuser@mio001 guide]$  module load StdEnv

Make the program:

[joeuser@mio001 guide]$ make
make
mpicc -DNODE_COLOR=node_color_  helloc.c color.o -lgfortran -lmpi_mpifh -o helloc
rm -rf *.o

At this point you should exit your interactive session by entering exit.

[joeuser@ppc002 guide]$ exit
exit

So, to run a parallel application on Mio Power nodes you would do the following:

[joeuser@mio001 guide]$  sbatch -p ppc power_script
Submitted batch job 1299071

The option -p ppc forces your job to run on the Power nodes. This can also be specified in the script.

Click here to see typical output.

There are a few special requirements for scripts for the Power nodes. Here is a slightly edited version of the run script with the important differences in red.

Script for Mio Power Nodes

#!/bin/bash 
#SBATCH --job-name="hybrid"
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=4
#SBATCH --ntasks=4
##SBATCH --exclusive
#SBATCH --time=00:05:00
#SBATCH -p ppc
#SBATCH --export=NONE
#SBATCH --get-user-env=10L


# Go to the directoy from 
# which our job was launched
cd $SLURM_SUBMIT_DIR

module purge
module load StdEnv

srun --mpi=pmi2 --export=ALL  ./helloc

Explanation of the differences

#SBATCH -p ppc
Forces the job to run on the Power nodes
#SBATCH --export=NONE
Prevents the environmental variables set on Mio from being used in the script
#SBATCH --get-user-env=10L
Sets up the environment similar to what you would get if you logged on the node.
--mpi=pmi2
Required to get the proper setting for MPI
--export=ALL
Makes all of the Power nodes variables available to your program

Example files:

info.html
this list
example.tgz
all of the files in this directory
color.f90
part of the hello world example
docol.f90
part of the hello world example
helloc.c
part of the hello world example
phostname.c
source for /opt/utility/phostname
phostone.c
same as phostname.c but pure C
out.dat
Example output from phostname with help file
makefile
makefile for the examples
slurm_script
a fancy run script (same as mio1_script)
aun_script
a fancy run script (same as mio1_script)
mio1_script
a fancy run script (same as aun_script)
mc2_script
a fancy run script with a few extras for Mc2
simple_slurm
runs on all platforms except the
ppc001 and ppc002 nodes of Mio
power_script
Script for running on the ppc001
and ppc002 nodes of Mio
set_alias
An alias for a command to get an interactive
session on the ppc001 and ppc002 nodes of Mio