Managing your scratch space usage.

Until recently we did not have a good way for people to monitor their usage of scratch space on Mio and AuN. We now can easily show total usage. With a bit more effort you can also show aging of your files and directories .

We have enabled the command mmlsquota which will show your usage.

You can do a

[joeuser@mio ~]$ man mmlsquota

to see the full description of the command or

[joeuser@mio ~]$ mmlsquota -h 
to get a short description.

When you run the mmlsquota command you will get more information than is useful. You will see two Filesystems listed, lb and sb. The one that describes your scratch usage is lb. The sb Filesystem report is not important. You may also see a line that lists a "sets" Fileset. Again, this is not important.

We have a command /opt/utility/scsize that filters out most of the unimportant information. For example:

[joeuser@mio ~]$ /opt/utility/scsize
                         Block Limits                                               
Filesystem Fileset  type     GB    quota     limit  in_doubt   grace 
lb         root     USR      19    76800    102400         0    none 
[joeuser@mio ~]$ 

This shows that joeuser has 19 Gbytes in scratch. The quota is a theoretical upper limit as to the amount of space you could use. In fact you will draw the attention of the HPC group long before you get anywhere (think small fraction) close to that limit.

As you know the HPC group reserves the right to remove files in scratch as necessary to keep the system running. Scratch by definition is for temporary storage of data. If you plan on keeping data it should be moved off of the machine.

There has been a question and debate about automatically removing files after they reach a certain age. Some institutions do that. We don't for three reasons. People are generally responsible about cleaning up after themselves. It is actually an expensive operation to routinely purge files. Finally, for those few that are not responsible, it is too easy to "game" the aging tests.

However, we now have the ability for users to show their file aging information. This is a multistep process. The first step can be time consuming and hits the file system pretty hard so it is not something you will want to do on a daily basis.

The new command is /opt/utility/agedu. Again, you can get the man page for this command.

For the first step cd to your scratch directory and then run the command

[joeuser@mio joeuser]$cd $SCRATCH

[joeuser@mio joeuser]$/opt/utility/agedu --no-progress -f $HOME/adedu.dat -s $SCRATCH

This will create an inventory of your scratch directory. It will create a file agedu.dat. This can take several minutes. In a recent test for a user with a large number of files this took about 20 minutes. For most users it should run in a minute or two.

Please delete your inventory file, $HOME/adedu.dat, after you are done with it. They can be rather large and become irrelevant after you have modified your directory. The file is binary and can only be viewed as discussed below.

Once the inventory is created there are many options for displaying the data. You can:

Here are some examples of generating a text report filtering by age. The first column is the amount of data in kilobytes in the given directory of that age or older.

Find data over 2 years old...

[joeuser@mio joeuser]$ /opt/utility/agedu -a 2y -f  $HOME/adedu.dat -t $SCRATCH
89247072    /scratch/joeuser/DMOL
42528       /scratch/joeuser/QuIET
48716960    /scratch/joeuser/Siesta
395154304   /scratch/joeuser

Find data over 1 years old...

[joeuser@mio joeuser]$ /opt/utility/agedu -a 1y -f  $HOME/adedu.dat -t $SCRATCH
89247072    /scratch/joeuser/DMOL
2170464     /scratch/joeuser/Octopus
42528       /scratch/joeuser/QuIET
48717024    /scratch/joeuser/Siesta
1952        /scratch/joeuser/ddscat
397326784   /scratch/joeuser

Find data over 1 month old...

[joeuser@mio joeuser]$ /opt/utility/agedu -a 1m -f  $HOME/adedu.dat -t $SCRATCH
89247072    /scratch/joeuser/DMOL
2170528     /scratch/joeuser/Octopus
512941760   /scratch/joeuser/Qchem
42528       /scratch/joeuser/QuIET
48717024    /scratch/joeuser/Siesta
1952        /scratch/joeuser/ddscat
910268608   /scratch/joeuser
[joeuser@mio joeuser]$ 

Notice the size changes as we change the reporting period. You can also specify subdirectories to get more detailed information.

[joeuser@mio joeuser]$ /opt/utility/agedu -a 9m -f  $HOME/adedu.dat -t $SCRATCH/Qchem
8726752     /scratch/joeuser/Qchem/Aniline
1931872     /scratch/joeuser/Qchem/Benzene
32448       /scratch/joeuser/Qchem/Coronene
27296       /scratch/joeuser/Qchem/H2
135328      /scratch/joeuser/Qchem/H2O
34905824    /scratch/joeuser/Qchem/TPA
96          /scratch/joeuser/Qchem/TPBoron
20947296    /scratch/joeuser/Qchem/TPCarbon
50214656    /scratch/joeuser/Qchem/TPP
9971424     /scratch/joeuser/Qchem/TPSilicon
40411488    /scratch/joeuser/Qchem/Trinapamine
4786048     /scratch/joeuser/Qchem/Triphenylarsenic
172090528   /scratch/joeuser/Qchem
[joeuser@mio joeuser]$ 

Create a static web page for offline viewing...

[joeuser@mio joeuser]$ /opt/utility/agedu -a 1y -f  $HOME/adedu.dat -H $SCRATCH/Qchem > agedu.html

You can then copy the file agedu.html to your local machine for viewing. This will give you a static very top level view of your directory structure.

The next option is much more interesting.

Create a navigable web page ...

Finally, maybe the most useful option is to create a navigable web page that allows you to dive into subdirectories. When the page is created you can view your directory as a tree structure and navigate to see the size and ages of directories and files.

[joeuser@mio joeuser]$ /opt/utility/agedu -a 2y -f  $HOME/adedu.dat -w --address mio001.mines.edu --auth basic
Username: agedu
Password: p35n1vnd94nmx9cy
URL: http://mio001.mines.edu:34372/

This command will block until you do a Control-C. The command shows a user name: agedu, a password and a URL. Agedu actually starts a mini web server. It will display your data via the given URL. You will need to enter the requested Password and username.

login
Fig1. - An example agedu dynmamic web page login screen.

On a live version of the page you can click on the directory name on the right to see details.

joeuser
Fig2. - A static screen dump of a navigable web page created with agedu.

Please note, this page is not updated if you delete files. You will need to regenerate the agedu.dat file to see your updates.

Finally, please delete your inventory file, $HOME/adedu.dat, after you are done with it. They can be rather large and become irrelevant after you have modified your directory. The file is binary and can only be viewed as discussed above.