Until recently we did not have a good way for people to monitor their usage of scratch space on Mio and AuN. We now can easily show total usage. With a bit more effort you can also show aging of your files and directories .
We have enabled the command mmlsquota which will show your usage.
You can do a
to see the full description of the command or
[joeuser@mio ~]$to get a short description.
When you run the mmlsquota command you will get more information than is useful. You will see two Filesystems listed, lb and sb. The one that describes your scratch usage is lb. The sb Filesystem report is not important. You may also see a line that lists a "sets" Fileset. Again, this is not important.
We have a command /opt/utility/scsize that filters out most of the unimportant information. For example:
/opt/utility/scsizeBlock Limits Filesystem Fileset type GB quota limit in_doubt grace lb root USR 19 76800 102400 0 none [joeuser@mio ~]$
This shows that joeuser has 19 Gbytes in scratch. The quota is a theoretical upper limit as to the amount of space you could use. In fact you will draw the attention of the HPC group long before you get anywhere (think small fraction) close to that limit.
As you know the HPC group reserves the right to remove files in scratch as necessary to keep the system running. Scratch by definition is for temporary storage of data. If you plan on keeping data it should be moved off of the machine.
There has been a question and debate about automatically removing files after they reach a certain age. Some institutions do that. We don't for three reasons. People are generally responsible about cleaning up after themselves. It is actually an expensive operation to routinely purge files. Finally, for those few that are not responsible, it is too easy to "game" the aging tests.
However, we now have the ability for users to show their file aging information. This is a multistep process. The first step can be time consuming and hits the file system pretty hard so it is not something you will want to do on a daily basis.
The new command is /opt/utility/agedu. Again, you can get the man page for this command.
For the first step cd to your scratch directory and then run the command
cd $SCRATCH[joeuser@mio joeuser]$
/opt/utility/agedu --no-progress -f $HOME/adedu.dat -s $SCRATCH
This will create an inventory of your scratch directory. It will create a file agedu.dat. This can take several minutes. In a recent test for a user with a large number of files this took about 20 minutes. For most users it should run in a minute or two.
Please delete your inventory file, $HOME/adedu.dat, after you are done with it. They can be rather large and become irrelevant after you have modified your directory. The file is binary and can only be viewed as discussed below.
Once the inventory is created there are many options for displaying the data. You can:
Here are some examples of generating a text report filtering by age. The first column is the amount of data in kilobytes in the given directory of that age or older.
/opt/utility/agedu -a 2y -f $HOME/adedu.dat -t $SCRATCH89247072 /scratch/joeuser/DMOL 42528 /scratch/joeuser/QuIET 48716960 /scratch/joeuser/Siesta 395154304 /scratch/joeuser
/opt/utility/agedu -a 1y -f $HOME/adedu.dat -t $SCRATCH89247072 /scratch/joeuser/DMOL 2170464 /scratch/joeuser/Octopus 42528 /scratch/joeuser/QuIET 48717024 /scratch/joeuser/Siesta 1952 /scratch/joeuser/ddscat 397326784 /scratch/joeuser
/opt/utility/agedu -a 1m -f $HOME/adedu.dat -t $SCRATCH89247072 /scratch/joeuser/DMOL 2170528 /scratch/joeuser/Octopus 512941760 /scratch/joeuser/Qchem 42528 /scratch/joeuser/QuIET 48717024 /scratch/joeuser/Siesta 1952 /scratch/joeuser/ddscat 910268608 /scratch/joeuser [joeuser@mio joeuser]$
Notice the size changes as we change the reporting period. You can also specify subdirectories to get more detailed information.
/opt/utility/agedu -a 9m -f $HOME/adedu.dat -t $SCRATCH/Qchem8726752 /scratch/joeuser/Qchem/Aniline 1931872 /scratch/joeuser/Qchem/Benzene 32448 /scratch/joeuser/Qchem/Coronene 27296 /scratch/joeuser/Qchem/H2 135328 /scratch/joeuser/Qchem/H2O 34905824 /scratch/joeuser/Qchem/TPA 96 /scratch/joeuser/Qchem/TPBoron 20947296 /scratch/joeuser/Qchem/TPCarbon 50214656 /scratch/joeuser/Qchem/TPP 9971424 /scratch/joeuser/Qchem/TPSilicon 40411488 /scratch/joeuser/Qchem/Trinapamine 4786048 /scratch/joeuser/Qchem/Triphenylarsenic 172090528 /scratch/joeuser/Qchem [joeuser@mio joeuser]$
/opt/utility/agedu -a 1y -f $HOME/adedu.dat -H $SCRATCH/Qchem > agedu.html
You can then copy the file agedu.html to your local machine for viewing. This will give you a static very top level view of your directory structure.
The next option is much more interesting.
Finally, maybe the most useful option is to create a navigable web page that allows you to dive into subdirectories. When the page is created you can view your directory as a tree structure and navigate to see the size and ages of directories and files.
/opt/utility/agedu -a 2y -f $HOME/adedu.dat -w --address mio001.mines.edu --auth basicUsername: agedu Password: p35n1vnd94nmx9cy URL: http://mio001.mines.edu:34372/
This command will block until you do a Control-C. The command shows a user name: agedu, a password and a URL. Agedu actually starts a mini web server. It will display your data via the given URL. You will need to enter the requested Password and username.
On a live version of the page you can click on the directory name on the right to see details.
Please note, this page is not updated if you delete files. You will need to regenerate the agedu.dat file to see your updates.
Finally, please delete your inventory file, $HOME/adedu.dat, after you are done with it. They can be rather large and become irrelevant after you have modified your directory. The file is binary and can only be viewed as discussed above.