Grid storage

In addition to computing elements, the grid has a storage system consisting of several storage elements (SEs). Usually all the sites with CEs have also an SE. If you need to transfer files with your job, and the total size of the files is more than 10MB, you need to use something else than inputfiles (and generally, it is beneficial to have small inputfiles, as they add to the overhead of the job management). One possibility is to utilise grid storage system for this purpose.

1. GFAL access to grid storage

The files on the grid disks in Durham can be viewed in your normal file-browser even on your laptop at the following link dav://se01.dur.scotgrid.ac.uk/dpm/dur.scotgrid.ac.uk/home/pheno or in your web-browser at http://se01.dur.scotgrid.ac.uk/dpm/dur.scotgrid.ac.uk/home/pheno. To make a directory on the grid server, you need a valid certificate and to issue a command like 
gfal-mkdir gsiftp://se01.dur.scotgrid.ac.uk/dpm/dur.scotgrid.ac.uk/home/pheno/MyUserName
It is much faster to get files from the grid storage directly than through the output sandbox of jobs. So try moving the output.root to the grid storage at the end each run, then you can even get them directly on your laptop by mounting the directory. A local file, e.g. ArcTools.tar.gz, is copied copied to the grid storage with the command 
gfal-copy file://`pwd`/ArcTools.tar.gz gsiftp://se01.dur.scotgrid.ac.uk/dpm/dur.scotgrid.ac.uk/home/pheno/MyUserName/
The job running on the node should then copy the output files you want to keep to an appropriate directory on the grid storage. This is much better than specifying the output file in the arc job description 
(outputFiles = ("output.root" "") )
since that requires the output to be downloaded as part of the job management process, and will slow this down. So use the grid storage! This is also updated in real time, so it is much quicker to follow progress in the job execution.