Table of Contents
Transferring data into the archive
Please note: The archive is operated as part of the service Archivierung and thus not part of the cluster system.
The archive can be used to store results and simulation data permanently. Each account has to be registered for archive use, before using it. This can be done on the BIAS-website after logging in with your user name and password. After clicking on the link entitled Ihren Benutzernamen für die Nutzung des Archivsystems zulassen it takes roughly an hour, before the archive can be used.
Quota
Archival storage in the archive system of Leibniz Universität Hannover is controlled by a quota mechanism. There is a quota on the amount of files as well as storage space. Please see the website of the archive service for further details at http://www.luis.uni-hannover.de/archivierung.html.
Transferring data into the archive
In order to transfer data into the archive of Leibniz Universität Hannover, it is recommended to use the cluster’s dedicated transfer node, see section.
Login with lftp
The archive can be reached at archiv.luis.uni-hannover.de
using the lftp
command.
username@transfer:~$ lftp <username>@archiv.luis.uni-hannover.de
After entering your cluster user name’s password the lftp
prompt appears.
lftp <username>@archiv.luis.uni-hannover.de:~>
Now you can use the ls
command to list your directory contents at the archive. At the same time this is to test an established connection to the archive.
lftp <username>@archiv.luis.uni-hannover.de:~> ls
At your first login to the archive system with your account the directory is empty. The ls
command will not return any listing. You can terminate the connection with exit
.
lftp <username>@archiv.luis.uni-hannover.de:~> exit <username>@transfer:~$
Aliases for exit
are quit
and bye
.
Copying files into the archive
On the cluster system’s transfer node change to the directory where the data to be copied are located.
username@transfer:~$ cd $BIGWORK/my_data_dir username@transfer:/bigwork/username/my_data_dir$
After logging in using lftp the put
command is used.
username@transfer:/bigwork/username/my_data_dir$ lftp <username>@archiv.luis.uni-hannover.de:~> lftp <username>@archiv.luis.uni-hannover.de:~> put myfile.tar.gz
The file myfile.tar.gz
is located inside the directory we previously changed to in this example. After using put
to transfer the file it is also available on the archive. The TAB
key works for completing file and directory names in lftp as well.
Saving multiple small files in the archive is not desired, because at least one copy of the data are kept on magnetic tape. Therefore a constant stream of data is desirable which can be achieved by some large files. It is recommended to use tar
or zip
to combine small files into one bigger file. This can also optimize your quota.
In order to transfer multiple (large) files at once, the mput
command can be used. This is short for multiple put. The mput
command understands the wildcard *
as it is used in bash.
lftp <username>@archiv.luis.uni-hannover.de:~> mput mydata*.tar.gz
Fetching files from the archive
In order to get fetch files from the archive, the get
command can be used.
lftp <username>@archiv.luis.uni-hannover.de:~> get myfile.tar.gz
This command puts the file at the location the lftp
command was issued from which transferred the file into the archive. For fetching more than one file the mget
command can be used (multiple get). Fetching the file may take some time until transfer starts. This time is needed by the storage robot to find the respective magnetic tape and wind the tape to the position the file is located at.
Some useful commands
Listing the current directories’ contents can be achieved by the command !ls
. An exclamation mark executes the command on the machine lftp was started on. On the contrary listing the current local directory can be done with lpwd
at the lftp prompt.
It is possible to create directories in the archive using the mkdir
command.
lftp <username>@archiv.luis.uni-hannover.de:~> mkdir myDir
Changing directories works in the usual way using cd
.
lftp <username>@archiv.luis.uni-hannover.de:~> cd myDir
And back up one directory.
lftp <username>@archiv.luis.uni-hannover.de:~> cd ..
A local directory can be changed using the lcd
command, short for local cd.
lftp <username>@archiv.luis.uni-hannover.de:~> lcd /bigwork/<username>/datadir
Further reading
- man page lftp,
man lftp
. Navigate using the arrow keys and exit with 'q' - Service Archivierung