User Tools

Site Tools


guide:transferring_files_into_the_archive

Transferring data into the archive


Please note: The archive is operated as part of the service Archivierung and thus not part of the cluster system.

The archive can be used to store results and simulation data permanently. Each account has to be registered for archive use, before using it. This can be done on the BIAS-website after logging in with your user name and password. After clicking on the link entitled Ihren Benutzernamen für die Nutzung des Archivsystems zulassen it takes roughly an hour, before the archive can be used.

Quota

Archival storage in the archive system of Leibniz Universität Hannover is controlled by a quota mechanism. There is a quota on the amount of files as well as storage space. Please see the website of the archive service for further details at http://www.luis.uni-hannover.de/archivierung.html.

Transferring data into the archive

In order to transfer data into the archive of Leibniz Universität Hannover, it is recommended to use the cluster’s dedicated transfer node, see section.

Login with lftp

The archive can be reached at archiv.luis.uni-hannover.de using the lftp command.

username@transfer:~$ lftp <username>@archiv.luis.uni-hannover.de

After entering your cluster user name’s password the lftp prompt appears.

lftp <username>@archiv.luis.uni-hannover.de:~>

Now you can use the ls command to list your directory contents at the archive. At the same time this is to test an established connection to the archive.

 lftp <username>@archiv.luis.uni-hannover.de:~> ls

At your first login to the archive system with your account the directory is empty. The ls command will not return any listing. You can terminate the connection with exit.

lftp <username>@archiv.luis.uni-hannover.de:~> exit
<username>@transfer:~$

Aliases for exit are quit and bye.

Copying files into the archive

On the cluster system’s transfer node change to the directory where the data to be copied are located.

username@transfer:~$ cd $BIGWORK/my_data_dir
username@transfer:/bigwork/username/my_data_dir$

After logging in using lftp the put command is used.

username@transfer:/bigwork/username/my_data_dir$ lftp
<username>@archiv.luis.uni-hannover.de:~>
lftp <username>@archiv.luis.uni-hannover.de:~> put myfile.tar.gz

The file myfile.tar.gz is located inside the directory we previously changed to in this example. After using put to transfer the file it is also available on the archive. The TAB key works for completing file and directory names in lftp as well.

Saving multiple small files in the archive is not desired, because at least one copy of the data are kept on magnetic tape. Therefore a constant stream of data is desirable which can be achieved by some large files. It is recommended to use tar or zip to combine small files into one bigger file. This can also optimize your quota.

In order to transfer multiple (large) files at once, the mput command can be used. This is short for multiple put. The mput command understands the wildcard * as it is used in bash.

lftp <username>@archiv.luis.uni-hannover.de:~> mput mydata*.tar.gz

Fetching files from the archive

In order to get fetch files from the archive, the get command can be used.

lftp <username>@archiv.luis.uni-hannover.de:~> get myfile.tar.gz

This command puts the file at the location the lftp command was issued from which transferred the file into the archive. For fetching more than one file the mget command can be used (multiple get). Fetching the file may take some time until transfer starts. This time is needed by the storage robot to find the respective magnetic tape and wind the tape to the position the file is located at.

Some useful commands

Listing the current directories’ contents can be achieved by the command !ls. An exclamation mark executes the command on the machine lftp was started on. On the contrary listing the current local directory can be done with lpwd at the lftp prompt.

It is possible to create directories in the archive using the mkdir command.

lftp <username>@archiv.luis.uni-hannover.de:~> mkdir myDir

Changing directories works in the usual way using cd.

lftp <username>@archiv.luis.uni-hannover.de:~> cd myDir

And back up one directory.

lftp <username>@archiv.luis.uni-hannover.de:~> cd ..

A local directory can be changed using the lcd command, short for local cd.

lftp <username>@archiv.luis.uni-hannover.de:~> lcd /bigwork/<username>/datadir

Further reading

  • man page lftp, man lftp. Navigate using the arrow keys and exit with 'q'
  • Service Archivierung
guide/transferring_files_into_the_archive.txt · Last modified: 2021/04/14 22:53 (external edit)