User Tools

Site Tools


guide:connecting_and_data_transfer

Connecting to the cluster system & file transfer


You can only connect to the cluster from “within” the LUH network. If you want to connect from the “outside” (e.g. from Home), you'll have to establish a VPN connection to the university’s network first. See VPN Service for details.

Please note: Working from outside the LUH may feel slower than from your office, depending on the quality (bandwidth, latency) of your internet connection. If graphical applications feel sluggish and respond slowly, it is usually due to this and not a technical problem with the cluster system. Quite frequently, you can drastically improve performance by experimenting with the compression settings of X2Go, or the use of OpenOnDemand or VirtualGL as described below in this guide. If you have a good internet connection, you should be able to work quite comfortably from Home, except perhaps if you are in countries that are really distant to Germany.

The following addresses should be used to connect to the cluster system:

  • login.cluster.uni-hannover.de in order to submit jobs.
  • transfer.cluster.uni-hannover.de whenever you need to transfer data.

Please note: Execution time is limited to 30 minutes on all login nodes. Thus, transfers on the login nodes will be aborted if the execution time limit is reached. On the transfer node, however, execution time is unlimited, but you can not submit batch jobs from the transfer node to avoid clogging the node.

SSH Connection to the cluster

SSH Connection from Linux and Mac OS

The most basic option to connect to the cluster system is via SSH. For this, an ssh client is required, which comes with most Linux distributions by default and also should be readily available under Mac OS (start a terminal, applications::utilities).

SSH Connection from Windows

Under Windows, you'll need to install an SSH Client like e.g. PuTTY. You could also install a Unix-like environment like Cygwin to benefit from some Unix-like functionality under Windows, but be aware that for many use cases, there may be easier ways to get you started (see below).

When you've found a terminal, the following command will establish a connection to the cluster system. Replace username with your cluster user name.

ssh username@login.cluster.uni-hannover.de

If you want to use graphical programs on the cluster system, add the option -X, which enables X11 forwarding (see below). Depending on your system, you may need to use -Y instead of -X.

ssh -X username@login.cluster.uni-hannover.de

File transfer to / from the cluster

There is a special node dedicated to data transfer with the cluster system. Whenever you transfer data with the cluster system, please use this node: transfer.cluster.uni-hannover.de

Files can be transferred to and from the cluster system in a number of ways. What has been said above about the availability of the tools on different operating systems also holds true here: the command line tools mentioned are usually already available on Linux and Mac OS. For Windows, you may need to install additional software.

scp can be used to copy files (use scp -r to recursively copy directories). The following command, e.g., copies a whole directory from the cluster to a local directory on your workstation:

[myworkstation]$  scp -r login.cluster.uni-hannover.de:mydir/data .

Another very efficient, but also very powerful and thus slightly more demanding tool is the rsync command, whith which you can keep your directories in sync across sites and also complete file transfers that got interrupted. Take care to review the correct syntax and use the --dry-run feature of rsync if you are unsure what you are about to do.

Alternatively, you can use FileZilla if you would like to use a graphical tool. Information on how to configure FileZilla for use with the cluster system can be found in section File Transfer under Windows using FileZilla.

Please note: Use the dedicated transfer node for file transfers, because processes that use more than 30 minutes of cpu time on the login machines will be aborted by the system. Since ssh uses encryption, this may use enough cpu time to get the transfer canceled.

File transfer using FileZilla

The FileZilla client may be used to exchange files on a machine running windows with the cluster system. In the following section we provide instructions on how to install and configure FileZilla client version 3.14.1_win64 on Windows, but the software is available for other platforms as well. FileZilla can be obtained from the following URL. After downloading, you can install the FileZilla client to a directory of your choice.

After installing, open the Site Manager and create a new server which you can then connect to. The following options have to be set (cf. the red boxes in figure figure 1).

  • Host: transfer.cluster.uni-hannover.de
  • Protocol: SFTP - SSH File Transfer Protocol
  • Logon Type: Ask for password
  • User: Your user name

 Site Manager: GeneralFig. 1: Site Manager: General

Furthermore, it is possible to open the remote connection directly to . Without further configuration, the remote directory will be set to . In order to configure this option, go to the Advanced tab and set Default remote directory accordingly, see figure figure 2.

 Site Manager: AdvancedFig. 2: Site Manager: Advanced

The first time a connection to the transfer node is made, you will need to confirm the authenticity of the node’s host-key - cf. figure figure 3.

 Host-key verification on first connection attemptFig. 3: Host-key verification on first connection attempt

After a connection is successfully established – cf. figure figure 4 – you can exchange data with the cluster system.

 Connection to transfer-node, established with FileZillaFig. 4: Connection to transfer-node, established with FileZilla

Graphical Connections to the cluster

There are three main methods to work with interactive graphic applications on the cluster:

  • X / X2Go
  • OpenOnDemand
  • VirtualGL

Graphical Connection via X / X11 / X2Go

One way to enable the usage of graphical programs on the cluster is to run an X-Server on your local machine, which means nothing more than running a piece of software that other software (called X-Clients in this context, but really just the graphic programs that you start) can make graphic requests to. Conveniently, an X-Server is usually the foundation for graphic environments under Linux and Mac OS, so it is usually already installed here, and graphical connections tunneled through ssh (via ssh -X or ssh -Y, see above) should work right out of the box. For more information about the concepts, cf. X Window System.

For improved performance, however, it may still be useful to install and configure the X2Go client on Linux and Mac OS, too, as connections made via X2Go are compressed and optimized. The standard “X” protocol is not designed for use over wide-area networks, and newer tools can improve performance quite a bit. Th X2Go client is part of most Linux distributions' package repositories.

On Windows, you'll need to install X2Go, anyway, if you want to use this way for interactive work, since Windows does not come with an X server by default. Windows X2Go-client version 4.0.2.1 hotfix1 has been tested to work with the cluster.

X2Go can be obtained here. During installation, please make sure to check the “full installation” box to install all fonts. Otherwise, you may experience strange error messages later on.

For additional comfort, we recommend using X2Go broker, see section x2go_broker.

Please configure the X2Go client as described in x2go_client_configuration.

X2Go client configuration

Please note: We recommend using X2Go in the broker mode for graphical connections, see section x2go_broker.

After starting the X2Go client, either using a desktop short-cut or using the start menu, a configuration dialogue is displayed. In this dialogue you should specify a session name and make the following four entries.

  1. Host: login.cluster.uni-hannover.de
  2. Login: Your user name
  3. SSH-Port: 22
  4. Session type: XFCE

The completed configuration dialogue is depicted in figure 5. Entries in the red boxes have to be set accordingly. Afterwards leave the configuration assistant by clicking the OK button.

X2Go configuration dialogue, entries in red boxes have to be seFig. 5: X2Go configuration dialogue, entries in red boxes have to be set

On the right side of the main window the newly created session name is displayed, see figure figure 6. You can start this session by clicking on the session name (in the upper right corner in figure figure 6) or by entering the session name in the dialogue box named session.

 Start a new session by clicking or entering the session name.Fig. 6: Start a new session by clicking or entering the session name

The first time a connection is established, the login nodes’ host-key is unknown. A notification will pop up and you need to accept the host-key (see figure figure 7) by pressing yes.

 Host key verification dialogueFig. 7: Host key verification dialogue

Please note: Current hashes of the host key fingerprints of login.cluster.uni-hannover.de are listed below:

 MD5: bf:aa:71:bd:d1:19:d4:4f:b5:60:e7:cb:c4:26:85:a7 (ECDSA)
 SHA256: cYIZZQC96J5bhannZnH2cGQLIBPLQVy29HNc6/vnyFg (ECDSA)
 SHA1: fb:a1:47:ad:97:99:b3:66:86:06:8d:9d:56:46:61:3a:44:9c:a6:cb (ECDSA)

 MD5: 3b:0a:17:0f:53:85:40:87:c2:be:ed:65:fb:40:59:8a (ED25519)
 SHA256: sG8ZYabQctyGjxPD7X8K2IBxJIE5xHHZ9mQqjlVcjxo (ED25519)
 SHA1: 51:f7:b9:0a:e3:84:8e:d8:2c:a4:e7:7d:42:14:d3:8b:62:2e:a6:5d (ED25519)

 MD5: bd:b2:04:f9:4e:36:c1:b6:36:8e:d9:03:cc:5d:75:c4 (RSA)
 SHA256: zkVF8Xyxmm7bOEKTNOlvRKe+nG1oHZDeOtY3Un60grg (RSA)
 SHA1: ed:e3:ed:e6:bf:4f:29:9a:2c:72:92:c1:b4:ff:a6:b9:81:f6:6b:45 RSA)

 MD5: e3:c7:80:67:68:5b:d9:78:df:50:d7:e1:c5:ae:bf:e7 (DSA)
 SHA256: DuWre1exwDsyTzD4yIMy6Oc2CFBCmx5o+l0LmWjMvUc (DSA)
 SHA1: 85:95:f5:c9:9b:2b:29:82:5f:13:70:b9:2b:43:44:84:22:8f:87:40 (DSA)

After a connection was successfully established an XFCE Desktop is displayed as depicted in figure figure 8.

 XFCE desktop with applications menu in the lower left cornerFig. 8: XFCE desktop with applications menu in the lower left corner

The Applications Menu in the bottom left corner can be used to start a console window and then load modules or submit jobs into the queue. You can open editors e.g. to write or edit batch scripts. Particularly interactive jobs which open graphical program windows can be run. To end your session either go to the Applications Menu or press the little green icon in the bottom right corner of your desktop.

Please note: Suspended X2Go sessions will be terminated after four weeks without prior notice.

X2Go Broker

Please note: Windows users must install the X2Go-Client version 4.1.2.0

If you would like to reconnect to a graphical session, use X2Go broker. For example you could start a session at the university and reconnect to it at home. In order to do this, you have to establish a connection through the X2Go broker. After installing an X2Go client, proceed as described below.

X2Go broker on Linux

Use the following command to establish a connection with X2Go broker (the following command should be on one line. Replace <username> with your cluster username).

x2goclient --broker-url=ssh://<username>@x2gobroker.cluster.uni-hannover.de/usr/bin/x2gobroker --broker-autologin

X2Go broker on Windows

Either edit the existing shortcut to X2Go or create a new one (chose “Eigenschaften” in Picture figure 9).

 Edit X2Go shortcutFig. 9: Edit X2Go shortcut

Extend the command given as “Ziel”, see Picture figure 10, with the following parameters (the following command should be on one line. Replace <username> with your cluster username).

"[...]x2goclient.exe" --broker-url=ssh://<username>@x2gobroker.cluster.uni-hannover.de/usr/bin/x2gobroker

 X2Go broker commandFig. 10: X2Go broker command

After providing your password, a session is listed in the X2Go window, see picture . Choose this session. You will get a desktop on the cluster system. You can reconnect to this session and continue working graphically.

 X2Go broker sessionFig. 11: X2Go broker session

Please note: Suspended X2Go sessions will be terminated after four weeks without prior notice.

Cluster Web Portal

The web interface powered by the software package Open OnDemand allows you to access the LUIS cluster resources using a web browser without installing additional software on your personal workstation. Currently, the portal works with newer versions of Chrome (22 ), Firefox (32 ) and Safari(13 ). Internet Explorer is not fully supported. Compatibility to mobile devices may be provided by the project in the future.

From within the Open OnDemand environment, you can:

  • Create, submit, cancel and monitor batch jobs
  • Open a terminal connection to the cluster login servers
  • Browse, edit, download and upload files in/to your HOME directory
  • Run noVNC Remote Desktop sessions connecting to compute nodes for interactive applications using a GUI
  • Run other preconfigured interactive applications like Jupyter Notebook, MATLAB or COMSOL

Please note: Do NOT use der weblogin features to transfer large amounts of data via your HOME directory. You will almost inevitably hit your HOME quota doing this, and that in turn will almost always have very undesirable side effects, like your jobs not terminating properly any more, you not being able to log in etc. Use one of the other methods described on this page, like rsync, scp, and the transfer node to move your data between sites. This also is way more efficient than first copying data from BIGWORK to your HOME and then transferring it from there - do not do that, please.

The Open OnDemand website contains additional information about the current and future directions of the project.

How to connect to the web portal

Please note: To access the portal, make sure you are connected to the University network, e.g. via the LUIS VPN Service.

Log on to the cluster web portal using your normal cluster login credentials by opening a new page in your web browser pointing to https://weblogin.cluster.uni-hannover.de (see figure figure 12).

 Cluster web portal login pageFig. 12: Cluster web portal login page

Once you have been connected to the portal, you will be presented with the main dashboard page, see figure figure 13. There, you will find several menus to enable access to the different Applications for File Browsing, Job Management and Interactive Computing. Tutorial videos in the section “Getting started with OnDemand” of the dashboard page explain further details of using the web portal.

 Cluster web portal dashboard pageFig. 13: Cluster web portal dashboard page

How to use modules for Jupyter from within the web portal

In order to load modules in the Jupyter application in the web portal, you first have to assemble the necessary modules in a collection on the command line (you will only need to do this once). To do that, please log in – either via the secure shell, SSH, or from within the webportal via the menu item Clusters → LUIS Computing Cluster Shell Access. On the command line, load the modules necessary for your application.

Please note: You will also need to load either JupyterLab OR IPython to make this work.

Example:

module load foss/2019b JupyterLab/2.2.8-Python-3.7.4 TensorFlow/2.3.1-Python-3.7.4

After this, save the modules loaded in a collection:

module save jupyter_modules

Use the following command to list the collections you already created:

module savelist

Collections are saved in your ${HOME}/.lmod.d directory

Now you can start a Jupyter session that loads the collection of modules you just created. To do this, go to Interactive Apps → Jupyter in the web portal and choose your module collection under the item “Jupyter/JupyterLab environment”. In case it is not yet listed there, try to reload the page, possibly while pressing the shift key to update the browser cache.

Using Graphical 3D-Applications with VirtualGL

When working with a 3d-application that interactively displays 3-dimensional representations of objects, you may experience awkwardly slow rendering. If your application uses OpenGL – which should be the case for many 3d-software packages running on a Linux system – you may try the following to speed up things:

To work from the command line of a Linux workstation

Get the VirtualGL package installed on your workstation. This provides the vglconnect command, which you use in the following way to connect to the cluster:

vglconnect -bindir /opt/VirtualGL/bin -s <your_account_name_here>@login.cluster.uni-hannover.de

Now set up and check the connection as described below.

Running a 3D-Application from within an X2Go-/X2Go-Broker Session

Start an X2Go session as described in the section above. Set up and check the connection as follows:

Test the connection and check whether the animations feel “smooth”:

vglrun -fps 60 glxgears
vglrun -fps 60 /opt/VirtualGL/bin/glxspheres64 -p 10000 -n 20

An example how to use VirtualGL with a “real” application:

module load ANSYS
vglrun -fps 60 cfx5

Compare this performance to just running glxgears or glxspheres64 without the vglrun command. You may or may not experience a difference, depending on your application. If things don’t work out yet and you want to try other settings, we recommend a look into chapter 15 of the VirtualGL documentation, which contains recipes for various applications. 1)

Hint: in case a software does not seem to work with vglrun, you may try to first load the following modules:

module load GCC/7.3.0-2.30 OpenMPI/3.1.1 Mesa/.18.1.1
guide/connecting_and_data_transfer.txt · Last modified: 2021/10/14 18:02