User Tools

Site Tools


guide:connecting_and_data_transfer

Connecting to the cluster system & file transfer


The following addresses should be used to connect to the cluster system:

  • login.cluster.uni-hannover.de in order to submit jobs.
  • transfer.cluster.uni-hannover.de whenever you need to transfer data.

Please note: Execution time is limited to 30 minutes on all login nodes.

Thus transfers will be aborted on all login nodes if the execution time limit is reached. On the transfer node execution time is unlimited, but on the other hand it is not possible to submit jobs on the transfer node.

Connecting from Linux, Mac OS or Cygwin under Windows

In order to connect to the cluster system, an ssh client is required, which comes with most distributions by default. The following command will establish a connection to the cluster system. Replace username with your cluster user name.

ssh username@login.cluster.uni-hannover.de

If you need to use a graphical program on the cluster system, use the option -X, which enables X11 forwarding.

ssh -X username@login.cluster.uni-hannover.de

Alternatively, you can connect to the cluster system graphically using X2Go. This client is part of most Linux distributions package repositories. Please configure your X2Go client as described in chapter. Furthermore you can use the X2Go broker mode with this client, see section X2Go Broker.

File transfer using Linux, Mac OS or Cygwin under Windows

There is a special node dedicated to data transfer with the cluster system. Whenever you transfer data with the cluster system use the special transfer node: transfer.cluster.uni-hannover.de

Files can be transferred to and from the cluster system in a number of ways. For single files, scp can be used (scp -r for directories). Example:

[myworkstation]$  scp -r login.cluster.uni-hannover.de:mydir/data .

Another very efficient, but also very powerful and thus slightly more demanding tool is the rsync command, whith which you can keep your directories in sync across sites and also complete file transfers that got interrupted. Take care to understand the syntax and use the --dry-run feature of rsync if you are not sure what you are about to do.

Alternatively, you can use FileZilla if you would like to use a graphical tool. Information on how to configure FileZilla for use with the cluster system can be found in section.

Please note: Use the dedicated transfer node for file transfers, because processes that use more than 30 minutes of cpu time on the login machines will be aborted by the system. Since ssh uses encryption, this may use enough cpu time to get the transfer canceled.

Connecting from Windows

In order to connect to the cluster system from Windows, additional software is needed. Using graphical programs on login nodes or compute nodes is possible with the help of an x-window client. Following are instructions on how to install and configure the X2Go client under Windows 7 for use with the cluster system. This guide is based on the X2Go-client version 4.0.2.1 hotfix1, which can be obtained from the following URL. After downloading you may install the program to a folder of your choice. Furthermore we recommend using X2Go broker, see section. If you want to try out a Unix-like environment under Windows and feel happy using a shell/command line, you may give https://cygwin.com/ a shot.

X2Go client configuration

Please note: We recommend using X2Go in the broker mode for graphical connections, see section.

After starting the X2Go client, either using a desktop short-cut or using the start menu, a configuration dialogue is displayed. In this dialogue you should specify a session name and make the following four entries.

  1. Host: login.cluster.uni-hannover.de
  2. Login: Your user name
  3. SSH-Port: 22
  4. Session type: XFCE

The completed configuration dialogue is depicted in figure 1. Entries in the red boxes have to be set accordingly. Afterwards leave the configuration assistant by clicking the OK button.

X2Go configuration dialogue, entries in red boxes have to be seFig. 1: X2Go configuration dialogue, entries in red boxes have to be set

On the right side of the main window the newly created session name is displayed, see figure figure 2. You can start this session by clicking on the session name (in the upper right corner in figure figure 2) or by entering the session name in the dialogue box named session.

 Start a new session by clicking or entering the session name.Fig. 2: Start a new session by clicking or entering the session name

The first time a connection is established, the login nodes’ host-key is unknown. A notification will pop up and you need to accept the host-key (see figure figure 3) by pressing yes.

 Host key verification dialogueFig. 3: Host key verification dialogue

Please note: Current hashes of the host key fingerprints of login.cluster.uni-hannover.de are listed below:

 MD5: bf:aa:71:bd:d1:19:d4:4f:b5:60:e7:cb:c4:26:85:a7 (ECDSA)
 SHA256: cYIZZQC96J5bhannZnH2cGQLIBPLQVy29HNc6/vnyFg (ECDSA)
 SHA1: fb:a1:47:ad:97:99:b3:66:86:06:8d:9d:56:46:61:3a:44:9c:a6:cb (ECDSA)

 MD5: 3b:0a:17:0f:53:85:40:87:c2:be:ed:65:fb:40:59:8a (ED25519)
 SHA256: sG8ZYabQctyGjxPD7X8K2IBxJIE5xHHZ9mQqjlVcjxo (ED25519)
 SHA1: 51:f7:b9:0a:e3:84:8e:d8:2c:a4:e7:7d:42:14:d3:8b:62:2e:a6:5d (ED25519)

 MD5: bd:b2:04:f9:4e:36:c1:b6:36:8e:d9:03:cc:5d:75:c4 (RSA)
 SHA256: zkVF8Xyxmm7bOEKTNOlvRKe+nG1oHZDeOtY3Un60grg (RSA)
 SHA1: ed:e3:ed:e6:bf:4f:29:9a:2c:72:92:c1:b4:ff:a6:b9:81:f6:6b:45 RSA)

 MD5: e3:c7:80:67:68:5b:d9:78:df:50:d7:e1:c5:ae:bf:e7 (DSA)
 SHA256: DuWre1exwDsyTzD4yIMy6Oc2CFBCmx5o+l0LmWjMvUc (DSA)
 SHA1: 85:95:f5:c9:9b:2b:29:82:5f:13:70:b9:2b:43:44:84:22:8f:87:40 (DSA)

After a connection was successfully established an XFCE Desktop is displayed as depicted in figure figure 4.

 XFCE desktop with applications menu in the lower left cornerFig. 4: XFCE desktop with applications menu in the lower left corner

The Applications Menu in the bottom left corner can be used to start a console window and then load modules or submit jobs into the queue. You can open editors e.g. to write or edit batch scripts. Particularly interactive jobs which open graphical program windows can be run. To end your session either go to the Applications Menu or press the little green icon in the bottom right corner of your desktop.

Please note: Suspended X2Go sessions will be terminated after four weeks without prior notice.

X2Go Broker

Please note: Windows users must install the X2Go-Client version 4.1.2.0

If you would like to reconnect to a graphical session, use X2Go broker. For example you could start a session at the university and reconnect to it at home. In order to do this, you have to establish a connection through the X2Go broker. After installing an X2Go client, proceed as described below.

X2Go broker on Linux

Use the following command to establish a connection with X2Go broker (the following command should be on one line. Replace <username> with your cluster username).

x2goclient --broker-url=ssh://<username>@x2gobroker.cluster.uni-hannover.de/usr/bin/x2gobroker --broker-autologin

X2Go broker on Windows

Either edit the existing shortcut to X2Go or create a new one (chose “Eigenschaften” in Picture figure 5).

 Edit X2Go shortcutFig. 5: Edit X2Go shortcut

Extend the command given as “Ziel”, see Picture figure 6, with the following parameters (the following command should be on one line. Replace <username> with your cluster username).

"[...]x2goclient.exe" --broker-url=ssh://<username>@x2gobroker.cluster.uni-hannover.de/usr/bin/x2gobroker

 X2Go broker commandFig. 6: X2Go broker command

After providing your password, a session is listed in the X2Go window, see picture . Choose this session. You will get a desktop on the cluster system. You can reconnect to this session and continue working graphically.

 X2Go broker sessionFig. 7: X2Go broker session

Please note: Suspended X2Go sessions will be terminated after four weeks without prior notice.

Cluster Web Portal

The web interface powered by the software package Open OnDemand allows you to access the LUIS cluster resources using a web browser without installing additional software on your personal workstation. Currently, the portal works with newer versions of Chrome (22 ), Firefox (32 ) and Safari(13 ). Internet Explorer is not fully supported. Compatibility to mobile devices may be provided by the project in the future.

From within the Open OnDemand environment, you can:

  • Create, submit, cancel and monitor batch jobs
  • Open a terminal connection to the cluster login servers
  • Browse, edit, download and upload files in/to your HOME directory
  • Run noVNC Remote Desktop sessions connecting to compute nodes for interactive applications using a GUI
  • Run other preconfigured interactive applications like Jupyter Notebook, MATLAB or COMSOL

Please note: Do NOT use der weblogin features to transfer large amounts of data via your HOME directory. You will almost inevitably hit your HOME quota doing this, and that in turn will almost always have very undesirable side effects, like your jobs not terminating properly any more, you not being able to log in etc. Use one of the other methods described on this page, like rsync, scp, and the transfer node to move your data between sites. This also is way more efficient than first copying data from BIGWORK to your HOME and then transferring it from there - do not do that, please.

The Open OnDemand website contains additional information about the current and future directions of the project.

How to connect to the web portal

Please note: To access the portal, make sure you are connected to the University network, e.g. via the LUIS VPN Service.

Log on to the cluster web portal using your normal cluster login credentials by opening a new page in your web browser pointing to https://weblogin.cluster.uni-hannover.de (see figure figure 8).

 Cluster web portal login pageFig. 8: Cluster web portal login page

Once you have been connected to the portal, you will be presented with the main dashboard page, see figure figure 9. There, you will find several menus to enable access to the different Applications for File Browsing, Job Management and Interactive Computing. Tutorial videos in the section “Getting started with OnDemand” of the dashboard page explain further details of using the web portal.

 Cluster web portal dashboard pageFig. 9: Cluster web portal dashboard page

How to use modules for Jupyter from within the web portal

In order to load modules in the Jupyter application in the web portal, you first have to assemble the necessary modules in a collection on the command line (you will only need to do this once). To do that, please log in – either via the secure shell, SSH, or from within the webportal via the menu item Clusters → LUIS Computing Cluster Shell Access. On the command line, load the modules necessary for your application.

Please note: You will also need to load either JupyterLab OR IPython to make this work.

Example:

module load foss/2019b JupyterLab/2.2.8-Python-3.7.4 TensorFlow/2.3.1-Python-3.7.4

After this, save the modules loaded in a collection:

module save jupyter_modules

Use the following command to list the collections you already created:

module savelist

Collections are saved in your ${HOME}/.lmod.d directory

Now you can start a Jupyter session that loads the collection of modules you just created. To do this, go to Interactive Apps → Jupyter in the web portal and choose your module collection under the item “Jupyter/JupyterLab environment”. In case it is not yet listed there, try to reload the page, possibly while pressing the shift key to update the browser cache.

Tipps for working with graphical 3d-applications

When working with a 3d-application that interactively displays 3-dimensional representations of objects, you may experience awkwardly slow rendering. If your application uses OpenGL – which should be the case for many 3d-software packages running on a Linux system – you may try the following to speed up things:

To work from the command line of a Linux workstation:

Get the VirtualGL package installed on your workstation. This provides the vglconnect command, which you use in the following way to connect to the cluster:

vglconnect -bindir /opt/VirtualGL/bin -s <your_account_name_here>@login.cluster.uni-hannover.de

Now set up and check the connection as described below.

Running a 3d-application from within an X2Go-/X2Go-Broker session:

Start an X2Go session as described in section 3.5. Set up and check the connection as described in the following paragraph.

Test the connection and check whether the animations feel “smooth”:

vglrun -fps 60 glxgears
vglrun -fps 60 /opt/VirtualGL/bin/glxspheres64 -p 10000 -n 20

An example how to use VirtualGL with a “real” application:

module load ANSYS
vglrun -fps 60 cfx5

Compare this performance to just running glxgears or glxspheres64 without the vglrun command. You may or may not experience a difference, depending on your application. If things don’t work out yet and you want to try other settings, we recommend a look into chapter 15 of the VirtualGL documentation, which contains recipes for various applications. 1)

Hint: in case a software does not seem to work with vglrun, you may try to first load the following modules:

module load GCC/7.3.0-2.30 OpenMPI/3.1.1 Mesa/.18.1.1

File transfer under Windows using FileZilla

The FileZilla client may be used to exchange files on a machine running windows with the cluster system. In the following section we provide instructions on how to install and configure FileZilla client version 3.14.1_win64 on Windows 7. FileZilla can be obtained from the following URL. After downloading, you can install the FileZilla client to a directory of your choice.

After installing, open the Site Manager and create a new server which you can then connect to. The following options have to be set (cf. the red boxes in figure figure 10).

  • Host: transfer.cluster.uni-hannover.de
  • Protocol: SFTP - SSH File Transfer Protocol
  • Logon Type: Ask for password
  • User: Your user name

 Site Manager: GeneralFig. 10: Site Manager: General

Furthermore, it is possible to open the remote connection directly to . Without further configuration, the remote directory will be set to . In order to configure this option, go to the Advanced tab and set Default remote directory accordingly, see figure figure 11.

 Site Manager: AdvancedFig. 11: Site Manager: Advanced

The first time a connection to the transfer node is made, you will need to confirm the authenticity of the node’s host-key - cf. figure figure 12.

 Host-key verification on first connection attemptFig. 12: Host-key verification on first connection attempt

After a connection is successfully established – cf. figure figure 13 – you can exchange data with the cluster system.

 Connection to transfer-node, established with FileZillaFig. 13: Connection to transfer-node, established with FileZilla

Connecting from outside the university’s network

The cluster system in not reachable from outside the university’s network. In order to connect from outside you have to establish a connection with the university’s network first, e.g. by using a VPN. The colleagues from the network team provide a VPN Service

After establishing a connection to the university’s network you can connect to the cluster system as usual.

Please note: Connection speed from outside networks will mostly be slower than from your office. This can cause applications which provide a graphical user interface to respond slowly and make working with them hard. If GUI windows respond slowly, it is due to the technical nature and not a technical problem with the cluster system.

guide/connecting_and_data_transfer.txt · Last modified: 2021/06/02 14:32 by zzzznana