Cluster Docs

“Project Jupyter exists to develop open-source software, open-standards, and services for interactive computing across dozens of programming languages.” ¹⁾

“The Jupyter notebook extends the console-based approach to interactive computing in a qualitatively new direction, providing a web-based application suitable for capturing the whole computation process: developing, documenting, and executing code, as well as communicating the results.” ²⁾

“JupyterLab is a web-based interactive development environment for Jupyter notebooks, code, and data.” ³⁾

Detailed information about various components of Project Jupyter can be accessed here.

In this section we explain Jupyter usage in the cluster.

On the cluster, Jupyter Notebook and JupyterLab are availabe from within the cluster web portal as OOD interactive application, which starts Jupyter sessions on a compute node by means of a SLURM job. To access the Jupyter app, login onto the portal and in the Interavitve Apps menu select the Jupyter server. This will open a web dialog page which allows you to start a Jupyter session in a SLURM job using the standard cluster-wide Conda and Module environments you can select from the drop-down Standard environment list. On this page, you can also specify the job parameters such as time limit, number of CPU cores, amount of allocated memory, number of GPU units and the cluster partition you want to start your Jupyter session on. Note that OOD Jupyter applications currently launch Jupyter sessions on a single compute node.

Fig. 1: Jupyter on the cluster

By selecting the standard Jupyter environment in addition to the IPython kernel providing many popular Python packages for Data Science, Machine Learning and Scientific Computing (Tensorflow, PyTorch, Pandas, NumPy, SciPy, Matplotlib, etc) you can also run the Jupyter kernels for R and MATLAB. If a standard environment of your choice does not contain packages you want to use or you would like more kernels to be included, please get in touch with the cluster group at cluster-help@luis.uni-hannover.de. You can also make the packages you need available in your JupyterLab session via a custom Jupyter kernel, see next subsection below. Check out the list of available Jupyter kernels.

If the standard JupyertLab environments do not contain packages you require for your project, you need a different python version or other programming languages you may provide them by creating JupyterLab kernels. In this section we explain how to install a custom python kernel for your Python virtualenv or Conda environments and how to switch between environments without resubmitting JupyterLab session SLURM job.

The installation of a new kernel is done in two steps:

1. Create an environment (Conda or Python virtualenv) and install required Python libraries and packages including ipykernel
2. Install a custom Python kernel for JupyterLab. The kernel will be located in a sub-directory of $HOME/.local/share/jupyter/kernels

Please note: since the installation of Python packages requires an access to internet, the configuration of a kernel must be done on a login node.

[username@login01 ~]$ module load Miniconda3
[username@login01 ~]$ conda create -n my_env

[username@login01 ~]$ conda activate my_env
(my_env)[username@login01 ~]$ conda install ipykernel
(my_env)[username@login01 ~]$ make-ipykernel --name my_conda_env --display-name "My Software Env"

(my_env)[username@login01 ~]$ conda install numpy matplotlib sympy

[username@login01 ~]$ module load GCC/10.2.0 Python/3.8.6

[username@login01 ~]$ virtualenv $HOME/myvenv

[username@login01 ~]$ source $HOME/myvenv/bin/activate
(myvenv)[username@login01 ~]$ pip install ipykernel
(myvenv)[username@login01 ~]$ make-ipykernel --name my_venv --display-name "My Software Env"

(myvenv)[username@login01 ~]$ pip install numpy sympy

A few seconds after creating the kernel, your software environment will appear in your active JupyterLab session listed as My Software Env on the launcher page. To make the kernel immediately available in JupyterLab, just refresh the browser window. Fig. 2: Python Jupyter kernel

You can switch kernels using the menu Kernel → Change Kernel... The kernel Python3 (ipykernel) corresponds to the standard cluster-wide environment you selected when submitting your JupyterLab session. Fig. 3: Change Kernel

Classic Jupyter notebook: Your kernel is listed in the menu New: Fig. 4: Python Jupyter kernel

Classic Jupyter notebook: To switch kernels visit the menu Kernel: Fig. 5: Switch kernels

If software packages provided by the default Jupyter environmnet are not suffcient for your application, you can assemble the necessary modules together with your application in a collection (you will only need to do this once) and access them in a Jupyter session. To do this, on the command line (you can start the command line from within the web portal via the menu item Clusters → LUIS Computing Cluster Shell Access), load the modules necessary for your application. s Please note: You will also need to load either JupyterLab OR IPython to make this work.

Example:

module load foss/2019b JupyterLab/2.2.8-Python-3.7.4 TensorFlow/2.3.1-Python-3.7.4

After this, save the modules loaded in a collection:

module save jupyter_tensorflow231

Use the following command to list the collections you already created:

module savelist

Collections are saved in your ${HOME}/.lmod.d directory

Now you can start a Jupyter session that loads the collection of modules you just created. To do this, go to Interactive Apps → Jupyter in the web portal and choose your module collection under the item “Jupyter/JupyterLab environment”. In case it is not yet listed there, try to reload the page, possibly while pressing the shift key to update the browser cache.

Alternatvelly, you can use Conda package manager to create environments and use them from within the OOD Jupyter interactive application. For details about how to create conda environments, see the conda usage on the cluster. Please note that your conda environment must contain either juputerlab or jupyter package to use it from the OOD Jupyter app. Once you have created an environment for you application, you can accesss it from the same drop-down Jupyter/JupyterLab environment list.