Jupyter on LUIS Cluster

“Project Jupyter exists to develop open-source software, open-standards, and services for interactive computing across dozens of programming languages.” 1)Project Jupyter webpage

“The Jupyter notebook extends the console-based approach to interactive computing in a qualitatively new direction, providing a web-based application suitable for capturing the whole computation process: developing, documenting, and executing code, as well as communicating the results.” 2)Jupyter Notebook documentation

“JupyterLab is a web-based interactive development environment for Jupyter notebooks, code, and data.” 3)JupyterLab documentation

Detailed information about various components of Project Jupyter can be accessed here.

In this section we explain Jupyter usage in the cluster.

How to start Jupyter on the cluster

On the cluster, Jupyter Notebook and JupyterLab are availabe from within the cluster web portal as OOD interactive application, which starts Jupyter sessions on a compute node by means of a SLURM job. To access the Jupyter app, login onto the portal and in the Interavitve Apps menu select the Jupyter server. This will open a web dialog page which allows you to start a Jupyter session in a SLURM job using the standard cluster-wide Conda and Module environments you can select from the drop-down Standard environment list. On this page, you can also specify the job parameters such as time limit, number of CPU cores, amount of allocated memory, number of GPU units and the cluster partition you want to start your Jupyter session on. Note that OOD Jupyter applications currently launch Jupyter sessions on a single compute node.

Jupyter on the cluster

By selecting the standard Jupyter environment in addition to the IPython kernel providing many popular Python packages for Data Science, Machine Learning and Scientific Computing (Tensorflow, PyTorch, Pandas, NumPy, SciPy, Matplotlib, etc) you can also run the Jupyter kernels for R and MATLAB. If a standard environment of your choice does not contain packages you want to use or you would like more kernels to be included, please get in touch with the cluster group at cluster-help@luis.uni-hannover.de. You can also make the packages you need available in your JupyterLab session via a custom Jupyter kernel, see next subsection below. Check out the list of available Jupyter kernels.

Creating custom Jupyter kernels

If the standard JupyertLab environments do not contain packages you require for your project, you need a different python version or other programming languages you may provide them by creating JupyterLab kernels. In this section we explain how to install a custom python kernel for your Python virtualenv or Conda environments and how to switch between environments without resubmitting JupyterLab session SLURM job.

The installation of a new kernel is done in two steps:

Create an environment (Conda or Python virtualenv) and install required Python libraries and packages including ipykernel

Install a custom Python kernel for JupyterLab. The kernel will be located in a sub-directory of $HOME/.local/share/jupyter/kernels

Please note: since the installation of Python packages requires an access to internet, the configuration of a kernel must be done on a login node.

the conda usage

1

[username@login01 ~]$ module load Miniforge3
[username@login01 ~]$ conda create -n my_env

2ipykernel

[username@login01 ~]$ conda activate my_env
(my_env)[username@login01 ~]$ conda install ipykernel
(my_env)[username@login01 ~]$ make-ipykernel --name my_conda_env --display-name "My Software Env"

kernel.jsonmy_conda_env$HOME/.local/share/jupyter/kernels/my_conda_env

Note--name

3

(my_env)[username@login01 ~]$ conda install numpy matplotlib sympy

1

[username@login01 ~]$ module load GCC/10.2.0 Python/3.8.6

2myvenv

[username@login01 ~]$ virtualenv $HOME/myvenv

3ipykernel

[username@login01 ~]$ source $HOME/myvenv/bin/activate
(myvenv)[username@login01 ~]$ pip install ipykernel
(myvenv)[username@login01 ~]$ make-ipykernel --name my_venv --display-name "My Software Env"

kernel.jsonmy_venv$HOME/.local/share/jupyter/kernels/my_venv

Note--name

4

(myvenv)[username@login01 ~]$ pip install numpy sympy

A few seconds after creating the kernel, your software environment will appear in your active JupyterLab session listed as My Software Env on the launcher page. To make the kernel immediately available in JupyterLab, just refresh the browser window. Python Jupyter kernel

You can switch kernels using the menu Kernel → Change Kernel... The kernel Python3 (ipykernel) corresponds to the standard cluster-wide environment you selected when submitting your JupyterLab session. Change Kernel

Classic Jupyter notebook: Your kernel is listed in the menu New: Python Jupyter kernel

Classic Jupyter notebook: To switch kernels visit the menu Kernel: Switch kernels