Jupyter

We may want to run a Jupyter Notebook on a powerful remote server, connecting to it from our laptop or local desktop. The best option to achieve that is to run the notebook on the remote machine, but opening it in a tab in our local browser. This is explained in the next section. If you have VSCode, you can use the instructions in the second section.

On a remote machine (browser)

Connect to the remote server and launch the notebook

In this example, change "your_username" by your actual username and "burro" for the name of the machine of your choosing.

Connect to the machine

ssh -X -Y your_username@burro

On the burro, we create a local folder in the /tmp/ directory where to store temporary files (this is a workaround for an issue with jupyter notebooks and remote filesystems mounted via NFS, such as the /home/ filesystem). We also define the JUPYTERLAB_DIR variable (if we are going to use the jupyter-lab command):

mkdir -p /tmp/${USER}
export JUPYTER_RUNTIME_DIR="/tmp/${USER}"
export JUPYTERLAB_DIR="/opt/python/python3.10/share/jupyter/lab"

We load the python 3.10 module:

module load python/3.10

Then we go to our work directory (where we have or want to create our notebooks) and launch the jupyter notebook. It may be a good idea to use a port value higher than the default 8888, to avoid potential conflicts with other notebooks running on the same server.

cd /path/to/the/work/directory/         # replace this by the actual path to your work directory
jupyter notebook --no-browser --port=8891         # or some other port number, like 8892, 8893, etc.

If you get a message that the selected port is already in use, kill it and try with another port number.

Warning

If not already, remember to start and interactive slurm session in the burro before launching jupyter. Otherwise, you may only be using 3/4 of a core in your session. See the default configuration of a burro here.

Take note of the URL associated to the notebook, which is something like: http://localhost:8891/?token=LONGTOKEN

Note

If you use your own python environment, for instance one created with conda, skip the module load python/3.10 step and activate your own environment instead.

Establish a tunnel to the remote machine

We create a tunnel to the remote machine, using the same port number we set in the previous step. We do it on a different terminal window:

ssh -N -f -L localhost:NNNN:localhost:NNNN your_username@burro

replacing NNNN by the actual port number used above. It's OK if it just returns the prompt without asking for a password. It's also OK if, after providing the password, it seems to do nothing. This is expected behaviour. You can kill the tunnel with CTRL-C once you have finished using the jupyter notebook.

Open the notebook in your browser

Open a new window or tab in your local browser (the one running on your laptop), and paste the URL associated to the notebook as noted above.

If you create a new notebook, select the "Python 3 (ipykernel)" option.

Setting Python 3.10 as default kernel

You might need to set the Python version 3.10 as default kernel, especially if, when starting the notebook, you get error messages like "ImportError: cannot import name 'constants'" and the traceback shows a mix of different python versions being used.

To set python 3.10 as the default kernel, we can do the following:

jupyter notebook --generate-config

which generates file /home/${USER}/.jupyter/jupyter_notebook_config.py

Then we edit such file, and change the default_kernel_name line to:

c.MultiKernelManager.default_kernel_name = 'python310'

uncommenting the above line if needed.

If still you have problems running the notebook, please let us know (sinfin [at] iac.es), since the remote server may need a minor modification to allow the jupyter notebook to run properly.

On a remote machine (VSCode)

Have VSCode installed locally

You must first have VSCode installed locally on your own machine (free and all platforms available!).

Once installed you only need a couple of extensions installed:

Load Remote SSH Extension

Go to "Extensions" (Ctr+Shift+X on Win or Cmd+Shift+X on Mac) and search for "SSH".

or use the "Extensions button" to the left bar of VSCode:

Find "Remote - SSH" (by Microsoft) and click on install.

Connect to remote server

By pressing "F1" a text box to enter commands appears. Start typing "Remote-SSH: Connect to Host..." and select it when it appears. Add a connection (first time only)

Click on "+ Add New SSH Host" option and type in the text box the host (like ssh your_user@remote_hostname).

It will ask you where to save the configuration, choose your /home/your_user/.config

Press Enter to connect with your account password.

If all goes well a new VSCode window will appear and you will have established a remote connection to the server (see bottom left text).

(Next time the connection will exist and it will be displayed in the drop-down list)

Load Jupyter (and Python) Extensions on new editor

The new instance of VSCode does not inherit all plugins, you must install them for this remote connection (only the first time).

As before, in this new editor instance press "F1" the text box to enter commands appears. Start typing "Jupyter" and select the Jupyter Notebook Support (from Microsoft)

The Python extension is also very useful for syntax highlighting, linting, syntax correction, etc. so be sure to install it as above, just search for "Python" in the extensions box.

Open an Jupyter Notebook page on the remote server

As this instance runs on the remote server, all files accessed by this server are now available. You can the open an existing file with "File -> Open File..." (or create a new one).

Establish a Remote Python path (or choose a Kernel)

Warning

On machines using Slurm (most at the IAC) if you choose a remote environment it will run but it will be limited to a 3/4 of a core and no GPU.

USING SLURM: setup a Jupyter server on remote machine

To setup a Jupyter server on remote machine first reserve resources on an interactive session in slurm (reserving a GPU also if needed) and then follow the instructions explained above.

Once you have the Jupyter Server running (and get the URL related to it), just tell your local VStudio to use it:

Choose "Existing Jupyter Server..."

Then paste the URL (staring with http://127.0.0.1:/...)

No SLURM: choose an existing Python Kernel (environment)

So the Jupyter Notebook page runs, you can select the path to run Python on the remote server or even an existing environment on this remote server.

Go to the "Select Kernel" icon on the upper right hand of the new instance of VSCode

and a list of available environments will drop-down to choose from.

(if not you can always create a new environment outside of VSCode and restart VSCode so it sees it)