Environment Modules¶
The environment modules package is a tool that provides the dynamic modification of the user’s environment for supporting several versions of an application or a library without any conflict.
The tool provides the command line “module” and its syntax is:
module [options] [command] [args ...]
The main commands are:
command |
meaning |
|---|---|
avail |
list all or matching available software |
load / add |
load a specific software |
unload / rm |
unload a specific software |
purge |
unload all loaded software |
list |
list all loaded software |
switch |
change version of software |
The options are
option |
meaning |
|---|---|
–help or -h |
Help infomation |
–version or -v |
Module version |
If you want to know other commands, you can run
module or module --help
The result is paginated: to move to the next page, press “space” or, if you quit, press button “q”
List/Search available software¶
You want to know all available software
module avail
In contrast, in order to know the latest or default version
# the latest version
module avail -L
# the default version
module avail -d
In this example, the default and latest versions are the same.
If you want to know the specific versions of a software,
# the specific versions
module avail gnuastro
Load/Unload specific software¶
When you want to use a specific software, you use the command “load / add”
# For a default version
module load gnuastro
module add gnuastro
# For a specific version
module load gnuastro/0.15
module add gnuastro/0.15
If you are done with a specific software, you can unload it with commands “unload/rm”
module unload gnuastro/0.15
module rm gnuastro/0.15
One thing is important, you cannot use two versions at the same time
For example, I’m loading the default gnuastro
module load gnuastro
But, I want to use the another version
module load gnuastro/0.15
WARNING: gnuastro/0.16 cannot be loaded due to a conflict.
HINT: Might try "module unload gnuastro" first.
Assuming the software is loaded, we can do it two ways: unload/load or switch/swap
# first way
module unload gnuastro
module load gnuastro/0.15
# second way
module switch gnuastro gnuastro/0.15 or
module swap gnuastro gnuastro/0.15
Maybe, to know what software versions are loaded, you can use the command “list”
module list
If you need to unload all software, you can use command “purge”
module purge
# check all unloaded software
module list
Save/Restore a specific environment¶
If you should load the same software several times, you can save current software list to a file, named collection
module load gnuastro
module load python
# save the environment to a file, (pythonastro)
module save pythonastro
This command creates a directory in your home, .module, where all files are stored.
For restoring the specific environment, you should use the command ‘restore’
# restore the softare list from a file, (pythonastro)
module restore pythonastro
# check if the software was loaded
module list
If you want to list all files, collections, you should use the command ‘savelist’
# list all file
module savelist
If you want to remember the software list saved, you should use the command ‘saveshow’
# restore the softare list from a file, (pythonastro)
module saveshow pythonastro
That’s it!
Python virtual environments¶
Note
Though I mentioned the “vaex” package in the slides, we are not using it in this example, as it is a big module and its installation (including dependencies) takes too long. We use a simpler web application, to explore galaxy spectra dataset from the SDSS.
Create new virtual environment¶
In this example, I work in “denso” (one of the public Linux computing servers).
First of all, if on an IAC Linux server or desktop, make sure no modules are loaded (as they might interfere or conflict with the virtual environment).
module purge
# clear all loaded modules
Then create a directory which will hold the Python virtual environments (useful if you install several virtual environments).
mkdir -p /scratch/ncaon/py38_virtualenvs
# -p: create parent directories as necessary; no error if
# directory already exists
Create the Python virtual environment inside the above directory.
cd /scratch/ncaon/py38_virtualenvs/
/usr/bin/python3.8 -m venv sdss_explor
# python3.8 is the highest python version installed in our
# Ubuntu 18.04 servers
This creates the /scratch/ncaon/py38_virtualenvs/sdss_explor/ directory, which includes directories bin/, include/, lib/. No python packages are present, except “pip” and “setuptools”, necessary to install new packages.
Activate environment, install packages¶
We first activate the environment
source /scratch/ncaon/py38_virtualenvs/sdss_explor/bin/activate
# Note that the prompt has changed.
Check that now command python3 is the one from the virtual
environment
which python3
# /scratch/ncaon/py38_virtualenvs/sdss_explor/bin/python3
# OK!
Then I usually install/update the basic packages (pip, setuptools and wheel) to avoid multiple warning messages reminding that new versions of pip and setuptools are available.
python3 -m pip install --upgrade pip
python3 -m pip install --upgrade setuptools
python3 -m pip install --upgrade wheel
And finally install all the packages required by the application.
python3 -m pip install numpy
python3 -m pip install pylru
python3 -m pip install packaging
python3 -m pip install pandas
python3 -m pip install jinja2==2.11.3
python3 -m pip install bokeh==1.4.0
python3 -m pip install markupsafe==1.1.1
Here we force specific, older versions of these three packages, as this application is incompatible with more recent releases (by default, the latest version is installed).
See what packages are now installed in this environment.
python3 -m pip list --format=columns
Save the package list into a file, which will be used to recreate the environment on a different system.
python3 -m pip freeze > /home/ncaon/requirements.txt
# quickly inspect its content:
more /home/ncaon/requirements.txt
# note that each package is listed with the specific installed
# version.
When we are done working in this environment, we should deactivate it:
deactivate
# the prompt is back to its previous setting
Recreate environment in another machine¶
Let’s say that we also want to install this environment in another machine, with a different Operating System, like a MacBook laptop or desktop.
In this example I will recreate the environment on my iMac, where I have Python 3.9 installed via MacPorts.
First of all I switch to my iMac account and copy the requirements.txt file.
scp ncaon@denso:/home/ncaon/requirements.txt /Users/ncaon/
Then I will create the directory where I keep the Python virtual environments, create the “sdss_explor” folder, and activate it.
mkdir -p /Users/ncaon/py39_virtualenvs
cd /Users/ncaon/py39_virtualenvs/
/opt/local/bin/python3.9 -m venv sdss_explor
# /opt/local/bin/python3.9 is the MacPorts python3.9 executable
source /Users/ncaon/py39_virtualenvs/sdss_explor/bin/activate
Again we install/upgrade first pip, setuptools and wheel.
python3 -m pip install --upgrade pip
python3 -m pip install --upgrade setuptools
python3 -m pip install --upgrade wheel
Then install the packages using the “requirements.txt” file.
python3 -m pip install -r /Users/ncaon/requirements.txt
# may take a little while ...
We make sure all has worked out as expected.
python3 -m pip list --format=columns
As a final check, we deploy the web application.
cd /Users/ncaon/sdss_explor/portal/
export BOKEH_ALLOW_WS_ORIGIN=161.72.201.103:5006
# this should allow connections from external hosts.
python3 portal.py
Now try to load http://161.72.201.103:5006/ on your browser (it may take a few seconds for the web app to show up).
Replicate in similar machine¶
If you wish to replicate then environment in a machine with the same characteristics (same OS distribution and release, Python version etc.) there is as simpler and faster procedure. Here we do it in dejar, under a different naming scheme.
We use rsync to copy the whole “sdss_explor” directory from denso to dejar. Here I log in on dejar and type:
from="/scratch/ncaon/py38_virtualenvs/sdss_explor/"
to="/scratch/ncaon/sdss_explor-py38/"
mkdir -p ${to}
rsync -auv --progress ncaon@denso:${from} ${to}
What happens if I try to activate the environment?
source /scratch/ncaon/sdss_explor-py38/bin/activate
python3 -m pip list --format=columns
As you can see, this is not the list of the packages we installed in the virtual environment, but a list of the system-installed packages.
The reason is that we need to change the value of variable VIRTUAL_ENV inside bin/activate to reflect the new path.
cd /scratch/ncaon/sdss_explor-py38/bin/
nano activate
# change: VIRTUAL_ENV="/scratch/ncaon/sdss_explor-py38/"
And we check if it works.
deactivate
# since the activation was done with a wrong path.
source /scratch/ncaon/sdss_explor-py38/bin/activate
python3 -m pip list --format=columns
OK!