.. |br| raw:: html

   <br>

.. role:: red



Environment Modules
===========================

:red:`The environment modules package is a tool that provides the dynamic 
modification of the user's environment for supporting several versions 
of an application or a library without any conflict.`


The tool provides the command line “module” and its syntax is:

.. sourcecode:: bash

    module [options] [command] [args ...]

The main commands are:

.. list-table::
   :header-rows: 1

   * - command
     - meaning
   * - avail
     - list all or matching available software
   * - load / add
     - load a specific software
   * - unload / rm
     - unload a specific software
   * - purge
     - unload all loaded software
   * - list
     - list all loaded software
   * - switch
     - change version of software

The options are

.. list-table::
   :header-rows: 1

   * - option
     - meaning
   * - --help or -h
     - Help infomation
   * - --version or -v
     - Module version

If you want to know other commands, you can run

.. sourcecode:: bash

    module or module --help

The result is paginated: to move to the next page, press "space" or, if you quit, press button "q"


List/Search available software
------------------------------

.. I'm working in "denso" (one of the public Linux computing servers).

You want to know all available software

.. sourcecode:: bash

    module avail

In contrast, in order to know the latest or default version

.. sourcecode:: bash

   # the latest version
   module avail -L
   # the default version
   module avail -d

In this example, the default and latest versions are the same.

If you want to know the specific versions of a software,

.. sourcecode:: bash

   # the specific versions
   module avail gnuastro


Load/Unload specific software
-----------------------------

When you want to use a specific software, you use the command "load / add"

.. sourcecode:: bash

    # For a default version
    module load gnuastro
    module add gnuastro
    # For a specific version
    module load gnuastro/0.15
    module add gnuastro/0.15

If you are done with a specific software, you can unload it with commands "unload/rm"

.. sourcecode:: bash

    module unload gnuastro/0.15
    module rm gnuastro/0.15

One thing is important, you cannot use two versions at the same time

For example, I'm loading the default gnuastro

.. sourcecode:: bash

    module load gnuastro

But, I want to use the another version

.. sourcecode:: bash

    module load gnuastro/0.15
    WARNING: gnuastro/0.16 cannot be loaded due to a conflict.
    HINT: Might try "module unload gnuastro" first.

Assuming the software is loaded, we can do it two ways: unload/load or switch/swap

.. sourcecode:: bash

   # first way
   module unload gnuastro
   module load gnuastro/0.15
   # second way
   module switch gnuastro gnuastro/0.15 or
   module swap gnuastro gnuastro/0.15

Maybe, to know what software versions are loaded, you can use the command "list"

.. sourcecode:: bash

   module list

If you need to unload all software, you can use command "purge"

.. sourcecode:: bash

   module purge
   # check all unloaded software
   module list


Save/Restore a specific environment
------------------------------------
If you should load the same software several times, you can save current software list to a file, named collection

.. sourcecode:: bash

    module load gnuastro
    module load python

    # save the environment to a file, (pythonastro)
    module save pythonastro

This command creates a directory in your home, .module, where all files are stored.

For restoring the specific environment, you should use the command 'restore'

.. sourcecode:: bash

    # restore the softare list from a file, (pythonastro)
    module restore pythonastro
    # check if the software was loaded
    module list

If you want to list all files, collections, you should use the command 'savelist'

.. sourcecode:: bash

    # list all file
    module savelist

If you want to remember the software list saved, you should use the command 'saveshow'

.. sourcecode:: bash

    # restore the softare list from a file, (pythonastro)
    module saveshow pythonastro

That's it!



|
|

Python virtual environments
===========================


.. note::

   Though I mentioned the "vaex" package in the slides, we are not
   using it in this example, as it is a big module and its 
   installation (including dependencies) takes too long. We use
   a simpler web application, to explore galaxy spectra dataset
   from the SDSS.


Create new virtual environment
------------------------------

In this example, I work in "denso" (one of the public Linux computing 
servers).

First of all, if on an IAC Linux server or desktop, make sure no
modules are loaded (as they might interfere or conflict with the
virtual environment).

.. sourcecode:: bash

   module purge
   # clear all loaded modules

Then create a directory which will hold the Python virtual environments 
(useful if you install several virtual environments).

.. sourcecode:: bash
   
   mkdir -p /scratch/ncaon/py38_virtualenvs
   # -p: create parent directories as necessary; no error if
   # directory already exists

Create the Python virtual environment inside the above directory.

.. sourcecode:: bash

   cd /scratch/ncaon/py38_virtualenvs/
   /usr/bin/python3.8 -m venv sdss_explor
   # python3.8 is the highest python version installed in our
   # Ubuntu 18.04 servers

This creates the */scratch/ncaon/py38_virtualenvs/sdss_explor/*
directory, which includes directories *bin/*, *include/*, *lib/*.
No python packages are present, except "pip" and "setuptools", 
necessary to install new packages.

..
   bare-bone python installation

Activate environment, install packages
--------------------------------------

We first activate the environment

.. sourcecode:: bash

   source /scratch/ncaon/py38_virtualenvs/sdss_explor/bin/activate
   # Note that the prompt has changed.

Check that now command ``python3`` is the one from the virtual 
environment

.. sourcecode:: bash
    
   which python3
   # /scratch/ncaon/py38_virtualenvs/sdss_explor/bin/python3
   # OK!

Then I usually install/update the basic packages (pip,
setuptools and wheel) to avoid multiple warning messages 
reminding that new versions of pip and setuptools are available.

.. 
   the above is not strictly necessary ...

.. sourcecode:: bash

   python3 -m pip install --upgrade pip
   python3 -m pip install --upgrade setuptools
   python3 -m pip install --upgrade wheel

..
   wheel package: compile source code and pack it for installation
   in python.
   wheel format: binary format similar to zip which only need 
   unpacking to be installed.
 
And finally install all the packages required by the application. 

.. sourcecode:: bash

   python3 -m pip install numpy
   python3 -m pip install pylru
   python3 -m pip install packaging
   python3 -m pip install pandas
   python3 -m pip install jinja2==2.11.3
   python3 -m pip install bokeh==1.4.0
   python3 -m pip install markupsafe==1.1.1

Here we force specific, older versions of these three packages, as 
this application is incompatible with more recent releases (by 
default, the latest version is installed).

.. 
   jinja2: package to write HTML in python-based
   web applictions
   bokeh: interactive data visualization in Python
   markupsafe: escapes characters so text is safe to use 
   in HTML and XML.
   pylru: implement a Python cache

See what packages are now installed in this environment.

.. sourcecode:: bash

   python3 -m pip list --format=columns

Save the package list into a file, which will be used to recreate the
environment on a different system.

.. sourcecode:: bash

   python3 -m pip freeze  > /home/ncaon/requirements.txt
   # quickly inspect its content:
   more /home/ncaon/requirements.txt
   # note that each package is listed with the specific installed
   # version.


When we are done working in this environment, we should deactivate it:

.. sourcecode:: bash

   deactivate
   # the prompt is back to its previous setting


Recreate environment in another machine
---------------------------------------

Let's say that we also want to install this environment in another
machine, with a different Operating System, like a MacBook laptop
or desktop.

In this example I will recreate the environment on my iMac, where I
have Python 3.9 installed via MacPorts.

First of all I switch to my iMac account and copy the requirements.txt 
file.

.. sourcecode:: bash

   scp ncaon@denso:/home/ncaon/requirements.txt /Users/ncaon/

Then I will create the directory where I keep the Python virtual
environments, create the "sdss_explor" folder, and activate it.

.. sourcecode:: bash
 
   mkdir -p /Users/ncaon/py39_virtualenvs    
   cd /Users/ncaon/py39_virtualenvs/
   /opt/local/bin/python3.9 -m venv sdss_explor
   # /opt/local/bin/python3.9 is the MacPorts python3.9 executable 
   source /Users/ncaon/py39_virtualenvs/sdss_explor/bin/activate

Again we install/upgrade first pip, setuptools and wheel.

.. sourcecode:: bash

   python3 -m pip install --upgrade pip
   python3 -m pip install --upgrade setuptools
   python3 -m pip install --upgrade wheel


Then install the packages using the "requirements.txt" file.

.. sourcecode:: bash

   python3 -m pip install -r /Users/ncaon/requirements.txt
   # may take a little while ...

We make sure all has worked out as expected.

.. sourcecode:: bash

   python3 -m pip list --format=columns


As a final check, we deploy the web application.

.. sourcecode:: bash

   cd /Users/ncaon/sdss_explor/portal/
   export BOKEH_ALLOW_WS_ORIGIN=161.72.201.103:5006
   # this should allow connections from external hosts.
   python3 portal.py

Now try to load http://161.72.201.103:5006/ on your browser 
(it may take a few seconds for the web app to show up).


Replicate in similar machine
----------------------------

If you wish to replicate then environment in a machine with the
same characteristics (same OS distribution and release, 
Python version etc.) there is as simpler and faster procedure. 
Here we do it in dejar, under a different naming scheme.

We use rsync to copy the whole "sdss_explor" directory from denso to
dejar. Here I log in on dejar and type:

.. sourcecode:: bash

   from="/scratch/ncaon/py38_virtualenvs/sdss_explor/"
   to="/scratch/ncaon/sdss_explor-py38/"
   mkdir -p ${to}
   rsync -auv --progress ncaon@denso:${from} ${to}

What happens if I try to activate the environment?

.. sourcecode:: bash

   source /scratch/ncaon/sdss_explor-py38/bin/activate
   python3 -m pip list --format=columns

As you can see, this is not the list of the packages we installed
in the virtual environment, but a list of the system-installed packages.

The reason is that we need to change the value of variable *VIRTUAL_ENV*
inside *bin/activate* to reflect the new path.

.. sourcecode:: bash

   cd /scratch/ncaon/sdss_explor-py38/bin/
   nano activate
   # change:   VIRTUAL_ENV="/scratch/ncaon/sdss_explor-py38/"

And we check if it works.

.. sourcecode:: bash

   deactivate
   # since the activation was done with a wrong path.
   source /scratch/ncaon/sdss_explor-py38/bin/activate
   python3 -m pip list --format=columns

OK!