Presentation Supercomputing Zulip @ IAC SIEpedia IT News

Index of all issues November 2021 June 2021 March 2021 December 2020 Summer 2018 March / April 2018 January / February 2018 November / December 2017 September / October 2017 May / June 2017 January / February 2017 November / December 2016 July / August 2016
For earlier issues, please browse the Index

SIENews

The newsletter of the SIE de Investigación y Enseñanza N. 62 - March/April 2018

IACTalks 10th anniversary

Ten years ago, the IAC started broacasting publicly in live streaming, for the worldwide astronomical community, virtually all the scientific colloquia and seminars given at the IAC, which were also archived on a dedicated server. To take advantage of this valuable repository, we developed a web-based application to organize, manage and promote the scientific talks given at the IAC. IACTalks is probably one of the most complete, free and open access astronomy seminars archive in the world.
IACTalks celebrated its 10th anniversary this last April, with the very first talk taking place on April 29 2008, given by Prof. Hans-G. Ludwig on Radiation-hydrodynamical model atmospheres across the Hertzsprung-Russell diagram. By now, the IACTalks archive has grown to more than one thousand talks, including colloquia by high profile astronomers and scientists and thematic series (IAC Winter Schools, software course, Virtual Observatory Workshop, etc.), thanks to the work of the Seminars Commission throughout all these years. Recently, we have adapted IACTalks to use the IAC's Youtube channel so as to get more visibility: we expect a large update once the new IAC's institutional website becomes public later this year. Meanwhile, don't forget to visit IACTalks from time to time, as we are sure that you will find many interesting talks there.

Workshop "Introduction to Supercomputing"

As part of the SIE training program, last week we gave an introductory 10-hour workshop about Supercomputing, addressed to those researchers who had no (or little) knowledge of Supercomputing. Besides learning basic notions in these topics, the 12 attenders worked on simple parallelizations using "de-facto" standards for both shared (OpenMP) and distributed memory (MPI), became more familiar with working with our Supercomputers, including best practices, and finally got an overview about how to execute jobs using HTCondor. Our intention is to continuously adapt this workshop to the current status and Supercomputing resources at IAC and teach it once or twice a year, depenging on the demand (please, let us know if you would like to attend a new edition of this workshop in September/October).

It's time to move to Python 3! (Python 2 obsoleted in 2020)

As you may have already heard, Python 2 will reach its end of life on 1st January 2020 (see https://devguide.python.org/#status-of-python-branche). This means that many fundamental packages such as numpy, matplotlib, ipython, etc. will "freeze" for Python 2 at that time (or even earlier), and new developments will only be done for Python 3. Please see http://python3statement.org/ for more details and for specific timelines.
This means that you should consider moving your Python 2 programs and applications to Python 3: no need to hurry up, but do start thinking about that. The only reason we see for using Python 2 is PyRAF, which can however be taken care of by installing and running Astroconda, or by setting up a virtual environment. Another important thing to note is that the latest version of astropy, 3.X, requires Python 3.5 or 3.6. Now, in Fedora 21, we have Python 3.4, so we cannot update astropy. If you wish to use astropy 3.X, an update to Fedora version 26 is required. Do get in touch with us if you have any Python related issues or want to discuss upgrade options.

Note: On Fedora-26 PCs, the environment is set up for Python 3. If you wish to use Python 2, please redefine PYTHONPATH as follows:
export PYTHONPATH="${PYTHONPATH2}"   (bash)   or
setenv PYTHONPATH "${PYTHONPATH2}"   (tcsh), or load the corresponding Python module

Using environment modules on our "burros" (Fedora 26)

The SIE keeps updating astronomical and other related software packages to allow our researchers to use the latest features. However, some users might need to use old versions due to compatibility issues or bugs. It's not always easy to achieve the coexistence of different versions of the same software on the same machine, since they usually set different paths to libraries, executable files, etc., among other problems. To deal with this, there are Software Environment Management applications, Modules being probably the most famous and widely used. For instance, "modules" is used in all our Supercomputers, and we are beginning to use it also on our burros with Fedora 26. Working with "modules" is very easy: some software packages aren't accessible by default, but must be loaded with the module load command.
To see what software packages are available, type module avail. For instance:
module avail (full list of all available software)
module avail python
  python/2.7.14 python/3.6.5
module load python/3.6.5

Here you can find the most common commands:
module help → Show help
module avail → Show all available modules
module load/add <md> → Load module <md>
module list → List all my loaded modules
module rm/unload <md> → Unload module <md>
module purge → Unload all loaded modules
module switch/swap <md1> <md2> → Unload module <md1> and load <md2>
module display/show <md> → Show detailed info about <md>
(If two slash-separated options are shown, you can use either.)

Right now we have included just a few software packages in the module list, while the remaining ones are loaded by default. But little by little this situation will change, so we encourage you to begin to work with modules and get familiar with it in the near future.

Python bindings for HTCondor

While HTCondor versions later than 8.4 (released in 2015) have added more powerful submission features, we are aware that many of our HTCondor users are still using scripts to generate customized submission files. Sometimes this blurs and complicates the process of submitting jobs; usually such scripts can be easily replaced by simple submission files. Nevertheless, if you like the idea of submitting jobs via scripts, then we highly recommend that you use the Python bindings, which not only allow you to submit your jobs directly from a python code, but also to control all jobs, check whether they have finished or not, if the results are correct or there were errors, and much more. Python bindings for HTCondor are available on all our Linux Desktops, with official documentation in https://research.cs.wisc.edu/htcondor/manual/current/6_7Python_Bindings.html.

Here goes a simple example:
import htcondor 
import classad 

# Number of Jobs 
N = 10 

# Get the Schedduler 
schedd = htcondor.Schedd() 

# Build the Submission info 
sub = htcondor.Submit() 

sub['FNAME'] = "test" 
sub['ID']   = "$(Cluster).$(Process)" 
sub['output']  = "$(FNAME).$(ID).out" 
sub['error']   = "$(FNAME).$(ID).err" 
sub['log']     = "$(FNAME).$(Cluster).log" 
sub['should_transfer_files']   = 'YES' 

sub['executable'] = '/bin/hostname' 
sub['arguments']  = '' 

# Submit! 
with schedd.transaction() as txn: 
    clusterId = sub.queue(txn, N) 

if not clusterId: 
    print("There was an error submitting jobs") 
else: 
    print("SUBMISSION OK!! ClusterId = " + str(clusterId))
SIE de Investigación y Enseñanza :: N. 62 - March/April 2018 - Contact: