IACTalks 10th anniversary
Ten years ago, the IAC started broacasting publicly in live streaming, for the
worldwide astronomical community, virtually all the scientific colloquia
and seminars given at the IAC, which were also archived on a dedicated server.
To take advantage of this valuable repository, we developed a web-based
application to organize, manage and promote the scientific talks given at the
IAC.
IACTalks is probably one of
the most complete, free and open access astronomy seminars archive in the world.
IACTalks celebrated its 10th anniversary this last April, with the very
first talk taking place on April
29 2008, given by Prof. Hans-G. Ludwig on
Radiation-hydrodynamical model atmospheres across the Hertzsprung-Russell diagram.
By now, the IACTalks archive has grown to more than one thousand talks,
including
colloquia by
high profile astronomers and scientists and
thematic series
(IAC Winter Schools, software course, Virtual Observatory Workshop, etc.),
thanks to the work of the Seminars Commission throughout all these years.
Recently, we have adapted IACTalks to use the
IAC's Youtube channel
so as to get more visibility: we expect a large update once the new IAC's
institutional website becomes public later this year. Meanwhile, don't forget
to visit IACTalks from time to time, as we are sure that you will find many
interesting talks there.
Workshop "Introduction to Supercomputing"
As part of the SIE training program, last week we gave an introductory
10-hour workshop about Supercomputing, addressed to those researchers who
had no (or little) knowledge of Supercomputing. Besides learning
basic notions in these topics, the 12 attenders worked on simple
parallelizations using "de-facto" standards for both shared (OpenMP) and
distributed memory (MPI), became more familiar with working with our
Supercomputers, including best practices, and finally got an overview
about how to execute jobs using HTCondor. Our intention is to continuously
adapt this workshop to the current status and Supercomputing resources at
IAC and teach it once or twice a year, depenging on the demand (please,
let us know if you would like to attend a new edition of this workshop in
September/October).
It's time to move to Python 3! (Python 2 obsoleted in 2020)
As you may have already heard, Python 2 will reach its end of life on 1st
January 2020 (see
https://devguide.python.org/#status-of-python-branche). This
means that many fundamental packages such as numpy, matplotlib, ipython, etc.
will "freeze" for Python 2 at that time (or even earlier), and new
developments will only be done for Python 3. Please see
http://python3statement.org/
for more details and for specific timelines.
This means that you should consider moving your Python 2 programs and
applications to Python 3: no need to hurry up, but do start thinking about
that. The only reason we see for using Python 2 is PyRAF, which can however be
taken care of by installing and running Astroconda, or by setting up a virtual
environment. Another important thing to note is that the latest version of
astropy, 3.X, requires Python 3.5 or 3.6. Now, in Fedora 21, we have Python
3.4, so we cannot update astropy. If you wish to use astropy 3.X, an update
to Fedora version 26 is required. Do get in touch with us if you have
any Python related issues or want to discuss upgrade options.
Note: On Fedora-26 PCs, the environment is set up for Python 3. If you
wish to use Python 2, please redefine
PYTHONPATH as follows:
export PYTHONPATH="${PYTHONPATH2}" (bash) or
setenv PYTHONPATH "${PYTHONPATH2}" (tcsh),
or load the corresponding Python module
Using environment modules on our "burros" (Fedora 26)
The SIE keeps updating astronomical and other related software
packages to allow our researchers to use the latest features. However, some
users might need to use old versions due to compatibility issues or bugs.
It's not always easy to achieve the coexistence of different versions of the
same software on the same machine, since they usually set different
paths to libraries, executable files, etc., among other problems. To deal
with this, there are
Software Environment Management applications,
Modules being probably the most
famous and widely used.
For instance, "modules" is used in all our Supercomputers, and we are beginning
to use it also on our burros with Fedora 26.
Working with "modules" is very easy: some software packages aren't
accessible by default, but must be loaded with the
module load command.
To see what software packages are available, type module avail. For instance:
module avail (full list of all available software)
module avail python
python/2.7.14 python/3.6.5
module load python/3.6.5
Here you can find the most common commands:
module help → Show help
module avail → Show all available modules
module load/add <md> → Load module <md>
module list → List all my loaded modules
module rm/unload <md> → Unload module <md>
module purge → Unload all loaded modules
module switch/swap <md1> <md2> → Unload module <md1> and
load <md2>
module display/show <md> → Show detailed info about <md>
(If two slash-separated options are shown, you can use either.)
Right now we have included just a few software packages in the module list, while
the remaining ones are loaded by default. But little by little this situation will
change, so we encourage you to begin to work with modules and get familiar with
it in the near future.
Python bindings for HTCondor
While HTCondor versions later than 8.4 (released in 2015) have added
more powerful submission features, we are aware that many of our
HTCondor users are still using scripts to generate customized submission
files. Sometimes this blurs and complicates the process of submitting jobs;
usually such scripts can be easily replaced by simple submission files.
Nevertheless, if you like the idea of submitting jobs via scripts, then we
highly recommend that you use the Python bindings, which not only allow
you to submit your jobs directly from a python code, but also to control all
jobs, check whether they have finished or not, if the results are correct or
there were errors, and much more. Python bindings for HTCondor are available
on all our Linux Desktops, with official documentation in
https://research.cs.wisc.edu/htcondor/manual/current/6_7Python_Bindings.html.
Here goes a simple example:
import htcondor
import classad
# Number of Jobs
N = 10
# Get the Schedduler
schedd = htcondor.Schedd()
# Build the Submission info
sub = htcondor.Submit()
sub['FNAME'] = "test"
sub['ID'] = "$(Cluster).$(Process)"
sub['output'] = "$(FNAME).$(ID).out"
sub['error'] = "$(FNAME).$(ID).err"
sub['log'] = "$(FNAME).$(Cluster).log"
sub['should_transfer_files'] = 'YES'
sub['executable'] = '/bin/hostname'
sub['arguments'] = ''
# Submit!
with schedd.transaction() as txn:
clusterId = sub.queue(txn, N)
if not clusterId:
print("There was an error submitting jobs")
else:
print("SUBMISSION OK!! ClusterId = " + str(clusterId))