Application Development Environment

The application development environment on foseres is primarily controlled through the modules environment. By loading and switching modules you control the compilers, libraries and software available.

This means that for compiling on foseres you typically set the compiler you wish to use using the appropriate modules, then load all the required library modules (e.g. numerical libraries, IO format libraries).

Additionally, if you are compiling parallel applications using MPI (or SHMEM, etc.) then you will need to load the MPI environment and use the appropriate compiler wrapper scripts.

The list of the modules loaded can be obtained by typing:

[user@foseres ~]$ module list

Currently Loaded Modules:
  1) autotools   2) prun/1.3   3) intel/19.0.5.281   4) impi/2019.5.281   5) ohpc

Basic usage of the module command on foseres is covered below. For full documentation please see:

Linux manual page on modules

Using the modules environment

Information on the available modules

Finding out which modules (and hence which compilers, libraries and software) are available on the system is performed using the module avail command:

[user@foseres ~]$ module avail
...

This will list all the names and versions of the modules available on the service. Not all of them may work in your account though due to, for example, licencing restrictions. You will notice that for many modules we have more than one version, each of which is identified by a version number. One of these versions is the default. As the service develops the default version will change.

You can list all the modules of a particular type by providing an argument to the module avail command. For example, to list all available versions of the Intel libraries, compilers and tools:

[user@foseres ~]$ module avail gnu

--------------------------------------------------------------------------------------- /opt/ohpc/pub/modulefiles ----------------------------------------------------------------------------------------
gnu/5.4.0    gnu7/7.3.0    gnu8/8.3.0

--

If you want more info on any of the modules, you can use the module help command:

[user@foseres ~]$ module help openmpi3/3.1.4

------------------------------------------------------------------------------- Module Specific Help for "openmpi3/3.1.4" --------------------------------------------------------------------------------

This module loads the openmpi3 library built with the intel toolchain.

Version 3.1.4

The simple module list command will give the names of the modules and their versions you have presently loaded in your envionment:

[user@foseres ~]$ module list

 Currently Loaded Modules:
 1) autotools   2) prun/1.3   3) intel/19.0.5.281   4) impi/2019.5.281   5) ohpc

Loading, unloading and swapping modules

To load a module to use module add or module load. For example, to load the default Intel compilers into the development environment:

module load intel

If you need a specific version of the module, you can add more information:

module load intel/18.0.1.163

will load version 18.0.1.163 for you, regardless of the default. If you want to clean up, module remove will remove a loaded module:

module remove intel

(or module rm intel or module unload intel) will unload what ever version of intel (even if it is not the default) you might have loaded. There are many situations in which you might want to change the presently loaded version to a different one, such as trying the latest version which is not yet the default or using a legacy version to keep compatibility with old data. This can be achieved most easily by using module swap oldmodule newmodule.

Available Compiler Suites

Note

As foseres uses dynamic linking by default you will generally also need to load any modules you used to compile your code in your job submission script when you run your code.

Intel Compiler Suite

The Intel compiler suite is accessed by loading the intel/18.0.1, intel/18.0.1.163, or intel/19.0.5.281 or the most recent intel oneAPI modules which are located in a separated section:/opt/ohpc/pub/modulefiles_intel_oneAPI. For example:

module load compiler/2021.2.0

Once you have loaded the module, the compilers are available as:

ifort - Fortran
icc - C
icpc - C++

GCC Compiler Suite

The GCC 4.8.5 compiler suite is available by default without loading any modules.

The compilers are available as:

gfortran - Fortran
gcc - C
g++ - C++

More recent version of the GCC compiler suite are available: gnu/5.4.0, gnu7/7.3.0 or gnu8/8.3.0.

Compiling MPI codes

foseres currently supports several MPI implementations. By default we recommend to use the Intel MPI library. It can be loaded by typing:

module load mpi/2021.2.0

You should also consult the chapter on running jobs through the batch system for examples of how to run jobs compiled against MPI.

Using Intel MPI

To compile MPI code with the oneAPI implementaion of Intel MPI you must first load the “compiler/2021.2.0” and “mpi/2021.2.0” module:

module load compiler/2021.2.0  mpi/2021.2.0

Using GCC Compilers and openmpi3

To load gnu8 and openmpi3, type

module load gnu8/8.3.0 openmpi3/3.1.4

The MPI compilers are then available as

mpif90 - Fortran with MPI
mpicc - C with MPI
mpicxx - C++ with MPI

Summary

The name of the wrapper script depends on the compiler suite you are using. In summary:

Language	Intel	GCC
Fortran	mpiifort	mpif90
C++	mpiicpc	mpicxx
C	mpiicc	mpicc

Compiler Information and Options

Help is available for the different compiler suites

GCC: Fortran gfortran --help , C/C++ gcc --help
Intel: Fortran man ifort , C/C++ man icc

Useful compiler options

Note

For best performance on foseres we currently advise that you should use the Intel compilers.

Whilst difference codes will benefit from compiler optimisations in different ways, for reasonable performance on foseres, at least initially, we suggest the following compiler options:

Intel: -O2
GNU: -O2 -ftree-vectorize -funroll-loops -ffast-math

TO UPDATE

Depending on how many users need to use the software, software requests are divided into three types, each of which corresponds to the location where the software is installed:

The user’s home directory Software packages installed here will be accessible only to the user. It is suitable for software packages that will be used by a single user. Python, Perl and R modules should be installed here. /project Software packages installed in /project can be accessed by a group of users. It is suitable for software packages that Need to be shared by users from the same research group, or are bigger than the quota on the home file syste. This type of request must be sent by the PI of the research group, who may be asked to apply for a storage allocation. /usr/local/packages Software packages installed under /usr/local/packages can be accessed by all users. It is suitable for software packages that will be used by users from multiple research groups. This type of request must be sent by the PI of a research group.