Software Development

Debugging Tools

• See https://cesrwww.lepp.cornell.edu/wiki/CESR/LinuxDebuggingTools for documentation (maintained by CESR developers) on available debugging tools.
• More detailed documentation on the TotalView debugger can be found on our TotalView Documentation wiki page.

Language Versions

As SL is built for stability and longevity, you may find the default software versions somewhat out of date. Many updated versions are available locally on each system in /opt/rh, with additional libraries in /nfs/opt . Please see below for instructions on setting up your environment for specific versions. Please note that if you want to automatically enable specific software versions in your login scripts (~/.bashrc), you must make sure it's only executed within interactive sessions. You can guarantee this by inserting the code below into a block such as :

if [ "$PS1" ]; then
   ...
fi

within your .bashrc, as described further in UnixLoginScripts.

Python

Please consider using the python virtualenv command to create portable and user-maintainable virtual environments where you can install whatever python modules you need. Please see your search engine of choice for docs on python virtual environments, and don't hesitate to open a ServiceRequest with any problems or questions you may have.

Python 2.7 - Default in SL7

As of this writing, the default version of python is 2.7.5, on SL7. To setup your environment to use libraries installed for you in /nfs/opt, execute:

export PYTHONPATH=/nfs/opt/python2.7/packages.sl7${PYTHONPATH:+:${PYTHONPATH}}

Additional python versions are available as software collections and can be used as follows.

Python 3.3

source /opt/rh/python33/enable
export PYTHONPATH=/nfs/opt/python3.3/packages.sl7${PYTHONPATH:+:${PYTHONPATH}}

After this, all python commands will use 3.3 from /opt/rh/python33.

Python 3.6.8

Since python 3.6.8 is the default python 3 package, simply type:

python3

Instead of the "python" command. To see what modules are available, type:

ls $PYTHONPATH

To setup your environment to use libraries installed for you in /nfs/opt, execute:

export PYTHONPATH=/nfs/opt/python3.6/packages.sl7${PYTHONPATH:+:${PYTHONPATH}}

Using a Python Virtual Environment, with Python 3.6.8

What's recommended is to set up a Python Virtual Environment in which you can install the packages you desire. You can even have multiple virtual environments. There are alternatives (Anaconda/Miniconda) but virtual environments are simple to use and manage.

The official Python Virtual Environment site (but there is lots of info on the web):

https://docs.python.org/3/library/venv.html

As an example:

• login to a CLASSE Linux box following the instructions on RemoteLinux
• change to a directory outside of your home directory. For example

  cd /nfs/user/username

  . Replace "username" With your CLASSE username, and see DataStewardship and HomeDisk for more informaiton.
• to create venv

  python3 -m venv <DIR>

• to activate

  source <DIR>/bin/activate

• to install a package (e.g. pyexpander), in venv

  pip3 install pyexpander

• to deactivate, in venv

  deactivate

Python 3.6.9

source /opt/rh/rh-python36/enable
export PYTHONPATH=/nfs/opt/python3.6/packages.sl7${PYTHONPATH:+:${PYTHONPATH}}

After this, all python commands will use 3.6.9 from /opt/rh/python36.

source /opt/rh/rh-python36/enable
export PYTHONPATH=/nfs/opt/python3.6/packages.sl7${PYTHONPATH:+:${PYTHONPATH}}

Python 3.8.13

source /opt/rh/rh-python38/enable
export PYTHONPATH=/nfs/opt/python3.8/packages.sl7${PYTHONPATH:+:${PYTHONPATH}}

After this, all python commands will use 3.8.13 from /opt/rh/python38.

To simplify enabling the python 3.8.13 environment, place a function in your ~/.bashrc, by using the follwing:

mypython3 () {
source /opt/rh/rh-python38/enable
export PYTHONPATH=/nfs/opt/python3.8/packages.sl7${PYTHONPATH:+:${PYTHONPATH}}
}

Then simply type the following to enable the python 3.8.13 environment:

mypython3

How to install Anaconda on a CLASSE Linux system to avoid interference with the Login Environment

We have had many Service tickets requesting assistance with repairing a Linux User's Login Environment, after the User had installed or updated Anaconda Python. This is especially true on systems where the login is shared amoung a group of users. The issue arises as the Anaconda installer installs an environment initialization into the User's .bashrc file. We have found that this Anaconda environment initialization is not required to use the Anaconda install and causes interference with other functions of the Login Environment.

Instead, either providing the full PATH to the installed Anaconda binaries or simply adding "aliases" or "functions" for the installed Anaconda binaries into the User's .bashrc file remedies the issue.

The first step is to NOT have the the installer place the initialization code into your ~/.bashrc, during the initial Anaconda Installation. This is done by answering "no" to the the following prompt at the end of the installation:

Do you wish the installer to initialize Anaconda3
by running conda init? [yes|no]
[no] >>> 

Then add an "alias" or "function" to your ~/.bashrc. So if you installed Anaconda3 in /nfs/chess/sw/anaconda3-amd275, the following example would let you run ipython:

myipython () {
export PATH=/nfs/chess/sw/anaconda3-amd275/bin${PATH:+:${PATH}}
ipython3
}

Then to run ipython, simply type:

myipython

If you have any questions as to how to install Anaconda on a CLASSE Linux system to avoid interference with the Login Environment, please submit a Service Request to service-classe@cornell.edu .

Red Hat Developer Toolset suite for GCC/GFORTRAN

The Scientific Linux build of the Red Hat Developer Toolset and Red Hat Software Collections, installed locally on every CLASSE Linux workstation, gives us a more supportable means of providing up-to-date compilers and development utilities. Using Red Hat's Software Collections (scl) bundling, the Developer Toolset is installed and used without replacing the default system installed compilers.

Red Hat Developer Toolset 1.1 (GCC version 4.7.2)

The Red Hat Developer Toolset 1.1 is still available via a network file share (just not installed locally on every CLASSE system). It provides the following tools:

• GNU Compiler Collection (GCC) version 4.7.2 (includes gfortran)
• GNU Debugger (GDB) version 7.5
• binutils version 2.23.51
• elfutils version 0.154
• System Tap version 1.8
• Valgrind version 3.8.1
• OProfile version 0.9.7
• dwz version 0.7

To use the Developer Toolset 1.1, type:

 . /nfs/opt/rh/devtoolset-1.1/enable

GCC version 4.8.5 is default is SL7

The Red Hat Developer Toolset 2.0 packages are not needed, since the default SL7 GCC version is 4.8.5

Red Hat Developer Toolset 4.1 (GCC version 5.3.1)

The Developer Toolset 4.1 packages, installed in Feb-2016, updated Jan-2017, provides the following updates:

• GNU Compiler Collection (GCC) version 5.3.1 (includes gfortran)
• GNU Debugger (GDB) version 7.11
• binutils version 2.25.1
• elfutils version 0.166
• System Tap version 2.1
• Valgrind version 4.2.2
• OProfile version 1.1.0
• dwz version 0.12
• Eclipse 4.5.0
• Strace 4.10

To use the updated Developer Toolset 4.1 packages, type:

 . /opt/rh/devtoolset-4/enable 

To list all Developer Toolset 4.1 packages, type:

 rpm -qa devtoolset-4-* | sort

For more info and alternate enable instructions, see the file:

/opt/rh/devtoolset-4/root/usr/share/doc/devtoolset-4-4.1/README

Red Hat Developer Toolset 6.1 (GCC version 6.3.1)

The Developer Toolset 6.1 packages, provides the following updates:

• GNU Compiler Collection (GCC) version 6.3.1 (includes gfortran)
• GNU Debugger (GDB) version 7.12.1
• binutils version 2.27
• elfutils version 0.168
• System Tap version 3.0-8s
• Valgrind version 3.12.0
• OProfile version 1.1.0
• dwz version 0.12
• Strace 4.12

To use the updated Developer Toolset 6.1 packages, type:

 . /opt/rh/devtoolset-6/enable 

To list all Developer Toolset 6.1 packages, type:

 rpm -qa devtoolset-6-* | sort

For more info and alternate enable instructions, see the file:

/opt/rh/devtoolset-6/root/usr/share/doc/devtoolset-6-6.1/README

Red Hat Developer Toolset 7.1 (GCC version 7.3.1)

The updated Developer Toolset 7.1 packages, provides the following updates:

• GNU Compiler Collection (GCC) version 7.3.1 (includes gfortran)
• GNU Debugger (GDB) version 8.0.1
• binutils version 2.28
• elfutils version 0.170
• System Tap version 3.4-4s
• Valgrind version 3.13.0
• OProfile version 1.2.0
• dwz version 0.12
• Strace 4.17

To use the updated Developer Toolset 7.1 packages, type:

 . /opt/rh/devtoolset-7/enable 

To list all Developer Toolset 7.1 packages, type:

 rpm -qa devtoolset-7-* | sort

For more info and alternate enable instructions, see the file:

/opt/rh/devtoolset-7/root/usr/share/doc/devtoolset-7-7.1/README

Red Hat Developer Toolset 8.1 (GCC version 8.3.1)

The updated Developer Toolset 8.1 packages, provides the following updates:

• GNU Compiler Collection (GCC) version 8.3.1 (includes gfortran)
• GNU Debugger (GDB) version 8.2
• binutils version 2.30
• elfutils version 0.176
• System Tap version 3.3-2
• Valgrind version 3.14.0
• OProfile version 1.3.0
• dwz version 0.12
• Strace 4.24

To use the updated Developer Toolset 8.1 packages, type:

 . /opt/rh/devtoolset-8/enable 

To list all Developer Toolset 8 packages, type:

 rpm -qa devtoolset-8-* | sort

For more info and alternate enable instructions, see the file:

/opt/rh/devtoolset-8/root/usr/share/doc/devtoolset-8-8.1/README

Intel Fortran Compilers

To see information about available Intel ifort Fortran compiler versions, please see our Fortran Compilers wiki page.

Continuous Integration (CI)

In the most basic sense, continuous integration is simply the practice of committing small code changes to a code base regularly and then verifying on each commit that everything still works by automatically building, testing and deploying the code. Along with CI is the practice of continuous deployment (CD) which piggy-backs on CI and regularly pushes code changes to the production environment.

Reasons for using CI:

• Avoid "merge hell" by keeping your local workspace up to date through committing small changes and then updating your local copy from the repository.
• Avoid "deployment hell"
  • Figure out your deployment steps ONCE and then let a machine take care of it.
  • Regularly deploy to a staging environment that is a clone of your production environment so you can verify that your build is actually deployable.
  • Avoid breaking the code base by writing tests that are part of an automated buid-test-deploy process, where builds with failed tests are rejected and not deployed.
  • Keep successful builds which have been verified as deployable and can be re-deployed on a moment's notice.

Any software project might have aspects that won't allow using every continuous integration idea. Maybe it takes your code base two hours to build or all of your tests need 8 hours to run. In that case, building and running tests on every single commit isn't feasible. So, you create a cron job to run your build-deploy-test once a day in the middle of the night. In the morning you know if you've got a deployable build. It doesn't matter how you do it. The goal is to check as frequently and as painlessly as possible that your code still works as you're changing it.

There are some very nice continuous integration tools that make automating the build-test-deploy process easier. Tools like Jenkins or GitLab's CI offer nice user interfaces, email notifications on job failures, automatic job running when changes to repositories are detected, and integration with other services.

CI Automation Tools

• Jenkins
• GitLab