Information for "Off-Site" Bmad Distribution or Conda-Forge Setup

Distribution Overview

• A Distribution is meant for use on machines that are not part of the CLASSE laboratory network.
• Distributions are freely available for download from the web. No CLASSE account is required.
• New Distributions are generated approximately weekly.
• Distributions contain a subset of the code in a Release. The code that is left out of Distributions is CESR specific and therefore not of interest to outside people.
• Distributions include the Tao program (as well as many other programs).
• Distributions can be built on Linux or Mac OS X (intel) computers. For Windows systems, WSL2 should be used.
• All " OFF-site" work is done using Distributions.

1. The Bmad source code to build a collection of libraries and executables. (The term " Bmad source code" includes not only the Bmad code library as well as the subsidiary libraries, like cesr_utils, but also the code for programs like Tao.)
2. Script files to build the libraries and executables.

• Linux / Intel ifort compiler (v 17.0.7.259 or later).
• Linux / gcc gfortran compiler (v 6.3 or later).
• Mac Intel / gcc gfortran compiler (v 6.3 or later).

Conda-Forge installation

1. Install the conda package manager according to your platform: https://docs.conda.io/en/latest/miniconda.html
2. Install Bmad via:

   conda install -c conda-forge bmad 

tao

conda search -c conda-forge bmad

conda install -c conda-forge bmad=20211013.0 

conda install -c conda-forge bmad=*=mpi_openmpi*

conda install -c conda-forge compilers  # this will install C, CXX and FC

conda uninstall -c conda-forge bmad

Bmad Distribution build steps

1. Install the required packages for your operating system and do any required setup
2. Download a distribution
3. Set up the distribution environment (this step will also need to be executed any time you use Bmad)
4. Compile the distribution

Downloading a Distribution tarball

1. Check compiler requirements for compiling Distributions.
2. Download a distribution tarball from the
   -------->> Distributions Directory <<--------
   The latest distribution is recommended.
3. Unpack the tarball (substitute the actual file name for bmad_dist_YYYY_MMDD-N.tgz):

    tar xzf bmad_dist_YYYY_MMDD-N.tgz 

Size of tarball:                        0.54 GB
Size with no builds:                    2.44 GB
Size with only production built:        6.42 GB
Size with debug and production builds: 13.56 GB
 

DIST_FILE=$(curl https://www.classe.cornell.edu/~cesrulib/downloads/latest_distribution_file_prefix)
wget https://www.classe.cornell.edu/~cesrulib/downloads/tarballs/$DIST_FILE.tgz

Updating a Distribution

svn update *

Installing Packages for Various Operating Systems

• macOS
• RedHat Enterprise Linux and Linux distributions derived from it (CentOS, Scientific Linux)
• Debian Linux and distributions derived from it (Ubuntu)
• Microsoft Windows

• gmake
• automake
• autoconf
• libtool
• m4

• readline
• ncurses
• termcap
• X11 Core and Development Libraries (libX11 and libX11-devel)

• cairo
• pango
• libXt-devel (usually installs as a dependency for cairo/pango)

macOS (formally Mac OS X)

Note: The SHELL for building Bmad is bash.

• If prompted, please don't follow Apple's warning to "upgrade" to zsh.
• If your SHELL set to zsh, please use the command below to switch back to bash:

  chsh -s /bin/bash

Note: There are various Xcode/OS problems. See the Troubleshooting page for details.

The following tools are needed in addition to those listed in the above Libraries and Tools section:

1. XQuartz -- provides core and development libraries for X11 X-Windows graphics support for tao and other packages that require graphical output.
2. Xcode -- Install the latest version from the Apple developer website or get it using the Mac App Store. Once you have Xcode installed, open a terminal, run

    sudo xcode-select --install

   Then click the Install button to install the required command line developer tools.
3. To complete the Apple Xcode License Agreement, please type:

   sudo xcodebuild -license

Setup with MacPorts Package Management System.

• The following commands will download the necessary packages.

  sudo port install gcc12       # Any version (> 6) of gcc may be used. Except! Use gcc 12 or above with M1 Mac to avoid compiler bug.
  sudo port select gcc mp-gcc12
  sudo port install wget
  sudo port install cmake
  sudo port install gmake
  sudo port install automake
  sudo port install autoconf
  sudo port install libtool
  sudo port install m4
  sudo port install pkgconfig
  sudo port install pango      # Note: Needed for PLplot.
  sudo port install zlib

Setup with Homebrew Package Management System.

• With Homebrew use the following the commands:

  brew install make
  brew install gcc@9  # A different version (> 6) of gcc may be used if desired
  brew install cmake
  brew install automake
  brew install autoconf
  brew install libtool
  brew install m4
  brew install pkgconfig
  brew install pango
  brew install zlib
  
  sudo ln -s /usr/local/bin/gcc-9 /usr/local/bin/gcc
  sudo ln -s /usr/local/bin/glibtool /usr/local/bin/libtool
  sudo ln -s /usr/local/bin/glibtoolize /usr/local/bin/libtoolize
  
  export PATH=/usr/local/opt/gettext/bin:/usr/local/opt/make/libexec/gnubin:/usr/local/opt/m4/bin:/usr/local/opt/sqlite/bin:$PATH
  export LDFLAGS="-L/usr/local/opt/gettext/lib -L/usr/local/opt/libffi/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/sqlite/lib"
  export CPPFLAGS="-I/usr/local/opt/gettext/include -I/usr/local/opt/readline/include -I/usr/local/opt/sqlite/include"
  export PKG_CONFIG_PATH="/usr/local/opt/libffi/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig:/usr/local/opt/sqlite/lib/pkgconfig"

• If ACC_ENABLE_SHARED is set to "Y", you must do the following:

  sudo mkdir /opt/local/
  sudo ln -s /usr/local/lib /opt/local/lib

• Discover the latest version of readline installed by Homebrew:

  ls /usr/local/Cellar/readline/

• Use the readline version name to build the symlinks in the command below. For example, if the Homebrew version of readline is 8.1.1:

  sudo ln -s /usr/local/Cellar/readline/8.1.1/lib/lib* /usr/local/lib/
  sudo ln -s /usr/local/Cellar/readline/8.1.1/lib/pkgconfig/*.pc /usr/local/lib/pkgconfig/
  sudo ln -s /usr/local/Cellar/readline/8.1.1/include/readline /usr/local/include

RHEL variants (RHEL, CentOS, Scientific Linux, CERN SLC)

yum install bc cmake gcc-c++ gcc-gfortran libtool make openssl-devel pango-devel readline-devel

yum install centos-release-scl

yum install yum-conf-softwarecollections

yum install cmake devtoolset-10-gcc-gfortran devtoolset-10-gcc-c++ devtoolset-10-make libtool openssl-devel pango-devel readline-devel

scl enable devtoolset-10 bash

Debain Variants (Ubuntu, etc.)

 sudo apt-get install gfortran g++ cmake libtool-bin libreadline-dev libpango1.0-dev libssl-dev bc

sudo apt-get install libxt-dev

MS Windows

1. Install WSL2, which will allow you to run a Linux distribution under Windows. For most users the best choice will be to install the latest Ubuntu distribution. This may not perform well on a system with a small amount of memory (you're effectively running a second computer on your hardware).
2. Build Bmad as described above on whatever Linux distribution you've chosen.
3. Install an X server. VcXsrv is the simplest choice if you aren't running one already or if your institution doesn't have a license for a commercial option.

• You may need to start the X server. With VcXsrv, an XLaunch application is installed which will do this.
• tao will need to connect to the X server. You need to do two things for this to work:
  1. You need to set the DISPLAY environment variable on Linux so that tao knows how to connect to the X server. The IP address of the server is the IP address given in /etc/resolv.conf, and it is display 0. So export DISPLAY=###.###.###.###:0 will set the environment variable, where ###.###.###.### is the IP address given in resolv.conf.
  2. The X server will need to allow connections from your Linux distribution. Much of the advice you find says to disable access control when you run XLaunch. This is a poor choice from a computer security perspective. Instead, from Linux, you can cause the X server to create an authorization "cookie" via xauth on the Windows end, then use xauth on the Linux end to add that cookie to its authentication database. The following can be inserted into your .bashrc to set this up (and the DISPLAY environment variable) every time you start a new shell (note C:\Users\xxx\Documents should be replaced by the path to the documents folder in your Windows profile):

     k=$('/mnt/c/Program Files/VcXsrv/xauth.exe' -f 'C:\Users\xxx\Documents\scratch.xauth' -i -n -q 2>/dev/null <<EOF
     generate localhost:0 . trusted timeout 604800
     list
     quit
     EOF
     ) 
     if [ -n "$k" ]
     then 
        export DISPLAY=$(sed '/^nameserver/ {s/^nameserver\s\s*\([0-9][0-9.]*\)[^0-9.]*$/\1/;p;};d' /etc/resolv.conf):0
        xauth -q add $DISPLAY . ${k##* }
        export LIBGL_ALWAYS_INDIRECT=true
     fi 
     unset k

  3. It's possible that connections to the X server may be blocked by a firewall, either the Windows firewall or a firewall installed with a security suite. To allow connections to the X server, allow port 6000 in, and if possible allow that connection only to the X server program (C:\Program Files\VcXsrv\vcxsrv.exe if VcXsrv is your X server). Unfortunately, you usually cannot easily restrict this connection to come from the Linux system, since its address will be unpredictable and will change between reboots.
  4. You may run into problems with the memory usage of the Linux distribuion getting so large that you have performance issues under Windows. The evidence that this is happening is that under Task Manager on Windows, you will see a "vmmem" process consuming a large fraction of your memory. An immediate solution to this problem if you see it happening is to execute sudo echo 3 > /proc/sys/vm/drop_caches on the Linux system. The long-term solution is to limit memory usage of the Linux system using the .wslconfig file in your user profile. To do this, create a file named .wslconfig in your Windows user profile folder (usually C:\Users\<yourUserName>), and insert the following (possibly replacing 8GB with by the amount of memory you want to allow for the Linux system; if possible leave at least 8 GB for Windows):

     [wsl2]
     memory=8GB

Distribution Environment Setup

1. Distributions are compiled using bash, sh, or ksh. If your default shell is not one of these, you have two options:
   • Switch the default shell to bash (recommended), or
   • Switch to the bash shell each time you want to compile or run using the command

     /bin/bash

2. Users should add the following text to their local account login or profile files (or put these commands in a command file and run the command file each time you create a new command window).
   • Note: Replace PATH-to-DISTRIBUTION with the full path to the directory where the Distribution archive was unpacked.
   • The following stuff should be put in your ~/.bashrc file (or ~/.bash or ~/.profile file whatever one is appropriate):

     export DIST_BASE_DIR="PATH-to-DISTRIBUTION"
     source ${DIST_BASE_DIR}/util/dist_source_me
     ulimit -S -c 0
     ulimit -S -s 10240 # Remove this line when using a Mac
     ulimit -d unlimited

     The first ulimit prevents core dump files from being generated (takes up space and no one ever looks at them anyway). The next two ulimit lines helps prevent programs from bombing due to software limits on internal stack and heap sizes.
   • If you are using the multiple Distribution scheme as discussed above, PATH-to-DISTRIBUTION should refer the bmad_dist soft link. For example, the definition of DIST_BASE_DIR might look like:

     export DIST_BASE_DIR="..../distribution_base_dir/bmad_dist"

   • Note: The default behavior is for the setup process to print out informational messages. To suppress this output, you may define the environment variable DIST_SETUP_QUIET to have the value "Y". This is entirely optional.
3. If you have put the initalization commands in a shell startup file, setup the environment by simply logging out and logging back in again (and using the /bin/bash command if needed). Otherwise, if not in a shell startup file, source the command file the initialization commands are in.
4. To see information about the Distribution that is being used by a terminal session, and to make sure the environment is properly setup, use either of the commands:

   distinfo
   accinfo

   Either command will display a summary of the active Distribution's information, build architecture, selected Fortran compiler and enviromental configuration. In particular, you should see a set of environment variables starting with the prefix ACC_. If you do not see this, the environment has not been properly setup.

Compiling a Distribution

1. Unzip and extract the tar archive into a working directory of your choice with the command:

   tar -zxf bmad_dist_YYYY_MMDD-N.tgz

2. Set up the Distribution environment as explained in the Distribution Environment Setup section. Notice that the environmental variable DIST_BASE_DIR points to the root directory (with a name like to bmad_dist_YYYY_MMDD-N).
3. Change to the root directory:

   cd $DIST_BASE_DIR

4. Edit the file:
   

   $DIST_BASE_DIR/util/dist_prefs

   This file contains the preferences for custimizing the build. Documentation for dist_prefs is contained in the file itself. In particular, one can set:
   • The compiler
   • plotting library back end
   • Whether a shared-object version of the libraries is built. Note: When using the PyTao Python-Tao interface , the shared object version needs to be built.
   • Whether programs that support MPI and/or OpenMP are built multi-threaded.
5. After editing dist_prefs, run your initialization file by loggin out and logging back in again.
6. The final step is to build the source. For building production libraries and executables, type:

   util/dist_build_production

   For building debug libraries and executables, type:

   util/dist_build_debug

• Note: If you ever want to restart the build from scratch, before using one of the build commands above, first remove all compiled files using the command:

  util/dist_clean

Organizing Multiple Distributions

              distribution_base_dir
                       |
     +--------------------------+------------------------+---------------------+
     |                          |                        |                     |
bmad_dist_2020_0507-0     bmad_dist_2020_0821-0    bmad_dist_2121_0107-0    bmad_dist (soft link)

ln -sfn bmad_dist_2021_0107-0 bmad_dist

Directories in a Distribution

Directories Containing Executables

Cornell Developed Library Directories:

Cornell Developed Directories containing programs:

Other Cornell Developed Directories:

3rd-Party Libraries ("Packages"):

Building your own code using a Distribution

Licensing Issues

Issues with building a Distribution: Troubleshooting Page

Getting Help

Old: Docker Setup