Tags

Setup Information for Bmad Distributions ( "Off-Site" Setup)



Distribution Overview

A Distribution, (also called a Bmad Distribution), is a stand-alone subset of the ACC Code Libraries, including the Bmad toolkit used for accelerator/X-ray simulations.
  • A Distribution is meant for use on machines that are not part of the CLASSE laboratory network.
  • Distributions are freely available for download from the web. No CLASSE account is required.
  • New Distributions are generated approximately weekly.
  • Distributions contain a subset of the code in a Release. The code that is left out of Distributions is CESR specific and therefore not of interest to outside people.
  • Distributions include the Tao program (as well as many other programs).
  • Distributions can be built on Linux or Mac OS X (intel) computers. For Windows systems, WSL2 should be used.
  • All " OFF-site" work is done using Distributions.
A Distribution contains:
  1. The Bmad source code to build a collection of libraries and executables. (The term " Bmad source code" includes not only the Bmad code library as well as the subsidiary libraries, like cesr_utils, but also the code for programs like Tao.)
  2. Script files to build the libraries and executables.

Distribution names are generally of the form: bmad_dist_YYYY_MMDD-N.tgz where YYYY_MMDD is the year_month-day tag of the release and N is the version number which starts with "0" for the first version generated on a given date, etc.

Bmad has been successfully been compiled on:
  • Linux / Intel ifort compiler (v 17.0.7.259 or later).
  • Linux / gcc gfortran compiler (v 6.3 or later).
  • Mac Intel / gcc gfortran compiler (v 6.3 or later).

Note: ifort and gcc versions earlier than what is stated above are known to fail.

Note: Bmad can be built on Windows by installing a Linux distribution via WSL2.

For more information on Bmad, please see http://www.lepp.cornell.edu/~dcs/bmad/



Conda-Forge installation

Pre-compiled Bmad executables (Tao, etc) can be installed via conda-forge (a Python package manager). When installing with conda, you do not have to download and build a Bmad Distribution. Conda can be used on Windows using WSL2. Using Conda is convenient since Conda takes care of the many package dependencies that Bmad based programs have.

The following instructions for conda are for installing under MacOS or Linux. For installing on Windows, you need to install a Linux distribution under WSL2. Follow the instructions under MS Windows for setting up and running under WSL2 (read that entire section), but replace the step of building Bmad with the following steps, run under the Linux distribution that you installed under WSL2. Do not install the conda package manger for Windows, install the Linux version while running in Linux.

Install instructions:
  1. Install the conda package manager according to your platform: https://docs.conda.io/en/latest/miniconda.html
  2. Install Bmad via:
    conda install -c conda-forge bmad 
    The executables will be added to your path. For example, type:
    tao
    to run Tao.

    By default this will install the latest distribution. All distributions can be listed using:
    conda search -c conda-forge bmad

    A specific distribution can be installed as:
    conda install -c conda-forge bmad=20211013.0 

Note: If you want to modify existing Bmad based program or create new ones under conda, you can link to the libraries at $CONDA_PREFIX/lib and use headers from $CONDA_PREFIX/include. The only drawback is that you will need a compatible compiler as the one used by conda-forge which can be installed with:

conda install -c conda-forge compilers  # this will install C, CXX and FC



Bmad Distribution build steps

To build a Bmad Distribution (as opposed to using Conda-Forge), do the following steps:
  1. Install the required packages for your operating system and do any required setup
  2. Download a distribution
  3. Set up the distribution environment (this step will also need to be executed any time you use Bmad)
  4. Compile the distribution



Downloading a Distribution tarball

  1. Check compiler requirements for compiling Distributions.
  2. Download a distribution tarball from the
    -------->> Distributions Directory <<--------
    The latest distribution is recommended.
  3. Unpack the tarball (substitute the actual file name for bmad_dist_YYYY_MMDD-N.tgz):
     tar xzf bmad_dist_YYYY_MMDD-N.tgz 

As of 2/2020, the Distribution has the following size:

Size of tarball:                        0.54 GB
Size with no builds:                    2.44 GB
Size with only production built:        6.42 GB
Size with debug and production builds: 13.56 GB
 

Note to script writers: The following will set a shell variable to the name of the latest Distribution file (without the .tgz suffix)  

and download the tarball

DIST_FILE=$(curl https://www.classe.cornell.edu/~cesrulib/downloads/latest_distribution_file_prefix)
wget https://www.classe.cornell.edu/~cesrulib/downloads/tarballs/$DIST_FILE.tgz



Updating a Distribution

Sometimes it is useful to update files directly from the repository as opposed to downloading a tarball. For example, if a bug fix has been uploaded to the repository but that fix is not yet available in any tarball. However, downloading a new Distribution tarball is to be preferred since code in the repository is generally not as stable as the code in the tarballs. Having multiple Distributions is advantageous if there are problems with the latest code which, while very rare, has happened.

To update directly from the Repository, go to the distribution root directory and issue the command:
svn update *

This does not require a CLASSE account.

Note: When doing "svn update *" for the first time, SVN might complain that you need to do an upgrade first to modify the format of the SVN information in the Distribution to match the version of the SVN client that you are using. Unfortunately, the "svn upgrade *" command is a bit brain dead and is not smart enough to skip directories that are not under SVN version control. The work around is to upgrade directories individually or in groups. For example, "svn upgrade x*" will upgrade directories whose name starts with "x", etc.

After updating a Distribution, the code needs to be recompiled.



Installing Packages for Various Operating Systems

Detailed instructions are given below for

For any other operating systems, here is some general guidance:

Besides cmake (v 2.8.5 or later) and GCC, there are several standard unix/linux tools needed for compiling the xraylib library. They are:
  • gmake
  • automake
  • autoconf
  • libtool
  • m4

The Tao program needs the following standard packages:
  • readline
  • ncurses
  • termcap
  • X11 Core and Development Libraries (libX11 and libX11-devel)

PLplot (if it is used instead of PGPLOT) requires two more packages installed on your system:
  • cairo
  • pango
  • libXt-devel (usually installs as a dependency for cairo/pango)

tip Use the appropriate package manager (yum or rpm for Red Hat based systems, etc.) to install any needed tools. Note: The name of the package can vary from platform to platform. For example, the readline package is installed with yum by installing readline-devel. A google search will generally reveal the appropriate name.

Note: The relevant compilers for your platform must appear in each user's default PATH , otherwise the scripts in the Distribution will not work!

macOS (formally Mac OS X)

Note: Supported macOS versions are Catalina v10.15, Mojave v10.14 and High Sierra v10.13 - older versions may work but we don't have the resources to help if you have trouble.

Note: The SHELL for building Bmad is bash.

  • If prompted, please don't follow Apple's warning to "upgrade" to zsh.
  • If your SHELL set to zsh, please use the command below to switch back to bash:
    chsh -s /bin/bash

Note: There are various Xcode/OS problems. See the Troubleshooting page for details.

The following tools are needed in addition to those listed in the above Libraries and Tools section:

  1. XQuartz -- provides core and development libraries for X11 X-Windows graphics support for tao and other packages that require graphical output.
  2. Xcode -- Install the latest version from the Apple developer website or get it using the Mac App Store. Once you have Xcode installed, open a terminal, run
     sudo xcode-select --install
    Then click the Install button to install the required command line developer tools.
  3. To complete the Apple Xcode License Agreement, please type:
    sudo xcodebuild -license

Setup with MacPorts Package Management System.

MacPorts documentation and download here.

  • The following commands will download the necessary packages.
    sudo port install gcc12       # Any version (> 6) of gcc may be used. Except! Use gcc 12 or above with M1 Mac to avoid compiler bug.
    sudo port select gcc mp-gcc12
    sudo port install wget
    sudo port install cmake
    sudo port install gmake
    sudo port install automake
    sudo port install autoconf
    sudo port install libtool
    sudo port install m4
    sudo port install pkgconfig
    sudo port install pango      # Note: Needed for PLplot.
    sudo port install zlib

Setup with Homebrew Package Management System.

Note: Using Homebrew and MacPorts on the same computer can lead to conflicts. It is highly recommended that only one be used.
  • With Homebrew use the following the commands:
    brew install make
    brew install gcc@9  # A different version (> 6) of gcc may be used if desired
    brew install cmake
    brew install automake
    brew install autoconf
    brew install libtool
    brew install m4
    brew install pkgconfig
    brew install pango
    brew install zlib
    
    sudo ln -s /usr/local/bin/gcc-9 /usr/local/bin/gcc
    sudo ln -s /usr/local/bin/glibtool /usr/local/bin/libtool
    sudo ln -s /usr/local/bin/glibtoolize /usr/local/bin/libtoolize
    
    export PATH=/usr/local/opt/gettext/bin:/usr/local/opt/make/libexec/gnubin:/usr/local/opt/m4/bin:/usr/local/opt/sqlite/bin:$PATH
    export LDFLAGS="-L/usr/local/opt/gettext/lib -L/usr/local/opt/libffi/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/sqlite/lib"
    export CPPFLAGS="-I/usr/local/opt/gettext/include -I/usr/local/opt/readline/include -I/usr/local/opt/sqlite/include"
    export PKG_CONFIG_PATH="/usr/local/opt/libffi/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig:/usr/local/opt/sqlite/lib/pkgconfig"

  • If ACC_ENABLE_SHARED is set to "Y", you must do the following:
    sudo mkdir /opt/local/
    sudo ln -s /usr/local/lib /opt/local/lib
  • Discover the latest version of readline installed by Homebrew:
    ls /usr/local/Cellar/readline/
  • Use the readline version name to build the symlinks in the command below. For example, if the Homebrew version of readline is 8.1.1:
    sudo ln -s /usr/local/Cellar/readline/8.1.1/lib/lib* /usr/local/lib/
    sudo ln -s /usr/local/Cellar/readline/8.1.1/lib/pkgconfig/*.pc /usr/local/lib/pkgconfig/
    sudo ln -s /usr/local/Cellar/readline/8.1.1/include/readline /usr/local/include


RHEL variants (RHEL, CentOS, Scientific Linux, CERN SLC)

The instructions here assume that ACC_PLOT_PACKAGE="plplot" in dist_prefs.

For RHEL8 and its derivatives (Rocky, CentOS-8), install the required packages with

yum install bc cmake gcc-c++ gcc-gfortran libtool make openssl-devel pango-devel readline-devel

then follow the build instructions.

For RHEL7 and its derivatives (CentOS, Scientific Linux), you will need a newer compiler than the default that is available, which will require the RHEL Developer Toolset. For RHEL7 itself, you will need to adjust your subscription accordingly. For the RHEL derivatives, you will ned need to first install the Software Collections repository as follows:

CentOS 7
yum install centos-release-scl
Scientific Linux 7
yum install yum-conf-softwarecollections
Next, you can install the required packages with

yum install cmake devtoolset-10-gcc-gfortran devtoolset-10-gcc-c++ devtoolset-10-make libtool openssl-devel pango-devel readline-devel

Before starting the build process, you will need to execute

scl enable devtoolset-10 bash

This will create a new subshell where your environment is set up to use the required compilers. If you quit and come back to the build process, you will need to re-execute this latter command to again give yourself the required build environment.


Debain Variants (Ubuntu, etc.)

Your Ubuntu installation should be at version 18.04 or higher. The required packages (for a PLPlot build) are installed by:

 sudo apt-get install gfortran g++ cmake libtool-bin libreadline-dev libpango1.0-dev libssl-dev bc

MS Windows

To install Bmad on Windows 10:
  1. Install WSL2, which will allow you to run a Linux distribution under Windows. For most users the best choice will be to install the latest Ubuntu distribution. This may not perform well on a system with a small amount of memory (you're effectively running a second computer on your hardware).
  2. Build Bmad as described above on whatever Linux distribution you've chosen.
  3. Install an X server. VcXsrv is the simplest choice if you aren't running one already or if your institution doesn't have a license for a commercial option.

A few notes on running:
  • You may need to start the X server. With VcXsrv, an XLaunch application is installed which will do this.
  • tao will need to connect to the X server. You need to do two things for this to work:
    1. You need to set the DISPLAY environment variable on Linux so that tao knows how to connect to the X server. The IP address of the server is the IP address given in /etc/resolv.conf, and it is display 0. So export DISPLAY=###.###.###.###:0 will set the environment variable, where ###.###.###.### is the IP address given in resolv.conf.
    2. The X server will need to allow connections from your Linux distribution. Much of the advice you find says to disable access control when you run XLaunch. This is a poor choice from a computer security perspective. Instead, from Linux, you can cause the X server to create an authorization "cookie" via xauth on the Windows end, then use xauth on the Linux end to add that cookie to its authentication database. The following can be inserted into your .bashrc to set this up (and the DISPLAY environment variable) every time you start a new shell (note C:\Users\xxx\Documents should be replaced by the path to the documents folder in your Windows profile):
      k=$('/mnt/c/Program Files/VcXsrv/xauth.exe' -f 'C:\Users\xxx\Documents\scratch.xauth' -i -n -q 2>/dev/null <<EOF
      generate localhost:0 . trusted timeout 604800
      list
      quit
      EOF
      ) 
      if [ -n "$k" ]
      then 
         export DISPLAY=$(sed '/^nameserver/ {s/^nameserver\s\s*\([0-9][0-9.]*\)[^0-9.]*$/\1/;p;};d' /etc/resolv.conf):0
         xauth -q add $DISPLAY . ${k##* }
         export LIBGL_ALWAYS_INDIRECT=true
      fi 
      unset k
    3. It's possible that connections to the X server may be blocked by a firewall, either the Windows firewall or a firewall installed with a security suite. To allow connections to the X server, allow port 6000 in, and if possible allow that connection only to the X server program (C:\Program Files\VcXsrv\vcxsrv.exe if VcXsrv is your X server). Unfortunately, you usually cannot easily restrict this connection to come from the Linux system, since its address will be unpredictable and will change between reboots.
    4. You may run into problems with the memory usage of the Linux distribuion getting so large that you have performance issues under Windows. The evidence that this is happening is that under Task Manager on Windows, you will see a "vmmem" process consuming a large fraction of your memory. An immediate solution to this problem if you see it happening is to execute sudo echo 3 > /proc/sys/vm/drop_caches on the Linux system. The long-term solution is to limit memory usage of the Linux system using the .wslconfig file in your user profile. To do this, create a file named .wslconfig in your Windows user profile folder (usually C:\Users\<yourUserName>), and insert the following (possibly replacing 8GB with by the amount of memory you want to allow for the Linux system; if possible leave at least 8 GB for Windows):
      [wsl2]
      memory=8GB



Distribution Environment Setup

Important! The distribution environment setup must be done before building or running programs. It is recommended that the setup commands be put in the appropriate shell startup file so that the setup is done automatically when logging in.

  1. Distributions are compiled using bash, sh, or ksh. If your default shell is not one of these, you have two options:
    • Switch the default shell to bash (recommended), or
    • Switch to the bash shell each time you want to compile or run using the command
      /bin/bash
  2. Users should add the following text to their local account login or profile files (or put these commands in a command file and run the command file each time you create a new command window).
    • Note: Replace PATH-to-DISTRIBUTION with the full path to the directory where the Distribution archive was unpacked.
    • The following stuff should be put in your ~/.bashrc file (or ~/.bash or ~/.profile file whatever one is appropriate):
      export DIST_BASE_DIR="PATH-to-DISTRIBUTION"
      source ${DIST_BASE_DIR}/util/dist_source_me
      ulimit -S -c 0
      ulimit -S -s 10240 # Remove this line when using a Mac
      ulimit -S -d 25165824
      The first ulimit prevents core dump files from being generated (takes up space and no one ever looks at them anyway). The next two ulimit lines helps prevent programs from bombing due to software limits on internal stack and heap sizes.
    • If you are using the multiple Distribution scheme as discussed above, PATH-to-DISTRIBUTION should refer the bmad_dist soft link. For example, the definition of DIST_BASE_DIR might look like:
      export DIST_BASE_DIR="..../distribution_base_dir/bmad_dist"
    • Note: The default behavior is for the setup process to print out informational messages. To suppress this output, you may define the environment variable DIST_SETUP_QUIET to have the value "Y". This is entirely optional.
  3. If you have put the initalization commands in a shell startup file, setup the environment by simply logging out and logging back in again (and using the /bin/bash command if needed). Otherwise, if not in a shell startup file, source the command file the initialization commands are in.
  4. To see information about the Distribution that is being used by a terminal session, and to make sure the environment is properly setup, use either of the commands:
    distinfo
    accinfo
    Either command will display a summary of the active Distribution's information, build architecture, selected Fortran compiler and enviromental configuration. In particular, you should see a set of environment variables starting with the prefix ACC_. If you do not see this, the environment has not been properly setup.



Compiling a Distribution

  1. Unzip and extract the tar archive into a working directory of your choice with the command:
    tar -zxf bmad_dist_YYYY_MMDD-N.tgz
  2. Set up the Distribution environment as explained in the Distribution Environment Setup section. Notice that the environmental variable DIST_BASE_DIR points to the root directory (with a name like to bmad_dist_YYYY_MMDD-N).
  3. Change to the root directory:
    cd $DIST_BASE_DIR
  4. Edit the file:
    $DIST_BASE_DIR/util/dist_prefs
    This file contains the preferences for custimizing the build. Documentation for dist_prefs is contained in the file itself. In particular, one can set:
  5. After editing dist_prefs, run your initialization file by loggin out and logging back in again.
  6. The final step is to build the source. For building production libraries and executables, type:
    util/dist_build_production
    For building debug libraries and executables, type:
    util/dist_build_debug
  • Note: If you ever want to restart the build from scratch, before using one of the build commands above, first remove all compiled files using the command:
    util/dist_clean



Organizing Multiple Distributions

It can be convenient to have multiple Distributions. Older Distributions can act as backups in case, for example, the results of simulation programs change. The recommended way to handle multiple distributions is to create a root directory to hold the different distributions. An example directory structure is:

              distribution_base_dir
                       |
     +--------------------------+------------------------+---------------------+
     |                          |                        |                     |
bmad_dist_2020_0507-0     bmad_dist_2020_0821-0    bmad_dist_2121_0107-0    bmad_dist (soft link)

Here the Distribution root directory is called distribution_base_dir. This directory hold three Distributions and a directory soft link called bmad_dist. The soft link will point to the distribution currently being used. For example, if the Distribution named bmad_dist_2021_0107-0 is to be used, the soft link is created by the command (do this in the distribution_base_dir):

ln -sfn bmad_dist_2021_0107-0 bmad_dist

The idea is that to switch from using one Distribution to another, only the bmad_dist soft link needs to be changed. The setup scripts discussed below do not have to be modified since they refer to the bmad_dist soft link.



Directories in a Distribution

Directories Containing Executables

production/bin Production executables are meant for ordinary use.
debug/bin Debug executables are meant for debugging code. Debug executables run slower but do more error checking and can be used with a debugger.

Cornell Developed Library Directories:

bmad Main simulation library.
cpp_bmad_interface C++ interface to Bmad
sim_utils Helper routines (matrix , plotting, file manipulation, etc.)

Cornell Developed Directories containing programs:

bsim Beam simulation programs
examples Example programs and example lattices.
regression_tests Bmad regression testing suite.
tao General simulation program.
util_programs Utility programs (EG lattice format converters, etc.)

Other Cornell Developed Directories:

bmad-doc Documentation of Bmad and associated programs.
util Utility scripts.
build_system Build system script.

3rd-Party Libraries ("Packages"):

fftw Discrete Fourier transform library
forest FPP/PTC Full Polymorphic Package/Particle Tracking Code from Etienne Forest
fgsl A Fortran interface to the GNU Scientific Library
gsl GNU Scientific Library
hdf5 library for storing and managing data.
lapack Linear Algebra PACKage
mad_tpsa A Library for TPSA (Truncated Power Series Algebra) manipulations.
PGPLOT "the Pretty Good Plotting Package". For setup information, see the Plotting Packages wiki page.
plplot A cross-platform software package for creating scientific plots. For setup information, see the Plotting Packages wiki page.
xraylib A library for X-ray matter interaction cross sections for X-ray fluorescence applications


Building your own code using a Distribution

To build your own code, after successfully installing a Distribution, please see the Code Development wiki page.

The following is the ordered list of libraries to link to. Different programs will need different subsets of this list for linking. The libraries that the linker uses is specified by the LINK_LIBS variable in the cmake.XXX file used to compile and link your program. See the file bmad_dist/tao/cmake.tao for an example. The linker at link time will notify you if there are any missing libraries from the LINK_LIBS list. The order of the libraries in the LINK_LIBS list is important.

cpp_bmad_interface C++ interface for Bmad
bmad http://www.lns.cornell.edu/~dcs/bmad/
sim_utils Bmad utility routines
${PLOT_LINK_LIBS} pgplot or plplot; see Plot Packages wiki
${ACC_BMAD_LINK_LIBS} List of third party libraries


Licensing Issues

The FOREST FPP/PTC (Polymorphic Tracking Code) package, from Etienne Forest, has no licensing issues.

The PGPLOT (pretty good plotting) package from Tim Pearson, states on the PGPLOT web page: PGPLOT is not public-domain software. However, it is freely available for non-commercial use. The source code and documentation are copyrighted by California Institute of Technology, and may not be redistributed or placed on public Web servers without permission. The software is provided ``as is'' with no warranty.



Issues with building a Distribution: Troubleshooting Page

Please see our wiki page on Troubleshooting ACC Code Issues



Getting Help

To request help, Please see our Help and Mailing List Information wiki page for email contacts.

Topic revision: r181 - 03 Aug 2022, DavidSagan



  • ACC/ACL Web

  • CLASSE Computing Info
  • CLASSE

Create personal sidebar

This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding CLASSE Wiki? Send feedback