Tags

Setup Information for Bmad Distributions ( "Off-Site" Setup)



Distribution Overview

A Distribution, (also called a Bmad Distribution), is a stand-alone subset of the ACC Code Libraries, including the Bmad toolkit used for accelerator/X-ray simulations.
  • A Distribution is meant for use on machines that are not part of the CLASSE laboratory network.
  • Distributions are freely available for download from the web. No CLASSE account is required.
  • New Distributions are generated approximately weekly.
  • Distributions contain a subset of the code in a Release. The code that is left out of Distributions is CESR specific and therefore not of interest to outside people.
  • Distributions include the Tao program (as well as many other programs).
  • Distributions can be built on Linux or Mac OS X (intel) computers. Note: The ACC Build System does not work well on Windows but it is possible to run in a Docker virtual environment.
  • All " OFF-site" work is done using Distributions.
A Distribution contains:
  1. The Bmad source code to build a collection of libraries and executables. (The term " Bmad source code" includes not only the Bmad code library as well as the subsidiary libraries, like cesr_utils, but also the code for programs like Tao.)
  2. Script files to build the libraries and executables.

Distribution names are generally of the form: bmad_dist_YYYY_MMDD-N.tgz where YYYY_MMDD is the year_month-day tag of the release and N is the version number which starts with "0" for the first version generated on a given date, etc.

Bmad has been successfully been compiled on:
  • Linux / Intel ifort compiler (v 17.0.7.259 or later).
  • Linux / gcc gfortran compiler (v 6.3 or later).
  • Mac Intel / gcc gfortran compiler (v 6.3 or later).

Note: ifort and gcc versions earlier than what is stated above are known to fail.

Note: Bmad can be built on Windows by installing a Linux distribution via WSL2.

For more information on Bmad, please see http://www.lepp.cornell.edu/~dcs/bmad/



Downloading a Distribution tarball

  1. Check compiler requirements for compiling Distributions.
  2. Download a distribution tarball from the
    -------->> Distributions Directory <<--------
    The latest distribution is recommended.
  3. Unpack the tarball (substitute the actual file name for bmad_dist_YYYY_MMDD-N.tgz):
     tar xzf bmad_dist_YYYY_MMDD-N.tgz 

As of 2/2020, the Distribution has the following size:
Size of tarball:                        0.54 GB
Size with no builds:                    2.44 GB
Size with only production built:        6.42 GB
Size with debug and production builds: 13.56 GB
 

Note to script writers: The following will set a shell variable to the name of the latest Distribution file (without the .tgz suffix)

DIST_FILE=$(curl https://www.classe.cornell.edu/~cesrulib/downloads/latest_distribution_file_prefix)



Updating a Distribution

Sometimes it is useful to update files directly from the repository as opposed to downloading a tarball. For example, if a bug fix has been uploaded to the repository but that fix is not yet available in any tarball. However, downloading a new Distribution tarball is to be preferred since code in the repository is generally not as stable as the code in the tarballs. Having multiple Distributions is advantageous if there are problems with the latest code which, while very rare, has happened.

To update directly from the Repository, go to the distribution root directory and issue the command:

svn update *

This does not require a CLASSE account.

Note: When doing "svn update *" for the first time, SVN might complain that you need to do an upgrade first to modify the format of the SVN information in the Distribution to match the version of the SVN client that you are using. Unfortunately, the "svn upgrade *" command is a bit brain dead and is not smart enough to skip directories that are not under SVN version control. The work around is to upgrade directories individually or in groups. For example, "svn upgrade x*" will upgrade directories whose name starts with "x", etc.

After updating a Distribution, the code needs to be recompiled.



Libraries and Tools Needed for Compiling

Besides cmake (v 2.8.5 or later) and GCC, there are several standard unix/linux tools needed for compiling the xraylib library. They are:
  • gmake
  • automake
  • autoconf
  • libtool
  • m4

The Tao program needs the following standard packages:
  • readline
  • ncurses
  • termcap
  • X11 Core and Development Libraries (libX11 and libX11-devel)

PLplot (if it is used instead of PGPLOT) requires two more packages installed on your system:
  • cairo
  • pango
  • libXt-devel (usually installs as a dependency for cairo/pango)

tip Use the appropriate package manager (yum or rpm for Red Hat based systems, etc.) to install any needed tools. Note: The name of the package can vary from platform to platform. For example, the readline package is installed with yum by installing readline-devel. A google search will generally reveal the appropriate name.

Note: The relevant compilers for your platform must appear in each user's default PATH , otherwise the scripts in the Distribution will not work!

Yum install

  sudo yum install gcc-gfortran 
  sudo yum install make 
  sudo yum install automake 
  sudo yum install autoconf 
  sudo yum install libtool 
  sudo yum install m4 
  sudo yum install ncurses 
  sudo yum install libX11 
  sudo yum install libX11-devel 
  sudo yum install readline-devel 
  sudo yum install cmake 
  sudo yum install gcc-c++ 
  sudo yum install xauth 
  sudo yum install libXt-devel
  sudo yum install xorg-xeyes

macOS (formally Mac OS X)

Note: Supported macOS versions are Catalina v10.15, Mojave v10.14 and High Sierra v10.13 - older versions may work but we don't have the resources to help if you have trouble.

Note: The SHELL for building Bmad is bash.

  • If prompted, please don't follow Apple's warning to "upgrade" to zsh.
  • If your SHELL set to zsh, please use the command below to switch back to bash:
    chsh -s /bin/bash

Note: There are various Xcode/OS problems. See the Troubleshooting page for details.

The following tools are needed in addition to those listed in the above Libraries and Tools section:

  1. XQuartz -- provides core and development libraries for X11 X-Windows graphics support for tao and other packages that require graphical output.
  2. Xcode -- Install the latest version from the Apple developer website or get it using the Mac App Store. Once you have Xcode installed, open a terminal, run
     sudo xcode-select --install
    Then click the Install button to install the required command line developer tools.
  3. To complete the Apple Xcode License Agreement, please type:
    sudo xcodebuild -license
  4. MacPorts -- package management system. See the installation site for downloads and instructions.

  • The following commands will download the necessary packages along with cmake and GCC using MacPorts.
    sudo port install gcc9       # A different version (> 6) of gcc may be used if desired
    sudo port select gcc mp-gcc9
    sudo port install wget
    sudo port install cmake
    sudo port install gmake
    sudo port install automake
    sudo port install autoconf
    sudo port install libtool
    sudo port install m4
    sudo port install pkgconfig
    sudo port install pango      # Only needed if PLplot is being used

Homebrew Users

  • With Homebrew use the following the commands:
    brew install make
    brew install gcc@9  # A different version (> 6) of gcc may be used if desired
    brew install cmake
    brew install automake
    brew install autoconf
    brew install libtool
    brew install m4
    brew install pkgconfig
    brew install pango
    
    sudo ln -s /usr/local/bin/gcc-9 /usr/local/bin/gcc
    sudo ln -s /usr/local/bin/glibtool /usr/local/bin/libtool
    sudo ln -s /usr/local/bin/glibtoolize /usr/local/bin/libtoolize
    
    export PATH=/usr/local/opt/gettext/bin:/usr/local/opt/make/libexec/gnubin:/usr/local/opt/m4/bin:/usr/local/opt/sqlite/bin:$PATH
    export LDFLAGS="-L/usr/local/opt/gettext/lib -L/usr/local/opt/libffi/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/sqlite/lib"
    export CPPFLAGS="-I/usr/local/opt/gettext/include -I/usr/local/opt/readline/include -I/usr/local/opt/sqlite/include"
    export PKG_CONFIG_PATH="/usr/local/opt/libffi/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig:/usr/local/opt/sqlite/lib/pkgconfig"

  • If ACC_ENABLE_SHARED is set to "Y", you must do the following:
    sudo mkdir /opt/local/
    sudo ln -s /usr/local/lib /opt/local/lib
  • Discover the latest version of readline installed by Homebrew:
    ls /usr/local/Cellar/readline/
    Then use that to build the symlinks in the command below - in macOS 10.15.5, the Homebrew version of readline is 8.0.1:
    sudo ln -s /usr/local/Cellar/readline/8.0.1/lib/* /usr/local/lib/

RHEL variants (RHEL, CentOS, Scientific Linux, CERN SLC)

If the version of the default GCC gfortran compiler is less than 6.3, you should install one of the RHEL Developer Toolset Software Collections, at least devtoolset-6 or greater.

To install devtoolset-6 (provides GCC gfortran version 6.3.1), see basic information below:
https://www.softwarecollections.org/en/scls/rhscl/devtoolset-6/

That page gives instructions for RHEL and CentOS. For straight Scientific Linux (not the CERN version), replace the first step of the instructions with the instructions from:
https://blog.abysm.org/2016/03/installing-developer-toolset-rhel-based-distributions/

For CERN SLC 6, follow the instructions here:
http://linux.web.cern.ch/linux/scientific6/docs/softwarecollections.shtml

Once devtoolset-6 is installed, the Bmad Distribution build system will find and initialize the devtoolset environment. If this fails, you can manually initialize the devtoolset environment (for bash shell users) by typing:

scl enable devtoolset-6 bash



Debain Variants (Ubuntu, etc.)

Your Ubuntu installation should be at version 18.04 or higher. The required packages (for a PLPlot build) are installed by:

 sudo apt-get install gfortran g++ cmake libtool-bin libreadline-dev libpango1.0-dev libssl-dev


Windows Building

To install Bmad on Windows 10:
  1. Install WSL2, which will allow you to run a Linux distribution under Windows. For most users the best choice will be to install the latest Ubuntu distribution. This may not perform well on a system with a small amount of memory (you're effectively running a second computer on your hardware).
  2. Build Bmad as described above on whatever Linux distribution you've chosen.
  3. Install an X server. VcXsrv is the simplest choice if you aren't running one already or if your institution doesn't have a license for a commercial option.
A few notes on running:
  • You may need to start the X server. With VcXsrv, an XLaunch application is installed which will do this.
  • tao will need to connect to the X server. You need to do two things for this to work:
    1. You need to set the DISPLAY environment variable on Linux so that tao knows how to connect to the X server. The IP address of the server is the IP address given in /etc/resolv.conf, and it is display 0. So export DISPLAY=###.###.###.###:0 will set the environment variable, where ###.###.###.### is the IP address given in resolv.conf.
    2. The X server will need to allow connections from your Linux distribution. Much of the advice you find says to disable access control when you run XLaunch. This is a poor choice from a computer security perspective. Instead, from Linux, you can cause the X server to create an authorization "cookie" via xauth on the Windows end, then use xauth on the Linux end to add that cookie to its authentication database. The following can be inserted into your .bashrc to set this up (and the DISPLAY environment variable) every time you start a new shell (note C:\Users\xxx\Documents should be replaced by the path to the documents folder in your Windows profile):
      k=$('/mnt/c/Program Files/VcXsrv/xauth.exe' -f 'C:\Users\xxx\Documents\scratch.xauth' -i -n -q 2>/dev/null <<EOF
      generate localhost:0 . trusted timeout 604800
      list
      quit
      EOF
      ) 
      if [ -n "$k" ]
      then 
         export DISPLAY=$(sed '/^nameserver/ {s/^nameserver\s\s*\([0-9][0-9.]*\)[^0-9.]*$/\1/;p;};d' /etc/resolv.conf):0
         xauth -q add $DISPLAY . ${k##* }
         export LIBGL_ALWAYS_INDIRECT=true
      fi 
      unset k
    3. It's possible that connections to the X server may be blocked by a firewall, either the Windows firewall or a firewall installed with a security suite. To allow connections to the X server, allow port 6000 in, and if possible allow that connection only to the X server program (C:\Program Files\VcXsrv\vcxsrv.exe if VcXsrv is your X server). Unfortunately, you usually cannot easily restrict this connection to come from the Linux system, since its address will be unpredictable and will change between reboots.
    4. You may run into problems with the memory usage of the Linux distribuion getting so large that you have performance issues under Windows. The evidence that this is happening is that under Task Manager on Windows, you will see a "vmmem" process consuming a large fraction of your memory. An immediate solution to this problem if you see it happening is to execute sudo echo 3 > /proc/sys/vm/drop_caches on the Linux system. The long-term solution is to limit memory usage of the Linux system using the .wslconfig file in your user profile. To do this, create a file named .wslconfig in your Windows user profile folder (usually C:\Users\<yourUserName>), and insert the following (possibly replacing 8GB with by the amount of memory you want to allow for the Linux system; if possible leave at least 8 GB for Windows):
      [wsl2]
      memory=8GB
An alternative on Windows is to use Docker, though WSL2 is likely to work better.

Organizing Multiple Distributions

It can be convenient to have multiple Distributions. Older Distributions can act as backups in case, for example, the results of simulation programs change. The recommended way to handle multiple distributions is to create a root directory to hold the different distributions. An example directory structure is:

              distribution_base_dir
                       |
     +--------------------------+------------------------+---------------------+
     |                          |                        |                     |
bmad_dist_2020_0507-0     bmad_dist_2020_0821-0    bmad_dist_2121_0107-0    bmad_dist (soft link)

Here the Distribution root directory is called distribution_base_dir. This directory hold three Distributions and a directory soft link called bmad_dist. The soft link will point to the distribution currently being used. For example, if the Distribution named bmad_dist_2021_0107-0 is to be used, the soft link is created by the command (do this in the distribution_base_dir):

ln -sfn bmad_dist_2021_0107-0 bmad_dist

The idea is that to switch from using one Distribution to another, only the bmad_dist soft link needs to be changed. The setup scripts discussed below do not have to be modified since they refer to the bmad_dist soft link.



Distribution Environment Setup

Important! The distribution environment setup must be done before building or running programs. It is recommended that the setup commands be put in the appropriate shell startup file so that the setup is done automatically when logging in.

  1. Distributions are compiled using bash, sh, or ksh. If your default shell is not one of these, you have two options:
    • Switch the default shell to bash (recommended), or
    • Switch to the bash shell each time you want to compile or run using the command
      /bin/bash
  2. Users should add the following text to their local account login or profile files (or put these commands in a command file and run the command file each time you create a new command window).
    • Note: Replace PATH-to-DISTRIBUTION with the full path to the directory where the Distribution archive was unpacked.
    • The following stuff should be put in your ~/.bashrc file (or ~/.bash or ~/.profile file whatever one is appropriate):
      export DIST_BASE_DIR="PATH-to-DISTRIBUTION"
      source ${DIST_BASE_DIR}/util/dist_source_me
      ulimit -S -c 0
      ulimit -S -s 10240 # Remove this line when using a Mac
      ulimit -S -d 25165824
      The first ulimit prevents core dump files from being generated (takes up space and no one ever looks at them anyway). The next two ulimit lines helps prevent programs from bombing due to software limits on internal stack and heap sizes.
    • If you are using the multiple Distribution scheme as discussed above, PATH-to-DISTRIBUTION should refer the bmad_dist soft link. For example, the definition of DIST_BASE_DIR might look like:
      export DIST_BASE_DIR="..../distribution_base_dir/bmad_dist"
    • Note: The default behavior is for the setup process to print out informational messages. To suppress this output, you may define the environment variable DIST_SETUP_QUIET to have the value "Y". This is entirely optional.
  3. If you have put the initalization commands in a shell startup file, setup the environment by simply logging out and logging back in again (and using the /bin/bash command if needed). Otherwise, if not in a shell startup file, source the command file the initialization commands are in.
  4. To see information about the Distribution that is being used by a terminal session, and to make sure the environment is properly setup, use either of the commands:
    distinfo
    accinfo
    Either command will display a summary of the active Distribution's information, build architecture, selected Fortran compiler and enviromental configuration. In particular, you should see a set of environment variables starting with the prefix ACC_. If you do not see this, the environment has not been properly setup.



Compiling a Distribution

  1. Unzip and extract the tar archive into a working directory of your choice with the command:
    tar -zxf bmad_dist_YYYY_MMDD-N.tgz
  2. Set up the Distribution envionment as explained in the Distribution Environment Setup section. Notice that the environmental variable DIST_BASE_DIR points to the root directory (with a name like to bmad_dist_YYYY_MMDD-N).
  3. Change to the root directory:
    cd $DIST_BASE_DIR
  4. Edit the file:
    $DIST_BASE_DIR/util/dist_prefs
    This file contains the preferences for custimizing the build. Documentation for dist_prefs is contained in the file itself. In particular, one can set:
  5. After editing dist_prefs, run your initialization file by loggin out and logging back in again.
  6. The final step is to build the source. For building production libraries and executables, type:
    util/dist_build_production
    For building debug libraries and executables, type:
    util/dist_build_debug
  • Note: To remove compiled files and restart from scratch run:
    util/dist_clean



Docker Virtual Environment

Docker is a program where software can be run in a virtual environment called a container. In particular, containers may be setup to run Bmad based programs. The advantage is that for people with no experience, it is easier to setup a container than it is to setup a Distribution. And on Windows, setting up a Distribution is problematical (due to lack of official support for Windows building). The drawback is that a container adds a layer between a program to be run and the user.

For setting up a Docker container for Bmad click here.



Directories in a Distribution

Directories Containing Executables

production/bin Production executables are meant for ordinary use.
debug/bin Debug executables are meant for debugging code. Debug executables run slower but do more error checking and can be used with a debugger.

Cornell Developed Library Directories:

bmad Main simulation library.
cpp_bmad_interface C++ interface to Bmad
sim_utils Helper routines (matrix , plotting, file manipulation, etc.)

Cornell Developed Directories containing programs:

bsim Beam simulation programs
examples Example programs and example lattices.
regression_tests Bmad regression testing suite.
tao General simulation program.
util_programs Utility programs (EG lattice format converters, etc.)

Other Cornell Developed Directories:

lattice Lattice files for various machines.
util Utility scripts.
build_system Build system script.

3rd-Party Libraries ("Packages"):

fftw Discrete Fourier transform library
forest FPP/PTC Full Polymorphic Package/Particle Tracking Code from Etienne Forest
fgsl A Fortran interface to the GNU Scientific Library
gsl GNU Scientific Library
hdf5 library for storing and managing data.
lapack Linear Algebra PACKage
mad_tpsa A Library for TPSA (Truncated Power Series Algebra) manipulations.
PGPLOT "the Pretty Good Plotting Package". For setup information, see the Plotting Packages wiki page.
plplot A cross-platform software package for creating scientific plots. For setup information, see the Plotting Packages wiki page.
recipes_f-90_LEPP A F90 double precision version of Numerical Recipes
xraylib A library for X-ray matter interaction cross sections for X-ray fluorescence applications


Building your own code using a Distribution

To build your own code, after successfully installing a Distribution, please see the Code Development wiki page.

The following is the ordered list of libraries to link to. Different programs will need different subsets of this list for linking. The libraries that the linker uses is specified by the LINK_LIBS variable in the cmake.XXX file used to compile and link your program. See the file bmad_dist/tao/cmake.tao for an example. The linker at link time will notify you if there are any missing libraries from the LINK_LIBS list. The order of the libraries in the LINK_LIBS list is important.
cpp_bmad_interface C++ interface for Bmad
bmad http://www.lns.cornell.edu/~dcs/bmad/
sim_utils Bmad utility routines
${PLOT_LINK_LIBS} pgplot or plplot; see Plot Packages wiki
${ACC_BMAD_LINK_LIBS} List of third party libraries



Licensing Issues

This distribution includes a modified version of the Numerical Recipes code. It is the responsibility of the end user to ensure that they have a valid Numerical Recipes license for any computer on which they build the Bmad distribution. Numerical Recipes licensing information (for individual computers or sites) can be obtained at www.nr.com.

The FOREST FPP/PTC (Polymorphic Tracking Code) package, from Etienne Forest, has no licensing issues.

The PGPLOT (pretty good plotting) package from Tim Pearson, states on the PGPLOT web page: PGPLOT is not public-domain software. However, it is freely available for non-commercial use. The source code and documentation are copyrighted by California Institute of Technology, and may not be redistributed or placed on public Web servers without permission. The software is provided ``as is'' with no warranty.



Issues with building a Distribution: Troubleshooting Page

Please see our wiki page on Troubleshooting ACC Code Issues



Getting Help

To request help, Please see our Help and Mailing List Information wiki page for email contacts.

Topic revision: r162 - 23 Sep 2021, DavidSagan



  • ACC/ACL Web

  • CLASSE Computing Info
  • CLASSE

Create personal sidebar

This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding CLASSE Wiki? Send feedback