SAXS/WAXS workflow tools

These instructions apply to the SAXS/WAXS workflow tools released in 2023.

Official documentation here: https://gitlab01.classe.cornell.edu/msn-c/saxswaxsworkflow/-/wikis/home

Before you start

1. Make sure you can access lnx201 on NoMachine
2. Make sure you have the saxswaxsworkflow environment installed in your Jupyter environment

Instructions for these steps are found on the RemoteComputingFMB page.

1. Make sure you can view data in the InstantPlot gui

To validate and run a workflow on a single .yaml file

1. Using a template provided by FMB staff, and the interactive InstantPlot gui, set up a .yaml file in a text editor window defining the integrations and corrections you want to perform. Save the .yaml file in your aux/reduced_data/.../analysis area.
   • Viewing InstantPlot is still easiest on NoMachine, but the text file may be edited on JupyterHub.
   • Checklist of fields you should look at:
     • "title" and "name" are yours to choose, but similar to the spec file makes sense
     • "station" is id3b and "experiment type" is SAXSWAXS
     • Make sure the spec file path is correct
     • Check the scan number. You can choose more than one value for scan_numbers if you have multiple scans to run with the same configuration in the same spec file, for example: [1, 3, 5, 7]
     • Check the independent dimensions- typically samx and samz for a 2D map, either samx or samz for a 1D linescan
     • Check the "name" under "presample_intensity"
       • Typically ic3 for microbeam work, and ic1 for bulk beam work
2. Open a terminal window. If on lnx201, connect to a higher powered computing node (32 cores).
   • Connect from lnx201 to a parallel processing node: qrsh -q interactive.q -pe sge_pe 32
3. Activate the saxswaxsworkflow environment
   • source /nfs/chess/sw/miniconda3_msnc/bin/activate; conda activate saxswaxsworkflow
4. Change directory to your analysis area.
   • For example: cd /nfs/chess/auxiliary/reduced_data/cycles/2023-2/id3b/stebner-3723-A/analysis/
5. Run the following:
   • saxswaxs validate workflow your_configuration.yaml
   • If the configuration .yaml file has no errors, it will be reported as a valid workflow object. If there are errors, fix them in the .yaml file, save, and try again.
6. When the workflow is valid and you're ready to run it:
   • saxswaxs convert your_configuration.yaml nexus_out/output_filename.nxs
   • (you're not required to put your output .nxs file in a subfolder, but we find it useful)

These steps can also be performed on JupyterHub, if you choose 32 core queue or the FMB queue.

Batch saxswaxs workflow

1. Using a template provided by FMB staff, and the interactive InstantPlot gui, set up "map" .yaml files for each scan, and "tool" .yaml files for each type of integration or correction you want to perform. Save the .yaml files in your aux/reduced_data/.../analysis area.
   • It is helpful if you put a unique string in the map filenames that will use the same workflow tools so you can call them as a group with * in the batch line
2. Open a terminal window. If on lnx201, connect to a higher powered computing node (32 cores).
   • Connect from lnx201 to a parallel processing node: qrsh -q interactive.q -pe sge_pe 32
3. Activate the saxswaxsworkflow environment
   • source /nfs/chess/sw/miniconda3_msnc/bin/activate; conda activate saxswaxsworkflow
4. Change directory to your analysis area.
   • For example: cd /nfs/chess/auxiliary/reduced_data/cycles/2023-2/id3b/stebner-3723-A/analysis/
5. Run the saxswaxs batch command using the following structure, entering your map names and tool names (multiple allowed):
   • saxswaxs batch -m mapname.yaml -t tool1.yaml -o output_directory/
   • output nexus files will be named automatically and saved to your output area

usage: saxswaxs batch [-h] -m map.yaml [map.yaml ...] -t tool.yaml [tool.yaml ...] [-i INPUTDIR] [-o OUTPUTDIR] [-f]
                      [-l {CRITICAL,FATAL,ERROR,WARN,WARNING,INFO,DEBUG,NOTSET}]

options:
  -h, --help            show this help message and exit
  -m map.yaml [map.yaml ...], --map_config_files map.yaml [map.yaml ...]
                        List of .yaml files containing map configurations to which all the tools should be applied
  -t tool.yaml [tool.yaml ...], --tool_config_files tool.yaml [tool.yaml ...]
                        List of .yaml files containing tool configurations to apply to the map configurations provided.
  -i INPUTDIR, --inputdir INPUTDIR
                        Path to a directory in which all map and tool configuration files are located.
  -o OUTPUTDIR, --outputdir OUTPUTDIR
                        Path to a directory in which all resulting workflow NeXus files will be placed
  -f, --force_overwrite
                        Use this flag to overwrite the output file if it already exists
  -l {CRITICAL,FATAL,ERROR,WARN,WARNING,INFO,DEBUG,NOTSET}, --log {CRITICAL,FATAL,ERROR,WARN,WARNING,INFO,DEBUG,NOTSET}
                        Specify a preferred logging level.

Viewing the nexus file with nexpy

1. On NoMachine, open a terminal window and connect to a node that is not lnx201.
2. Activate the saxswaxsworkflow environment
   • source /nfs/chess/sw/miniconda3_msnc/bin/activate; conda activate saxswaxsworkflow
3. Change directory to your analysis area.
   • For example: cd /nfs/chess/auxiliary/reduced_data/cycles/2023-2/id3b/stebner-3723-A/analysis/
4. Run: nexpy
5. In the gui that opens, go to File > Open, select a .nxs file from your nexus_out area
6. Unpack the nexus file with the arrow drop downs and double click any of the "data" arrays to view in the central window.

-- LouisaSmieska - 29 Jun 2023