Instant Plot: FMB SAXS/WAXS data viewing and reduction

Before you start- add path to .bashrc

You only need to do this once- you're setting up a shortcut for yourself.

1. Connect to NoMachine and open a terminal window.
2. Make sure you're in your home directory: "pwd" should return "/home/your_userid"
   1. if you're somewhere else, typing "cd ~" should return you to your home area
3. Enter > pluma .bashrc (or open .bashrc in your favorite text editor).
   1. This should open a text editor window withe the line "# .bashrc" at the top
4. Below the section

   # Source global definitions
   if [ -f /etc/bashrc ]; then
   . /etc/bashrc
   fi

   
   Add the lines:
   

   ### InstantPlot ###

   alias activate_instantplot="source /nfs/chess/opt/miniconda3_msnc/bin/activate; conda activate saxswaxs-viewer"

   export PATH="/nfs/chess/user/chess_id3b/git_repos/saxswaxs-viewer:$PATH"

5. Save the file and close the text editor.

Set up terminal for viewing data or for batch processing

• Connect to NoMachine, lnx201.classe.cornell.edu

• Open a terminal window

• In the terminal window, you need to run a qrsh command to move from lnx201 to another node on the cluster:

  • For viewing data only, no batch processing, this command is enough: qrsh -q interactive.q

  • If you will want to batch process, choose a node with parallel cores by running: qrsh -q interactive.q -pe sge_pe 32

• To make file finding easier, change directories to your analysis area

  • cd /nfs/chess/auxiliary/reduced_data/cycles/YYYY-#/id3b/username-id/
  • where YYYY-# is the year and cycle number, e.g. 2022-2
  • username-id is PI name and project number, e.g. smieska-3222-A
• Activate the python environment you set up in the .bashrc by entering:
• • activate_instantplot (or whichever name you chose when you set up your shortcut)
• The terminal window should now show the text "(saxs-waxs-viewer)" to the left of the cursor prompt, indicating that the correct python environment is now active.

InstantPlot: Required and optional inputs

• Required: PONI files created with the pyFAI calibration tool are necessary for viewing detector data in reciprocal space
• Optional but recommended: Mask files from pyFAI calibration tool (save as .tif)
• Optional: Background detector images are optional for viewing data in InstantPlot

InstantPlot viewing data

Option 1: Open InstantPlot without specifying config file

From your prepared terminal window (i.e., on a different node from lnx201 and with the saxs-waxs-viewer python environment active), enter: InstantPlot

The InstantPlot gui should open. The default should be that the "Calibration, Background" tab on the upper right is selected.

Option 1A: Manual setup (or first time setup)

• REQUIRED: Under the "Calibration, Background" tab on the upper right, select the poni files for each detector: Click the buttons labeled PIL5, PIL9, PIL11 to locate poni files for each detector. These are usually saved in the "setup" folder for your beamtime. Highlight the needed .poni file and "open," or double click the needed file. When all paths are updated, click "load PONI files."

• OPTIONAL: Load background tiff snapshots (locate background snapshot .tiffs for each detector). Click one of the buttons labeled PIL5, PIL9, PIL11 and navigate to a file you want to use for a background image for that detector. An option to auto-populate the remaining detector images from the same scan should pop up, usually it's convenient to say yes. Click "load background files." NOTE: when batch processing data, the background images must come from a scan that is not in the list to be processed.

• OPTIONAL BUT RECOMMENDED: Using the pyfai calibration tool, save mask files (.tiff or .tif) for each of the three detectors. Click the buttons labeled PIL5, PIL9, PIL11 to locate each of the mask files. Click "load mask files."

Proceed to viewing 2D and 1D data.

Use the “save config” button at the bottom right to save the selected poni, background, and mask files in a text file in your setup area when finished, to make viewing data faster the next time you open InstantPlot (use option 1B or 2).

Option 1B: Load an existing config file from inside InstantPlot

• Press “clear config file”

• Press “read config” and navigate to your config file (e.g., /nfs/chess/aux/reduced_data/cycles/2021-3/id3b/[username-BTR-ID]/setup)

• Press “load config file”

• Press “load PONI files”

• Press “load background files” if desired

• Press "load mask files" if desired

Option 2: Open InstantPlot with an existing config file specified

If you have a config file saved already, you can call it and load it when you open InstantPlot:

> InstantPlot configfilename.txt

• Either first change directories to your "setup" area, where your config files are stored, or specify the full file path to configfilename.txt

To view 2D data:

• Press “select path” at bottom left and navigate to scan directory on /nfs/chess/raw/[YYYY-X]/id3b/[username-BTR-ID]/
  • YYYY-X = Year-cycle number, e.g. 2021-3 or 2022-2, etc.
  • You need to open the actual scan folder (e.g. scanname_001/ ) by double clicking on it. It will look empty. Click "OK." This should populate the “tiff files” window for you to view the individual snapshots
  • If you click somewhere outside the "Choose directory" popup window, it can get lost and InstantPlot can appear frozen- check the bar at the top of the window to locate the popup window again.
• “Plot2D” tab on the right has options for changing the data display here

To view 1D data:

• Choose 1D SAXS or 1D WAXS tabs at top left. “Plot 1D” tab at top right has options for viewing these plots.
• Good to know: if you change the q/chi ranges in the Plot 1D area, you can see the result (what will be integrated) on the reciprocal space SAXS/WAXS tabs at top left.

Batch processing: use InstantPlot to configure a custom batch (run with parallel_processing.py script)

Batch processing tools are available on the "batch processing" tab.

Open InstantPlot as normal, described above (> InstantPlot configfilename.txt)

Load your config file and make any desired adjustments in the Plot 1D views (q ranges, number of points, etc).

When you're happy with the integration, go to the Batch Processing tab and choose your options:

• check the boxes for integrating saxs, waxs, or both
• choose your plot type (I vs. q, I vs. chi, I vs. q vs. chi)
• choose your correction type (flux, absorption options)
  • We do not redommend using the "flux- absorption- and background-corrected data" option; we find better results subtracting the background manually after integrating the data
• choose to overwrite or skip existing output files
• check the box for "Traverse child directories" if you want to process multiple folders with this configuration
• check the box for "process in parallel" if you're running on a specialized computing node and you have mapping data
• click "Generate output path" and then edit if desired

When you're happy with all of the above, then click "Save config for parallel processing"

This should first show you a popup window where you can choose which scans to process with the selected conditions. You can select multiple scans by holding "CTRL." Click Continue.

Next, you will be asked to enter a filename to save the batch processing file. We recommend saving this file in your setup folder. You must add the extension .yaml to the filename.

In a separate terminal window from the one running InstantPlot, connect to a node with multiple cores and activate the saxs-waxs-viewer python environment:

• qrsh -q interactive.q -pe sge_pe 40
  • OR: qrsh -q interactive.q -pe sge_pe 32 (fewer cores but more of these nodes are available)
  • Note: if you're on the station analysis computer lnx-id3b-1, you'll need to first ssh to lnx201 as yourself
    • > ssh -Y your_CLASSE_ID@lnx201.classe.cornell.edu
• activate_instantplot
• cd to your setup area (/nfs/chess/auxiliary/reduced_data/cycles/YYYY-N/id3b/userid-BTR-ID/setup)

Now you can run the stand-alone parallel processing script, using the .yaml file you set up as an input argument:

• python parallel_processing.py mybatch.yaml

This should start the batch processing. It may not look like anything is happening for a moment, but then you should start to see reports of scans being integrated start rolling in.

If the parallel_processing.py script is not already present in the setup folder, ask one of the FMB scientists for help obtaining a copy.

No longer recommended: Using InstantPlot to run batch processing directly

Batch processing tools are available on the "batch processing" tab.

If you're running on the cluster, we recommend starting InstantPlot from a specialized node:

• In a new terminal window, hop to a higher computing power node with parallel processing environment invoked:
  • qrsh -q interactive.q -pe sge_pe 40
    • OR: qrsh -q interactive.q -pe sge_pe 32 (fewer cores but more of these nodes are available)
  • Note: if you're on the station analysis computer lnx-id3b-1, you'll need to first ssh to lnx201 as yourself
    • > ssh -Y your_CLASSE_ID@lnx201.classe.cornell.edu

Open InstantPlot as normal, described above (> InstantPlot configfilename.txt)

Load your config file and make any desired adjustments in the Plot 1D views (q ranges, number of points, etc).

When you're happy with the integration, go to the Batch Processing tab and choose your options:

• check the boxes for integrating saxs, waxs, or both
• choose your plot type (I vs. q, I vs. chi, I vs. q vs. chi)
• choose your correction type (flux, absorption, background correction options)
  • We do not redommend using the "flux- absorption- and background-corrected data" option; we find better results subtracting the background manually after integrating the data
• choose to overwrite or skip existing output files
• check the box for "Traverse child directories" if you want to process multiple folders with this configuration
• check the box for "process in parallel" if you're running on a specialized computing node and you have mapping data
• click "Generate output path" and then edit if desired

When you're happy with all of the above, then click "Begin Batch Processing"

• If you chose "Traverse child directories," you should now see a pop-up window asking you to select directories. Choose the ones you want to process with the current configuration and press "continue"
• There may be a delay before you see output in the terminal window where InstantPlot is running, but give it a minute or two and you should start seeing results being written to text files

-- LouisaSmieska - 21 Feb 2023