Instant Plot: FMB SAXS/WAXS data viewing and reduction

Adding shortcut and path to .bashrc

If you are using the analysis computer at the beamline, you may skip this step. The follow steps apply only if you are logged in with your own CLASSE ID.

If you are logged in with your own CLASSE ID and would like to create a shortcut that places you in the appropriate environment for using InstantPlot, follow the steps below. This setup is only required once, as you are creating a personal shortcut for future use.

1. On NoMachine, connect to lnx201 and open a terminal window.
   
   
2. Make sure you are in your home directory. Running pwd should return /home/[your CLASSE ID]. If you are somewhere else, running cd ~ should return you to your home area.
   
   
3. Run code .bashrc to open .bashrc with VS code or open .bashrc in your favorite text editor. This should open a text editor window with the line "# .bashrc" at the top.
   
   
4. Below the section

   # Source global definitions
   if [ -f /etc/bashrc ]; then
   . /etc/bashrc
   fi

   
   Add the lines:

   ### InstantPlot ###

   alias activate_instantplot="source /nfs/chess/opt/miniconda3_msnc/bin/activate; conda activate saxswaxs-viewer"

   export PATH="/nfs/chess/user/chess_id3b/git_repos/saxswaxs-viewer:$PATH"

5. IMPORTANT: Save the file!
   
   
6. Close the text editor.
   
   

Setting up terminal for viewing data or for batch processing

1. On NoMachine, connect to lnx201 and open a terminal window.
   
   
2. Move to a CLASSE Compute Farm node.
   1. For almost all use cases: Running qrsh -q interactive.q should be sufficient for viewing data in InstantPlot
   2. Rare: If you will want to batch process in InstantPlot, run qrsh -q interactive.q -pe sge_pe 32 to access a node with parallel cores.
      
      
3. To make file finding easier, change directories to your analysis area by running cd /nfs/chess/aux/reduced_data/cycles/[YYYY-X]/id3b/[BTR-ID]/ *. YYYY-X is the year and cycle number, e.g. 2025-1 BTR-ID is the first eight charactes of PI's last name and project number, e.g. smieska-3222-A.
   
   
4. Enter the "saxswaxs-viewer" Python environment:
   1. On the analysis computer: Open a terminal window and run activate_instantplot.
   2. On a compute farm node:
      • If you have set up a shortcut for entering this environment using the instructions above, run activate_instantplot.
      • Otherwise, run source /nfs/chess/opt/miniconda3_msnc/bin/activate; conda activate saxswaxs-viewer.
        
        
5. The terminal window should now show the text (saxswaxs-viewer) to the left of the cursor prompt, indicating that the correct Python environment is now active.
   
   

InstantPlot: Required and optional inputs

• Required: PONI files created with the pyFAI calibration tool (saved as .poni) are necessary for viewing detector data in reciprocal space
• Optional but recommended: Mask files from pyFAI calibration tool (save as .tif)
• Optional: Background detector images are optional for viewing data in InstantPlot

InstantPlot viewing data

Option 1: Open InstantPlot without specifying config file

From your prepared terminal window (i.e., on a different node from lnx201 and with the saxs-waxs-viewer Python environment active), run InstantPlot.

The InstantPlot GUI should open. The default should be that the "Calibration, Background" tab on the upper right is selected, as seen below.

InstantPlot quirks

• To quit InstantPlot, click Quit button on the GUI, not the "x" button.
• The File Open dialog may appear behind the main GUI window. While this dialog is open, no other buttons or tabs in the GUI are accessible, which can make the interface appear unresponsive. To resolve this, locate the File Open dialog on the Workspace toolbar, and select it to bring it to the foreground.

Option 1A: Manual setup (or first time setup)

• REQUIRED: Under the "Calibration, Background" tab in the upper right, select the .poni file for each detector:
  1. Click the button corresponding to the detector for which .poni file is to be loaded (e.g. PIL5).
  2. Navigate to the directory where the .poni file is located. These files are typically in /nfs/chess/aux/reduced_data/cycles/[YYYY-X]/id3b/[BTR-ID]/setup.
  3. Select the desired .poni file and then click Open, or double-click the file to update the field for the detector.
  4. Repeat the steps above for the remaining detectors.
  5. Click Load PONI files button. The message below each field should say "Status: load successful".

• OPTIONAL BUT RECOMMENDED: Under the "Calibration, Background" tab in the upper right, select the mask file for each detector:
  1. Click the button corresponding to the detector for which .poni file is to be loaded (e.g. PIL5).
  2. Navigate to the directory where the .poni file is located. These files are typically in /nfs/chess/aux/reduced_data/cycles/[YYYY-X]/id3b/[BTR-ID]/setup.
  3. Select the desired .tif file and then click Open, or double-click the file to update the field for the detector.
  4. Repeat the steps above for the remaining detectors.
  5. Click Load detector mask files button. The message below each field should say "Status: load successful".

• OPTIONAL: Load background tiff snapshots (locate background snapshot .tiffs for each detector). Click one of the buttons labeled PIL5, PIL9, PIL11 and navigate to a file you want to use for a background image for that detector. An option to auto-populate the remaining detector images from the same scan should pop up, usually it's convenient to say yes. Click "load background files." NOTE: when batch processing data, the background images must come from a scan that is not in the list to be processed.

Click Save Config at the bottom right to save the inputs from the steps above. This will streamline the setup process the next time InstantPlot is launched with the same set of PONI files, detector masks, and, if applicable, background files (see Option 1B and Option 2 below). It is recommended that you save the config file as .txt in the same directory as the .poni files.

Once the above steps are completed, proceed to viewing 2D and 1D data.

Option 1B: Load an existing config file from inside InstantPlot

1. Click Clear Config file button to remove all entries for PONI files, mask files, and if applicable, background files.
2. Click Read Config button and navigate, navigate to the directory where the config file is located.
3. Select the desired config file and then click Open, or double-click the file to update the field for the config file.
4. Click Load Config file button to populate the fields for PONI files, mask files, and if applicalbe, background files.
5. IMPORTANT: Click Load PONI files, Load detector mask files, and if applicable, Load background files buttons. Loading the config file does NOT load PONI, detector mask or background files automatically. The message below each field should say "Status: load successful".

Option 2: Open InstantPlot with an existing config file specified

If a config file already exists, you can launch InstantPlot with the config file pre-loaded with the following steps, assuming the config file is in /nfs/chess/aux/reduced_data/cycles/[YYYY-X]/id3b/[BTR-ID]/setup:

1. cd /nfs/chess/aux/reduced_data/cycles/[YYYY-X]/id3b/[BTR-ID]/setup (or other location where the config file is located)
2. InstantPlot config_filename.txt

• IMPORTANT: Click Load PONI files, Load detector mask files, and if applicable, Load background files buttons. Loading the config file does NOT load PONI, detector mask or background files automatically. The message below each field should say "Status: load successful".

To view 2D data:

1. Click Select path (navigate to a scan directory) button.
2. Navigate to the directory containing detector images.
3. The directory may appear empty. This is expected. Click OK to continue.
   • The detector image path filed will be update to show the selected directory.
   • The panel under Detector data files will populated with list of detector files.
   • Selecting an entry from the list will update the image display in the Detector view tab.
4. The Plot 2D tab contains options for adjusting image display parameters. After making adjustments, click Update Plot button to refresh the images shown in Detector view tab.

To view 1D data:

1. Navigate to the 1D SAXS or 1D WAXS tab.
2. Select an entry in the panel with a list of detector data files.
3. Plot 1D tab contains options for adjusting plot parameters. After making adjustments, click Update Plot button to refresh the plot shown in 1D SAXS or 1D WAXS tab.

Batch integration

As of 2023-3, we recommend using the SAXSWAXSWorkFlow methods outlined here: https://wiki.classe.cornell.edu/CHESS/FMB/SAXSWAXSWorkFlow

Quitting InstantPlot

Click Quit button at the bottom-right corner. Clicking the "x" button at the top-right corner will cause an error (a known bug).

---

The following information is for reference only (last updated in early 2024).

Pre-2023-3 batch processing workflows

No longer recommended: Using InstantPlot to configure a custom batch (run with parallel_processing.py script)

Batch processing tools are available on the "batch processing" tab.

Open InstantPlot as normal, described above (> InstantPlot configfilename.txt)

Load your config file and make any desired adjustments in the Plot 1D views (q ranges, number of points, etc).

When you're happy with the integration, go to the Batch Processing tab and choose your options:

• check the boxes for integrating saxs, waxs, or both
• choose your plot type (I vs. q, I vs. chi, I vs. q vs. chi)
• choose your correction type (flux, absorption options)
  • We do not recommend using the "flux- absorption- and background-corrected data" option; we find better results subtracting the background manually after integrating the data
• choose to overwrite or skip existing output files
• check the box for "Traverse child directories" if you want to process multiple folders with this configuration
• check the box for "process in parallel" if you're running on a specialized computing node and you have mapping data
• click "Generate output path" and then edit if desired

When you're happy with all of the above, then click "Save config for parallel processing"

This should first show you a popup window where you can choose which scans to process with the selected conditions. You can select multiple scans by holding "CTRL." Click Continue.

Next, you will be asked to enter a filename to save the batch processing file. We recommend saving this file in your setup folder. You must add the extension .yaml to the filename.

In a separate terminal window from the one running InstantPlot, connect to a node with multiple cores and activate the saxs-waxs-viewer python environment:

• qrsh -q interactive.q -pe sge_pe 40
  • OR: qrsh -q interactive.q -pe sge_pe 32 (fewer cores but more of these nodes are available)
  • Note: if you're on the station analysis computer lnx-id3b-1, you'll need to first ssh to lnx201 as yourself
    • > ssh -Y your_CLASSE_ID@lnx201.classe.cornell.edu
• activate_instantplot
• cd to your setup area (/nfs/chess/auxiliary/reduced_data/cycles/YYYY-N/id3b/userid-BTR-ID/setup)

Now you can run the stand-alone parallel processing script, using the .yaml file you set up as an input argument:

• python parallel_processing.py mybatch.yaml

This should start the batch processing. It may not look like anything is happening for a moment, but then you should start to see reports of scans being integrated start rolling in.

If the parallel_processing.py script is not already present in the setup folder, ask one of the FMB scientists for help obtaining a copy.

No longer recommended: Using InstantPlot to run batch processing directly

Batch processing tools are available on the "batch processing" tab.

If you're running on the cluster, we recommend starting InstantPlot from a specialized node:

• In a new terminal window, hop to a higher computing power node with parallel processing environment invoked:
  • qrsh -q interactive.q -pe sge_pe 40
    • OR: qrsh -q interactive.q -pe sge_pe 32 (fewer cores but more of these nodes are available)
  • Note: if you're on the station analysis computer lnx-id3b-1, you'll need to first ssh to lnx201 as yourself
    • > ssh -Y your_CLASSE_ID@lnx201.classe.cornell.edu

Open InstantPlot as normal, described above (> InstantPlot configfilename.txt)

Load your config file and make any desired adjustments in the Plot 1D views (q ranges, number of points, etc).

When you're happy with the integration, go to the Batch Processing tab and choose your options:

• check the boxes for integrating saxs, waxs, or both
• choose your plot type (I vs. q, I vs. chi, I vs. q vs. chi)
• choose your correction type (flux, absorption, background correction options)
  • We do not redommend using the "flux- absorption- and background-corrected data" option; we find better results subtracting the background manually after integrating the data
• choose to overwrite or skip existing output files
• check the box for "Traverse child directories" if you want to process multiple folders with this configuration
• check the box for "process in parallel" if you're running on a specialized computing node and you have mapping data
• click "Generate output path" and then edit if desired

When you're happy with all of the above, then click "Begin Batch Processing"

• If you chose "Traverse child directories," you should now see a pop-up window asking you to select directories. Choose the ones you want to process with the current configuration and press "continue"
• There may be a delay before you see output in the terminal window where InstantPlot is running, but give it a minute or two and you should start seeing results being written to text files

-- LouisaSmieska - 9 Jan 2024