[this is a raw cut&paste of an email from jed]

Below is an attempt to list all the steps that need to occur for Pass2 in an attempt to get a more systematic approach in place, as well as examples, and a list of tools that have been useful to me.

I think I've got all the steps, but I am currently a bit confused about the move from p2-postprocess to physics-unchecked. The steps I have are a move from p2-unchecked to p2-postprocess, followed by injection of PP2 constants into physics-unchecked - is this correct, or is there a move step I have forgotten about? I've also tried to include words about what I find confusing at each step.

Anyway - Valentin, can you scan through this and see if I have forgotten any steps or have syntax trouble? I would guess that the physics-unchecked to physics move has an old syntax that needs updating.

I'll update the file once I get comments and then make it public somehow.

thanks jean

************************

Steps in talking to EventStore for Pass2

Below is a list of steps that are required to Pass2 to run and write to the eventstore. Included are examples of how these steps are accomplished. These examples are not meant to illustrate the full power of the GroupEventStoreToolkit, but rather the features I have used.

Note that all work is to be done on the master -mysql lnx151.lns.cornell.edu -esdb EventStore Note also that one can test on -mysql lnx151.lns.cornell.edu -esdb EventStoreTest EventStoreTest is synched everyday to the master and serves as a test bed to avoid screw ups. Once one is happy with the results, then the slave EventStore on lnx150 can be updated (by Valentine)

Separate steps are indicated by "******"

******************************** 0) Documenting what is done:

At each step the P2 czar should try to document what he is trying to accomplish (ie move runs in data36 from p2-unchecked to p2-postprocess, with a list of why certain runs might have been dropped.)

Injecting a note into the EventStore example: /home/pass2/p2/EventStore/moveGrades/data36_vs1/add_note_p2move_20050405.scr ${myGroupEventStoreToolkit}/addComment.py -grade p2-postprocess -timeStamp 20041207 -comment myComment.txt -date 20050405 -esdb EventStore -mysql lnx151.lns.cornell.edu

Reading notes in the EventStore example: /home/pass2/p2/EventStore/moveGrades/data36_vs1/add_note_p2move_20050405.log ${myGroupEventStoreToolkit}/getComment.py -date 20050101 20050413 -esdb EventSto re -mysql lnx151.lns.cornell.edu

************ 1) move raw data into the eventstore under grade daq-unchecked. Currently this is done automatically, no P2 czar intervention required.

************ 2) move data from daq-unchecked to daq

This requires manual intevervention by a person to be determined. Should the list of runs flagged as not worth considering by the CLEOSoft/RunMan/Bill Love be excluded at this stage ? If not then the move from daq-unchecked to daq can be done automatically by the same person who does the daq-unchecked injection.

The P2 Czar's work for this should be done in /home/pass2/p2/EventStore/moveGrades/

/home/pass2/p2/EventStore/moveGrades/data36_vs1/move_data36_from_p2-unchecked_to_p2-postprocess.scr (this example is for p2-postprocess NOT daq grade.) ${myToolKit}/modVersionDB.py -grade ${OLDGRADE} ${NEWGRADE} -timeStamp ${MYTIMESTAMPOLD} ${MYTIMESTAMPNEW} -mysql ${MyServer} -esdb ${MYEVENTSTOREDB} -runRange ${MYMINRUN} ${MYMAXRUN} -runList ${BADRUNLIST} -verbose

The env variables above that need to be supplied are: -grade ${OLDGRADE} ${NEWGRADE} Set OLDGRADE to daq-unchecked, NEWGRADE to daq -timeStamp ${MYTIMESTAMPOLD} ${MYTIMESTAMPNEW} For the daq move both MYTIMESTAMPOLD and MYTIMESTAMPNEW are 0 -runRange ${MYMINRUN} ${MYMAXRUN} This gives the first and last run in the full dataXY_vsZ dataset -runList ${BADRUNLIST} This is an ascii list of runs that should not be moved - one per line. Remember to note why these runs did not get moved.

need to add a -logDir argument here to send logfiles to one place. Should we create /nfs/objy/EventStoreDB/ESDBDir/daq/ for this ????

************ 3) run pass2 Pass2 does not write out directly to the EventStore although it gets its input from it. As an addendum to the above - currently the pass2 setup script is the one that decides which runs should or should not be P2ed by looking at the file /home/pass1/cleo-c/Luminosities/runlists/dataXY.runlist The script that does this is /home/pass2/p2/Pass2Scripts/parallelPass2/setup.pl It looks for runs with quality "1" in the fourth column of the dataXY.runlist file, requires in addition at least 1 K events for low energy runs and more than 20K events for high energy runs. This selection touches files on ~pass2/p2/dataXY_vsZ/runstate/todo - these files are the ones P2 looks at to know what to run on.

************ 4) inject Pass2 output pds files as p2-unchecked

Work for this should be done in /home/pass2/p2/EventStore/injection/pass2/

example: /home/pass2/p2/EventStore/injection/pass2/data36_vs1/inject_p2_data36_vs1_5.scr $MyESBuilder -mysql $MyServer -esdb $MyDB -output $MyOutputArea -time $MyTimeStamp -grade $MyGrade -dataVersionName $MydataVersionName -add /nfs/cleoc/data1/event/P2-20050224-20041104_P2/206359/206502/*_hot_*.pds

-output $MyOutputArea This is where the lindex files and such go - set it to /nfs/objy/EventStoreDB/ESDBDir/pass2/dataXY_vsZ/ -time $MyTimeStamp This is currently set to 20041207 - ie the latest "cycle" of pass2 done was on this date. How do we decide that we have started a new cycle and need to change this ? -grade $MyGrade Set this to p2-unchecked -dataVersionName $MydataVersionName This one is tough... Currently set to P2-20050225-20041104_P2-data36_vs1 (ie P2-(date P2 first ran on data36_vs1)_(Code release)-dataXY_vsZ ) -add /nfs/cleoc/data1/event/P2-20050224-20041104_P2/206359/206502/*_hot_*.pds This is the path to the hot store pds output for each run. When assembling these, make sure the file actually exists, and that there is exactly one for each run.

need to add a -logDir argument here to send logfiles to one place. We should use /nfs/objy/EventStoreDB/ESDBDir/pass2/dataXY_vsZ ?

Points of possible confusion: a) Timestamp to use b) dataVersionName to use

Note that one should inject all runs that completed running in pass2 just for accounting sanity. (What happens if some run is could not complete pass2 ?)

************ 5) inject Pass2 views (bhagam, 2photon, qcd) (in the form of idxa files) into p2-unchecked

Work happens in /home/pass2/p2/EventStore/injection/pass2/dataXY_vsZ

example: /home/pass2/p2/EventStore/injection/pass2/data36_vs1/move_to_2photon.scr ${myGroupEventStoreToolkit}/ESBuilder.py -output ${MYOUTPUT} -time ${MYTIMESTAMP} -grade ${MYGRADE} -view ${MYVIEW} -add $MYVIEWLIST -dataVersionName ${MYSVNAME} -esdb ${MYEVENTSTORE} -mysql lnx151.lns.cornell.edu -logDir ${MYLOGAREA}

-output ${MYOUTPUT} Set this to /nfs/objy/EventStoreDB/ESDBDir/pass2/dataXY_vsZ -time ${MYTIMESTAMP} use same TIMESTAMP as the p2-unchecked injection in step 5 -grade ${MYGRADE} set this to p2-unchecked -view ${MYVIEW} set this to either 2photon, qcd, or bhagam -add $MYVIEWLIST set this to a file that has on each line the path to the idxa file for this view, eg: /nfs/cleoc/data1/event/P2-20050224-20041104_P2/206359/206363/bhagam_skim_206363.idxa -dataVersionName ${MYSVNAME} same as the p2-unchecked version eg P2-20050225-20041104_P2-data36_vs1 -logDir ${MYLOGAREA} logfiles for failed work go to /nfs/objy/EventStoreDB/ESDBDir/pass2/dataXY_vsZ

Points of confusion: a) Should have one and only one file injected for each run that is in p2-unchecked

************ 6) move pass2 output from p2-unchecked to p2-postprocess work for this should be done in /home/pass2/p2/EventStore/moveGrades/

example: /home/pass2/p2/EventStore/injection/pass2/data36_vs1/move_data36_from_p2-unchecked_to_p2-postprocess.scr ${myToolKit}/modVersionDB.py -grade ${OLDGRADE} ${NEWGRADE} -timeStamp ${MYTIMESTAMPOLD} ${MYTIMESTAMPNEW} -mysql ${MyServer} -esdb ${MYEVENTSTOREDB} -runRange ${MYMINRUN} ${MYMAXRUN} -runList ${BADRUNLIST} -verbose

-grade ${OLDGRADE} ${NEWGRADE} Set these to p2-unchecked, p2-postprocess -timeStamp ${MYTIMESTAMPOLD} ${MYTIMESTAMPNEW} These are both set to the same TIMESTAMP as the original p2-unchecked time stamp (if for data36_vs1 20041207) -runRange ${MYMINRUN} ${MYMAXRUN} These are the lowest and highest run number in this move -runList ${BADRUNLIST} An ascii list of runs to exclude in this move

need to add a -logDir argument here to send logfiles to one place. We should use /nfs/objy/EventStoreDB/ESDBDir/pass2/dataXY_vsZ

Points of possible confusion: a) What timestamps to use b) Can we do partial run ranges without too much complication ? This should be rarely done.

************ 7) Run PostP2 jobs and create PP2 pds files These jobs do not write out to the EventStore directly. These jobs require that calibrators be able to look at the p2-unchecked output. Thus the slave lnx150 EventStore needs to be updated before this occurs. Currently calibrators must write out enough to get the following: {beginrun {CcRunGain CcHotList BeamSpot RunStats Luminosity {Luminosity GamGam} } } {startrun {BeamEnergy LabNet4Momentum MagneticField} } {event {FATable FATable FATable FATable} } endrun

a) CcRunGain, CcHotList (BKH) b) BeamSpot (Watkins) c) Runinfo (BKH) d) MagFudge * I think this is assumed in RunInfo (Batbold/BKH for runinfo) This constant is very constant and its variation can be safely neglected. e) Beam Energy corrections (DLK) f) dedx (Gocha)

Once all these are done and the relevant ascii files and constants DB are updated, scripts in Pass2Scripts/extract_p2postinfo/ can be used to create the PP2 output pds files. One thing to be careful of here is that the constants tag used for this must be ANALYSIS-C_5 - the postp2 constants tag does not have dedx built into it (!) and is used only for CC calibration.

************ 8) Inject PP2 pds files into physics-unchecked grade

work is done in /home/pass2/p2/EventStore/injection/p2postprocess/dataXY_vsZ

example: /home/pass2/p2/EventStore/injection/p2postprocess/data35_vs1/p2_data35_vs1_postp2_inject_1.scr ${MyESBuilder} -add /cdat/sol514/disk3/pass2/data35_vs1/run205136/p2post_205136_vs2.pds -grade ${MyGrade} -time ${MyTimeStamp} -dataVersionName ${MysvName} -listOfParents ${MyParentName} -output ${MyOutputArea} -esdb ${MyDB} -mysql ${MyServer} -no-key

-add /cdat/sol514/disk3/pass2/data35_vs1/run205136/p2post_205136_vs2.pds This is the PP2 pds file you are adding (one for each run) -grade ${MyGrade} physics-unchecked -time ${MyTimeStamp} for data35_vs we set this to 20050324 - this was used because we had screwed up in our first round, and need to create a new time stamp. (This new time stamp forced us to move all other datasets to this new time stamp as well.) -dataVersionName ${MysvName} Up to now we have used P2-20050320-20050120_FULL-data35_vs1-PP2 - ie the same datasetversion name as the injection of the pds files into p2-unchecked in step 4 with a "-PP2" added on. For data35 since we tried to fix a problem we did a second injection with a "-PP2-EBeam" -listOfParents ${MyParentName} This should be the dataVersionName of p2-unchecked stuff we are trying to associate these PP2 files with (ie what was used in step 4) -output ${MyOutputArea} /nfs/objy/EventStoreDB/ESDBDir/p2postprocess/dataXY_vsZ

need to add a -logDir argument here to send logfiles to one place. We should use /nfs/objy/EventStoreDB/ESDBDir/p2postprocess/dataXY_vsZ

************ 9) move runs from physics-unchecked to physics

** The following command may be wrong as it comes from an old file I was using *******

${myToolKit}/modVersionDB.py -grade ${OLDGRADE} ${NEWGRADE} -timeStamp ${MYTIMESTAMPOLD} ${MYTIMESTAMPNEW} -mysql lnx151.lns.cornell.edu -esdb ${MYEVENTSTOREDB} -runRange ${MYMINRUN} ${MYMAXRUN} -runList ${BADRUNLIST} -verbose -grade ${OLDGRADE} ${NEWGRADE} set to physics-unchecked, and physics -timeStamp ${MYTIMESTAMPOLD} ${MYTIMESTAMPNEW} Again we should use the time stamps appropriate for this cycle of pass2 -runRange ${MYMINRUN} ${MYMAXRUN} lowest and highest run numbers in this dataset -runList ${BADRUNLIST} A list of runs to exclude in this full runrange

Points of confusion: a) TIMESTAMP b) is listofParents needed here?

need to add a -logDir argument here to send logfiles to one place. We should use /nfs/objy/EventStoreDB/ESDBDir/p2postprocess/dataXY_vsZ ????

************ 10) release to users

Note that dtag are done after this by someone else. Also, some users might still be using constants that we might wish to integrate into the eventstore - things like electron ID - but these are only available after data gets into physics grade at this point.

************

****** Useful tools: I am attempting to keep some useful scripts in /home/pass2/p2/EventStore/myTools. Among them are: getSuezRunList_official_lnx150.scr getSuezRunList_official_lnx151.scr If you supply a grade, this script will dump out a list of runs available as of today in the evenstore

myESDumpOfficial_lnx150.scr myESDumpOfficial_lnx151.scr myESDumpTest_lnx151.scr These script do a summary dump of the EventStore contents for today's date.

myFileContent.scr Writes out the content of a given input pds file.

myesdump_backup.scr Allows us to make a local backup of the eventstore before we do anything else. Requires a username and password.

-- DanRiley - 22 Apr 2005