Tags

Event Store graph and injection policies

This document describes the Event Store injection policies and provides outline of existing Event Store tables. The graph snapshot of data collected by Event Store is described by a concrete example of various injection steps.

Event Store tables:

Here we represent a list of tables used by Event Store. The bold names are primary keys in those tables and italic names represents the group index (for faster lookup).

FileID
fileId fileName typeId

Here fileId is an unique file identifier in EventStore associated with every stored file.

KeyFile
graphId view run uid keyFileId

Here graphId is an id assigned to the graph; view represents a user skim (e.g. qcd); uid is unique id of underlying file(data), e.g. for real data it is alwasy 1, for MC it's unique id. This uid is a part of syncValue=(run#,event#,uid). keyFileId is a key file id.
RunUID
run uid

Location
id graphId run uid locationFileId

id represents the internal unique identifier which allows store multiple entries for the same (graphId,run,uid) group index. locationFileId is a location file id.
FileType
id type description

id represents the internal unique identifier, type is a file type (e.g. pds, ikey, lpds, etc.)
OrphanFile
id dateTime user

SpecificVersion
svName svid

svName stands for Specific Version Name (or data version name) and svid its id.
GraphPath
graphId svid

PathDepend
parentId childId

both parentId and childId are svid's, but show a dependencies among the graph.
Version
id grade timeStamp minRunNumber maxRunNumber graphId state

Available grades

Available grades in Event Store
write read
daq-unchecked daq
p2-unchecked p2-postprocess
physics-unchecked physics

Injection steps

The Greek letters will be used to describe graphId's and roman letters stand for paths within a graph. A graph represents a complete set of data visible to users from a client application. As an example we will start injection process by injecting a raw data whose parentId=0, timeStamp=0.

Injection I

The first injection attaches data to the daq graph. The injection step affects the following tables: "Version", "GraphPath", "PathDepend", "KeyFile" and "Location" and it starts by updating the "Version" table. In order to add a new graphId, the timeStamp and grade need to be unique or can be the same for a disabled state. For demonstration purposes, we add data which are represented by a path ("0" in a graph represents the parent node and "a" the child node). This path becomes the first available graph (α). At this step, new key and location files are created. As will be shown further, the keyFileId is an independent quantity. The locationFileId's would be related to the graphId.

Version
1 grade1 timeStamp1 minR maxR α active
GraphPath
α a
PathDepend
a NULL
KeyFile
α view run uid key
Location
id1 α run uid La

A concrete example for this type of graph would be: "0" represents raw data assigned to daq grade, "a" represents one pass2 dataset which was injected as "p2-unchecked" grade and "α" is the new graphId.

Injection II

Now we inject a new piece of data represented by node (b). We would like to use this data in conjunction with the original data represented by node (a). To do that we disable the original graphId=α and inject our data using the new graphId=β. After that, our data snapshot looks like

where β represents a full graph.

Version
1 grade1 timeStamp1 minR maxR α disable
2 grade1 timeStamp1 minR maxR β active
GraphPath
α a
β a
β b
PathDepend
a NULL
b a
KeyFile
α view run uid key
β view run uid key
Location
id1 α run uid La
id2 β run uid La
id2 β run uid Lb

Example: "0" represents data from daq grade, "a" represents pass2 data assigned to "p2-unchecked" grade, "b" represents a post-pass2 correction injected as "physics-unchecked" grade and "β" is the new graphId.

Injection III

Let's assume we made a mistake and found that data (b) need to be replaced by data (b'). To reflect this change we choose another timeStamp.

Version
1 grade1 timeStamp1 minR maxR α disable
2 grade1 timeStamp1 minR maxR β active
3 grade1 timeStamp2 minR maxR γ active
GraphPath
α a
β a
β b
γ a
γ b'
PathDepend
a NULL
b a
b' a
KeyFile
α view run uid key
β view run uid key
γ view run uid key
Location
id1 α run uid La
id2 β run uid La
id2 β run uid Lb
id3 γ run uid La
id3 γ run uid Lb'

Example: "0" represents data from daq grade, "a" represents pass2 data assigned to "p2-unchecked" grade, "b" represents a post-pass2 correction assigned to "physics-unchecked" grade with "PP2-dataXX-version1" specificVersionName, "b'" is post-pass2 data assigned to "physics-unchecked" grade with "PP2-dataXX-version2" specificVersionName and "γ" is the new graphId.

Injection IV

Once new data (c) are available we would like to assign them to the same timeStamp as the original processed data (a). To do that we choose another grade. The new graph (δ) represent (0,a,c) nodes.

Version
1 grade1 timeStamp1 minR maxR α disable
2 grade1 timeStamp1 minR maxR β active
3 grade1 timeStamp2 minR maxR γ disable
4 grade2 timeStamp1 minR maxR δ active
GraphPath
α a
β a
β b
γ a
γ b'
δ a
δ c
PathDepend
a NULL
b a
b' a
c a
KeyFile
α view run uid key
β view run uid key
γ view run uid key
δ view run uid key
Location
id1 α run uid La
id2 β run uid La
id2 β run uid Lb
id3 γ run uid La
id3 γ run uid Lb'
id4 δ run uid La
id4 δ run uid Lb

Example: "0" represents data from daq grade, "0" represents pass2 data assigned to p2-unchecked grade, "b" is post-p2 data assigned to physics-unchecked grade with "PP2-dataXX-version1" specificVersionName, "b'" is post-p2 data assigned to physics-unchecked grade with "PP2-dataXX-version2" specificVersionName, "c" is another post-p2 data assigned to physics-unchecked grade with "PP2-dataYY-version7" specificVersionName and "δ" is some a new graphId.

Injection V

Finally we decided to combine data (0), (a), (b') and (c) into unique graph (ε). Since data (b') was injected using timeStamp2, we assign new graph with timeStamp2.

Version
1 grade1 timeStamp1 minR maxR α disable
2 grade1 timeStamp1 minR maxR β active
3 grade1 timeStamp2 minR maxR γ disable
4 grade2 timeStamp1 minR maxR δ active
5 grade1 timeStamp2 minR maxR ε active
GraphPath
α a
β a
β b
γ a
γ b'
δ a
δ c
ε a
ε b'
ε c
PathDepend
a NULL
b a
b' a
c a
KeyFile
α view run uid key
β view run uid key
γ view run uid key
δ view run uid key
ε view run uid key
Location
id1 α run uid La
id2 β run uid La
id2 β run uid Lb
id3 γ run uid La
id3 γ run uid Lb'
id4 δ run uid La
id4 δ run uid Lb
id5 ε run uid La
id5 ε run uid Lb'
id5 ε run uid Lc

Example: "0" represents data from daq grade, "0" represents pass2 data assigned to p2-unchecked grade, "b" is post-p2 data assigned to physics-unchecked grade with "PP2-dataXX-version1" specificVersionName, "b'" is post-p2 data assigned to physics-unchecked grade with "PP2-dataXX-version2" specificVersionName, "c" is another post-p2 data assigned to physics-unchecked grade with "PP2-dataYY-version7" specificVersionName and "ε" is a new graphId which declared to be usefull for physics analysis.

-- ValentinKuznetsov - 14 Feb 2005

  • version_graph1:
    version graph1

  • version_graph2:
    version graph2

  • version_graph3:
    version graph3

  • version_graph4:
    version graph4

  • version_graph5:
    version graph5
Topic revision: r5 - 20 Jul 2007, GregorySharp
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding CLASSE Wiki? Send feedback