Wire-Cell Toolkit Configuration
1 Outline
We will cover these aspects of Wire-Cell Toolkit (WCT) configuration:
- Some intro
- C++ layers
- Jsonnet
- Current overall structure
- Development tips
- Future plans
2 Configuration requirements (why configure?)
While not re-compiling we wish to:
- Define initial application state precisely and stably.
- Compose sets of components into different applications.
- Provide variety value for a parameter across multiple applications.
- Share a parameter value with multiple components.
- Set parameters independently across the components.
- Independence between code and configuration representations.
- Manage configuration complexity.
- Seek brevity, simplicity, clarity.
Yes, there are some seemingly self-contradictions….
3 High-level, Layered Configuration Model
| configurable component |
|---|
| application of configuration |
| transient representation (object) |
| representation conversion |
| persistent representation (files) |
| sources of configuration information |
- objects
- a JSON object model (currently JsonCPP)
- files
- JSON (plain text or compressed) or Jsonnet
Only the object model is exposed to configurable components.
4 Two same but different categories of files
- data
- larger files, typically generated and compressed
-
.json.bz2from programs inwire-cell-python. - Official versions in
wire-cell-data. - eg: field response, wire geometry, noise spectra
-
- cfg
- smaller files, typically hand-made plain-text
-
.jsonnetfrom human developers and users - Official versions in
wire-cell-toolkit/cfg. - eg: component configuration
-
The distinction is somewhat artificial.
- One may provide
.jsonnet“data” files or.json.bz2“cfg” files.- latter may include pre-compiling
.jsonnetto.json.bz2
- latter may include pre-compiling
- An algorithmic “wire geometry” in a
.jsonnetfile is a reasonable thing.
But, for the rest of the slides, we focus mostly on “cfg” type.
5
C++ layers
6 WCT Main Configuration Sequence
WireCell::Main class expects an array of objects like:
[ // Jsonnet language {type:"A", name:"a", data: {...}}, {type:"B", name:"b", data: {...}}, // ... {type:"Z", name:"z", data: {...}}, ]
- One object per component instance
- Identify component type and instance name
- The data is a type-specific data structure.
7 One example component configuration object
{ type: "Drifter",
data: {
DL: 7.2 * wc.cm2/wc.s,
DT: 12.0 * wc.cm2/wc.s,
lifetime: 8 * wc.ms,
drift_speed: 1.6 * wc.mm/wc.us,
fluctuate: true, } }
The name is optional, default is "".
Will talk more about where wc comes from later.
But how do we know what to provide for data???
- Frankly, for now one must read the component’s C++ source….
- Will give some “tips” to reduce this pain.
- Future development plans will improve this issue.
8
Continuing with C++ layer….
9 WireCell::Persist namespace
Low-level object/file interface is provided by:
Functions for:
- Locating configuration files.
- Uses WCT’s only env var:
WIRECELL_PATH
- Uses WCT’s only env var:
- Loading file or string to product an object.
- and vice-versa.
- Provides a C++ interface to C or GO Jsonnet libraries.
- May pass Jsonnet “top level arguments” and “external variables”.
- Component developers usually need not encounter this level.
- A new type of “data” file will need a new component that does.
10 WireCell::Configuration namespace
Generally used throughout all components:
Mainly provides a thin:
typedef Json::Value Configuration;
This exposes the JsonCPP type to components.
- dynamic object so one must always inspect carefully.
- helpful functions to prefer to using JsonCPP methods directly:
// Convert to C++ type T convert<T>(const Configuration& cfg); // Get object attribute as C++ type T get<T>(const Configuration& cfg, const std::string& name); x = "foo"
JSONCPP will be replaced in WCT with “Modern JSON” (aka
nlohmann::json) which will require refactoring of all components!
11 WCT IConfigurable
class MyFilter : public virtual IConfigurable { public: // IConfigurable virtual void configure(const WireCell::Configuration& cfg); virtual WireCell::Configuration default_configuration() const; private: int m_a{42}; // a "sane" default float m_b{0}; // always initialize }
WCT will:
- Call
default_configuration()to get an initial object, treating it as opaque other than it is a JSON “object” (not scalar nor array). - Merge in to it the
data: {...}attribute. - Provide result to
configure()
12 Implementing IConfigurable
Current best practice: “filter” class data members through configuration.
WireCell::Configuration MyFilter::default_configuration() { WireCell::Configuration cfg; cfg["a"] = m_a; cfg["b"] = m_b; return cfg; }
void configure(const WireCell::Configuration& cfg) { m_a = get<int>(cfg, "a", m_a); m_b = get<float>(cfg, "b", m_b); }
- Can do error checking and throw exceptions here, early before any processing occurs.
- Currently, we must read this kind of code to discover
what to provide in
data:{...}.
13 Implementation issues
- A lot of code is devoted to these two methods.
- It tends to enact the same pattern across all components.
- Patterns can be abstracted and (meta)parameterized by schema.
- At end, will talk about plans to enact this to make life easier for developers (and users).
14
Jsonnet
15 Jsonnet data templating language.
15.1 :Bcolumns:
15.1.1 :BMCOL:
Pure functional programming language
- No side effects, lazy evaluation
- LISP/Scheme-like standard library
Compiles to JSON text by default
- or YAML, INI, Python (
dict), XML(svg), TOML - Any text format can developed in Jsonnet code.
Very simple, familiar syntax
- one of very few languages where (sometimes) a file “works” on first try
- but, side-effect-free functional programming is an unfamiliar new paradigm for many
15.1.2 :BMCOL:
From https://jsonnet.org/
Plus comments, trailing commas, quote-free object keys.
16 Hello Jsonnet command line
Expression strings
jsonnet -e '{hello:"world",} // quick CLI test'
{
"hello": "world"
}
Source files
echo '2 + 2' > sum.jsonnet jsonnet sum.jsonnet
4
17 Jsonnet Types
- Null
- only one value:
null, equal only to itself - Boolean
- two values
trueandfalse - Number
3,-0.1000001, 64 bit IEEE754 FP.- exact integers \(\in [-2^{53},2^{53}]\)
- String
- Unicode codepoints
- may use
'single quotes',"or double" - may index:
s[2]but strings are not arrays
- may use
- Array
- Finite sequence, possibly with heterogeneous element types
-
[1,2],[4, "you", [1, ["nest"]]]
-
- Object
- Collection of pairs: key of string, value of any type.
-
{a:1, b:2, c:{greet: "hi!"}} - ordered by key
{b:2, a:1}→{"a":1, "b":2}
-
- Function
- A calculation on arguments and environment
-
{a::1, f::function() self.a, b:self.f()}→ {“b”:1} - A function is a first-class instance.
-
18 String operations
Interpolation, (in)equalities, indexing, slicing:
local a = "%0.5f"%(1/19); local cat = "meow"; {a: a, c: cat[0:2], equal: a == cat, isarray: cat == ["m","e","o","w"], lexical_less_than: cat < "dog"}
{
"a": "0.05263",
"c": "me",
"equal": false,
"isarray": false,
"lexical_less_than": false
}
- Also many string functions in
std. - Will more fully introduce
locallater.
19 Array operations
19.1 :Bcolumns:
19.1.1 :BMCOL:
Length, indexing, slicing, appending, concatenating.
- All similar to what one expects from Python.
- Many more Array functions under
std.
19.1.2 :BMCOL:
local a = [1,2,3]; local b = [40,50,60]; { len: std.length(a), index: b[0], append: a + [4], concat: a + b, slice: self.concat[3:5], }
{
"append": [
1,
2,
3,
4
],
"concat": [
1,
2,
3,
40,
50,
60
],
"index": 40,
"len": 3,
"slice": [
40,
50
]
}
20 Jsonnet local keyword
Defines file-local value which may be directly referenced.
local x = 42; local y = x + 1; // semicolon at top level { local z = y - x, // comma inside object result: z }
{
"result": 1
}
Use = to assign local variable, use : with object attributes.
21 Relative referencing
{
a: $.sub.c, // reference from top of structure
b: self.sub.c, // reference from current context
sub: {
c: self.d,
d: "all the same",
}
}
{
"a": "all the same",
"b": "all the same",
"sub": {
"c": "all the same",
"d": "all the same"
}
}
Lazy evaluation means reference can precede definition.
22 Self-referencing in object context
Via local
local obj = { name: "Alice", lower: std.asciiLower(self.name), upper: std.asciiUpper($.name), greeting: "Hello, " + obj.name }; obj
{
"greeting": "Hello, Alice",
"lower": "alice",
"name": "Alice",
"upper": "ALICE"
}
23 File referencing
{ // a.jsonnet
x = 1
}
// b.jsonnet local myvar = import "a.jsonnet"; { y = myvar.x }
- The
b.jsonnetcompiles to{"y":1} - The file name to
importmust be a literal string, not a computed value.- In part, this assures results are stable.
- Files located in usual “path” manner
-
jsonnet -J /path/to/my/libjsonnet ... wire-cellandwcsonnetuse~-Pand honorsWIRECELL_PATH
-
24 Lazy evaluation and no side effects
Lazy evaluation:
- Jsonnet will not evaluate any code that is not used for the final result.
- In previous example, the
a.jsonnetmay be a gigantic file butb.jsonnetused onlymyvar.xand so onlya.xis evaluated.
No side effects:
- There is absolutely no way to modify any “variable”.
- But we can make new values based on existing ones.
25 Object inheritance
The + operator on Object types is inheritance, or really shallow merge.
{a:1, b:2, c:{x:"hi", y:"there"}} + {b:20, c:30, d:40}
{
"a": 1,
"b": 20,
"c": 30,
"d": 40
}
- Attributes of latter object “win”.
- There is std.mergePatch() which operates deeply.
26 Object inheritance, more
The + operator is optional and usually omitted.
Look for two or more objects “bumping against” each other.
local one = {a:1, b:2, c:{x:"hi", y:"there"}}; one {b:20, c:30, d:40} {d:400}
{
"a": 1,
"b": 20,
"c": 30,
"d": 400
}
27 Array and Object Comprehensions
Similar to the same language idiom in Python:
local squares = [x*x for x in std.range(1,3)]; local sqrts = {["%d"%xx]:std.sqrt(xx) for xx in squares}; sqrts
{
"1": 1,
"4": 2,
"9": 3
}
- Need to put the computed key inside
[]. - Keys need to be strings so we use string interpolation
- Notice
std.range(Nmin, Nmax)returns inclusive range!!-
std.range(1,3)→[1,2,3] - This is unlike Python
range()or C++std::iota().
-
28 if/then/else switch
// give example
- Balance its use as it tends to hide the variance it produces.
- Instead, consider to write a non-switched function and then explicitly enact whatever variance is needed by modifying the return
- A switch on a TLA is acceptable as the whole point there is to deliver a ready-to-consume data structure with no further processing.
29 Use of : vs :: in object
Jsonnet will use but not emit any attribute defined with ::
{
see_me : "yes",
me_too :: "no",
and_me : self.me_too + ", I mean yes!",
}
{
"and_me": "no, I mean yes!",
"see_me": "yes"
}
- Mostly used to assure we do not try to render a
function()as JSON.
30 Functions - basics
Functions may be defined in many contexts.
local bye = function(name) "bye " + name; local hi(name) = "hi " + name; // simpler syntax { local backwards(n) = bye(n) + ", " + hi(n), bw: backwards("who"), hb :: function(name) hi(name) + ", " + bye(name), me: self.hb("me"), you: self.hb("you"), }
{
"bw": "bye who, hi who",
"me": "hi me, bye me",
"you": "hi you, bye you"
}
31 Function returning internal values
Sometimes you need to a “working space” object and then return some part of it:
local f(x) = { local a = x*x, res: a + 2, }.res; f(42)
1766
- Construct some attribute value and then immediately reference it from the otherwise “returned” object.
32 Function arguments
Required positional and optional keyword with default (like Python)
local f(x, y=2) = { local a = x*x, res: a + y, }.res; f(42)
1766
- Unlike Python it is safe to give an array as an argument’s default value!
33 Function environment
local a=1; { f:: function(x) a + self.b + x, b:: " + ", zz: self.f("2 = 3") }.zz
"1 + 2 = 3"
- The function
f(x)“captures” (in C++ terms) thelocalvariables and the object context in which it was defined. - Not heavily used in WCT config, one can extend this to construct objects resembling C++/Python
classinstances.
34 “Classes” (avoid this idiom)
One can pretend Jsonnet has classes, but it’s not worth it!
local myclass(x,y=2) = { x:x, y:y, tot:: function() self.x + self.y, }; myclass(4).tot()
6
Due to lazy loading (and no side effects) this idiom is more quirk. Better:
local myclass(x,y=2) = { x:x, y:y, tot: self.x + self.y, }; myclass(4).tot
If .tot is never accessed, the sum is never calculated.
35 Top-level Function Arguments (TLAs)
A Jsonnet file may produce a function:
// tla.jsonnet function(x, y=2) {x:x, y:y}
jsonnet -A x=1 -A y="foo" tla.jsonnet jsonnet --tla-code 'x=22' tla.jsonnet
{
"x": "1",
"y": "foo"
}
{
"x": 22,
"y": 2
}
- CLI processing makes values given by
-A/--tla-strinto strings. - TLAs with default values need not be specified.
- The
--tla-codecan set a TLA to arbitrary Jsonnet code, take care with shell quoting. - There is also
--tla-code-fileto put this code in a secondary file.
36 The power of TLAs
We can subsum a Jsonnet producing a top-level function into more Jsonnet.
local tla = import "./tla.jsonnet"; tla(42, 24)
jsonnet tla2.jsonnet
{
"x": 42,
"y": 24
}
TLAs let us break out otherwise “internal” structure so we may more easily develop and testing it while still able to call it in other code, and in some cases to also use it as our “main” file.
37 TLAs in applications
WCT’s CLI args are similar to what jsonnet takes:
$ wire-cell -A x=1, --tla-code y='{a:2}' [...] -c foo.jsonnet
$ wcsonnet -A x=1, --tla-code y='{a:2}' [...] foo.jsonnet
- No
-cwithwcsonnetand not-tla-code-filefor either. - But, some big caveats:
- Most of WCT config was structured before TLAs. FIXME!
larwirecelllacks TLA support. FIXME!
38 The old bad way: std.extVar()
// extvar.jsonnet { x: std.extVar("x") }
jsonnet -V x=2 extvar.jsonnet jsonnet --ext-code 'x=2' extvar.jsonnet
{
"x": "2"
}
{
"x": 2
}
- Same CLI args for
wire-celland is supported bylarwirecell - But, there are major problems with
std.extVar()- No way to provide a default value, the caller must provide all
extVar’s std.extVar()calls may sprinkled anywhere, hard to spot.- Can not set them from Jsonnet, so full composition is not possible.
- No way to provide a default value, the caller must provide all
39 stdlib - full feature but “small” library of functions
- enumerate or test for object keys/values
- map, filter, reduce, sort, unique, set operations on arrays
- expected math, string and array functions
- type reflection:
std.type()andstd.parseXXX(),std.manifestXXX()
There is enough support that it was fairly easily to develop elaborate functions for graph construction, rewriting and topological sort!
40
WCT configuration structure
41 File layout
Everything is under wire-cell-toolkit/cfg/.
wirecell.jsonnet- WCT system of units and basic helper functions
vector.jsonnet- simple vector arithmetic
pgraph.jsonnet- support for describing WCT data flow graphs
pgrapher/- directory structure holding experiment configs
common/holds generic “base” data structuresexperiment/<name>/holds per-experiment derived structures
Currently: dune-vd, dune10kt-1x2x6, icarus, iceberg, pdps, sbnd, uboone.
- This structure really needs a revamp!
42 Main steps to building a WCT config:
- Produce a number of component structures.
- the
{type:"...", name:"...", data:{...}}
- the
- If component is also a DFP node, embed its structure in “node”.
- Form the graph from nodes culminating in a single aggregate node.
- Define
PgrapherorTbbFlowcomponent and give it its graphedges. - Optionally define special
wire-cellcomponent to avoid having to specify a bunch of CLI args. - Emit the WCT “main sequence”.
We now take each in trun.
43 1. Component structures - high level parameters
We may write each by hand like we saw with the Drifter example but,
the parameter…
drift_speed: 1.6 * wc.mm/wc.us,
…is needed in a few places. We setit once in a shared params data structure…
local wc = import "wirecell.jsonnet"; local params = import "pgrapher/common/params.jsonnet"; params.lar.drift_speed/(wc.mm/wc.us)
…and we override that value per each experiment:
local wc = import "wirecell.jsonnet"; local params = import "pgrapher/experiment/uboone/params.jsonnet"; params.lar.drift_speed/(wc.mm/wc.us)
44 1. Component structures - construction tools
To apply params consistently we make set of helper tools
and from them a set of component makers
local wc = import "wirecell.jsonnet"; local tool_maker = import "pgrapher/common/tools.jsonnet"; local params = import "pgrapher/experiment/uboone/params.jsonnet"; local tools = tool_maker(params); // uses TLAs! local sim_maker = import "pgrapher/common/sim/nodes.jsonnet"; local sim = sim_maker(params, tools); sim.drifter.uses[1].data.drift_speed/(wc.mm/wc.us)
- Do not worry about the form of this last line, we normally need not
dig into what
sim.driftholds as it is already a graph node. - Construction is automated but using it is more complicated than it should be.
- Diving into the guts it can be hard to understand what and how to change something.
45 2. Form graph nodes - primitive nodes
local wc = import "wirecell.jsonnet"; local g = import "pgraph.jsonnet"; local random={type:"Random"}; // normally tools.random local drifter = g.pnode({type:"Drifter", name:"drifter"}, nin=1, nout=1, uses=[random]); wc.tn(drifter)
- First arg is a configuration structure.
- Must give
ninandnoutwhich is the input/output port cardinality.- Used to catch graph construction errors.
- The
useslists dependencies to assure they are emitted in the final configuration sequence.
46 3. Form graph connections - aggregate nodes
A subgraph of nodes can be reformed into a single node. All unattached ports of constituent nodes become ports on an aggregate node.
local node1 = make_node1(); // assume nin=1, nout=2 local node2 = make_node2(); // assume nin=1, nout=1 local node3 = make_node3(); // assume nin=1, nout=1 local pl1 = g.intern(innodes = [node1], centernodes=[node2], outnodes=[node3], edges=[ g.edge(node1, node2) ]); local pl2 = g.pipeline([node1, node2, node3]); local star = g.intern(innodes=[node1], outnodes=[node2,node3], edges=[ g.edge(node1, node2, 0), g.edge(node2, node3, 1) ]);
intern()can connect any subgraph.-
edge(n1, n2, p1=0, p2=0)selects ports from nodes to connect.
-
pipeline()is for stringing end to end through port 0 (and only 0)- also
fanin(),fanout(),tap()andinsert_one()/insert_node()
- also
- Here, example aggregates all have
nin=1andnout=2. Do you see why?
47 4. Define main “app”
local graph = make_graph(); // top aggregate node local app = { type: 'Pgrapher', // or "TbbFlow" for multithread data: { edges: g.edges(graph), }, };
- The edges are simply lists of pairs of node “type:name” and port
numbers, all calculated in Jsonnet from the top aggragate node
graph. - The
PgrapherorTbbFlow“app” will usetype:nameto lookup the C++INodecomponents and perform edge connections given their port numbers.
48 5. Special wire-cell component
local cmdline = { type: "wire-cell", data: { plugins: ["WireCellGen", "WireCellPgraph", "WireCellSio", "WireCellSigProc", "WireCellRoot"], apps: ["Pgrapher"] // or "TbbFlow" } };
- This special “component” configuration is purely to avoid giving CLI options.
- It is used by
WireCell::Mainso useful for bothwire-cellandlarwirecell.
- It is used by
- Here or on CLI, one must give all plugins providing any referenced component “type”.
49 6. The last line: config sequence
Finally, the actual Jsonnet “program” is almost trivial!
[cmdline] + g.uses(graph) + [app]
uses() will iterate through the graph to produce a properly ordered
list of component configuration structures and we prepend cmdline and
append app by hand.
50 Configuration complexity
The WCT configuration is nominally very simple but
- Many parameters are needed for multiple componets.
- We want to provide “sane defaults” for parameters for:
- All uses of a given detector and all detectors
- And which let us customize on a detector or job basis.
It’s here the lack of TLAs in early development led to unnecessary complexity.
51
Development tips
52
Future plans