Wire-Cell Toolkit Configuration

1 Outline

We will cover these aspects of Wire-Cell Toolkit (WCT) configuration:

  • Some intro
  • C++ layers
  • Jsonnet
  • Current overall structure
  • Development tips
  • Future plans

2 Configuration requirements (why configure?)

While not re-compiling we wish to:

  • Define initial application state precisely and stably.
  • Compose sets of components into different applications.
  • Provide variety value for a parameter across multiple applications.
  • Share a parameter value with multiple components.
  • Set parameters independently across the components.
  • Independence between code and configuration representations.
  • Manage configuration complexity.
  • Seek brevity, simplicity, clarity.

Yes, there are some seemingly self-contradictions….

3 High-level, Layered Configuration Model

configurable component
application of configuration
transient representation (object)
representation conversion
persistent representation (files)
sources of configuration information
objects
a JSON object model (currently JsonCPP)
files
JSON (plain text or compressed) or Jsonnet

Only the object model is exposed to configurable components.

4 Two same but different categories of files

data
larger files, typically generated and compressed
  • .json.bz2 from programs in wire-cell-python.
  • Official versions in wire-cell-data.
  • eg: field response, wire geometry, noise spectra
cfg
smaller files, typically hand-made plain-text
  • .jsonnet from human developers and users
  • Official versions in wire-cell-toolkit/cfg.
  • eg: component configuration

The distinction is somewhat artificial.

  • One may provide .jsonnet “data” files or .json.bz2 “cfg” files.
    • latter may include pre-compiling .jsonnet to .json.bz2
  • An algorithmic “wire geometry” in a .jsonnet file is a reasonable thing.

But, for the rest of the slides, we focus mostly on “cfg” type.

5

C++ layers

6 WCT Main Configuration Sequence

WireCell::Main class expects an array of objects like:

[    // Jsonnet language
    {type:"A", name:"a", data: {...}},
    {type:"B", name:"b", data: {...}},
    // ...
    {type:"Z", name:"z", data: {...}},
]
  • One object per component instance
  • Identify component type and instance name
  • The data is a type-specific data structure.

7 One example component configuration object

{ type: "Drifter",
  data: {
      DL:          7.2 * wc.cm2/wc.s,
      DT:         12.0 * wc.cm2/wc.s,
      lifetime:      8 * wc.ms,
      drift_speed: 1.6 * wc.mm/wc.us,
      fluctuate:  true,                } }

The name is optional, default is "". Will talk more about where wc comes from later.

But how do we know what to provide for data???

  • Frankly, for now one must read the component’s C++ source….
  • Will give some “tips” to reduce this pain.
  • Future development plans will improve this issue.

8

Continuing with C++ layer….

9 WireCell::Persist namespace

Low-level object/file interface is provided by:

Functions for:

  • Locating configuration files.
    • Uses WCT’s only env var: WIRECELL_PATH
  • Loading file or string to product an object.
    • and vice-versa.
  • Provides a C++ interface to C or GO Jsonnet libraries.
    • May pass Jsonnet “top level arguments” and “external variables”.
  • Component developers usually need not encounter this level.
    • A new type of “data” file will need a new component that does.

10 WireCell::Configuration namespace

Generally used throughout all components:

Mainly provides a thin:

typedef Json::Value Configuration;

This exposes the JsonCPP type to components.

  • dynamic object so one must always inspect carefully.
  • helpful functions to prefer to using JsonCPP methods directly:

// Convert to C++ type
T convert<T>(const Configuration& cfg);
// Get object attribute as C++ type
T get<T>(const Configuration& cfg, const std::string& name);
x = "foo"

JSONCPP will be replaced in WCT with “Modern JSON” (aka nlohmann::json) which will require refactoring of all components!

11 WCT IConfigurable

class MyFilter : public virtual IConfigurable {
  public:
    // IConfigurable
    virtual void configure(const WireCell::Configuration& cfg);
    virtual WireCell::Configuration default_configuration() const;
  private:
    int m_a{42};                // a "sane" default
    float m_b{0};               // always initialize
}

WCT will:

  1. Call default_configuration() to get an initial object, treating it as opaque other than it is a JSON “object” (not scalar nor array).
  2. Merge in to it the data: {...} attribute.
  3. Provide result to configure()

12 Implementing IConfigurable

Current best practice: “filter” class data members through configuration.

WireCell::Configuration MyFilter::default_configuration() {
    WireCell::Configuration cfg;
    cfg["a"] = m_a;
    cfg["b"] = m_b;
    return cfg;
}
void configure(const WireCell::Configuration& cfg) {
    m_a = get<int>(cfg, "a", m_a);
    m_b = get<float>(cfg, "b", m_b);
}
  • Can do error checking and throw exceptions here, early before any processing occurs.
  • Currently, we must read this kind of code to discover what to provide in data:{...}.

13 Implementation issues

  • A lot of code is devoted to these two methods.
  • It tends to enact the same pattern across all components.
  • Patterns can be abstracted and (meta)parameterized by schema.
  • At end, will talk about plans to enact this to make life easier for developers (and users).

14

Jsonnet

15 Jsonnet data templating language.

15.1 :Bcolumns:

15.1.1 :BMCOL:

Pure functional programming language

  • No side effects, lazy evaluation
  • LISP/Scheme-like standard library

Compiles to JSON text by default

  • or YAML, INI, Python (dict), XML(svg), TOML
  • Any text format can developed in Jsonnet code.

Very simple, familiar syntax

  • one of very few languages where (sometimes) a file “works” on first try
  • but, side-effect-free functional programming is an unfamiliar new paradigm for many

15.1.2 :BMCOL:

jsonnet-venn.png

From https://jsonnet.org/

Plus comments, trailing commas, quote-free object keys.

16 Hello Jsonnet command line

Expression strings

jsonnet -e '{hello:"world",} // quick CLI test'
{
   "hello": "world"
}

Source files

echo '2 + 2' > sum.jsonnet
jsonnet sum.jsonnet
4

17 Jsonnet Types

Null
only one value: null, equal only to itself
Boolean
two values true and false
Number
3, -0.1000001, 64 bit IEEE754 FP.
  • exact integers \(\in [-2^{53},2^{53}]\)
String
Unicode codepoints
  • may use 'single quotes', "or double"
  • may index: s[2] but strings are not arrays
Array
Finite sequence, possibly with heterogeneous element types
  • [1,2], [4, "you", [1, ["nest"]]]
Object
Collection of pairs: key of string, value of any type.
  • {a:1, b:2, c:{greet: "hi!"}}
  • ordered by key {b:2, a:1} → {"a":1, "b":2}
Function
A calculation on arguments and environment
  • {a::1, f::function() self.a, b:self.f()} → {“b”:1}
  • A function is a first-class instance.

18 String operations

Interpolation, (in)equalities, indexing, slicing:

local a = "%0.5f"%(1/19);
local cat = "meow";
{a: a,
 c: cat[0:2],
 equal: a == cat,
 isarray: cat == ["m","e","o","w"],
 lexical_less_than: cat < "dog"}
{
   "a": "0.05263",
   "c": "me",
   "equal": false,
   "isarray": false,
   "lexical_less_than": false
}

  • Also many string functions in std.
  • Will more fully introduce local later.

19 Array operations

19.1 :Bcolumns:

19.1.1 :BMCOL:

Length, indexing, slicing, appending, concatenating.

  • All similar to what one expects from Python.
  • Many more Array functions under std.

19.1.2 :BMCOL:

local a = [1,2,3]; local b = [40,50,60];
{
    len: std.length(a),
    index: b[0],
    append: a + [4],
    concat: a + b,
    slice: self.concat[3:5],
}
{
   "append": [
      1,
      2,
      3,
      4
   ],
   "concat": [
      1,
      2,
      3,
      40,
      50,
      60
   ],
   "index": 40,
   "len": 3,
   "slice": [
      40,
      50
   ]
}

20 Jsonnet local keyword

Defines file-local value which may be directly referenced.

local x = 42;
local y = x + 1;      // semicolon at top level 
{
    local z = y - x,  // comma inside object
    result: z
}
{
   "result": 1
}

Use = to assign local variable, use : with object attributes.

21 Relative referencing

{
    a: $.sub.c,    // reference from top of structure
    b: self.sub.c, // reference from current context
    sub: { 
        c: self.d,
        d: "all the same",
    }
}
{
   "a": "all the same",
   "b": "all the same",
   "sub": {
      "c": "all the same",
      "d": "all the same"
   }
}

Lazy evaluation means reference can precede definition.

22 Self-referencing in object context

Via local

local obj = {
    name: "Alice",
    lower: std.asciiLower(self.name),
    upper: std.asciiUpper($.name),
    greeting: "Hello, " + obj.name
}; obj
{
   "greeting": "Hello, Alice",
   "lower": "alice",
   "name": "Alice",
   "upper": "ALICE"
}

23 File referencing

{ // a.jsonnet
    x = 1
}
// b.jsonnet
local myvar = import "a.jsonnet";
{
    y = myvar.x
}

  • The b.jsonnet compiles to {"y":1}
  • The file name to import must be a literal string, not a computed value.
    • In part, this assures results are stable.
  • Files located in usual “path” manner
    • jsonnet -J /path/to/my/libjsonnet ...
    • wire-cell and wcsonnet use ~-P and honors WIRECELL_PATH

24 Lazy evaluation and no side effects

Lazy evaluation:

  • Jsonnet will not evaluate any code that is not used for the final result.
  • In previous example, the a.jsonnet may be a gigantic file but b.jsonnet used only myvar.x and so only a.x is evaluated.

No side effects:

  • There is absolutely no way to modify any “variable”.
  • But we can make new values based on existing ones.

25 Object inheritance

The + operator on Object types is inheritance, or really shallow merge.

{a:1, b:2, c:{x:"hi", y:"there"}} + {b:20, c:30, d:40}
{
   "a": 1,
   "b": 20,
   "c": 30,
   "d": 40
}

  • Attributes of latter object “win”.
  • There is std.mergePatch() which operates deeply.

26 Object inheritance, more

The + operator is optional and usually omitted.

Look for two or more objects “bumping against” each other.

local one = {a:1, b:2, c:{x:"hi", y:"there"}};
one {b:20, c:30, d:40} {d:400}
{
   "a": 1,
   "b": 20,
   "c": 30,
   "d": 400
}

27 Array and Object Comprehensions

Similar to the same language idiom in Python:

local squares = [x*x for x in std.range(1,3)];
local sqrts = {["%d"%xx]:std.sqrt(xx) for xx in squares};
sqrts
{
   "1": 1,
   "4": 2,
   "9": 3
}

  • Need to put the computed key inside [].
  • Keys need to be strings so we use string interpolation
  • Notice std.range(Nmin, Nmax) returns inclusive range!!
    • std.range(1,3) → [1,2,3]
    • This is unlike Python range() or C++ std::iota().

28 if/then/else switch

// give example

  • Balance its use as it tends to hide the variance it produces.
  • Instead, consider to write a non-switched function and then explicitly enact whatever variance is needed by modifying the return
  • A switch on a TLA is acceptable as the whole point there is to deliver a ready-to-consume data structure with no further processing.

29 Use of : vs :: in object

Jsonnet will use but not emit any attribute defined with ::

{
    see_me : "yes",
    me_too :: "no",
    and_me : self.me_too + ", I mean yes!",
}
{
   "and_me": "no, I mean yes!",
   "see_me": "yes"
}

  • Mostly used to assure we do not try to render a function() as JSON.

30 Functions - basics

Functions may be defined in many contexts.

local bye = function(name) "bye " + name;
local hi(name) = "hi " + name;  // simpler syntax
{
    local backwards(n) = bye(n) + ", " + hi(n),
    bw: backwards("who"),
    hb :: function(name) hi(name) + ", " + bye(name),
    me: self.hb("me"),
    you: self.hb("you"),
}
{
   "bw": "bye who, hi who",
   "me": "hi me, bye me",
   "you": "hi you, bye you"
}

31 Function returning internal values

Sometimes you need to a “working space” object and then return some part of it:

local f(x) = {
    local a = x*x,
    res: a + 2,
}.res;
f(42)
1766

  • Construct some attribute value and then immediately reference it from the otherwise “returned” object.

32 Function arguments

Required positional and optional keyword with default (like Python)

local f(x, y=2) = {
    local a = x*x,
    res: a + y,
}.res;
f(42)
1766

  • Unlike Python it is safe to give an array as an argument’s default value!

33 Function environment

local a=1;
{
    f:: function(x) a + self.b + x,
    b:: " + ",
    zz: self.f("2 = 3")
}.zz
"1 + 2 = 3"

  • The function f(x) “captures” (in C++ terms) the local variables and the object context in which it was defined.
  • Not heavily used in WCT config, one can extend this to construct objects resembling C++/Python class instances.

34 “Classes” (avoid this idiom)

One can pretend Jsonnet has classes, but it’s not worth it!

local myclass(x,y=2) = {
    x:x, y:y,
    tot:: function() self.x + self.y,
};
myclass(4).tot()
6

Due to lazy loading (and no side effects) this idiom is more quirk. Better:

local myclass(x,y=2) = {
    x:x, y:y,
    tot: self.x + self.y,
};
myclass(4).tot

If .tot is never accessed, the sum is never calculated.

35 Top-level Function Arguments (TLAs)

A Jsonnet file may produce a function:

// tla.jsonnet
function(x, y=2) {x:x, y:y}
jsonnet -A x=1 -A y="foo" tla.jsonnet
jsonnet --tla-code 'x=22' tla.jsonnet      
{
   "x": "1",
   "y": "foo"
}
{
   "x": 22,
   "y": 2
}

  • CLI processing makes values given by -A/--tla-str into strings.
  • TLAs with default values need not be specified.
  • The --tla-code can set a TLA to arbitrary Jsonnet code, take care with shell quoting.
  • There is also --tla-code-file to put this code in a secondary file.

36 The power of TLAs

We can subsum a Jsonnet producing a top-level function into more Jsonnet.

local tla = import "./tla.jsonnet";
tla(42, 24)
jsonnet tla2.jsonnet      
{
   "x": 42,
   "y": 24
}

TLAs let us break out otherwise “internal” structure so we may more easily develop and testing it while still able to call it in other code, and in some cases to also use it as our “main” file.

37 TLAs in applications

WCT’s CLI args are similar to what jsonnet takes:

$ wire-cell -A x=1, --tla-code y='{a:2}' [...] -c foo.jsonnet
$ wcsonnet  -A x=1, --tla-code y='{a:2}' [...] foo.jsonnet

  • No -c with wcsonnet and not -tla-code-file for either.
  • But, some big caveats:
    • Most of WCT config was structured before TLAs. FIXME!
    • larwirecell lacks TLA support. FIXME!

38 The old bad way: std.extVar()

// extvar.jsonnet
{ x: std.extVar("x") }
jsonnet -V x=2 extvar.jsonnet      
jsonnet --ext-code 'x=2' extvar.jsonnet      
{
   "x": "2"
}
{
   "x": 2
}
  • Same CLI args for wire-cell and is supported by larwirecell
  • But, there are major problems with std.extVar()
    • No way to provide a default value, the caller must provide all extVar’s
    • std.extVar() calls may sprinkled anywhere, hard to spot.
    • Can not set them from Jsonnet, so full composition is not possible.

39 stdlib - full feature but “small” library of functions

  • enumerate or test for object keys/values
  • map, filter, reduce, sort, unique, set operations on arrays
  • expected math, string and array functions
  • type reflection: std.type() and std.parseXXX(), std.manifestXXX()

There is enough support that it was fairly easily to develop elaborate functions for graph construction, rewriting and topological sort!

40

WCT configuration structure

41 File layout

Everything is under wire-cell-toolkit/cfg/.

wirecell.jsonnet
WCT system of units and basic helper functions
vector.jsonnet
simple vector arithmetic
pgraph.jsonnet
support for describing WCT data flow graphs
pgrapher/
directory structure holding experiment configs
  • common/ holds generic “base” data structures
  • experiment/<name>/ holds per-experiment derived structures

Currently: dune-vd, dune10kt-1x2x6, icarus, iceberg, pdps, sbnd, uboone.

  • This structure really needs a revamp!

42 Main steps to building a WCT config:

  1. Produce a number of component structures.
    • the {type:"...", name:"...", data:{...}}
  2. If component is also a DFP node, embed its structure in “node”.
  3. Form the graph from nodes culminating in a single aggregate node.
  4. Define Pgrapher or TbbFlow component and give it its graph edges.
  5. Optionally define special wire-cell component to avoid having to specify a bunch of CLI args.
  6. Emit the WCT “main sequence”.

We now take each in trun.

43 1. Component structures - high level parameters

We may write each by hand like we saw with the Drifter example but, the parameter…

drift_speed: 1.6 * wc.mm/wc.us,

…is needed in a few places. We setit once in a shared params data structure…

local wc = import "wirecell.jsonnet";
local params = import "pgrapher/common/params.jsonnet";
params.lar.drift_speed/(wc.mm/wc.us)

…and we override that value per each experiment:

local wc = import "wirecell.jsonnet";
local params = import "pgrapher/experiment/uboone/params.jsonnet";
params.lar.drift_speed/(wc.mm/wc.us)

44 1. Component structures - construction tools

To apply params consistently we make set of helper tools and from them a set of component makers

local wc = import "wirecell.jsonnet";
local tool_maker = import "pgrapher/common/tools.jsonnet";
local params = import "pgrapher/experiment/uboone/params.jsonnet";
local tools = tool_maker(params); // uses TLAs!
local sim_maker = import "pgrapher/common/sim/nodes.jsonnet";
local sim = sim_maker(params, tools);
sim.drifter.uses[1].data.drift_speed/(wc.mm/wc.us)

  • Do not worry about the form of this last line, we normally need not dig into what sim.drift holds as it is already a graph node.
  • Construction is automated but using it is more complicated than it should be.
  • Diving into the guts it can be hard to understand what and how to change something.

45 2. Form graph nodes - primitive nodes

local wc = import "wirecell.jsonnet";
local g = import "pgraph.jsonnet";
local random={type:"Random"};   // normally tools.random
local drifter = g.pnode({type:"Drifter", name:"drifter"},
                        nin=1, nout=1, uses=[random]);
wc.tn(drifter)

  • First arg is a configuration structure.
  • Must give nin and nout which is the input/output port cardinality.
    • Used to catch graph construction errors.
  • The uses lists dependencies to assure they are emitted in the final configuration sequence.

46 3. Form graph connections - aggregate nodes

A subgraph of nodes can be reformed into a single node. All unattached ports of constituent nodes become ports on an aggregate node.

local node1 = make_node1();     // assume nin=1, nout=2
local node2 = make_node2();     // assume nin=1, nout=1
local node3 = make_node3();     // assume nin=1, nout=1
local pl1 = g.intern(innodes = [node1], centernodes=[node2],
   outnodes=[node3], edges=[ g.edge(node1, node2) ]);
local pl2 = g.pipeline([node1, node2, node3]);
local star = g.intern(innodes=[node1], outnodes=[node2,node3],
   edges=[ g.edge(node1, node2, 0), g.edge(node2, node3, 1) ]);
  • intern() can connect any subgraph.
    • edge(n1, n2, p1=0, p2=0) selects ports from nodes to connect.
  • pipeline() is for stringing end to end through port 0 (and only 0)
    • also fanin(), fanout(), tap() and insert_one()/insert_node()
  • Here, example aggregates all have nin=1 and nout=2. Do you see why?

47 4. Define main “app”

  local graph = make_graph(); // top aggregate node 
  local app = {
  type: 'Pgrapher',  // or "TbbFlow" for multithread
  data: {
    edges: g.edges(graph),
  },
};  

  • The edges are simply lists of pairs of node “type:name” and port numbers, all calculated in Jsonnet from the top aggragate node graph.
  • The Pgrapher or TbbFlow “app” will use type:name to lookup the C++ INode components and perform edge connections given their port numbers.

48 5. Special wire-cell component

local cmdline = {
    type: "wire-cell",
    data: {
        plugins: ["WireCellGen", "WireCellPgraph",
                  "WireCellSio", "WireCellSigProc",
                  "WireCellRoot"],
        apps: ["Pgrapher"]      // or "TbbFlow"
    }
};
  • This special “component” configuration is purely to avoid giving CLI options.
    • It is used by WireCell::Main so useful for both wire-cell and larwirecell.
  • Here or on CLI, one must give all plugins providing any referenced component “type”.

49 6. The last line: config sequence

Finally, the actual Jsonnet “program” is almost trivial!

[cmdline] + g.uses(graph) + [app]

uses() will iterate through the graph to produce a properly ordered list of component configuration structures and we prepend cmdline and append app by hand.

50 Configuration complexity

The WCT configuration is nominally very simple but

  • Many parameters are needed for multiple componets.
  • We want to provide “sane defaults” for parameters for:
    • All uses of a given detector and all detectors
    • And which let us customize on a detector or job basis.

It’s here the lack of TLAs in early development led to unnecessary complexity.

51

Development tips

52

Future plans

Author: Brett Viren

Created: 2021-10-02 Sat 11:37

Validate