


\begin{frame}
  \frametitle{High level \worch design}

  \begin{center}
    \textcolor{blue}{\worch = \textbf{w}af + \textbf{orch}estration}
  \end{center}

  

\end{frame}


\begin{frame}[fragile]
  \frametitle{What is \waf?}

  \waf is at \url{http://code.google.com/p/waf/}
  \begin{itemize}
  \item is a Python-based (v2.3-v3.1) framework for building build systems
  \item can be used as a \texttt{make/autoconf/cmake} replacement or as a layer
    on top to orchestrate a variety of ``native'', per-package build systems
  \item Provides a task scheduling and dependency engine
  \item has support for low level build patterns (C/C++ compilations, library linkage, etc)
  \item packaged as a single, self-contained, downloaded Python executable
  \end{itemize}
  Typical \waf installation procedure:
{\footnotesize
\begin{verbatim}
$ wget http://waf.googlecode.com/files/waf-1.7.12
$ chmod +x waf-1.7.12
$ alias waf=$(pwd)/waf-1.7.12
$ waf --version
waf 1.7.12 (c4b82f4337ebdf5236e844791a9faa346770d6db)
$ waf --help
\end{verbatim}
}
Or, just include a copy of \waf along with a project.
\end{frame}

\begin{frame}
  \frametitle{Aside: Synergy with ATLAS}
  
  Effort in ATLAS to replace CMT/LCGCMT.

  \begin{itemize}
  \item Work being done by \textbf{Sebastien Binet}
  \item A command line user interface: \hwaf
  \item Replacing CMT with \waf
  \item Migrating Gaudi + LCGCMT (in process) ``\texttt{tdaq}'' subsystem (done)
  \item Its use for bootstrapping/creating analysis packages,
  \item SB has made contributions to \worch itself.
  \end{itemize}

  \vspace{2mm}
  \footnotesize
  Note of caution for LBNE and Fermilab: 
  \begin{itemize}
  \item ATLAS investigated CMake to replace CMT, found it unsuitable
    after two months of effort.
  \item 
    I still support replacing SRT with CMake in LArSoft, but for
    large-scale meta-build systems we should take heed.
  \item 
    We should investigate \hwaf/\waf to see if would be a worthy basis
    for LBNE ``native'' package builds.
  \end{itemize}
\end{frame}

\begin{frame}[fragile]
  \frametitle{\waf basics}

  \waf looks for a \texttt{wscript} file (think \texttt{Makefile}).

{\footnotesize
\begin{verbatim}
def configure(cfg):
    cfg.find_program('root', var='ROOTEXE')
def build(bld):
    bld(rule="${ROOTEXE} -l -b -q ${SRC}",
        source = "myscript.C", target = "myexpectedout.pdf")
\end{verbatim}
}
  \begin{itemize}
  \item Connect input and output files through rules.
  \item Rules are shell command strings or Python functions.
  \item Rules run in parallel governed by their dependencies.
  \item Built in ``tools'' and many other features.
  \end{itemize}

  Run:
{\footnotesize
\begin{verbatim}
$ waf --prefix=/path/to/install configure 
$ waf
$ waf install
# or all in one go:
$ waf --prefix=/path/to/install configure build install
\end{verbatim}
}
\end{frame}

\begin{frame}
  \frametitle{What does \worch add to \waf?}

  \worch adds a configuration layer on top of \waf in order to:

  \begin{itemize}
  \item Determine installation layout policy, Eg:
    \begin{itemize}
    \item match UPS ``product area'' conventions
    \item share/interleave different build options (\texttt{opt/debug}), platform labels
    \item build on fast local disk/ramfs, install to AFS/NFS
    \end{itemize}
  \item Specify the list of packages to install
  \item Group them into atomically installed sets
  \item Define ``features'' implementing installation ``steps''
    \begin{description}
    \item [features] \textit{tarball, vcs, patch, autoconf, cmake, makemake}
    \item [steps] \texttt{seturl, download, unpack, patch, prepare, build, install}
    \end{description}
  \item Set parameters governing details of their build and installation
  \item Declare explicit inter-step dependencies
  \item Set package build and use environment variables
  \end{itemize}

\end{frame}

\begin{frame}[fragile]
  \frametitle{Basic configuration file syntax}

  Standard Python (\texttt{ConfigParser}) configuration file syntax \\
  (a.k.a. ``INI'' format).

  \begin{lstlisting}[language=bash,emph={section,section2}]
# this is a comment
[section]
key = value

[section2]
key = newvalue
key2 = value2
key3: equals-or-colons
  \end{lstlisting}

  Consists of a number of named \textbf{sections} containing a number
  of configuration \textbf{items} consisting of
  \textbf{key}/\textbf{value} pairs separated \\
  by ``\texttt{=}'' or ``\texttt{:}''.   Sections, keys and values are interpreted as strings.

\end{frame}

\begin{frame}[fragile]
  \frametitle{Extension: hierarchy of configuration parameters} 

  \begin{columns}
    \begin{column}{0.45\textwidth}
  \begin{lstlisting}[language=bash,emph={start,group,package,keytype},emphstyle=\color{red},emph={[2]group1,package1},emphstyle={[2]\color{blue}},emph={[3]groups,packages},emphstyle={[3]\color{magenta}}]
[start]
groups = group1, group2
key1 = val1
key2 = val2
key3 = val3

[group group1]
packages = package1, package2
key2 = grpval2

[package package1]
key1 = pkg1val1

[package package2]
key1 = pkg2val1

[keytype]
groups = group
packages = package
  \end{lstlisting}
    \end{column}
    \begin{column}{0.5\paperwidth}
      \small
      Hierarchy connected by section names
      \begin{itemize}
      \item Results in \textcolor{blue}{\texttt{package1}} having keys: 
        \begin{itemize}
        \item \texttt{key1 = pkg1val1}
        \item \texttt{key2 = grpval2}
        \item \texttt{key3 = val3}      
        \end{itemize}
      \item Likewise for \textcolor{blue}{\texttt{package2}} but with \\
        \texttt{key1 = pkg2val1}
      \item Special \textcolor{red}{\texttt{keytype}} section defines special variables to be
        interpreted as references to typed/named sections.
      \item Additional keytypes can be defined and parsed but \worch uses just \textcolor{magenta}{\texttt{groups}}
        and \textcolor{magenta}{\texttt{packages}}.
      \end{itemize}
    \end{column}
  \end{columns}

\end{frame}

\begin{frame}[fragile]
  \frametitle{Parameters in configuration}

  Configuration items form variables that can be referenced in other items.

  Reference a variable like ``\texttt{\{keyname\}}''.

  \begin{lstlisting}[language=bash,emph={package,version,gcc_version},emph={[2]PREFIX,tagsdashed},emphstyle={[2]\color{blue}}]
[start]
tags = debug
install_dir = {PREFIX}/{package}/{version}/{tagsdashed}
gcc_version = 4.8.1
groups = buildtools, compiler, externals
#...
[group compiler]
install_dir = {PREFIX}/gcc/{gcc_version}/{tagsdashed}
packages = gmp, ppl, mpfr, mpc, isl, cloog, gcc
#...
[package gcc]
version = {gcc_version}
#...
  \end{lstlisting}
  Variables are resolved late so ``flow down'' the hierarchy to take on meanings specific to each ``leaf'' keytype (\textcolor{red}{\texttt{package}} in this case).

  \textcolor{blue}{Some variables} and some other's default values are provided by \worch.
\end{frame}


\begin{frame}[fragile]
  \frametitle{Simple example - main file}
    \begin{lstlisting}[language=bash,emph={start,keytype}]
# worch/examples/simple/simple.cfg
[start]
groups = buildtools, gnuprograms
includes = buildtools.cfg, gnuprograms.cfg
group = gnuprograms
tags = debug
features = tarball autoconf makemake
download_dir = downloads
source_dir = sources
build_dir = builds/{package}-{version}-{tagsdashed}
install_dir = {PREFIX}/{package}/{version}/{tagsdashed}

[keytype]
groups = group
packages = package
    \end{lstlisting}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Simple example - buildtools group file}
    \begin{lstlisting}[language=bash,emph={start,keytype}]
# worch/examples/simple/buildtools.cfg
[group buildtools]
packages = cmake
build_target = bin/{package}
install_target = bin/{package}

[package cmake]
version = 2.8.8
source_url = http://www.cmake.org/files/v{version_2digit}/{source_package}
unpacked_target = bootstrap
prepare_cmd = ../../{source_dir}/{source_unpacked}/bootstrap
prepare_cmd_options = --prefix={install_dir}
prepare_target = cmake_install.cmake
export_PATH = prepend:{install_dir}/bin
    \end{lstlisting}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Simple example - gnuprograms group file}
    \begin{lstlisting}[language=bash,emph={start,keytype}]
# worch/examples/simple/gnuprograms.cfg
[group gnuprograms]
packages = hello, bc
source_url = http://ftp.gnu.org/gnu/{package}/{source_package}
environment = group:buildtools
unpacked_target = configure
prepare_target = config.status
build_target = bin/{package}
install_target = bin/{package}

[package hello]
version = 2.8
depends = prepare:bc_install

[package bc]
version = 1.06 
    \end{lstlisting}
\end{frame}



\begin{frame}
  \frametitle{User Environment with ``worch''}
  \begin{itemize}
  \item Unified installation and environment from single configuration file set
  \item Multiple environment setup systems supported.  So far have:
    \begin{description}
    \item[UPS] FNAL-centric system for user environment setup
      \begin{itemize}
      \item Used for user environment setup on fnal.gov machines
      \item Somewhat cryptic configuration language
      \item Sprinkled dependencies in FNAL installation ``system''
      \item Essentially is required to build Art.
      \end{itemize}
    \item[modules] Environment modules (\url{http://modules.sf.net/})
      \begin{itemize}
      \item Free Software, community supported
      \item Uses a ``real'' language (well, at least TCL)
      \item Already in use by some LBNE collaborators (PDSF)
      \end{itemize}
    \end{description}
  \end{itemize}

  Worch does not require a user environment system to build.  They are
  added as just another build ``feature''.

  \textbf{However....}
\end{frame}

\begin{frame}
  \frametitle{The problem with UPS and Art}
  \begin{itemize}
  \item Art is the de facto software framework for LBNE offline (and probably
    online/DAQ?) at this time.
  \item It is built with CMake.  That's great, however:
  \item Art's CMake files require UPS.
  \item This greatly hampers building it outside of FNAL
  \item Building Art, itself, has posed more problems than all other externals combined.
  \end{itemize}
  Possible solutions:
  \begin{itemize}
  \item Rewrite Art's native build system to be independent from UPS
    \begin{itemize}
    \item Art developers are not interested
    \item Large effort for an outsider, always needing to chase differences 
    \end{itemize}
  \item ``Fake'' UPS and Art is non-the-wiser.  
    \begin{itemize}
    \item Probably more effort than the next solution:
    \end{itemize}
  \item Provide for UPS to be used by worch installation elements
    \begin{itemize}
    \item \textbf{This is the approach I'm talking now.}
    \end{itemize}
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Status of Worch w.r.t UPS and Art}
  Worch status w.r.t. UPS and Art
  \begin{description}
  \item[done] Building UPS itself, setting up a UPS ``database'' directory
  \item[done] Producing per-package UPS ``database'' files (doesn't require UPS itself)
  \item[done] Using UPS-setup environment when building a package
  \item[began] Building Art itself.
  \end{description}

  UPS ``taints'' Worch only to the extent that one must choose a
  directly layout consistent with (largely unwritten) UPS norms.

\end{frame}

