# Installing larsoft under Spack

*[ Instructions created by Marc Mengel (FNAL). ]*

## Prerequisites

At the moment we need to run this onsite at Fermilab, or with VPN
turned on; once various parts are moved to GitHub we can do this
wherever. You're going to need a directory/partition with 40GB or so
of disk free; we'll refer to this as /my/spack/area in the
instructions below, but it can be whatever path is convenient for you.

## Setup spack area

Grab the spack-infrastructure bootstrap file from this repository and run it:

```
sh bootstrap /my/spack/area
```

## Get a compiler

We have 3 options here, you can

- install the prebuilt gcc
- point spack at an existing gcc/clang install
- install from source

### Installing prebuilt gcc with spack

```
source /my/spac/area/setup-env.sh
spack buildcache list --allarch gcc
```

pick the hash of the one you want, in our case

```
-- linux-scientific7-x86_64 / gcc@4.8.5 -------------------------
4s22i2p gcc@8.2.0  w6auovd gcc@8.2.0  arqo2nn gcc@9.3.0
```

argo2nn is the hash for gcc 9.3.0

```
spack load patchelf
spack buildcache install -o /arqo2nn
```

and get a cup of coffee...
its going to install the dependencies and then finally gcc.

BUG Workaround

At the moment, spack isn't relocating this gcc quite right; so..

```
spack find --paths gcc
cd /path/to/gcc/libexec/gcc/x86_64-pc-linux-gnu/9.3.0
patchelf --print-rpath cc1plus > /tmp/f1
```

edit /tmp/f1 and change '/scratch/spack' to your spack area

```
for f in cc1 cc1plus collect2 f951 lto1 ; do patchelf --set-rpath `cat /tmp/f1` $f; done
```

Now that we've installed it, we need to tell spack to use it:

```
spack load gcc@9.3.0
spack compiler find --scope=site
```

### Pointing spack at an existing compiler

First do whatever you do to make the compiler available (ups setup, update PATH, etc). Then

```
spack compiler find --scope=site
```

### Installing from source

```
spack install gcc@9.3.0w
spack load gcc@9.3.0
spack compiler find --scope=site
```

## Minor fixups

Our bootstrap script currently marks "curl" as not buildable, but larsoft requires us to build
a newer version, so edit $SPACK_ROOT/etc/scientific7/packages.yaml, find the 'curl' stanza:

```
curl:
  version: [7.29.0]
  target: [x86_64]
  compiler: [gcc@4.8.5]
  externals:
  - spec: curl@7.29.0%gcc@4.8.5 os=scientific7
    prefix: /usr
  buildable: False
```

and comment out the buildable line with a '#':

```
curl:
  version: [7.29.0]
  target: [x86_64]
  compiler: [gcc@4.8.5]
  externals:
  - spec: curl@7.29.0%gcc@4.8.5 os=scientific7
    prefix: /usr
  #buildable: False
```

and save that file.

You may also want to edit $SPACK_ROOT/etc/spack/config.yaml and add:

```
build_stage:
  - /some-path/$user/tmp/spack-stage
```

To move the build areas out of /tmp, because some of our packages are
Really Large and can fill up /tmp.

### Concretizing the spec(ification)

It's a long invocation, so I recommend making a file with the options:
`larsoft_spec.opts`

```
larsoft@mwm1 cxxstd=17 %gcc@9.3.0 arch=x86_64
  ^art@3.09.01  cxxstd=17
  ^artg4tk@10.02.01.01  cxxstd=17
  ^art-root-io@1.08.01  cxxstd=17
  ^boost@1.73.0  cxxstd=17
  ^bxdecay0@1.0.7 cxxstd=17
  ^canvas@3.12.02  cxxstd=17
  ^canvas-root-io@1.09.02  cxxstd=17
  ^cetlib@3.13.02  cxxstd=17
  ^cetlib-except@1.07.02  cxxstd=17
  ^cetmodules@2.16.02
  ^cetpkgsupport@1.14.01
  ^clhep@2.4.1.3  cxxstd=17
  ^cmake@3.19.6~qt
  ^cppunit@1.15.1  cxxstd=17
  ^cry@1.7  cxxstd=17
  ^davix@0.7.6 +thirdparty cxxstd=17
  ^dk2nudata@01.10.00  cxxstd=17
  ^dk2nugenie@1.10.00  cxxstd=17
  ^eigen@3.3.9  cppflags=-std=c++17
  ^fftw@3.3.8 ~mpi~openmp precision=float,double,long_double,quad
  ^fhicl-cpp@4.15.01 cxxstd=17
  ^g4abla@3.1
  ^g4emlow@7.9.1
  ^g4incl@1.0
  ^g4particlexs@2.1
  ^g4pii@1.3
  ^gallery@1.18.02  cxxstd=17
  ^geant4@10.6.1 ~motif~opengl~qt+threads~vecgeom~x11 cxxstd=17
  ^geant4-data@10.6.1
  ^genie@3.00.06  cxxstd=17
  ^gsl@2.6  cppflags=-std=c++17
  ^h5cpp@1.10.4.5
  ^hdf5@1.10.7+hl+cxx~mpi
  ^hep-concurrency@1.07.00 cxxstd=17
  ^ifbeam@2.4.8  cxxstd=17 cxxflags=-Wno-catch-value
  ^ifdh-art@2.12.02.01  cxxstd=17
  ^ifdhc@2.5.12  cxxstd=17
  ^intel-tbb@2021.1.1 cxxstd=17
  ^jsoncpp@1.7.7  cxxstd=17
  ^jsonnet@0.12.1  cxxstd=17
  ^larana@mwm1 cxxstd=17
  ^larcore@mwm1 cxxstd=17
  ^larcorealg@mwm1 cxxstd=17
  ^larcoreobj@09.24.01.01  cxxstd=17
  ^lardata@mwm1 cxxstd=17
  ^lardataalg@mwm1 cxxstd=17
  ^lardataobj@mwm1 cxxstd=17
  ^lareventdisplay@mwm1 cxxstd=17
  ^larevt@mwm1 cxxstd=17
  ^larexamples@mwm1 cxxstd=17
  ^larg4@mwm1 cxxstd=17
  ^larpandora@mwm1 cxxstd=17
  ^larpandoracontent@mwm1 cxxstd=17
  ^larreco@mwm1 cxxstd=17
  ^larrecodnn@mwm1 cxxstd=17
  ^larsim@mwm1 cxxstd=17
  ^larsimrad@mwm1 cxxstd=17
  ^larsoft-data@1.02.01
  ^larsoftobj@mwm1  cxxstd=17
  ^larwirecell@mwm1 cxxstd=17
  ^lhapdf@6.2.3  cxxstd=17
  ^libwda@2.29.1
  ^libxml2@2.9.10 +python
  ^log4cpp@1.1.3  cxxstd=17
  ^marley@1.2.0  cxxstd=17
  ^messagefacility@2.08.02  cxxstd=17
  ^mysql@5.5.62+client_only cxxstd=17
  ^nucondb@2.4.8  cxxstd=17 cxxflags=-Wno-catch-value
  ^nuevdb@mwm1 cxxstd=17
  ^nug4@mwm1 cxxstd=17
  ^nugen@mwm1 cxxstd=17
  ^nurandom@mwm1 cxxstd=17
  ^nusimdata@1.22.03  cxxstd=17
  ^nutools@mwm1  cxxstd=17
  ^openblas@0.3.10
  ^opencv@4.2.0
  ^pandora@03.16.00  cxxstd=17
  ^pdfsets@5.9.1
  ^postgresql@12.3 +client_only+python+perl+threadsafe+gssapi cflags=-I/usr/include/mit-krb5 ldflags=-L/usr/lib/x86_64-linux-gnu/mit-krb5
  ^protobuf@3.12.3
  ^py-numpy@1.18.0
  ^py-pybind11@2.5.0
  ^py-tensorflow@2.2.2 ~cuda ~nccl cxxstd=17
  ^pythia6@6.4.28
  ^python@3.7.10 +shared~libxml2~readline
  ^py-torch@1.6.0~cudnn~cuda~nccl
  ^range-v3@0.10.0  -doc cxxstd=17
  ^root@6.22.06  ~root7+tmva+mlp+davix+fits+fortran+mysql+sqlite+postgres+python+ssl+xrootd+fftw+pythia6+vmc cxxstd=17
  ^spdlog@1.8.2
  ^sqlite@3.32.03
  ^triton@2.3.0 ~cuda cxxstd=17
  ^wirecell@0.14.0 cxxstd=17
  ^xerces-c@3.2.2  cxxstd=17
  ^xrootd@4.12.3  cxxstd=17
  ^zlib@1.2.7
  ^mesa~llvm
```

[You'll notice there are a bunch of "mwm1" versions in there, those are using my patched forks on github.]

then you can concretize that spec:

```
spack spec --yaml `cat larsoft_spec.opts` > larsoft_spec.yaml
```

(note that if there were version conflicts, etc. at this point spack would yell at you)
This takes a few minutes.

## Installing the concretized spec

```
spack install --no-checksum -f larsoft_spec.yaml
```

In theory, this should just build; but this is where life is likely to get interesting...
