# Shifter usage for Cori

While NERSC's Cori system does support using Docker images directly,
users are encouraged to convert Docker images directly to Shifter
images, which are tailored to be efficient on HPC systems (see
[here](https://docs.nersc.gov/programming/shifter/overview/) for
details).

## Check if `larwire-kokkos` is available

To check if the Shifter image is available, type:

```console
$ shifterimg lookup hepcce2/larwire-kokkos:2.2
aa30d5698c59afd5c35dd36e75482e99165895f0c2888b6f752e407a39a65484
```

If the image is available, a checksum will be printed and the return
code (`$?`) will be 0, indicating success.  If the image is not
available, nothing will be printed to the screen and the return code
will be 1.

# Running an interactive `larwire-kokkos` container

In the same directory you checked out the `wire-cell-toolkit` repositories, type
```console
$ shifter --image=hepcce2/larwire-kokkos:2.2 <full-path-to>/PPSwork/Wire-Cell/entrypoint.sh
```
where the `entrypoint.sh` is provided from this repository.

At this point, you will be in an interactive Bash shell, allowing you
access to all `larwire` products (located in `/products`).

To gain access the Kokkos libraries within the container (located in /opt/kokkos_*), type:

```console
$ enable_kokkos_omp
 => or
$ enable_kokkos_cuda
```

Note that you will not be able to generally compile and execute with
the Kokkos/CUDA backend unless you are on a GPU node (see below).

# Accessing CUDA inside of the container

To use (or build with) CUDA inside the Shifter container, you must be
running on a GPU node.  To do this, type:

```console
$ module load cgpu
$ salloc -A m2845 -C gpu -N 1 --gres=gpu:1 --image=docker:hepcce2/larwire-kokkos:2.2 --module=gpu -t 60
 => Which should print out something like:
salloc: Granted job allocation 934891
salloc: Waiting for resource configuration
salloc: Nodes cgpu01 are ready for job
```

Once on the GPU node, type the following to start a container:

```console
$ srun -n 1 -c 1 --pty shifter <full-path-to>/PPSwork/Wire-Cell/entrypoint.sh
 => Takes a little time
```

Inside the container, type:

```
enable_kokkos_cuda
```

which will make available all CUDA executables and libraries supported
by CUDA 11.0.2 for either direct use or through Kokkos.
