# Compiling Wire-Cell against Kokkos



## Kokkos comipling options

- CUDA + OPENMP back-end:

```bash
srcdir=<path_to_kokkos_src>
path_to_install=<path_to_install_location>
path_to_compiler=<path_to_nvcc_wrapper>
cmake $srcdir \
  -DCMAKE_CXX_COMPILER=$path_to_compiler \
  -DCMAKE_INSTALL_PREFIX=$path_to_install \
  -DCMAKE_POSITION_INDEPENDENT_CODE=On \ # seems needed to build wcp shared lib
  -DKokkos_CXX_STANDARD=14 \
  -DKokkos_ENABLE_SERIAL=On \
  -DKokkos_ENABLE_OPENMP=On \
  -DKokkos_ENABLE_CUDA=On \
  -DKokkos_ARCH_TURING75=On \ # RTX 2080Ti
  -DKokkos_ENABLE_CUDA_LAMBDA=On \ # Needed
```

- OpenMP back-end

```bash
srcdir=<path_to_kokkos_src>
path_to_install=<path_to_install_location>
path_to_compiler=<path_to_g++>
cmake $srcdir \
  -DCMAKE_CXX_COMPILER=$path_to_compiler \
  -DCMAKE_INSTALL_PREFIX=$path_to_install \
  -DCMAKE_POSITION_INDEPENDENT_CODE=On \
  -DKokkos_CXX_STANDARD=14 \
  -DKokkos_ENABLE_SERIAL=On \
  -DKokkos_ENABLE_OPENMP=On \
```

## `wire-cell-gen-kokkos` Options

Corresponding modification of the `waf` based building system:
https://github.com/WireCell/waf-tools/pull/9



Usage: set `--kokkos-options=="cuda"` for Kokkos with cuda backend

```bash
./wcb configure --prefix=$INSTALL \
--with-tbb="$TBBROOT" \
--with-jsoncpp="$JSONCPP_FQ_DIR" \
--with-jsonnet="$JSONNET_FQ_DIR" \
--with-eigen-include="$EIGEN_DIR/include/eigen3" \
--with-root="$ROOTSYS" \
--with-fftw="$FFTW_FQ_DIR" \
--with-fftw-include="$FFTW_INC" \
--with-fftw-lib="$FFTW_LIBRARY" \
--with-fftwthreads="$FFTW_FQ_DIR" \
--boost-includes="$BOOST_INC" \
--boost-libs="$BOOST_LIB" \
--boost-mt \
--with-cuda="$CUDA_PATH/" \
--with-cuda-lib="$CUDA_PATH/lib64" \
--with-wct=$WIRECELL_FQ_DIR/ \
--with-wct-lib=$WIRECELL_LIB \
--with-kokkos=$KOKKOS_PATH/ \
--with-kokkos-include=$KOKKOS_INC/ \
--with-kokkos-lib=$KOKKOS_LIB/ \
--kokkos-options="cuda" \ # for Kokkos with cuda backend
--with-spdlog-lib=$SPDLOG_LIB \
--with-spdlog-include=$SPDLOG_INC \
```

**exercise 3 test results**:

- OMP

```bash
$./build/test_exercise_3_lib 
  Total size S = 4194304 N = 4096 M = 1024
  Computed result for 4096 x 1024 is 4194304.000000
  N( 4096 ) M( 1024 ) nrepeat ( 100 ) problem( 33.5954 MB ) time( 0.0813354 s ) bandwidth( 41.3048 GB/s )
```

- CUDA

```bash
$./build/test_exercise_3_lib 
  Total size S = 4194304 N = 4096 M = 1024
  Computed result for 4096 x 1024 is 4194304.000000
  N( 4096 ) M( 1024 ) nrepeat ( 100 ) problem( 33.5954 MB ) time( 0.0123353 s ) bandwidth( 272.352 GB/s )
```

