# Notes from porting of FastCaloSim/CUDA to Kokkos

## Build configurations
There are 3 ways to build (with CUDA/Kokkos)
* pure CUDA, where `CXX=nvcc`
* CUDA, built with Kokkos environment, where `CXX=nvcc_wrapper` but
`-DUSE_KOKKOS=Off`
* Kokkos w/ CUDA backend, where `CXX=nvcc_wrapper` and `-DUSE_KOKKOS=On`

When using the CUDA backend, Kokkos uses the `nvcc_wrapper` script
wrap nvcc / g++, providing extra options.

We need to pass separate lists of sources to cmake depending on
`USE_KOKKOS=On/Off`. This is because the build system / cmake treats different
extensions in different ways. Files with the `.cu` suffix will always be
built with nvcc.

If a file with a `.cu` suffix includes any Kokkos headers, even when
building with `USE_KOKKOS=On`, a compilation error will ensue. This is
one I often see:
```
/opt/kokkos/cuda/3.1-shlib/include/impl/Kokkos_Atomic_MinMax.hpp(117): error: no instance of overloaded function "atomicMin" matches the argument list
            argument types are: (unsigned long long *, const unsigned long long)
```
To avoid this, wrap the `FILE.cu` file in a `FILE.cxx` that merely
does a `#include FILE.cu`. This will ensure that the file is compiled
by `nvcc_wrapper` instead of `nvcc` directly.



## Shared libraries, relocatable code
building Kokkos with `-DBUILD_SHARED_LIBS=On` is incompatible with
`-DKokkos_ENABLE_CUDA_RELOCATABLE_DEVICE_CODE=ON` or
`-DKokkos_ENABLE_HIP_RELOCATABLE_DEVICE_CODE=On`, though the kokkos
build process will not complain about it. This means that if you want
to link as a shared library instead of a static one (which may be
required by a pre-existing build infrastructure), you can't use
relocatable device code. As a result, all symbols must be visible from
within the same compilation unit.

The only way to get this to work is to have one large file that has
all the (kokkos) code in it, or very carfully split up the files so
you don't call functions that aren't defined outside of the file.

A way to organize this is to keep the code in the
individual files as they currently exist, then wrap it with a single
file that merely `#include` all these files. Only tell the build
system about this file. See
[KokkosWrapper.cxx](https://github.com/meifeng/FastCaloSim-GPU/blob/kk/FastCaloSimAnalyzer/FastCaloGpu/src/KokkosWrapper.cxx).

## Moving memory between host and device
Kokkos would like to own/manage all the memory on both host and
device, by means of `View<>`s. However it's not neccessary to do it
this way, and often you don't have a choice when porting code, as the
memory may be allocated somewhere you have no control of. Kokkos
allows you to wrap Views around bare pointers, but then leaves it up
to you to manage the cleanup of the memory.

### Memory already allocated on host, want to transfer to device
```
Foo *ptr = new Foo[N];   // Array of N Foos is allocated on host, outside of your control

Kokkos::View<Foo*, Kokkos::HostSpace> host_view(ptr, N);  // wrap host ptr in a View

Kokkos::View<Foo*> dev_view("device view of Foo",N);      // allocate mem on device

Kokkos::deep_copy(dev_view, host_view);                   // copy host mem to device
```

you can use the same technique to copy back from the device to a
pre-allocated buffer on the host.

### Memory already allocated on device, use it in a kernel
```
Foo *dev_ptr = get_from_somewhere();               // retrieve previously allocated device ptr

Kokkos::View<Foo*> dev_view(dev_ptr,N);            // wrap device ptr in a view

Kokkos::parallel_for(N, KOKKOS_LAMBDA(int num) {   // use View in a kernel
        dev_view(num) += 10;
      });


// copy back to host
Kokkos::View<Foo*>::host_mirror host_view = create_mirror_view(dev_view);
Kokkos::deep_copy(host_view, dev_view);

```

## Data structures: jagged arrays

Kokkos doesn't like jagged arrays, ie where you have an array of
pointers, each of which points to an array, which are not all the same
size. One would think that this would translate into a
`View< View< Foo* >* >`, but that is not the case. See the notes in
the Kokkos manual about Views of Views. Some solutions are to
convert the structure into a padded 2D array (if the sizes of the
individual sub arrays are not too dissimilar), or flatten it into
singe 1D array, keeping track of where each of the sub-arrays
start. You can use `Kokkos::subviews<Foo>` to target the individual sub-arrays.

## File organization

Kokkos w/ CUDA backend will happily interoperate with CUDA code. One can call
CUDA kernels and functions from Kokkos code without issue. However this
limits portability, as you are then tied to the CUDA backend. If you wish
to make the Kokkos code portable, but still maintain the CUDA codebase, and
re-use as much code as possible, one solution is to extract all the C++
functions from the CUDA code, and save them in `.cxx` files which can be
included by both the CUDA and Kokkos wrappers. CUDA macros such as `__device__`
and `__host__` can be replaced with a wrapper that chooses the appropriate
functionality depending on whether a Kokkos or CUDA build is enabled. eg
```
#ifdef USE_KOKKOS
#  include <Kokkos_Core.hpp>
#  include <Kokkos_Random.hpp>
#  define __HOSTDEV__ KOKKOS_INLINE_FUNCTION
#else
#  ifdef __CUDACC__
#    define __HOSTDEV__ __host__ __device__
#  else
#    define __HOSTDEV__
#  endif
#endif
```
