# Notes from exploration of Kokkos with Patatrack code

## 2020-02-27 [makortel/pixel-standalone#34](https://github.com/makortel/pixel-standalone/pull/34)

- Even with the Inline Builds a static Kokkos library gets built
  * https://github.com/kokkos/kokkos#inline-builds-vs-installed-package
  * I did not find a way for a pure header-only build
- Need to decide at build configuration (Makefile generation) time which execution spaces (backends) to support
  * According to [documentation](https://github.com/kokkos/kokkos/wiki/Compiling#45-execution-space-restrictions), the current policy is to allow users to select at most one execution space from each of the following three categories
    - Host Serial: A top-level `Serial` execution space with no parallelism or concurrency
    - Host Parallel: Typically a threading model for CPUs, currently: `OpenMP`, `Threads`, and `QThreads`.
    - Device Parallel: Typically an attached GPU, currently: `CUDA`, `OpenMPTarget`, and `ROCm`.
  * Supporting multiple "device parallel" execution spaces dynamically in a single program will therefore be painful
    - Basically we would need one Kokkos library build for each device type `X`, that contains `Serial`, `TBB`, and `X`
    - In CMSSW we would need to build plugins for each `X` flavor linking to the Kokkos-`X`
  * Running a CPU version and a device version of an algorithm from a single program would still be possible, but two different different devices would (presumably) not be possible.
    - On the other hand, if I recall correctly, Kokkos supports exactly one device per process anyway, i.e. already two devices of the same type would not work
- If CUDA is enabled, need to build everything with Kokkos with `nvcc_wrapper` (or `nvcc`)
  * In fact the Kokkos' `Makefile.kokkos` (supposed to be included from the application Makefile) overrides `$(CXX)` to be `nvcc_wrapper` if CUDA is enabled, leading to **everything being built with `nvcc_wrapper`**. Furthermore the build flags contain a mixture of options for `g++` and `nvcc` that `nvcc_wrapper` takes care of, so it is not easy to just replace `nvcc_wrapper` with `g++` and `nvcc`.
  - I suppose it might be possible to craft one's own build flags and ignore `Makefile.kokkos`, but then one would have to maintain those rules.
- Need to have `nvcc` in `$PATH` for `nvcc_wrapper`

## 2020-03-26 [cms-patatrack/pixeltrack-standalone#6](https://github.com/cms-patatrack/pixeltrack-standalone/pull/6), [cms-patatrack/pixeltrack-standalone#7](https://github.com/cms-patatrack/pixeltrack-standalone/pull/7)

- Seems that Kokkos (3.0.00) library can be built only with CMake, see https://github.com/kokkos/kokkos/blob/3.0.00/BUILD.md and https://github.com/kokkos/kokkos/issues/2332
- Apparently can only give one GPU architecture, otherwise CMake complains
```
CMake Error at cmake/kokkos_arch.cmake:361 (MESSAGE):
  Multiple GPU architectures given! Already have KEPLER35, but trying to add
  PASCAL60.  If you are re-running CMake, try clearing the cache and running
  again.
```
  * Confirmed by Christian Trott in the Kokkos slack "No you can only compile for one at a time. Part of the problem is that we do special stuff under the hoods per architecture."

## 2020-03-27 [makortel/pixel-standalone#40](https://github.com/makortel/pixel-standalone/pull/40)

- Managed to make an example (following an earlier example in Alpaka) where the backend is chosen at run time.
- Whether or not to use CUDA is still only a configure-time choice (i.e. must be explicitly excluded from the build if one wants to run without a GPU), because Kokkos 3.0.00 does not allow initializing the execution spaces separatly.
  * Recently such mechanism has been added in `develop`, see [kokkos/kokkos#2658](https://github.com/kokkos/kokkos/issues/2658) and [kokkos/kokkos#2790](https://github.com/kokkos/kokkos/pull/2790).

## 2020-04-13 [cms-patatrack/pixeltrack-standalone#13](https://github.com/cms-patatrack/pixeltrack-standalone/pull/13)

- I verified that if the Kokkos runtime library is built with CUDA enabled, every file `#include`ing any Kokkos header must be compiled with `nvcc`
  * There are bare CUDA language extensions (`__device__` etc.) visible to the compiler
- `nvcc_wrapper` appears to not be strictly necessary, I was able to use `nvcc` directly
  * had to add `-Xcudafe --diag_suppress=esa_on_defaulted_function_ignored` to CUDA compilation flags
- Apparently there is a way to build a shared object of the Kokkos library, `-DBUILD_SHARED_LIBS=On` configure flag for CMake
  * See [kokkos/kokkos#2288](https://github.com/kokkos/kokkos/pull/2288) and [kokkos/kokkos#2283](https://github.com/kokkos/kokkos/issues/2283)
  * `-DBUILD_SHARED_LIBS=On` and `-DKokkos_ENABLE_CUDA_RELOCATABLE_DEVICE_CODE=On` together, however, lead to `nvcc` to complain from duplicate `--relocatable-device-code` and `-arch` arguments. Removing either CMake configure parameters fixes the error, but it seems that both are needed. Without `BUILD_SHARED_LIBS`, a static library is created instead of a shared, and turned out to be painful for linking the CUDA device code (symbols missing). Without `Kokkos_ENABLE_CUDA_RELOCATABLE_DEVICE_CODE` the `KOKKOS_ENABLE_CUDA_RELOCATABLE_DEVICE_CODE` macro is not defined in `KokkosCore_config.h`, which leads to some symbols being declared file-`static` instead of `extern`, which further leads to `nvcc -dlink` complains about duplicate symbols. This needs to be followed up with the Kokkos developers (I did not do that yet).
  * I patched the `nvcc_wrapper` to ignore the duplicate `--relocatable-device-code` and `-arch` arguments. The setup seems to work, at least.
- It seems that with the CUDA backend, the CPU version of the kernels are called via the same `__host__ __device__` functions that the `nvcc` compiles. This is actually very understandable given how the Kokkos system has been set up otherwise, but I think it was still worth to note it explicitly

## 2020-04-16 [kokkos/kokkos#2958](https://github.com/kokkos/kokkos/issues/2958)

I finally got to ask about the `-DBUILD_SHARED_LIBS=On` and `-DKokkos_ENABLE_CUDA_RELOCATABLE_DEVICE_CODE=On` problem described above, appears to be a genuine bug because Christian Trott suggested to open an issue. He also commented

> this is another thing I don't like about nvcc/cmake combination - cmake doesn't filter out duplicated arguments and nvcc doesn't like duplicated arguments
> 
> and the way multi library projects are designed you almost have to end up with duplicated arguments ...
>
> maybe patching up nvcc_wrapper is actually the right way to go

## 2020-05-08 [cms-patatrack/pixeltrack-standalone#29](https://github.com/cms-patatrack/pixeltrack-standalone/pull/29)

- Crafting SoA with `Kokkos::View` as columns feels laborious. The fact that write access and for read-only access need a different SoA definitions (`const` inside the column declaration or not) does not help. On the other hand, crafting a SoA with raw pointer types has the same nuisance.
- SoA with `Kokkos::View` as columns can not, in general, be passed to `KOKKOS_LAMBDA`, because the lambda body is executed in the device execution space. Instead, each column must be stored into a temporary variable that can then be passed to the lambda.
  * The SoA can be passed to lambda if all the members are either `Kokkos::View` or trivial, and the column accessors are declared as `KOKKOS_INLINE_LAMBDA`.
    - No, it can not. This approach works in CPU backends, but fails in CUDA with "illegal memory access".
- Can not `deep_copy()` Views of different size

## 2020-05-08 [cms-patatrack/pixeltrack-standalone#30](https://github.com/cms-patatrack/pixeltrack-standalone/pull/30)

- Nothing specific to report
- Second kernel was pretty quick to port

## 2020-05-11 [cms-patatrack/pixeltrack-standalone#33](https://github.com/cms-patatrack/pixeltrack-standalone/pull/33)

- Version update itself was seamless, except [the test on compiled CUDA version was fixed](https://github.com/kokkos/kokkos/pull/2861).
  * I can't anymore use Pascal 6.0 as the "lowest common denominator", but will have to bump it to Volta 7.0. Fortunately that covers all the current test machines.
    - Actually even that does not work if one runs on Turing (7.5), because Kokkos checks the target architecture and it seems to me that the compute capability minor version check is reversed. See [kokkos/kokkos#3026](https://github.com/kokkos/kokkos/issues/3026).
  * For a single build deployment this requirement would be painful. Especially given that `nvcc` itself is capable in building fat binaries (that CMS currently does, for 3.5, 6.0, and 7.0)
- Initializing only the selected backends was straightforward (after finding the issue in Kokkos GitHub that described the API)
- The need to give the execution space object as a parameter to `Kokkos::deep_copy()` was a bit surprising at first, but makes sense.
  * Note that, when given, it is used to call `fence()` on that execution space instead of global `Kokkos::fence()`, not for making the copy asynchronous
- The PR in the heading is closed, a small follow-up was done in [cms-patatrack/pixeltrack-standalone#37](https://github.com/cms-patatrack/pixeltrack-standalone/pull/37)
  * The rest will be done with Kokkos 3.2

## 2020-05-19 [makortel/pixel-standalone#43](https://github.com/makortel/pixel-standalone/pull/43)

- Managed to replace structs of arrays with structs of Kokkos Views:
  * Views not initialized before use will compile but result in run-time segmentation fault.
  * Only Views with run-time dimension can be used as member variables, compile time dimension will result in
```
terminate called after throwing an instance of 'std::runtime_error'
  what():  cudaDeviceSynchronize() error( cudaErrorIllegalAddress): an illegal memory access was encountered ...
Traceback functionality not available
```
- Applying structs of Views brings little change in terms of performance.

## 2020-05-20 [cms-patatrack/pixeltrack-standalone#47](https://github.com/cms-patatrack/pixeltrack-standalone/pull/47)

- PR [cms-patatrack/pixeltrack-standalone#46](https://github.com/cms-patatrack/pixeltrack-standalone/pull/46) failed to build with
  ```
  nvlink error   : Undefined reference to '_ZN6Kokkos4Impl25g_device_cuda_lock_arraysE' in '/home/mkortela/pixeltrack-standalone/obj/kokkos/test/kokkos/AtomicPairCounter_t.cc.cuda.o' (target: sm_35)
  ```
  The primary reason is that the Kokkos runtime library was missing from the CUDA device link step. Adding the shared object, however, does not help, because the device link [ignores shared objects](https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html#libraries). Therefore, the Kokkos runtime library must be built as a static librar.
  * I don't remember the exact hurdles I had [before](), but this time the use of the statically linked Kokkos runtime was straightforward to set up
- My next attempt was to build Kokkos as a static library (still with relocatable device code). That allowed [cms-patatrack/pixeltrack-standalone#46](https://github.com/cms-patatrack/pixeltrack-standalone/pull/46) to build and run, but then `./kokkos --cuda` fails with
  ```
  terminate called after throwing an instance of 'std::runtime_error'
    what():  cudaFuncGetAttributes( &attr, cuda_parallel_launch_local_memory<DriverType>) error( cudaErrorInvalidDeviceFunction): invalid device function .../pixeltrack-standalone/external/kokkos/install/include/Cuda/Kokkos_Cuda_KernelLaunch.hpp:440
  Traceback functionality not available
  ```
- We have had similar failures in CMSSW in some cases with multiple dynamic libraries with device code. Our current best guess for the cause is that multiple shared libraries define same symbol that CUDA uses to find device code (kernels), but we have not been able fully pinpoint the problem.
- My next attempt was to package all code using Kokkos into a single "plugin" dynamic library (in the PR of the title). This approach appears to work both for `kokkos` executable and the `AtomicPairCounter_t.cuda` unit test.
- Christian Trott confirmed that
  > that is the only way if you need relocatable device code
  >
  > a limitation in CUDA for which there is no workaround
  >
  > sorry
- And then we went to discuss whether relocatable device code is really needed. I need to try without.


## 2020-05-20 [cms-patatrack/pixeltrack-standalone#48](https://github.com/cms-patatrack/pixeltrack-standalone/pull/48)

- I managed to go back to "plugin" library per package by disabling the relocatable device code (keeping Kokkos as a static library)
  * Required some work on the build rules
- Below is a list of things that would need relocatable device code (thanks to Andrea Bocci)
  * call device functions defined in different translation unit
  * dynamic parallelism
  * cooperative groups


## 2020-06-09 [cms-patatrack/pixeltrack-standalone#56](https://github.com/cms-patatrack/pixeltrack-standalone/pull/56)

- Update of Kokkos to 3.1.1 went smoothly
- Now the target CUDA architecture needs to be specified explicitly when building (7.0 being the default)


## 2020-06-09 [cms-patatrack/pixeltrack-standalone#57](https://github.com/cms-patatrack/pixeltrack-standalone/pull/57)

- New attempt of [cms-patatrack/pixeltrack-standalone#3](https://github.com/cms-patatrack/pixeltrack-standalone/pull/33), this time no issues


## 2020-06-11 [cms-patatrack/pixeltrack-standalone#58](https://github.com/cms-patatrack/pixeltrack-standalone/pull/58)

- First prototype of the "device framework components for asynchronous execution"
- It seems to me that in order to use `Kokkos::Cuda` execution space as a replacement of CUDA stream, one has to manage the CUDA streams explicitly and wrap them to the `Kokkos::Cuda` objects.
  * It follows that I imported the `StreamCache` and `EventCache` from `cudatest` program, and basically manage all that stuff with my own "portable" abstractions and just return the `Kokkos::Cuda` object whenever I need to talk to Kokkos
  * In other words, Kokkos does not specifically help me to build what I want, but also does not get in the way.


## 2020-06-12 [cms-patatrack/pixeltrack-standalone#59](https://github.com/cms-patatrack/pixeltrack-standalone/pull/59)

- Porting `PixelCPEFastESProducer` went pretty smoothly
- Only (very minor) hurdle was that
```
template <typename T>
class SOAFrame {
public:
  SOAFrame() {};
private:
  T px, py, pz;
};
Kokkos::View<SOAFrame>
```
failed to compile
```
.../pixeltrack-standalone/src/kokkos/DataFormats/SOARotation.h(79): error: "SOAFrame<T>::SOAFrame() [with T=float]" provides no initializer for:
            const member "SOAFrame<T>::px [with T=float]"
            const member "SOAFrame<T>::py [with T=float]"
            const member "SOAFrame<T>::pz [with T=float]"
          detected during:
            instantiation of "SOAFrame<T>::SOAFrame() [with T=float]"
```
Defining the constructor as `SOAFrame() = default()` seemed to be sufficient to fix the compilation error.

## 2020-07-15 [cms-patatrack/pixeltrack-standalone#62](https://github.com/cms-patatrack/pixeltrack-standalone/pull/62)

- This includes porting most of the `HistoContainer` class from CUDA to Kokkos.
- The related unit tests were also ported: `HistoContainer_t`, `OneHistoContainer_t`, and `OneToManyAssoc_t`.
- This class included many CUDA specific warp-level operations that I replaced with with `Kokkos::parallel_scan` operations. 
- In addition, the class functions for these warp-ops typically recieved arrays called `ws` and there is the `HistoContainer::wsSize()` function (which returns zero for non-CUDA devices). These `ws` arrays appear to be related to the book-keeping needed while using these warp-ops to perform the scan. I don't think we need them in Kokkos if we use the `parallel_scan` instead.
- During the port it became clear that the unit tests were long CUDA kernels with many for-loops inside that used counting by grid and block sizes to ensure different threads operated on different data. Generally, I found it more efficient in Kokkos to divide these for-loops into separate `Kokkos::parallel_for` calls.
- Here is an example in `OneToManyAssoc_t` where the function `fillBulk` used on [line 82 in the CUDA version] (https://github.com/cms-patatrack/pixeltrack-standalone/blob/master/src/cuda/test/OneToManyAssoc_t.h#L82):
```C++
template <typename Assoc>
__global__ void fillBulk(AtomicPairCounter* apc, TK const* __restrict__ tk, Assoc* __restrict__ assoc, int32_t n) {
  int first = blockDim.x * blockIdx.x + threadIdx.x;
  for (int k = first; k < n; k += gridDim.x * blockDim.x) {
    auto m = tk[k][3] < MaxElem ? 4 : 3;
    assoc->bulkFill(*apc, &tk[k][0], m);
  }
}
```
- This method takes multiple on device objects and was being called inside a CUDA kernal. It immediately gets its global thread ID, then looks based on that ID. Many threads will do nothing because `k` will be greater than `n`. It made much more sense in Kokkos to do this on [line 122 in the Kokkos version](https://github.com/cms-patatrack/pixeltrack-standalone/blob/master/src/kokkos/test/kokkos/OneToManyAssoc_t.cc#L122):
```C++
template <typename Assoc, typename ExecSpace>
void fillBulk(Kokkos::View<AtomicPairCounter, ExecSpace> apc,
              Kokkos::View<uint16_t**, ExecSpace> const tk,
              Kokkos::View<Assoc, ExecSpace> assoc,
              const uint32_t& n,
              ExecSpace const& execSpace) {
  Kokkos::parallel_for(
      "fillBulk", Kokkos::RangePolicy<ExecSpace>(execSpace, 0, n), KOKKOS_LAMBDA(const int& i) {
        auto m = tk(i, 3) < MaxElem ? 4 : 3;
        // printf("01 i = %06d m = %06d tk(i,3) = %06d tk(i,0) = %06d\n",i,m,tk(i,3),tk(i,0));
        auto x = assoc().bulkFill(apc, &tk(i, 0), m);
        // printf("01  x = %06d\n",x);
      });
}
```
- This directly creates one thread per iteration of the loop with no complicated math necessary. 
- The CUDA `HistoContainer` uses the `blockPrefix()` function which on a non CUDA device defaults to a serial prefix scan implimentation on [line 79 of `prefixScan.h`](https://github.com/cms-patatrack/pixeltrack-standalone/blob/master/src/cuda/CUDACore/prefixScan.h#L79):
```C++
  co[0] = ci[0];
  for (uint32_t i = 1; i < size; ++i)
    co[i] = ci[i] + co[i - 1];
```
- So I replaced this call, which also was being executed inside a CUDA kernel and utilized counting based on Block/Thread sizes to avoid duplicate work, with a `Kokkos::parallel_scan` on [line 312 of the kokkos version of `HistoContainer`](https://github.com/cms-patatrack/pixeltrack-standalone/blob/master/src/kokkos/KokkosCore/HistoContainer.h#L312)
```C++
  template <typename Histo, typename ExecSpace>
  static KOKKOS_INLINE_FUNCTION void finalize(Kokkos::View<Histo, ExecSpace> histo, ExecSpace const& execSpace) {
    Kokkos::parallel_scan(
        "finalize",
        Kokkos::RangePolicy<ExecSpace>(execSpace, 0, Histo::totbins()),
        KOKKOS_LAMBDA(const int i, uint32_t& update, const bool final) {
          update += histo().off[i];
          if (final)
            histo().off[i] = update;
        });
  }
```

## 2020-07-20 [cms-patatrack/pixeltrack-standalone#61](https://github.com/cms-patatrack/pixeltrack-standalone/pull/61), [cms-patatrack/pixeltrack-standalone#67](https://github.com/cms-patatrack/pixeltrack-standalone/pull/67)
- Use of scratch memory is cryptic: 6744 bytes of scratch memory is required for `splitVerticesKernel` (https://github.com/cms-patatrack/pixeltrack-standalone/pull/67/files#diff-18407984c8e0d48ea474bf07a82fbb07R222), but `set_scratch_size(0, Kokkos::PerTeam(6744)` will result in runtime error shown below. I can reproduce the same error with 1 double-precision variable also. A temporary solution is always setting a memory space larger than required: i.e. `set_scratch_size(0, Kokkos::PerTeam(8192)` in this case.
```
terminate called after throwing an instance of 'std::runtime_error'
  what():  cudaMemcpyAsync(dst, src, n, cudaMemcpyDefault, instance.cuda_stream()) error( cudaErrorIllegalAddress): an illegal memory access was encountered
```
- The CUDA `blockPrefixScan` can be easily rewritten by using the team-wide `parallel_scan` (https://github.com/kokkos/kokkos/blob/master/core/unit_test/TestTaskScheduler.hpp#L369), but it's not supported yet in hierarchical parallelism. I've created an issue to track this (https://github.com/kokkos/kokkos/issues/3197). Similarly, rewriting `warpPrefixScan` looks feasible by using `parallel_for( Kokkos::ThreadVectorRange ...)` once it's ready.
- Atomic operations listed on the Kokkos Wiki page are not fully supported: `atomic_load`, `atomic_store`, `atomic_add_fetch`, etc. are not available to use. Someone filed an issue on this already: https://github.com/kokkos/kokkos/issues/3179. Kokkos doesn't provide an abstraction of CUDA `atomicInc(ptr, MAX)`, so the current implementation in Kokkos is not safe (https://github.com/cms-patatrack/pixeltrack-standalone/pull/67/files#diff-c2f9f345999d9e7a4b109432fae5c57aR64).
- CUDA specific functions like `__syncthreads_or` and `__syncthreads_and` requires some maunal efforts: Instead of `while (__syncthreads_or(more)) { ... }`, one can use `while(more) { ... }` and create `uint16_t lmore[teamSize]` thus each thread updates `lmore[teamRank]` only. In the end, a team-wide `parallel_scan` is performed over `lmore` and stores the result into `more`. The CUDA implementation declares `more` as `bool` or `uint8_t`. In Kokkos we use `uint16_t` instead since reduction over `uint8_t` will result in overflows if teamSize = 256 or larger.
- `continue` is not allowed in parallel_for + TeamThreadRange

## 2020-08-02 [cms-patatrack/pixeltrack-standalone#70](https://github.com/cms-patatrack/pixeltrack-standalone/pull/70)
- Porting fitting unit tests was straightforward
- Use RangePolicy instead of TeamPolicy whenever possible for code simplicity
- An elegant way to `memset` Kokkos Views: `Kokkos::deep_copy(KokkosExecSpace(), d_data, 0)`
  * not always working, e.g., an initialization loop is required for `Rfit::line_fit` which is a `struct` of `Eigen::Matrix2d` and `Eigen::Vector2d`

## 2020-08-04 [cms-patatrack/pixeltrack-standalone#72](https://github.com/cms-patatrack/pixeltrack-standalone/pull/72)

- Christian Trott suggested to replace `__constant__` CUDA variables with functor member variables (assuming they are not "too big". Those should end up in the constant memory.

## 2020-08-07 [cms-patatrack/pixeltrack-standalone#75](https://github.com/cms-patatrack/pixeltrack-standalone/pull/75)

- I'm starting to think that maybe for the pattern
  ```cpp
  auto first = blockIdx.x * blockDim.x + threadIdx.x;
  for (int idx = first, ; idx < n; idx += gridDim.x * blockDim.x) {
    if (something(idx)) {
      break;
    }
    ...
  }
  ```
  it would be better to use `Kokkos::TeamPolicy` instead of
  `Kokkos::RangePolicy`. That would allow actually doing a `break`
  instead of effective `continue`. Maybe not worth to do now, but
  something to keep in mind.

## 2020-08-18 Porting Note
- Came across this pattern in the `findClus` algorithm:
```cpp
while(__syncthreads_or(flag)){
  if(...){flag = true;}
  else{flag =false;}
}
```
- After consulting Christian Trott he suggested this pattern which seemed useful to document:
```cpp
int flag = 1;
while(flag) {
  if(...) flag = 1;
  else flag = 0;
  team.team_reduce(Kokkos::Sum<typeof(flag)>(flag));
}
```

## 2020-09-04 [cms-patatrack/pixeltrack-standalone#76](https://github.com/cms-patatrack/pixeltrack-standalone/pull/76)
- Constant-size arrays as member variables of the functor class will likely utilize constant memory (`__constant__`) on CUDA devices.
- Kokkos provides `Kokkos::TeamPolicy<Kokkos::LaunchBounds<MaxThreads,MinBlocks>>` to handle the CUDA `__launch_bounds__` qualifier.
- One generic solution for all backends? The key idea of coding in Kokkos is to have one parallel solution that can run properly on all backends including CUDA, Serial, OpenMP, etc. Current parallel solutions in patatrack are based on CUDA implementations thus may not be generic enough in all use cases. For example, the original CUDA implementation may hardcode the kernel launch parameter as 128 blocks with 1024 threads per block. Apparently there won't be so many hardware threads/cores supported on CPUs. In this case, we need to make sure the underlying algorithm doesn't depend on specific kernel launch parameters thus can be applied in all backends.

## 2020-09-15 [cms-patatrack/pixeltrack-standalone#90](https://github.com/cms-patatrack/pixeltrack-standalone/pull/90)

Enabling RecHit module demonstarted, again, the need to use `#ifdef`
to specify 1 thread/team for Serial backend. In addition, for some
reason, the use of `assert()` inside a specific `parallel_for()` lead to
```
Caught std::runtime_error
cudaFuncGetAttributes( &attr, cuda_parallel_launch_local_memory<DriverType>) error( cudaErrorInvalidDeviceFunction): invalid device function .../pixeltrack-standalone/external/kokkos/install/include/Cuda/Kokkos_Cuda_KernelLaunch.hpp:448
Traceback functionality not available
```

For future reference below are listed other attempts to fix or understand the issue
* Packaging all plugins into a single plugin (essentially reverting 57eb24233c49fd2e2bca90ebab1ca8d9433a1479 of #48) did not help
* Packaging all code using Kokkos into a single plugin did not help
* Artificially avoiding loading `pluginSiPixelClusterizer.so` did not help
* Commenting out the kernels using `TeamPolicy` did not help
* Using CUDA 11.0.2 instead CUDA 10.2 did not help
* Was not able to reproduce in a unit test executable with exactly the same `parallel_for()` loop

## 2020-09-21 [cms-patatrack/pixeltrack-standalone#92](https://github.com/cms-patatrack/pixeltrack-standalone/pull/92)
- Use `const` and `__restrict__` qualifiers for pointers to replace the CUDA device-only intrinsic: `__ldg`.
- Kokkos scratch memory (shared memory): allocating the exact number of bytes for target data may result in the error shown below: 
  ```
  terminate called after throwing an instance of 'std::runtime_error'
    what():  cudaFuncGetAttributes( &attr, cuda_parallel_launch_local_memory<DriverType>) error( cudaErrorIllegalAddress): an illegal memory access was encountered 
  ```
  Adding extra 4 or 8 bytes during allocation can solve this issue (mentioned in the `2020-07-20` notes as well).
- `Kokkos::View` as member variables cannot be directly passed to Kokkos lambda functions since `this` pointer is in the host memory. A local copy of all member `View`s is required before `parallel_for`.

## 2020-10-22 Profiling Note
- Adding labels for `Kokkos::View` and parallel algorithms like `Kokkos::parallel_for` are **important** for profiling. [kokkos-tools](https://github.com/kokkos/kokkos-tools) will use these labels to name NVTX markers thus help kernel identification in `nsight-sys` or `nvvp`.
- An example use case of [kokkos-tools](https://github.com/kokkos/kokkos-tools) with profiler (on Cori GPU):
  ```bash
  export KOKKOS_PROFILE_LIBRARY=/PATH/TO/kokkos-tools/kp_nvprof_connector.so
  # Nsight Systems
  srun nsys profile ./kokkos --cuda --maxEvents 10
  # nvprof
  srun nvprof -o kokkos.nvvp ./kokkos --cuda --maxEvents 10
  ```
  where the `*.qdrep` file generated by `nsys` can be opened by `nsight-sys` and `kokkos.nvvp` is visualized via `nvvp`.
- `Nsight Systems` and `nvprof` provide similar profiling information, but `Nsight Systems` is more user-friendly due to better zoom support.
- Unlike `Intel Advisor` or `VTune`, `nsight-sys` and `nvvp` cannot link profiling result to its source code so kernel indentification (even with NVTX markers) requires extra manual efforts. For example, `Kokkos::View` initialization is only marked as `Kokkos::View::Initialization` in NVTX and it provides no information like `View` data type or which file/line it corrresponds to.
- On Cori GPU, one can also use the prebuilt [timemory](https://github.com/NERSC/timemory) library for Kokkos profiling:
  ```bash
  module load cgpu
  module load timemory/cgpu-develop-gcc-py37-openmpi
  export KOKKOS_PROFILE_LIBRARY=/global/common/cori_cle7/software/timemory/cray-cnl7-haswell/view/cgpu-develop-gcc-py37-openmpi/lib64/timemory/kokkos-tools/kp_timemory.so
  srun ./kokkos --cuda --maxEvents 10
  ```
  the output folder is `timemory-*-output` and default profiling components are `wall_clock`, `peak_rss` and `kokkos_memory`. Use `timemory-avail` to check all available components and ```export TIMEMORY_KOKKOS_COMPONENTS="..."``` to customize each use case if needed. The `cupti_counters` component is set via ```export TIMEMORY_CUPTI_EVENTS="..."``` where valid entries can be obtained via ```srun timemory-avail -H```.

## 2020-10-22 [cms-patatrack/pixeltrack-standalone#114](https://github.com/cms-patatrack/pixeltrack-standalone/pull/114)
- Profiling of the naive Kokkos implementation shows that `Kokkos::View` initialization takes about 80% of overall kernel run time. Allocating `View` without initializing can get rid of this issue:
  ```cpp
  Kokkos::View<int*> a (Kokkos::ViewAllocateWithoutInitializing ("label"), num);
  ```
  By doing this, the throughput is improved from 12 events/s to 35 events/s on NVIDIA Tesla V100.


## 2020-11-03 [cms-patatrack/pixeltrack-standalone#116](https://github.com/cms-patatrack/pixeltrack-standalone/pull/116)

- Generalized the build rules and added a possibility to use the pthreads backend
- The pthreads backend is fundamentally incompatible of using TBB tasks to run the event loop. It leads to sporadic
  ```
  ThreadsExec::start FAILED : Called by a worker thread, can only be called by the master process.
  ```
  being thrown at [core/src/Threads/Kokkos_ThreadsExec.cpp#L269-L275](https://github.com/kokkos/kokkos/blob/3.1.01/core/src/Threads/Kokkos_ThreadsExec.cpp#L269-L275) following a check in [core/src/Threads/Kokkos_ThreadsExec_base.cpp#L118-L122](https://github.com/kokkos/kokkos/blob/3.1.01/core/src/Threads/Kokkos_ThreadsExec_base.cpp#L118-L122). I.e. the pthreads backend can only be used from one thread during the job, which will not work with TBB which can (and will) change worker threads.
- `gcc` directly appeared to be more picky with some warnings/errors than `gcc`-via-`nvcc`
- Many uses of thread teams had to be generalized
- Apparently using 1 thread/team gives better performance than >1 thread/team
- Apparently pinning the threads to the same NUMA node is needed for the best performance

## 2020-11-03 [cms-patatrack/pixeltrack-standalone#119](https://github.com/cms-patatrack/pixeltrack-standalone/pull/119)

- The idea is to mimic a team-wide prefix scan with a usual "grid-wide" prefix scan, and some post-processing
  * record the value of the last bin for each block
  * subtract the value of last bin of the "previous block" from each value of the current block
- Improved the throughput for
  * CUDA backend: from 37 events/s to 92 events/s on NVIDIA Tesla V100
  * Threads backend: from 22 events/s to 26 events/s (both peak with 7 threads)


## 2020-11-13 [cms-patatrack/pixeltrack-standalone#134](https://github.com/cms-patatrack/pixeltrack-standalone/pull/134)

- The reference CUDA code update included `__threadfence()`. Poking around in Kokkos source showed that `Kokkos::memory_fence()` does that in a portable way, but I'm uncertain if that is guaranteed to work.
- Improved the throughput for
  * Serial backend: from 13 events/s to 19 events/s
  * Threads backend: from 26 events/s (on 7 threads) to 46 events/s (on 13 threads)
  * CUDA backend: throughput stayed at 92 events/s

## 2020-11-18 [cms-patatrack/pixeltrack-standalone#135](https://github.com/cms-patatrack/pixeltrack-standalone/pull/135)

- In the end I merged [cms-patatrack/pixeltrack-standalone#134](https://github.com/cms-patatrack/pixeltrack-standalone/pull/134) too quickly, when building only the serial backend, running `./kokkos --serial --maxEvents 1` fails with
  ```
  kokkos: /home/mkortela/pixeltrack-standalone/src/kokkos/plugin-PixelTriplets/kokkos/../GPUCACell.h:326: void GPUCACell::find_ntuplets(const Hits&, GPUCACell*, GPUCACell::CellTracksVector&, GPUCACell::HitContainer&, cms::kokkos::AtomicPairCounter&, GPUCACell::Quality*, GPUCACell::TmpTuple&, unsigned int, bool) const: Assertion `tmpNtuplet.size() <= 4' failed.
  ```
  Curiously when building Serial+CUDA backends, the same command works.

- The problem appears to be caused by the two `Kokkos::atomic_compare_exchange()` in `GPUCACell::addOuterNeighbor()` and `GPUCACell::addTrack()`, more specifically the first one does not seem to actually exchange the content. Doing pure assignment
  ```cpp
  theOuterNeighbors = &cellNeighbors[i];
  ```
  does work, as does addition of an explicit fence
  ```cpp
  std::atomic_thread_fence(std::memory_order_seq_cst);
  ```
  after the `Kokkos::atomic_compare_exchange()`.
- I digged a bit further by compiling an affected translation unit with `-E` (to see the preprocessor output)
  * If Kokkos runtime is build with Serial backend only, the `atomic_compare_exchange()` is defined as
    ```cpp
    template <typename T>
    inline T atomic_compare_exchange(volatile T* const dest_v,
                                     const T compare, const T val) {
      T* dest = const_cast<T*>(dest_v);
      T retval = *dest;
      if (retval == compare) *dest = val;
      return retval;
    }

    ```
  * If Kokkos runtime is built with Serial+Threads backend, the `atomic_compare_exchange()` is defined as
    ```cpp
    template <typename T>
    inline T atomic_compare_exchange(
        volatile T* const dest, const T& compare,
        typename std::enable_if<sizeof(T) != sizeof(int) &&
                                    sizeof(T) == sizeof(long),
                                const T&>::type val) {
      union U {
        long i;
        T t;
        inline U() {}
      } tmp;
      tmp.i = __sync_val_compare_and_swap((long*)dest, *((long*)&compare),
                                          *((long*)&val));
      return tmp.t;
    }
    ```
- I was not able to reproduce the behavior with a simple test program

## 2020-11-18 [cms-patatrack/pixeltrack-standalone#136](https://github.com/cms-patatrack/pixeltrack-standalone/pull/136)

- Update to Kokkos 3.2 was smooth
- Improved the throughput for
  * Serial backend: stayed at 19 events/s
  * Threads backend: from 46 events/s (on 11 threads) to 45 events/s (on 16 threads), i.e. practically the same
  * CUDA backend: from 92 events/s to 110 events/s

## 2020-11-25 [cms-patatrack/pixeltrack-standalone#139](https://github.com/cms-patatrack/pixeltrack-standalone/pull/139)

- I realized that plain `--numberOfStreams 2` implies two threads because of `tbb::task::enqueue()` call used to enqueue TBB tasks for other than the first EDM stream. TBB will, in practice, start a second thread to guarantee that the enqueued work eventually completes.
  * So all observed failures with `--numberOfStreams 2` are, in fact, also caused by calling Kokkos from two threads
- Impact on throughput
  * Serial backend: stayed at 19 events/s
  * Threads backend: stayed at 45 events/s (on 16 threads)
  * CUDA backend: stayed at 110 events/s

## 2020-12-11 [cms-patatrack/pixeltrack-standalone#148](https://github.com/cms-patatrack/pixeltrack-standalone/pull/148)

- While working on [cms-patatrack/pixeltrack-standalone#152](https://github.com/cms-patatrack/pixeltrack-standalone/pull/152) I noticed he Eigen version was actually wrong. Need to check the impact of the update on physics and technical performance (should be none)

## 2020-12-28 [cms-patatrack/pixeltrack-standalone#149](https://github.com/cms-patatrack/pixeltrack-standalone/pull/149)

- Update to Kokkos 3.3 was smooth
- Improved the throughput for
  * Serial backend: from 19 events/s to 20 events/s, i.e. practically the same
  * Threads backend: from 45 events/s (on 16 threads) to 45 events/s (on 18 threads), i.e. practically the same
  * CUDA backend: from 110 events/s to 120 events/s, i.e. practically the same

## 2021-01-05 [cms-patatrack/pixeltrack-standalone#152](https://github.com/cms-patatrack/pixeltrack-standalone/pull/152)

- HIP requires devtoolset-7
- Kokkos 3.2 lead to compilation failures with HIP, updated to 3.3 release candidate, which compiled, but link failed
  ```
  .../ld: .../pixeltrack-standalone/external/kokkos/install/lib/libkokkoscore.a(Kokkos_Core.cpp.o): relocation R_X86_64_32 against `.bss' can not be used when making a shared object; recompile with -fPIC
  .../ld: .../pixeltrack-standalone/external/kokkos/install/lib/libkokkoscore.a(Kokkos_Error.cpp.o): relocation R_X86_64_32 against symbol `_ZTISt13runtime_error@@GLIBCXX_3.4' can not be used when making a shared object; recompile with -fPIC
  .../ld: .../pixeltrack-standalone/external/kokkos/install/lib/libkokkoscore.a(Kokkos_Profiling.cpp.o): relocation R_X86_64_32 against `.bss' can not be used when making a shared object; recompile with -fPIC
  .../ld: .../pixeltrack-standalone/external/kokkos/install/lib/libkokkoscore.a(Kokkos_Serial.cpp.o): relocation R_X86_64_32 against `.data' can not be used when making a shared object; recompile with -fPIC
  .../ld: .../pixeltrack-standalone/external/kokkos/install/lib/libkokkoscore.a(Kokkos_SharedAlloc.cpp.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
  .../ld: .../pixeltrack-standalone/external/kokkos/install/lib/libkokkoscore.a(Kokkos_HIP_Space.cpp.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
  .../ld: .../pixeltrack-standalone/external/kokkos/install/lib/libkokkoscore.a(Kokkos_HostSpace.cpp.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
  .../ld: .../pixeltrack-standalone/external/kokkos/install/lib/libkokkoscore.a(Kokkos_HostSpace_deepcopy.cpp.o): relocation R_X86_64_32 against `.hipFatBinSegment' can not be used when making a shared object; recompile with -fPIC
  .../ld: .../pixeltrack-standalone/external/kokkos/install/lib/libkokkoscore.a(Kokkos_HostThreadTeam.cpp.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
  .../ld: .../pixeltrack-standalone/external/kokkos/install/lib/libkokkoscore.a(Kokkos_MemorySpace.cpp.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
  .../ld: .../pixeltrack-standalone/external/kokkos/install/lib/libkokkoscore.a(Kokkos_HIP_Instance.cpp.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
  .../ld: .../pixeltrack-standalone/external/kokkos/install/lib/libkokkoscore.a(Kokkos_HIP_Locks.cpp.o): relocation R_X86_64_32 against symbol `_ZN6Kokkos4Impl22g_host_hip_lock_arraysE' can not be used when making a shared object; recompile with -fPIC
  .../ld: .../pixeltrack-standalone/external/kokkos/install/lib/libkokkoscore.a(Kokkos_HostBarrier.cpp.o): relocation R_X86_64_32 against `.hipFatBinSegment' can not be used when making a shared object; recompile with -fPIC
  .../ld: final link failed: Nonrepresentable section on output
  ```
  Does this mean that with HIP backend the Kokkos runtime library needs to be built as a shared library (while with CUDA we need a static library?)
- Building Kokkos runtime as a shared library indeed worked around of those problems. Next problem (still link) problem is
  ```
  .../hipcc .../pixeltrack-standalone/obj/kokkostest/plugin-Test1/IntESProducer.cc.o  .../pixeltrack-standalone/obj/kokkostest/plugin-Test1/kokkos/kokkosAlgo1.cc.serial.o .../pixeltrack-standalone/obj/kokkostest/plugin-Test1/kokkos/TestProducer.cc.serial.o .../pixeltrack-standalone/obj/kokkostest/plugin-Test1/kokkos/kokkosAlgo1.cc.device.o .../pixeltrack-standalone/obj/kokkostest/plugin-Test1/kokkos/TestProducer.cc.device.o -O2 -fPIC -pthread -Wl,-E -lstdc++fs -shared -Wl,-z,defs -L.../pixeltrack-standalone/lib/kokkostest -ldl -Wl,-rpath,.../pixeltrack-standalone/lib/kokkostest  -lFramework -lDataFormats -lKokkosCore  -L.../pixeltrack-standalone/external/tbb/lib -ltbb -L.../pixeltrack-standalone/external/kokkos/install/lib -lkokkoscore -ldl  -L.../pixeltrack-standalone/lib/kokkostest -o .../pixeltrack-standalone/lib/kokkostest/pluginTest1.so
  .../pixeltrack-standalone/obj/kokkostest/plugin-Test1/kokkos/TestProducer.cc.serial.o: In function `_GLOBAL__sub_I_TestProducer.cc':
  TestProducer.cc:(.text.startup+0xc1): undefined reference to `edm::PluginFactory::impl::Registry::add(std::string const&, std::unique_ptr<edm::PluginFactory::impl::MakerBase, std::default_delete<edm::PluginFactory::impl::MakerBase> >)'  .../pixeltrack-standalone/obj/kokkostest/plugin-Test1/kokkos/TestProducer.cc.device.o: In function `_GLOBAL__sub_I_TestProducer.cc':
  TestProducer.cc:(.text.startup+0xc1): undefined reference to `edm::PluginFactory::impl::Registry::add(std::string const&, std::unique_ptr<edm::PluginFactory::impl::MakerBase, std::default_delete<edm::PluginFactory::impl::MakerBase> >)'
  ```
- With a handcrafted build command, I built the `Framework` and `DataFormats` with `hipcc`, after which the link above succeeded. So I can't mix `g++` and `hipcc` that much?
  * I also tried to build all externals with `hipcc`, but building TBB failed because of unknown compiler parameters. I did not investigate that further.
  * Building and linking `Framework` only with `hipcc` was sufficient for the link aboce to succeed.
- Inspecting the `Framework` shared object files with `nm` show a difference in the symbol names
  * `gcc`: `_ZN3edm13PluginFactory4impl8Registry3addERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt10unique_ptrINS1_9MakerBaseESt14default_deleteISC_EE`
  * `hipcc`: `_ZN3edm13PluginFactory4impl8Registry3addERKSsSt10unique_ptrINS1_9MakerBaseESt14default_deleteIS6_EE`
- Diving deeply in `hipcc` I've tried the following but they had no effect
  * Dropping `-std=c++11` from the call to clang (leaving only `c++17`)
  * Explicitly specifying `-stdlib=libstdc++` (gives the same symbol as `-stdlib=libc++`)
- Charles rebuilt the gcc-8 making sure `ld` is built as well, and using the hipcc-clang to do the build, and that worked
- Had to move `approx_atan2.h` from `DataFormats` to `KokkosDataFormats` because two `constexpr` functions call `std::round` which is not `constexpr` in the standard, and not `constexpr` in the HIP math library
  * `libstdc++` declares `std::round` as `constexpr`, I guess `nvcc --expt-relaxed-constexpr` just streats `constexpr` functions as `__host__ __device__`
- Had to fix issues with `this` pointer being captured by lambdas passed to Kokkos
- Getting compiler errors
  ```
  In file included from .../pixeltrack-standalone/external/kokkos/install/include/Kokkos_View.hpp:501:
  .../pixeltrack-standalone/external/kokkos/install/include/impl/Kokkos_ViewMapping.hpp:2851:7: error: reference to __host__ function 'operator new' in __host__ __device__ function
        new (ptr + i) ValueType();
  ``` 
  * Started to take the approach of decorating more Eigen functions with `EIGEN_DEVICE_FUNC`
- That got me to be able to compile the code, then run into crash when running with `--hip`
  ```
  :0:rocdevice.cpp            :2296: 490464232399 us: Device::callbackQueue aborting with status: 0x100f
  
  Thread 2 "kokkos" received signal SIGABRT, Aborted.
  Thread 2 (Thread 0x7fffed12e700 (LWP 235054)):
  #0  0x00007ffff5bb9387 in raise () from /lib64/libc.so.6
  #1  0x00007ffff5bbaa78 in abort () from /lib64/libc.so.6
  #2  0x00007ffff4dd882b in ?? () from /opt/rocm-3.10.0/lib/libamdhip64.so.3
  #3  0x00007ffff46476bf in rocr::AMD::AqlQueue::DynamicScratchHandler(long, void*) () from /opt/rocm-3.10.0/lib/libhsa-runtime64.so.1
  #4  0x00007ffff467c69b in rocr::core::Runtime::AsyncEventsLoop(void*) () from /opt/rocm-3.10.0/lib/libhsa-runtime64.so.1
  #5  0x00007ffff462d6c7 in rocr::os::ThreadTrampoline(void*) () from /opt/rocm-3.10.0/lib/libhsa-runtime64.so.1
  #6  0x00007ffff5f58ea5 in start_thread () from /lib64/libpthread.so.0
  #7  0x00007ffff5c8196d in clone () from /lib64/libc.so.6
  
  Thread 1 (Thread 0x7ffff7fd5880 (LWP 235046)):
  #0  0x00007ffff5c66777 in sched_yield () from /lib64/libc.so.6
  #1  0x00007ffff4db0605 in ?? () from /opt/rocm-3.10.0/lib/libamdhip64.so.3
  #2  0x00007ffff4dbda20 in ?? () from /opt/rocm-3.10.0/lib/libamdhip64.so.3
  #3  0x00007ffff4ce3b4a in ?? () from /opt/rocm-3.10.0/lib/libamdhip64.so.3
  #4  0x00007ffff4ce75af in hipFree () from /opt/rocm-3.10.0/lib/libamdhip64.so.3
  #5  0x00007ffee434176b in Kokkos::Experimental::Impl::HIPParallelLaunch<Kokkos::Impl::ParallelFor<kokkos_hip::pixelgpudetails::SiPixelRawToClusterGPUKernel::makeClustersAsync(bool, Kokkos::View<SiPixelFedCablingMapGPU const, Kokkos::Experimental::HIP> const&, Kokkos::View<unsigned char const*, Kokkos::Experimental::HIP> const&, SiPixelGainForHLTonGPU<Kokkos::Experimental::HIP> const&, kokkos_hip::pixelgpudetails::SiPixelRawToClusterGPUKernel::WordFedAppender const&, std::map<unsigned int, std::vector<SiPixelRawDataError, std::allocator<SiPixelRawDataError> >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::vector<SiPixelRawDataError, std::allocator<SiPixelRawDataError> > > > >&&, unsigned int, unsigned int, bool, bool, bool, Kokkos::Experimental::HIP const&)::{lambda(unsigned long)#1}, Kokkos::RangePolicy<Kokkos::Experimental::HIP, Kokkos::Schedule<Kokkos::Static>, void, unsigned int, void, Kokkos::LaunchBounds<0u, 0u>, Kokkos::Experimental::WorkItemProperty::ImplWorkItemProperty<1ul>, Kokkos::Experimental::MaximizeOccupancy>, Kokkos::Experimental::HIP>, Kokkos::LaunchBounds, (Kokkos::Experimental::Impl::HIPLaunchMechanism)4>::HIPParallelLaunch(Kokkos::RangePolicy<Kokkos::Experimental::HIP, Kokkos::Schedule<Kokkos::Static>, void, unsigned int, void, Kokkos::LaunchBounds<0u, 0u>, Kokkos::Experimental::WorkItemProperty::ImplWorkItemProperty<1ul>, Kokkos::Experimental::MaximizeOccupancy> const&, dim3 const&, dim3 const, int, Kokkos::Experimental::Impl::HIPInternal const*, bool) (this=0x7fffffffafb8, driver=..., grid=..., block=..., shmem=0, hip_instance=0x70ba40)
      at .../pixeltrack-standalone/external/kokkos/install/include/HIP/Kokkos_HIP_KernelLaunch.hpp:216
  #6  0x00007ffee4341190 in Kokkos::Impl::ParallelFor<kokkos_hip::pixelgpudetails::SiPixelRawToClusterGPUKernel::makeClustersAsync(bool, Kokkos::View<SiPixelFedCablingMapGPU const, Kokkos::Experimental::HIP> const&, Kokkos::View<unsigned char const*, Kokkos::Experimental::HIP> const&, SiPixelGainForHLTonGPU<Kokkos::Experimental::HIP> const&, kokkos_hip::pixelgpudetails::SiPixelRawToClusterGPUKernel::WordFedAppender const&, std::map<unsigned int, std::vector<SiPixelRawDataError, std::allocator<SiPixelRawDataError> >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::vector<SiPixelRawDataError, std::allocator<SiPixelRawDataError> > > > >&&, unsigned int, unsigned int, bool, bool, bool, Kokkos::Experimental::HIP const&)::{lambda(unsigned long)#1}, Kokkos::RangePolicy<Kokkos::Experimental::HIP, Kokkos::Schedule<Kokkos::Static>, void, unsigned int, void, Kokkos::LaunchBounds<0u, 0u>, Kokkos::Experimental::WorkItemProperty::ImplWorkItemProperty<1ul>, Kokkos::Experimental::MaximizeOccupancy>, Kokkos::Experimental::HIP>::execute() const (
      this=0x7fffffffb010) at .../pixeltrack-standalone/external/kokkos/install/include/HIP/Kokkos_HIP_Parallel_Range.hpp:120
  #7  0x00007ffee4340f30 in Kokkos::parallel_for<Kokkos::RangePolicy<Kokkos::Experimental::HIP, Kokkos::Schedule<Kokkos::Static>, void, unsigned int, void, Kokkos::LaunchBounds<0u, 0u>, Kokkos::Experimental::WorkItemProperty::ImplWorkItemProperty<1ul>, Kokkos::Experimental::MaximizeOccupancy>, kokkos_hip::pixelgpudetails::SiPixelRawToClusterGPUKernel::makeClustersAsync(bool, Kokkos::View<SiPixelFedCablingMapGPU const, Kokkos::Experimental::HIP> const&, Kokkos::View<unonGPU<Kokkos::Experimental::HIP> const&, kokkos_hip::pixelgpudetails::SiPixelRawToClusterGPUKernel::WordFedAppender const&, std::map<unsigned int, std::vector<SiPixelRawDataError, std::allocator<SiPixelRawDataError> >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::vector<SiPixelRawDataError, std::allocator<SiPixelRawDataError> > > > >&&, unsigned int, unsigned int, bool, bool, bool, Kokkos::Experimental::HIP const&)::{lambda(unsigned long)#1}> (policy=..., functor=..., str=...)
      at .../pixeltrack-standalone/external/kokkos/install/include/Kokkos_Parallel.hpp:171
  #8  0x00007ffee4340459 in Kokkos::parallel_for<Kokkos::RangePolicy<Kokkos::Experimental::HIP, Kokkos::Schedule<Kokkos::Static>, void, unsigned int, void, Kokkos::LaunchBounds<0u, 0u>, Kokkos::Experimental::WorkItemProperty::ImplWorkItemProperty<1ul>, Kokkos::Experimental::MaximizeOccupancy>, kokkos_hip::pixelgpudetails::SiPixelRawToClusterGPUKernel::makeClustersAsync(bool, Kokkos::View<SiPixelFedCablingMapGPU const, Kokkos::Experimental::HIP> const&, Kokkos::View<unsigned char const*, Kokkos::Experimental::HIP> const&, SiPixelGainForHLTonGPU<Kokkos::Experimental::HIP> const&, kokkos_hip::pixelgpudetails::SiPixelRawToClusterGPUKernel::WordFedAppender const&, std::map<unsigned int, std::vector<SiPixelRawDataError, std::allocator<SiPixelRawDataError> >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::vector<SiPixelRawDataError, std::allocator<SiPixelRawDataError> > > > >&&, unsigned int, unsigned int, bool, bool, bool, Kokkos::Experimental::HIP const&)::{lambda(unsigned long)#1}>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Kokkos::RangePolicy<Kokkos::Experimental::HIP, Kokkos::Schedule<Kokkos::Static>, void, unsigned int, void, Kokkos::LaunchBounds<0u, 0u>, Kokkos::Experimental::WorkItemProperty::ImplWorkItemProperty<1ul>, Kokkos::Experimental::MaximizeOccupancy> const&, kokkos_hip::pixelgpudetails::SiPixelRawToClusterGPUKernel::makeClustersAsync(bool, Kokkos::View<SiPixelFedCablingMapGPU const, Kokkos::Experimental::HIP> const&, Kokkos::View<unsigned char const*, Kokkos::Experimental::HIP> const&, SiPixelGainForHLTonGPU<Kokkos::Experimental::HIP> const&, kokkos_hip::pixelgpudetails::SiPixelRawToClusterGPUKernel::WordFedAppender const&, std::map<unsigned int, std::vector<SiPixelRawDataError, std::allocator<SiPixelRawDataError> >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::vector<SiPixelRawDataError, std::allocator<SiPixelRawDataError> > > > >&&, unsigned int, unsigned int, bool, bool, bool, Kokkos::Experimental::HIP const&)::{lambda(unsigned long)#1} const&) (str=..., policy=..., functor=...)
      at .../pixeltrack-standalone/external/kokkos/install/include/Kokkos_Parallel.hpp:207
  #9  0x00007ffee433f222 in kokkos_hip::pixelgpudetails::SiPixelRawToClusterGPUKernel::makeClustersAsync (this=0x6d088b0, isRun2=true, cablingMap=..., modToUnp=..., gains=..., wordFed=..., errors=..., wordCounter=48316, fedCounter=108, useQualityInfo=true, 
      includeErrors=true, debug=false, execSpace=...) at plugin-SiPixelClusterizer/kokkos/SiPixelRawToClusterGPUKernel.cc:610
  #10 0x00007ffee433c582 in kokkos_hip::SiPixelRawToCluster::produce(edm::Event&, edm::EventSetup const&) () at /opt/gcc/8.4.0/lib/gcc/x86_64-pc-linux-gnu/8.4.0/../../../../include/c++/8.4.0/bits/basic_string.h:6465
  #11 0x00007ffee433e6f7 in edm::WorkerT<kokkos_hip::SiPixelRawToCluster>::doWorkAsync(edm::Event&, edm::EventSetup const&, edm::WaitingTask*)::{lambda(std::__exception_ptr::exception_ptr const*)#1}::operator()(std::__exception_ptr::exception_ptr const*) ()
      at /opt/gcc/8.4.0/lib/gcc/x86_64-pc-linux-gnu/8.4.0/../../../../include/c++/8.4.0/bits/basic_string.h:6465
  #12 0x00007ffee433e68e in edm::FunctorWaitingTask<edm::WorkerT<kokkos_hip::SiPixelRawToCluster>::doWorkAsync(edm::Event&, edm::EventSetup const&, edm::WaitingTask*)::{lambda(std::__exception_ptr::exception_ptr const*)#1}>::execute() ()
      at /opt/gcc/8.4.0/lib/gcc/x86_64-pc-linux-gnu/8.4.0/../../../../include/c++/8.4.0/bits/basic_string.h:6465
  #13 0x00007ffff6d9970d in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop (this=this@entry=0x7fffec5e3200, context_guard=..., t=t@entry=0x7fffec5e7640, isolation=isolation@entry=0) at ../../src/tbb/custom_scheduler.h:388
  #14 0x00007ffff6d99a05 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fffec5e3200, parent=..., child=<optimized out>) at ../../include/tbb/task.h:992
  #15 0x000000000041127b in edm::EventProcessor::runToCompletion() () at ../../../../../src/libstdc++-v3/src/filesystem/std-ops.cc:135
  #16 0x0000000000410cbb in main () at ../../../../../src/libstdc++-v3/src/filesystem/std-ops.cc:135
  ```
- After quite some poking around I realized I had enabled `-DKokkos_ENABLE_DEBUG_BOUNDS_CHECK=On` for Kokkos, and that was causing the failure
- Removed the bounds check, still got the same error, but from different place now
- Was missing `if defined __HIP_DEVICE_COMPILE__` in `kokkos_assert.h`
- Next failure is `Device::callbackQueue aborting with status: 0x1016`
- Found some more `#ifdef KOKKOS_BACKEND_CUDA` that had to be amended with `KOKKOS_BACKEND_HIP`
- Next failure is `Memory access fault by GPU node-2 (Agent handle: 0x205f840) on address 0x7f44cacb0000. Reason: Page not present or supervisor privilege.`
  * It occurs randomly...
  * Reorganizing code helped...
- Next failure is again `Device::callbackQueue aborting with status: 0x100f`, now in `GPUCACell` code
- Trying to make a PR now, running with CUDA fails now with the asserts
  * I don't seem to be able to make the `kokkos_assert.h` to work

## 2021-01-28 [cms-patatrack/pixeltrack-standalone#166](https://github.com/cms-patatrack/pixeltrack-standalone/pull/166)

- Team scan on CUDA backend requires the number of threads in a team to be a power of two
- `Kokkos::AUTO()` does not guarantee team size of power of two
- Impact on performance (from now on with a proper measurement of 8 trials, see [ThroughputKokkos notebook](ThroughputKokkos.ipynb)
  * Serial: 20.0 +- -> 20.68 +- 0.06 events/s (3 % improvement)
  * Pthread: 44.22 +- 0.09 -> 45.0 +- 0.2 events/s on 18 threads (2 % improvement)
  * CUDA: 114.7 +- 0.3 -> 115.2 +- 0.7 events/s (0.5 % improvement, but not statistically significant by eye)

## 2021-02-05 [cms-patatrack/pixeltrack-standalone#171](https://github.com/cms-patatrack/pixeltrack-standalone/pull/171)

- Update to Kokkos 3.3.1 was smooth
- Effectively no impact on performance
  * Serial: 20.68 +- 0.06 -> 20.5 +- 0.05 events/s (1 % degradation, but not statistically significant by eye)
  * Pthread: 45.0 +- 0.2 -> 45.0 +- 0.3 events/s on 18 threads (stays same)
  * CUDA: 115.2 +- 0.7 -> 115.7 +- 0.3 events/s (0.4 % improvement, but not statistically significant by eye
