# Notes from exploration of HIP with Patatrack code

## Note on Eigen

[`Eigen/src/Core/util/Macros.h`](https://gitlab.com/libeigen/eigen/-/blob/master/Eigen/src/Core/util/Macros.h) has a note
```cpp
// NVCC is not supported as the target platform for HIPCC
// Note that this also makes EIGEN_CUDACC and EIGEN_HIPCC mutually exclusive
```
so we can not easily use HIP today as a portable frontend for both AMD and NVIDIA GPUs.


## 2021-01-08 [cms-patatrack/pixeltrack-standalone#156](https://github.com/cms-patatrack/pixeltrack-standalone/pull/156)

- Easier to use `hipcc` to link code depending only on the HIP runtime than using `gcc` and guessing necessary link flags

## 2021-01-13 [cms-patatrack/pixeltrack-standalone#161](https://github.com/cms-patatrack/pixeltrack-standalone/pull/161)

- Code asks for amount of allocated dynamic memory size, I didn't find quickly how to do that with HIP (if any). I "commented out" the pieces with `#ifdef TODO`.
- `threadIdx.x` and `blockDim.x` are different types
- `__ballot_sync()` needs to be replaced with `__ballot()`, and `__shfl_up_sync()` with `__shfl_up()`
  * Notable the HIP versions do not take mask parameter
  * Quoting from https://rocmdocs.amd.com/en/latest/Programming_Guides/HIP-GUIDE.html#warp-cross-lane-functions
  > The hardware guarantees that all warp lanes will execute in lockstep, so additional synchronization is unnecessary, and the instructions use no shared memory.
- Building `OneToManyAssoc_t.cu`, `HistoContainer_t.cu`, `prefixScan_t.cu` lead to
  ```
  fatal error: error in backend: failed to annotate CFG
  clang-12: error: clang frontend command failed with exit code 70 (use -v to see invocation)
  ```
  I'll just disable the test for now.
- Running fails with
  ```
  Memory access fault by GPU node-2 (Agent handle: 0xb784e0) on address 0x7f5c27faf000. Reason: Page not present or supervisor privilege.
  ```
  somewhere in raw-to-cluster code
  * Starting to debug, removing other code I'm back to `:0:rocdevice.cpp            :2303: 891843668389 us: Device::callbackQueue aborting with status: 0x100f`
- I noticed the AMD setup uses warp size 64, but much of the code assumes warp size of 32. Generalizing all the places I found allowed me make progress.
- Then I noticed the aforementioned `failed to annotate CFG` compilation error could be worked around by removing one specific `assert()`
