# Standalone version of the Patatrack application

## Introduction

The full Patatrack pixel tracking application has been extracted into
a standalone program to exercise various portability technologies in
an environment that approximates an experiment software framework.

The code and more documentation can be found from https://github.com/cms-patatrack/pixeltrack-standalone/ .

## Instructions for Cori

### Building

```bash
module load gcc/8.3.0
module load cuda/11.0.2

# Download
git clone https://github.com/cms-patatrack/pixeltrack-standalone.git
cd pixeltrack-standalone

# Build the cuda test program
make -j 8 cuda CUDA_BASE=$CUDA_ROOT

# Build the kokkos test program
make -j 8 kokkotest CUDA_BASE=$CUDA_ROOT
```

### Running

```bash
module load cgpu
module load cuda/11.0.2
salloc -A m2845 -C gpu -N 1 --gres=gpu:1 -t 30
srun -n 1 -c 1 --pty /bin/bash -i

# Source environment
source env.sh

# Run the cuda program, use -h to check command line parameters
./cuda

# Run the kokkos program, use -h to check command line parameters
./kokkos --cuda
```

### Benchmarking

#### Throughput

The aim is to run the program with N events about 10 minutes with 1 CPU thread and concurrent event, and then multiply the N with the number of concurrent events. For the various test programs this translates to about
* `cuda`: 400k events
* `cuda --transfer`: 350k events

##### Interactively

```bash
module load cgpu
module load cuda/11.0.2
salloc -A m2845 -C gpu --nodes 1 --gres=gpu:1 --exclusive -t 04:00:00
srun -n 1 -c 80 --pty /bin/bash -i

# Source environment
source env.sh

# Run a scan over the number of threads and concurrent events
# Results are stored in result.json
./run-scan.py -o result --maxThreads 10 ./cuda
```

##### Batch job

```bash
module load cgpu
module load cuda/11.0.2
sbatch -A m2845 submit_patatrack.sh
```

#### Profile

For profiling it makes sense to process only the 1000 events (or even less).

```bash
module load cgpu
module load cuda/11.0.2
salloc -A m2845 -C gpu --nodes 1 --gres=gpu:1 --exclusive -t 04:00:00

# Source environment
source env.sh

# Run nvprof
srun -n 1 -c 80 nvprof -o cuda_prof.nvvp ./cuda
```

Reference timings for the kernels are given in https://github.com/cms-patatrack/pixeltrack-standalone/blob/master/doc/CodeStructure.md, and a corresponding `nvprof` profile can be found on Cori at `/global/cfs/cdirs/m2845/profiles/patatrack_cuda_transfer_20200321.nvvp`.
