# Roofline Analysis of FastCaloSim

## Build on Cori

The building process is detailed in https://github.com/meifeng/FastCaloSim-GPU/blob/master/ReadmeCori.txt


## Collect Roofline Data

Profiling 1000 events on 1 GPU for Roofline analysis:

```bash
module load cuda
module load esslurm
salloc -C gpu --nodes 1 --gres=gpu:1 --exclusive -t 1:00:00

# Copy profiling script to the working directory
cp profiling.sh /FASTCALOSIM/BUILD/DIRECTORY/x86_64-linux15-gcc8-opt/bin
cd /FASTCALOSIM/BUILD/DIRECTORY/x86_64-linux15-gcc8-opt/bin

# Source environment
source ../setup.sh

# Run Nsight profiling
./profiling.sh
```
The output data file `fastcalosim_roofline.nsight-cuprof-report` can be found at `/global/cfs/cdirs/m2845/roofline/fastcalosim`

### Convert Nsight output to csv

```bash
module load cuda

cd /global/cfs/cdirs/m2845/roofline/fastcalosim
nv-nsight-cu-cli --csv -i fastcalosim_roofline.nsight-cuprof-report > fastcalosim_roofline.csv
```


## Roofline Analysis

1. [ ] `process_nsight.ipynb`: process raw profiling data, compute FLOPS & AI for Roofline plot
2. [ ]  `roofline_plot.ipynb`: plot HBM Roofline for individual kernels & Hierarchical Roofline for the whole application
