# Patatrack application witin full CMSSW

## Introduction

Please see https://gitlab.com/NESAP/cms-gpu-tracking/benchmark/blob/master/NESAP.md#description

## Instructions for Cori

Only a quick recipe is given below, more information can be from
https://gitlab.com/NESAP/cms-gpu-tracking/benchmark/blob/master/NESAP.md#recipe

### One-time setup

```bash
cori$ module load esslurm
cori$ salloc -A m2845 -C gpu -N 1 --gres=gpu:1 --image=docker:makortel/cmssw_patatrack:11_0_0_pre7_v1 --module=cvmfs,gpu -t 60

# Start the container
cori$ srun -n 1 -c 1 --pty shifter /bin/bash -i
# Source necessary environment
cori (container)$ source /opt/cms/cmsset_default.sh
# Create a CMSSW developer area, e.g. in $SCRATCH
cori (container)$ cd $SCRATCH
cori (container)$ cmsrel CMSSW_11_0_0_pre7_Patatrack
cori (container)$ cd $SCRATCH/CMSSW_11_0_0_pre7_Patatrack/src

# Copy config files
cori (container)$ cp /global/cscratch1/sd/matti/patatrack_data/config_11_0_0_pre7/*.py .

# Following are needed for code development, for running it can be ignore
cori (container)$ cmsenv
cori (container)$ git cms-init -x cms-patatrack --upstream-only
cori (container)$ git branch CMSSW_11_0_X_Patatrack --track cms-patatrack/CMSSW_11_0_X_Patatrack
``` 

### Building

```bash
cori$ module load esslurm
cori$ salloc -A m2845 -C gpu -N 1 --gres=gpu:1 --image=docker:makortel/cmssw_patatrack:11_0_0_pre7_v1 --module=cvmfs,gpu -t 60

# Start the container, source environment if not done yet
cori$ srun -n 1 -c 1 --pty shifter /bin/bash -i
cori (container)$ source /opt/cms/cmsset_default.sh
cori (container)$ cd $SCRATCH/CMSSW_11_0_0_pre7_Patatrack/src

# Checkout code of some package, e.g. HeterogeneousCore/CUDACore
cori (container)$ git cms-addpkg HeterogeneousCore/CUDACore
# If you edit any header file, check out any other packages depending on that header
cori (container)$ git cms-checkdeps -a
# Compile with N threads
cori (container)$ scram b -j <N>
```

### Running

```bash
cori$ module load esslurm
cori$ salloc -A m2845 -C gpu -N 1 --gres=gpu:1 --image=docker:makortel/cmssw_patatrack:11_0_0_pre7_v1 --module=cvmfs,gpu -t 60

# Start the container, source environment if not done yet
cori$ srun -n 1 -c 1 --pty shifter /bin/bash -i
cori (container)$ source /opt/cms/cmsset_default.sh
cori (container)$ cd $SCRATCH/CMSSW_11_0_0_pre7_Patatrack/src
cori (container)$ cmsenv

# Run the application on GPU
cori (container)$ cmsRun profile_daq.py
# Run the application on CPU
cori (container)$ cmsRun profile_daq_cpu.py
```

Note that first couple of runs may be slow (take several minutes), if
the needed conditions data have been evicted from the squid caches at NERSC. 


Messages like below can be safely ignored.
```
%MSG-e EvFDaqDirector:  AfterModConstruction 27-Feb-2020 08:35:16 PST pre-events
problem with creating filedesc for fuwritelock -: tmp/run321177/fu.lock create:0 error:No such file or directory
%MSG
%MSG-e EvFDaqDirector:  AfterModConstruction 27-Feb-2020 08:35:16 PST pre-events
problem with opening fuwritelock file stream -: Bad file descriptor
%MSG
```
