set ENVs PPSwork=/home/zdong/PPS/git/PPSwork source /home/zdong/PPS/git/PPSwork/Wire-Cell/entrypoint.sh enable_kokkos_cuda in /home/zdong/PPS/git/wire-cell-gen-kokkos/example Warning: LBR backtrace method is not supported on this platform. DWARF backtrace method will be used. Collecting data... The following module labels are either not assigned to any path, or they have been assigned to ignored path(s): 'out', 'plopper' %MSG-i MF_INIT_OK: Early 25-Feb-2021 21:10:43 EST JobSetup Messagelogger initialization complete. %MSG [21:10:44.731] D [ sys ] loaded plugin #1 "WireCellPgraph" from library "libWireCellPgraph.so": 0x1c63920 [21:10:44.731] D [ sys ] loaded plugin #2 "WireCellGen" from library "libWireCellGen.so": 0x1c65d20 [21:10:44.736] D [ sys ] loaded plugin #3 "WireCellGenKokkos" from library "libWireCellGenKokkos.so": 0x1c65ae0 [21:10:44.739] D [ sys ] loaded plugin #4 "WireCellSio" from library "libWireCellSio.so": 0x1c6f620 [21:10:44.919] D [ sys ] loaded plugin #5 "WireCellRoot" from library "libWireCellRoot.so": 0x1ceefa0 [21:10:44.995] D [ sys ] loaded plugin #6 "WireCellLarsoft" from library "libWireCellLarsoft.so": 0x1d265e0 [21:10:45.002] D [ sys ] loaded plugin #7 "WireCellTbb" from library "libWireCellTbb.so": 0x1d21300 [21:10:45.009] D [ sys ] loaded plugin #8 "WireCellImg" from library "libWireCellImg.so": 0x1d1fdc0 [21:10:45.035] D [ sys ] loaded plugin #9 "WireCellHio" from library "libWireCellHio.so": 0x1a985f0 Kokkos::initializing Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads For best performance with OpenMP 3.1 set OMP_PROC_BIND=true For unit testing set OMP_PROC_BIND=false Kokkos::initialized [21:10:45.266] D [ sim ] Drifter: time offset: 0 ms, drift speed: 1.6 mm/us [21:10:45.266] I [ wct ] loading compressed json file: /opt/wire-cell-data/protodune-wires-larsoft-v4.json.bz2 [21:10:47.574] D [ geom ] AnodePlane: X planes: "cathode"@ -0.0254m, "response"@-3.49414m, "anode"@-3.57985m [21:10:47.577] D [ geom ] AnodePlane: face:0, plane:0, origin:(-3494.14 3071.4 1153.36) mm [21:10:47.579] D [ geom ] AnodePlane: face:0, plane:1, origin:(-3494.14 3069.73 1153.36) mm [21:10:47.579] D [ geom ] AnodePlane: face:0, plane:2, origin:(-3494.14 3068.05 1153.29) mm [21:10:47.580] D [ geom ] AnodePlane: face:0 with 3 planes and sensvol: [(-3579.85 76.1 3.35) --> (-25.4 6060 2303.37)] [21:10:47.580] D [ geom ] anode 0 face 1 is not sensitive [21:10:47.580] D [ geom ] AnodePlane: X planes: "cathode"@ 0m, "response"@0m, "anode"@0m [21:10:47.581] D [ geom ] AnodePlane: face:1, plane:0, origin:(0 3071.4 1153.36) mm [21:10:47.583] D [ geom ] AnodePlane: face:1, plane:1, origin:(0 3069.73 1153.36) mm [21:10:47.583] D [ geom ] AnodePlane: face:1, plane:2, origin:(0 3068.05 1153.44) mm [21:10:47.583] D [ geom ] AnodePlane: face:1 with 3 planes and sensvol: [(0 0 0) --> (0 0 0)] [21:10:47.584] I [ wct ] loading compressed json file: /opt/wire-cell-data/garfield-1d-boundary-path-rev-dune.json.bz2 [21:10:48.724] D [ geom ] PIR: short response ElecResponse has different number of samples (6125) than expected (200) [21:10:48.726] D [ geom ] PIR: long response RCResponse has different number of samples (6125) than expected (6174) [21:10:48.730] D [ geom ] PIR: short response ElecResponse has different number of samples (6125) than expected (200) [21:10:48.730] D [ geom ] PIR: long response RCResponse has different number of samples (6125) than expected (6174) [21:10:48.732] D [ geom ] PIR: short response ElecResponse has different number of samples (6125) than expected (200) [21:10:48.732] D [ geom ] PIR: long response RCResponse has different number of samples (6125) than expected (6174) [21:10:48.734] D [ sim ] Gen::Reframer: using anode: "AnodePlane:apa0" [21:10:48.734] D [ sim ] EmpiricalNoiseModel: using anode AnodePlane:apa0, chanstat [21:10:48.734] I [ wct ] loading compressed json file: /opt/wire-cell-data/protodune-noise-spectra-v1.json.bz2 [21:10:48.739] D [ sim ] AddNoise: using IRandom: "Random", IChannelSpectrum: "EmpiricalNoiseModel:empericalnoise-apa0" [21:10:48.739] D [ sim ] Gen::Digitizer: tag="orig0", resolution=12 bits, maxvalue=4096 counts, gain=1, fullscale=[200,1600] mV, baselines=[1003,1003,508] mV [21:10:48.743] D [ pgraph ] connecting: 10 edges Inputer: "wclsSimDepoSource:" 25-Feb-2021 21:10:50 EST Initiating request to open input file "g4-1-event.root" 25-Feb-2021 21:10:50 EST Opened input file "g4-1-event.root" Begin processing the 1st record. run: 1 subRun: 0 event: 1 at 25-Feb-2021 21:10:50 EST SimDepoSource got 578267 depos from art tag "InputTag: label = IonAndScint, instance = " returns: okay SimDepoSource: ready with 578267 depos spanning: [-3257.21, 2995.78]us [21:10:51.443] D [ pgraph ] executing with 11 nodes [21:11:08.565] D [ sim ] xregion: anode: -3579.85 mm, response: -3494.14 mm, cathode: -25.4 mm, flushing 36809 [21:11:08.571] D [ sim ] xregion: anode: 0 mm, response: 0 mm, cathode: 0 mm, flushing 0 [21:11:08.571] D [ sim ] xregion: anode: 3579.85 mm, response: 3494.14 mm, cathode: 25.4 mm, flushing 64866 [21:11:08.605] D [ sim ] at EOS, dropped 3144 / 578267 depos from stream, outside of all 3 drift regions [21:11:10.109] D [ sim ] anode: 0, face: 0, processing 45073 depos spanning t:[-0.31249,2.74997]ms, bb:[(-357.985 7.61 0.335)-->(-2.54 606 230.337)]cm [21:11:10.109] D [ sim ] anode: 0, face: 0, dropped 204940 depos spanning t:[-0.312484,2.75]ms, outside bb:[(-357.985 7.61 0.335)-->(-2.54 606 230.337)]cm get_charge_vec() : get_charge_vec() part1 running time : 5.53308e-05 get_charge_vec() : get_charge_vec() part2 running time : 0.00284214 get_charge_vec() : get_charge_vec() part3 running time : 1.82969 get_charge_vec() : set_sampling() running time : 1.34964, counter : 45073 get_charge_vec() : m_fluctuate : 0x7e2e620 get_charge_vec() CUDA : set_sampling() part1 time : 0.088017, part2 (CUDA) time : 1.24842 GaussianDiffusion::sampling_CUDA() part3 time : 0.199616, part4 time : 0.202914, part5 time : 0.190503, part6 time : 0.522436 GaussianDiffusion::sampling_CUDA() : g_total_sample_size : 10050087 [21:11:12.053] D [ sim ] ImpactTransform::ImpactTransform() : get_charge_vec() Total running time : 1.83578 [21:11:14.533] D [ sim ] ImpactTransform: # of channels: 1280 # of ticks: 6655 get_charge_vec() : get_charge_vec() part1 running time : 6.03911e-05 get_charge_vec() : get_charge_vec() part2 running time : 0.00570858 get_charge_vec() : get_charge_vec() part3 running time : 3.879 get_charge_vec() : set_sampling() running time : 2.72372, counter : 45073 get_charge_vec() : m_fluctuate : 0x7e2e620 get_charge_vec() CUDA : set_sampling() part1 time : 0.183477, part2 (CUDA) time : 2.51356 GaussianDiffusion::sampling_CUDA() part3 time : 0.399215, part4 time : 0.405877, part5 time : 0.378726, part6 time : 1.02524 GaussianDiffusion::sampling_CUDA() : g_total_sample_size : 20100886 [21:11:17.013] D [ sim ] ImpactTransform::ImpactTransform() : get_charge_vec() Total running time : 3.89159 [21:11:19.124] D [ sim ] ImpactTransform: # of channels: 1120 # of ticks: 6655 get_charge_vec() : get_charge_vec() part1 running time : 6.50114e-05 get_charge_vec() : get_charge_vec() part2 running time : 0.00691368 get_charge_vec() : get_charge_vec() part3 running time : 6.26843 get_charge_vec() : set_sampling() running time : 4.1873, counter : 45073 get_charge_vec() : m_fluctuate : 0x7e2e620 get_charge_vec() CUDA : set_sampling() part1 time : 0.281694, part2 (CUDA) time : 3.86565 GaussianDiffusion::sampling_CUDA() part3 time : 0.625288, part4 time : 0.610573, part5 time : 0.570463, part6 time : 1.52879 GaussianDiffusion::sampling_CUDA() : g_total_sample_size : 29899128 [21:11:21.928] D [ sim ] ImpactTransform::ImpactTransform() : get_charge_vec() Total running time : 6.28425 [21:11:22.802] D [ sim ] ImpactTransform: # of channels: 512 # of ticks: 6655 [21:11:22.988] D [ sim ] anode 0 face 1 is marked insensitive, skipping [21:11:23.110] D [ sim ] Gen::Reframer: frame 100 2560 traces, 6000 ticks ========== KokkosTestFrameFilter: start ========== Computed result for 4096 x 1024 is 4194304.000000 N( 4096 ) M( 1024 ) nrepeat ( 100 ) problem( 33.5954 MB ) time( 0.0238594 s ) bandwidth( 140.806 GB/s ) ========== KokkosTestFrameFilter: end ========== [21:11:23.664] D [ io ] AnodePlane:apa0: t: 0 - 5999; c: 0 - 2559 [21:11:23.664] D [ io ] HDF5FrameTap: chunking ncols=0 nrows=0 [21:11:23.664] D [ io ] HDF5FrameTap: gzip_level 2 [21:11:23.664] D [ io ] HDF5FrameTap: see frame #100 with 2560 traces with frame tags: "orig0" and trace tags: looking for tags: "orig0" [21:11:23.664] D [ io ] HDF5FrameTap: save 2560 tagged as orig0 [21:11:23.664] D [ io ] HDF5FrameTap: saving ncols=6000 nrows=2560 [21:11:23.664] D [ io ] HDF5FrameTap: chunking ncols=6000 nrows=2560 [21:11:25.098] D [ io ] HDF5FrameTap: saved /0/frame_orig0 with 2560 channels 6000 ticks @t=-0.25 ms qtot=2.76306e+10 [21:11:25.101] D [ glue ] sink frame: #100 @-0.25 with 2560 traces, frame tags:[orig0], trace tags:[] [21:11:25.101] D [ sim ] Gen::Digitizer: EOS [21:11:25.101] D [ io ] HDF5FrameTap: EOS [21:11:25.101] D [ glue ] frame sink sees EOS [21:11:25.101] I [ timer ] Timer: WireCell::Gen::DepoTransform : 14.51 sec [21:11:25.101] I [ timer ] Timer: WireCell::Hdf5::HDF5FrameTap : 2.24 sec [21:11:25.102] I [ timer ] Timer: WireCell::GenKokkos::KokkosTestFrameFilter : 1.73 sec [21:11:25.102] I [ timer ] Timer: WireCell::Gen::Drifter : 1.33 sec [21:11:25.102] I [ timer ] Timer: WireCell::Gen::DepoBagger : 1.11 sec [21:11:25.102] I [ timer ] Timer: WireCell::Gen::Digitizer : 1.06 sec [21:11:25.102] I [ timer ] Timer: WireCell::Gen::AddNoise : 1.03 sec [21:11:25.102] I [ timer ] Timer: WireCell::Gen::Reframer : 0.959999 sec [21:11:25.102] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0.929999 sec [21:11:25.102] I [ timer ] Timer: WireCell::Gen::DepoFanout : 0.77 sec [21:11:25.102] I [ timer ] Timer: wcls::SimDepoSource : 0.26 sec [21:11:25.102] I [ timer ] Timer: Total node execution : 25.93 sec 25-Feb-2021 21:11:25 EST Closed input file "g4-1-event.root" TrigReport ---------- Event Summary ------------ TrigReport Events total = 1 passed = 1 failed = 0 TimeReport ---------- Time Summary ---[sec]---- TimeReport CPU = 35.791451 Real = 34.341117 MemReport ---------- Memory Summary ---[base-10 MB]---- MemReport VmPeak = 10157.3 VmHWM = 1733.27 Kokkos::finalizing Kokkos::finalized Art has completed and will exit with status 0. Processing events... Capturing symbol files... Saving temporary "/tmp/nsys-report-503d-6a30-9e33-ceca.qdstrm" file to disk... Creating final output files... Processing [0% ] Processing [1% ] Processing [2% ] Processing [3% ] Processing [4% ] Processing [=5% ] Processing [=6% ] Processing [==7% ] Processing [===8% ] Processing [===9% ] Processing [===10% ] Processing [====11% ] Processing [====12% ] Processing [=====13% ] Processing [======14% ] Processing [======15% ] Processing [=======16% ] Processing [========17% ] Processing [========18% ] Processing [=========19% ] Processing [==========20% ] Processing [==========21% ] Processing [===========22% ] Processing [============23% ] Processing [============24% ] Processing [=============25% ] Processing [==============26% ] Processing [==============27% ] Processing [===============28% ] Processing [================29% ] Processing [================30% ] Processing [=================31% ] Processing [==================32% ] Processing [==================33% ] Processing [===================34% ] Processing [====================35% ] Processing [====================36% ] Processing [=====================37% ] Processing [======================38% ] Processing [======================39% ] Processing [=======================40% ] Processing [========================41% ] Processing [========================42% ] Processing [=========================43% ] Processing [==========================44% ] Processing [==========================45% ] Processing [===========================46% ] Processing [============================47% ] Processing [============================48% ] Processing [=============================49% ] Processing [==============================50% ] Processing [==============================51% ] Processing [===============================52% ] Processing [===============================53% ] Processing [================================54% ] Processing [=================================55% ] Processing [=================================56% ] Processing [==================================57% ] Processing [===================================58% ] Processing [===================================59% ] Processing [====================================60% ] Processing [=====================================61% ] Processing [=====================================62% ] Processing [======================================63% ] Processing [=======================================64% ] Processing [=======================================65% ] Processing [========================================66% ] Processing [=========================================67% ] Processing [=========================================68% ] Processing [==========================================69% ] Processing [===========================================70% ] Processing [===========================================71% ] Processing [============================================72% ] Processing [=============================================73% ] Processing [=============================================74% ] Processing [==============================================75% ] Processing [==============================================================100%] Saved report file to "/tmp/nsys-report-503d-6a30-9e33-ceca.qdrep" Exporting 1091594 events: [1% ] Exporting 1091594 events: [2% ] Exporting 1091594 events: [3% ] Exporting 1091594 events: [4% ] Exporting 1091594 events: [5% ] Exporting 1091594 events: [=6% ] Exporting 1091594 events: [=7% ] Exporting 1091594 events: [==8% ] Exporting 1091594 events: [==9% ] Exporting 1091594 events: [==10% ] Exporting 1091594 events: [==11% ] Exporting 1091594 events: [===12% ] Exporting 1091594 events: [===13% ] Exporting 1091594 events: [====14% ] Exporting 1091594 events: [====15% ] Exporting 1091594 events: [=====16% ] Exporting 1091594 events: [=====17% ] Exporting 1091594 events: [======18% ] Exporting 1091594 events: [======19% ] Exporting 1091594 events: [=======20% ] Exporting 1091594 events: [=======21% ] Exporting 1091594 events: [========22% ] Exporting 1091594 events: [========23% ] Exporting 1091594 events: [=========24% ] Exporting 1091594 events: [=========25% ] Exporting 1091594 events: [==========26% ] Exporting 1091594 events: [==========27% ] Exporting 1091594 events: [===========28% ] Exporting 1091594 events: [===========29% ] Exporting 1091594 events: [============30% ] Exporting 1091594 events: [============31% ] Exporting 1091594 events: [=============32% ] Exporting 1091594 events: [=============33% ] Exporting 1091594 events: [==============34% ] Exporting 1091594 events: [==============35% ] Exporting 1091594 events: [===============36% ] Exporting 1091594 events: [===============37% ] Exporting 1091594 events: [================38% ] Exporting 1091594 events: [================39% ] Exporting 1091594 events: [=================40% ] Exporting 1091594 events: [=================41% ] Exporting 1091594 events: [==================42% ] Exporting 1091594 events: [==================43% ] Exporting 1091594 events: [===================44% ] Exporting 1091594 events: [===================45% ] Exporting 1091594 events: [====================46% ] Exporting 1091594 events: [====================47% ] Exporting 1091594 events: [=====================48% ] Exporting 1091594 events: [=====================49% ] Exporting 1091594 events: [======================50% ] Exporting 1091594 events: [=======================51% ] Exporting 1091594 events: [=======================52% ] Exporting 1091594 events: [========================53% ] Exporting 1091594 events: [========================54% ] Exporting 1091594 events: [=========================55% ] Exporting 1091594 events: [=========================56% ] Exporting 1091594 events: [==========================57% ] Exporting 1091594 events: [==========================58% ] Exporting 1091594 events: [===========================59% ] Exporting 1091594 events: [===========================60% ] Exporting 1091594 events: [============================61% ] Exporting 1091594 events: [============================62% ] Exporting 1091594 events: [=============================63% ] Exporting 1091594 events: [=============================64% ] Exporting 1091594 events: [==============================65% ] Exporting 1091594 events: [==============================66% ] Exporting 1091594 events: [===============================67% ] Exporting 1091594 events: [===============================68% ] Exporting 1091594 events: [================================69% ] Exporting 1091594 events: [================================70% ] Exporting 1091594 events: [=================================71% ] Exporting 1091594 events: [=================================72% ] Exporting 1091594 events: [==================================73% ] Exporting 1091594 events: [==================================74% ] Exporting 1091594 events: [===================================75% ] Exporting 1091594 events: [===================================76% ] Exporting 1091594 events: [====================================77% ] Exporting 1091594 events: [====================================78% ] Exporting 1091594 events: [=====================================79% ] Exporting 1091594 events: [=====================================80% ] Exporting 1091594 events: [======================================81% ] Exporting 1091594 events: [======================================82% ] Exporting 1091594 events: [=======================================83% ] Exporting 1091594 events: [=======================================84% ] Exporting 1091594 events: [========================================85% ] Exporting 1091594 events: [========================================86% ] Exporting 1091594 events: [=========================================87% ] Exporting 1091594 events: [=========================================88% ] Exporting 1091594 events: [==========================================89% ] Exporting 1091594 events: [==========================================90% ] Exporting 1091594 events: [===========================================91% ] Exporting 1091594 events: [===========================================92% ] Exporting 1091594 events: [============================================93% ] Exporting 1091594 events: [============================================94% ] Exporting 1091594 events: [=============================================95% ] Exporting 1091594 events: [=============================================96% ] Exporting 1091594 events: [==============================================97% ] Exporting 1091594 events: [==============================================98% ] Exporting 1091594 events: [===============================================99% ] Exporting 1091594 events: [===============================================100%] Exported successfully to /tmp/nsys-report-503d-6a30-9e33-ceca.sqlite Generating CUDA API Statistics... CUDA API Statistics (nanoseconds) Time(%) Total Time Calls Average Minimum Maximum Name ------- -------------- ---------- -------------- -------------- -------------- -------------------------------------------------------------------------------- 60.8 2101638926 270458 7770.7 3200 23481815 cudaMemcpy 31.7 1097105713 270550 4055.1 3600 2522428 cudaLaunchKernel 3.8 132716770 9 14746307.8 880 132476946 cudaDeviceSynchronize 2.5 86411545 39 2215680.6 350 74371930 cudaFree 0.7 23360192 203 115074.8 790 235753 cudaStreamSynchronize 0.4 13983618 27 517911.8 1820 3179305 cudaMalloc 0.0 690808 1 690808.0 690808 690808 cudaHostAlloc 0.0 571696 2 285848.0 13500 558196 cudaFreeHost 0.0 39310 3 13103.3 5820 21200 cudaMemcpyToSymbol 0.0 16060 2 8030.0 4890 11170 cudaMemset 0.0 11541 3 3847.0 2890 5181 cudaThreadSynchronize 0.0 5220 1 5220.0 5220 5220 cudaEventCreate 0.0 4370 1 4370.0 4370 4370 cudaMallocHost 0.0 2460 1 2460.0 2460 2460 cudaEventDestroy 0.0 740 1 740.0 740 740 cuInit Generating CUDA Kernel Statistics... CUDA Kernel Statistics (nanoseconds) Time(%) Total Time Instances Average Minimum Maximum Name ------- -------------- ---------- -------------- -------------- -------------- -------------------------------------------------------------------------------------------------------------------- 49.4 305506227 135219 2259.3 1983 3424 ker_sampling(float*, float*, unsigned long, double, double) 39.4 243435865 135219 1800.3 1631 2944 ker_patching(double*, unsigned long, double*, unsigned long, float*, double) 6.9 42921309 3 14307103.0 12709720 16196290 void gen_sequenced(curandStateXORWOW*, normal_args_st)), rng_config >(curandStateXORWOW*, float2*, unsigned long, unsigned long, normal_args_st) 3.7 22906936 100 229069.4 226429 238236 void Kokkos::Impl::cuda_parallel_launch_local_memory const&, std::shared_ptr&)::{lambda(int, double&)#1}, Kokkos::RangePolicy, double, voi 0.6 3445384 3 1148461.3 1026159 1290219 generate_seed_pseudo(unsigned long long, unsigned long long, unsigned long long, curandOrdering, curandStateXORWOW*, unsigned int*) 0.0 102815 3 34271.7 1152 100031 void Kokkos::Impl::cuda_parallel_launch_local_memory, Kokkos::RangePolicy >, Kokkos::Cuda> >(Kokkos::Impl::ParallelFor, Kokkos::RangePolicy