Instructions
for E949 Shift Takers
VERSION 1.2.6
Benji Lewis
March 2002
Please send comments & suggestions to benjil@unm.edu
How to read this manual:
words in <...> are menu selections or buttons within an open window.
words in {...} are titles of an open window.
Table of
Contents
I. In
the beginning... In a counting house far far away...
II. Collect
Data (aka the joy of DAQ)
III. Logging
procedure. What to write in the log
book.
I. How
to do a standard run in DAQ
III. How
to deal with DAQ errors
Appendix
C. dp0 & data streams
IV. Monitoring
tape information
II. Viewing
the present voltage
III. To
turn OFF or ON a (all) Mainframe(s)
IV. To
change the high voltage file
II. Automated
entries into Elog
What
triggers & SSP files being used.
When
we have had SEB
Open an account on bnlku9.
>lpr -Pe787lp2r
/e787s/numao/gas.ps -To print a new gas
log
* If you are unfamiliar with how to handle the DAQ, consult Appendix B. *
{Toolchest}: <High Voltage> -> <CsI HV> -> <status>
If CsI HV is OFF, then
{Toolchest}: <High Voltage> -> <CsI HV> -> <ON>
If there are current jobs, then do them first (if possible).
If there are NO current jobs, then do standard mix trigger/ssp
Do not take tddac with beam on or about to
come on.
Do at least 1 per shift.
Be sure to place Fera, FB, and CCD ped outputs in binders.
After you are done with pedestals, do a CAMAC by {hardware}: <CAMAC> -> <ALL>
You may or may not want to write the cosmics to tape, it's the shift-leaders decision.
Do logging procedure in section III.
We are now using the MIDAS Electronic Logbook instead of the paper logbook. So only follow the below procedure if the ELog is not working.
It is located at http://bnlku33.phy.bnl.gov:8081/EL/?exp=E949
Check out Electronic Logbook in Appendix F.
Beginning-of-Run (BOR) time
Run number
SSP file
Trigger file
End-of-Run (EOR) time
Run number
Number of events in run (approximate # is fine)
If an error stopped the run, look at the 4 PPC windows and try to find the error message and write or print out the details of the error.
trips in separator, magnets, high voltage
when you paused a run & why
errors that popup from DeToX, QOD, RSMon, etc.
anything that people do upstairs
etc.
You may have to call an expert in the middle of the night (don't tell them I put you up to it).
Pause or end the run, if the data will be corrupted. Check Appendix-E for who is an expert in given areas.
If you are
unfamiliar with QOD, read Appendix-G.
At some points during the running this process maybe automated, however
be sure. Having QOD for every run is
important!
phone numbers - support group x2024
MCR x4662
Most of the time the separator will reset itself, and the voltage will start to rise.
Beam magnet trip.
HV for the RSSC's comes from the Bertan Supplies (red panels) in
the NIM bin above the LeCroy HV Mainframes. There are six NIM modules each
having two channels of HV and each channel has two sectors worth of chambers on
it.
Do NOT leave the switch down in the "Auto Reset" position. This can damage the chambers.
HV for the UTC comes from the Bertan Supplies (red panels) in the NIM bin above the LeCroy HV Mainframes.
Do not proceed with out an experts
advice, the UTC is very delicate!
Current expert is Toshio Numao - TRIUMF - (604)222-7345 toshio@triumf.ca,
Former expert is George Redlinger - BNL - x3056
Many things can happen.
However most of the electronics have steps on what to do. Just look around the area of interest, most
likely an expert has taped the steps onto the readout panel, monitor, etc.
**********************************************************************************************
* used operations in {Toolchest}
*----------------------------------------------------------------------------------
*<DAQ> -starts a production version of data acquisition.
*<DAQ new> -starts a "beta" version of the data acquisition.
*<Cleanup DAQ> -kills (most/all) programs DAQ is running
*<PPC> -Power PCs
* <Reboot PPCs>
* <PPC Windows> -creates ppc01, ppc02, ppc03, ppc04
*
*<High Voltage>
* <DeMon> -Starts DeMon, the high voltage controling program
* <CsI HV>
* <status> -Pops up a window stating if the CsI HV is on or off
* <ON> -Turns CsI on
* <OFF> -Turns CsI off
*
*<QOD> -Quality Of Data
* <DAQ production>
* <Run QOD> -Pops up a QOD window, for use during a normal run
* <Run QOD (cosmic)> -Pops up a QOD window, for use during a cosmic run
* <REPLAY on QOD>
* <REPLAY QOD with Print> -Prints QOD file
*
*<Photo>
* <production>
* <Photo (1sec duration)> -Allows users monitor the events visually
*
*<Shift summary>
* <Last 8 hours > -An emacs window appears, for user comments
* <Last 4 hours > after his/her shift
* <Last 24 hours>
*
*<EOS-MON> -End-of-Spill Monitor of Scalars
*
**********************************************************************************************
**********************************************************************************************
* {E787 Data Acquisition System} or {DAQ}
*-----------------------------------------------------------------------------------------
*<Control> <Processes> <Messages> <Help>
* | | | |
* V V V V
*<Control Panel> <data input > <Clear > <about>
*<run limits > <data reducer > <beep on update > <help >
*<set music > <scaler monitor> *<output to log file>
* <verbose >
* <event builer > -> <dd.eb >
* <e787daq>
* <fifo monitor > -> <dd.eb >
* <e787daq>
* <PPC consoles > -> <e787ppc01>
* <e787ppc02>
* <e787ppc03>
* <e787ppc04>
*
*-----------------------------------------------------------------------------------------
* run status date
* ------------------------------------------------------------
* Run number Spill number kbytes/event
* Events in run Events in spill Event limit
* Mbytes in run kbytes in spill Volume limit (tape limit)
* Running time Spill time Spill timeout
*
* DI state :
* Output volume:
* SSP init stat:
* TRIGintt stat:
*-----------------------------------------------------------------------------------------
*(The 3 rows of < ... > below are in the same location,)
*(but are different at different stages of the run )
*
* <hardware> <begin run> < exit > -before a run
* <comment > < pause > <end run > -duringa run
* <comment.> <continue > <end run > -during a pause
*
*-----------------------------------------------------------------------------------------
* Updated information every ~2 seconds.
*
*
**********************************************************************************************
**********************************************************************************************
* {hardware initialization}
*-----------------------------------------------------------------------------------------
* <setup> <pedestals> <CAMAC> <tests> <help>
* | | | |
* V V V V
*<SSP init > <feraped> <fera > <tdctest>
*<SSP config > <fbped > <predet >
*<trig > <ccd ped> <cfd >
*<reboot PPCs> <tddac > <scf >
* <flash >
* <level 1.2>
* <all >
*
*-----------------------------------------------------------------------------------------
* <clear> <OK>
**********************************************************************************************
**********************************************************************************************
* {dp0}
*-------------------------------------------------------------------------------------------
* Defined Streams
*-------------------------------------------------------------------------------------------
* stream device status label triggers
*-------------------------------------------------------------------------------------------
*C_stdmix /dev/mx0nrv open 30076.2.1 ALL
*C_stdmix /dev/mx1nrv open 30076.2.2 ALL
*
*
*
*
*-------------------------------------------------------------------------------------------
*status file close /dev/mx2nrv 0 -an example of what
*device /dev/mx2nrv now closed is here
*Cycling down the cycle info; dp0_idx = 2, pid = 18469
*stream # 0 deleted
*-------------------------------------------------------------------------------------------
* Configure Streams Open/Close Streams
* --------------------------------------- -------------------------------------------
* | --------------- ---------------- | | ------------------- ---------------- |
* | |One Stream | | One Stream | | | | Open selected | |Close Selected| |
* | | to tape | | to disk | | | | Stream | | Stream | |
* | --------------- ---------------- | | ------------------- ---------------- |
* | --------------- ---------------- | | ------------------- ---------------- |
* | | Default | | Change | | | |Open all closed | | Close all | |
* | | Pass 0 | |Configuration | | | | Streams | | Open Streams | |
* | --------------- ---------------- | | ------------------- ---------------- |
* --------------------------------------- -------------------------------------------
*
* ------------- ------------------------- -----------
* | | | | | |
* | Monitor | | Data Taking Enabled | | Help |
* | | | | | |
* ------------- ------------------------- -----------
*
**********************************************************************************************
*********************************************************
* {Change Stream Configuration} *
*-------------------------------------------------------*
* *
* ------------------------- *
* | | *
* | Add Stream | *
* | | *
* ------------------------- *
* *
* ------------------------- *
* | | *
* | Delete Stream | *
* | | *
* ------------------------- *
* *
* ------------------------- *
* | | *
* | Delete All Streams | *
* | | *
* ------------------------- *
* *
* ----------------- --------------------- *
* | Save || Read | *
* | Configuration || Configuration file| *
* ----------------- --------------------- *
* ----------------- --------------------- *
* | Enable | | Disable | *
* | Stream | | Stream | *
* ----------------- --------------------- *
* *
* ------------------------------------------*
* | OK | *
* ------------------------------------------*
*********************************************************
****************************************************************************************************
* {DeMon}
*---------------------------------------------------------------------------------------------------
* <File> <Help>
* |
* V
* <Quit>
*---------------------------------------------------------------------------------------------------
* ------------ DeMon Event Log
* Enter channel name: | | ---------------------------------
* ------------ |... |
* |Mainframe 7 loaded ... |
* Subsystem Viewer |... |
* <Error><RS><BV><BL><TT><BM><UV<RV><CV>AD><CO><MC><IC><VC> | |
* ---------------------------------
*----------------------------------------------------------------------------------------------------
* Mainframe Control Panels
* ------------------------
* Mainframe 1 ... ... ... ... ... ... Mainframe 8
* ----------- -----------
* <Power> ...
* [ON/OFF] ----- ON (Red background) ...
* <View > OFF (Green background) ...
* Channels
* in use:
* ---------
* | 256 |
* ---------
* Channels
* in error:
* ---------
* | 0 |
* ---------
*
******************************************************************************************************
Usually the DAQ is up and running (in some form-or-fashion) when one arrives on shift. So you most likely start at step I.i.4. TO SAVE TIME - step (9) can be done while step (5) completes.
Connect to bnlku9.phy.bnl.gov
open an bnlonl account on bnlku9 and type
>toolchest
{TC}: <PPC> -> <PPC Windows> -wait for the prompts in the ppcs before going to step D.
{TC}: <DAQ> -this will create the {dp0} and
{DAQ}: <start> {E787 Data
Acquisition System} ({DAQ} from now on).
{TC}: <High Voltage> ->
<DeMon>
{TC}: <High Voltage> -> <CsI> -> <status>
{TC}: <High Voltage> -> <CsI> -> <ON>
{DAQ}: <hardware> -this will pop-up the {hardware} window.
{hardware}: <setup> ->
<SSP init> -select the desired SSP
file, click <OK>
Load trigger.
{DAQ}: <hardware> -window should be open from step (5)
{hardware}: <setup> -> <trigger> -select the desired SSP file, click <OK>
{DAQ}: <hardware> -hardware window open from steps above
{hardware}: <CAMAC> ->
<all> -Look for errors shown
in the output (on screen)
If they are already open go to step (10).
{dp0}: <open closed streams> -A pop-up menu will appear {tape labels}
{tape labels} -Erase ALL of the field's content.
Type the tape number (only 5 numbers)
{tape labels}: <change all> -Another pop-up window will appear
Click on <append>.
Before going on to step (10) be sure the status of all the streams are open (like example below).
C_stdmix /dev/mx0nrv open 30076.2.1 ALL
C_stdmix /dev/mx1nrv open
30076.2.2 ALL
Start the run.
{DAQ}: <begin run>
{TC}: <QOD> -> <QOD (DAQ Production)>
-> <Run QOD> -for a standard run
-> <Run QOD (cosmic)> -for a cosmic run
KOFIA>an # -type "an" with # = to a few thousand
{TC}: <Photo> -> <production> -> <Photo (1 sec duration)>
KOFIA>an -short for analyze
A run ends for various reasons.
In good situations: tape is full, Error: "Tape volume limit"
too many events in run, Error: "Event limit size"
you manually <end run> due to job size, etc.
In bad situations: Some type of DAQ error.
If it is a bad situation, a Cleanup maybe required. Look in Appendix-B.III. "When troubles arise in DAQ."
{DAQ}: <end run>
Otherwise, DAQ has ended the run on
its own accord
KOFIA>save -type "save"
Enter save file ID name>{trig}_ -current QOD file format is: 'trigger name'_
For example: trigger used file ID name
smx_v12j "smx_"
km21.ps100 "km21_"
open an bnlonl account on bnlku9
$ONLINE_BIN/dd_cleanup
goto B.I.ii.2.a
You will have to do a cleanup after this is done! (B.III.3.a)
It is best to do pedestals when we have NO beam. That way we maximize the pnn data we
collect. So if the shift-taker knows
that there will be a time during shift when no beam will occur, wait until then
(unless told by experts otherwise). If
are expecting down time in the AGS (or C-line), go ahead at some point during
your shift and do feraped, tdped, ccd ped.
But DO NOT DO A TDDAC WITH BEAM ON!
{DAQ}: <hardware>
{hardware}: <pedestals>
-> <feraped>
{DAQ}: <hardware>
{hardware}: <pedestals> -> <tdped>
{DAQ}: <hardware>
{hardware}: <pedestals>
-> <ccd ped> -> "all_crates" <OK>
{hardware}: <pedestals> ->
<tddac> -> "all" <OK>
Do II.4.b until the channel/board does not show a "Bad channel". NOTE: a "not-so-bad" channel is acceptable.
log in as online on bnlku9
>cd $TDDAC_BIN
>do_tddac_ch_board CRATE BOARD -get CRATE address and BOARD number from {hardware} screen.
Requires a: Cleanup ppc-reboot CAMAC
spill count mismatch Fera board off (noticed by an unrecoverable trigger hang)
*DC2 header word wrong
Event count mismatch
Event number mismatch
Close data streams.
{dp0}: <close all open streams>
{Toolchest}: <Cleanup DAQ> -this will kill the DAQ programs and reboot ppcs *if ppcs do not reboot, do it manually. Go to step Appedix-B.I.2
Control-x on each {ppc##} window. -Manual reboot
OR {hardware}: <setup> -> <reboot PPCs>
OR {TC}: <PPC> -> <Reboot PPCs>
Appendix-B.I.7
{dp0}: <Open all closed
Streams>
{dp0}: <Open all Open Streams>
To modify the streams:
{dp0}:
<Change Configuration> -This
will popup {Change Stream Configuration} referred to {CSC} from now on.
{CSC}: <Delete All Streams>.
{CSC}: <Add Stream>
{CH_CONF_SR_DS}: select the desired file a click <OK>.
For current running with 2 tapes the files to use are:
std_mix_01_0.1 std_mix_01_1.0 std_mix_01_2.0 std_mix_01_3.0
std_mix_01_0.2 std_mix_01_1.2 std_mix_01_2.1 std_mix_01_3.1
std_mix_01_0.3 std_mix_01_1.3 std_mix_01_2.3 std_mix_01_3.2
The above files are all variations of using 2 tapes. The format is trigger_year_"tape 1's device"."tape 2's device"
Open a bnlonl account on bnlku9
>cd /e787d/bnlonl/online_conffiles
>emacs 'newfile_name' &
Current format for a file is:
C_stdmix /dev/mx2nrv ALL - stream device trigger
C_stdmix /dev/mx3nrv ALL
C_ must be the prefix of the stream
{dp0}: <Monitor> -Big button on bottom-left
These popup windows will show how much tape is used and the rate of transfer to each tape.
If the streams are not opening/closing, check to see if the streams in {dp0} match the devices that are being used. If not, change the streams to the correct devices, C.III.
If all else fails, do a cleanup and restart DAQ, thus restarting dp0, see Appendix-B.111
Do NOT operate the 'handle' on the DLT drive unless the "operate handle light is ON"
When you close the stream a pop-up window will ask if you want to print the label for this tape. Click <yes>. The tape's label will be printed out. If you are writing to two tapes, you will have to click <yes> twice and get 2 tape labels. Cut out the larger label and keep it for step 8.
>mt -t /dev/mx0nrv unload -Substitute the relevant DLT drive name above. The drives names are:
mx0nrv mx2nrv -this is the way they are
mx3nrv mx1nrv geometrically arranged.
On a new entry, place ~bnlonl/online_logfiles/tape_labels/{tape #}_{01 or 02}.ps for Attachment...
For example, your two tapes are 30204.2.1 and 30204.2.2 do the following:
Attachment1: ~bnlonl/online_logfiles/tape_labels/30204_01.ps
Attachment2:
~bnlonl/online_logfiles/tape_labels/30204_02.ps
The label sheet should be near the tape drives.
It will have 4 columns of numbers such as
30078._.1_ 30078._.2_ 30078._.3_ 30078._.4_
If we are still using two tapes during running, the two labels needed would For example: 30078._.1_ 30078._.2_
The print-out will have a large and small summary table. Place the large summary table in the plastic case with the tape. Place the small summary table in the log book.
If you don't click <ok> on the popup window OR if you notice a tape without a label:
Open a bnlonl account on bnlku9
>cd ~bnlonl/online_logfiles/tape_labels
>latex {DLT tape #} -creates a .dvi file
>dvips {DLT tape #} -creates a .ps file
>lpr -Pe787lp2r {DLT tape #}.ps -prints the .ps file
"xxx monitor" -xxx will be the monitor-number used below
mx0nrv mx2nrv mx3nrv mx1nrv
It will have 4 columns of numbers such as
mon01.00081 mon01.00101
mon01.00121 mon01.00141
Make sure the 'Operate Handle'
light is on.
Cut out the two tables. Place one in the case with the monitor tape (the other label was placed in the logbook, which we no longer use). So place the tape summary file as an attachment to an ELOG entry (automatically done). Example: Attachment1: /e949d/e787/monitors/backup_new/log/label/mon01_xxx_tex_label.ps
Push a tab on the side with the label, to the left. An orange square will be showing if it is write-protected.
DeMon a program that controls and monitors the high voltage to the detector elements (except CsI aka end-caps).
To open Demon: -{TC}: <High Voltage> -> <DeMon>
OR -open a bnlonl account on bnlku9: and >dm
At some
point an expert may tell a shift taker to 'load' a new high voltage file. Most
likely the expert has made the changes he/she wanted and all they want you to
do is to Restart DeMon. Do the following
to restart DeMon.
{DeMon}: <File> -> <Quit> -This will exit DeMon
{TC}: <High Voltage> -> <DeMon> -This will startup DeMon
Use the "Enter channel name" field. The format of the channels is:
Range Stack (RS), Barrel Veto (BV), Target (TT)
Error - channels that DeMon thinks
are out of tolerance
RS-range stack BV - barrel veto BL-veto liner
TT-target BM-
UV- RV-
CV- AD- CO-
MC- IC- VC-
> move down one channel
< move up one channel
>> move down one page
<< move up one page
>>| move to the bottom
|<< move to the top
It will take a few minutes to load the files.
bnlku9_e787[n]>setup online
bnlku9_e787[n]>cd $DEMON_DATA -this will take you to
/e787/local/online/demon/data.2001
For example: you have created a new range stack high voltage
file (hv.rs.45447), then do the following:
bnlku9_e787[n]>rm hv.rs
bnlku9_e787[n]>ln -s hv.rs.45447
hv.rs
Make a note in the log book when, why, and to what file.
Restart DeMon. (E.I)
>set PRIV
Password>system
>logout port n -n = crate number 1 through 8 (9
= CsI)
Many of the elog entries are now automated. However try to keep an eye on the entries during your shift. It is located at http://bnlku33.phy.bnl.gov:8081/EL/?exp=E949
The time of the entry is automatic and is the time at which you hit <submit>.
Automatically enters current run number, but this can be modified.
Any person that was involved with the entry is entered here, such as benjil for Benji Lewis.
Anything standard during the running
Something has corrupted the data, i.e. the data is not useable for analysis.
Enter highlights during your shift.
Something has changed to files, electronics, etc.
When the DAQ crashes, use this type. Even though the DAQ has crashed routinely in the past do not put it under Routine.
Used by the online script
Anything dealing with DAQ
If you have modified some sub-detector in the experiment use this system.
If you have modified some electronics in the experiment use this system.
Changes/problems in the beamline
If the AGS, has told you something
?
Place a summary of the Text field.
This is the body of the entry. Place as much detail as to inform other collaborators of what has happen.
Associated files with the entry.
Beginning-of-Run (BOR) time, Run number, SSP file, Trigger file, stored to disk/tape/trash can.
End-of-Run (EOR) time, Run number, Number of events
Prints the tape label file and stores as an attachment here.
Automatically entered now, however still needs the users input for the highlights, etc.
The pedestals feraped, fbped, ccdped, tddac enters into elog that it has been performed. The pedestal also enters the file that is also printed out.
trips in separator, magnets, high voltage
when you paused a run & why
errors that popup from DeToX, QOD, RSMon, etc.
anything that people do upstairs
etc.