BNL WCWC
Table of Contents
- 1. Introduction
- 2. Enrollment
- 3. User
- 4. Developer
- 5. Admin
- 5.1. Workstation prerequisites
- 5.2. Hostname
- 5.3. Basic configuration
- 5.4. Provide
/wcwc - 5.5. User accounts
- 5.6. Migrating a pre-existing
/home - 5.7. Add user to IDM
- 5.8. Enroll workstation in the IDM
- 5.9. Configure host to IDM
- 5.10. UID on
$HOMEprovider - 5.11.
$HOMEfor the homeless - 5.12. Annexed
$HOMEprovider - 5.13. Autofs overview
- 5.14. Debugging autofs
- 5.15. Walk throughs
- 5.16. Internal data
- 5.17. External data
- 5.18. Build host
- 5.19. External sync host
- 5.20. Misc checks
- 5.21. Installing Spack packages
- 5.22. Developing Spack packages
- 5.23. Updating wire-cell-spack
- 5.24. The
/srvmount - 5.25. CVMFS
1. Introduction
This document uses terms from and extends wcwc.html with BNL-specific information. In particular, review that document for definitions of local, remote, internal, external, etc. Unless stated otherwise, WCWC refers here to the BNL instance. The first section describes user and workstation enrollment in the BNL WCWC. Then sections with specific information intended for BNL WCWC users and admins, respectively. Specifications follow RFC 2119 language.
The WCWC project has all (meta) files in the brettviren/wcwc GitHub repo. Users of general WCWC features or the BNL WCWC instance are encouraged to report problems via that repository’s GitHub Issues. Any problems specific to non-BNL WCWC instances, should be reported elsewhere.
2. Enrollment
2.1. Workstation enrollment
Enrollment of a workstation into the BNL WCWC is initiated by its BNL custodian (“owner”) or a designee. These steps shall be followed:
- Owner contacts the BNL WCWC admin to discuss any special needs for the workstation and or user(s) it brings along.
- Remaining steps may be performed by the owner or with negotiation by the BNL WCWC admin.
- A supported OS is installed (see section 2.3)
- The OS is configured to use a static IP address that has been allocated from BNL ITD.
- A copy of the BNL WCWC admin SSH public keys is placed in
/root/.ssh/authorized_keys.
These requirements are for member workstations. WCWC users need not enroll a workstation in order to use the WCWC. The only requirement for non-member computers is to provide the user an SSH client.
2.2. User enrollment
Every user must perform these tasks to enroll in the BNL WCWC:
- Obtain a Physics department SSH account.
- Determine the home provider workstation.
- User and admin coordinate on adjusting account IDs (see section 5.10)
2.3. Supported operating systems
BNL WCWC currently supports the Debian version 12 distribution of GNU/Linux (aka “stable” aka “bookworm”). BNL WCWC workstations will be upgraded to Future releases of Debian in coordination with the workstations custodians and WCWC users.
3. User
This section provides guidance specifically for users of the BNL WCWC. See also wcwc.html#user.
3.1. Software packages
A goal of WCWC is to provide all software a user needs in a form that is consistent across all workstations in the cluster. Such software is provided either by OS packages or by Spack. If you need some software follow these guidelines to find/get it:
Check what is installed on OS or Spack:
$ dpkg -l | grep name $ wcwc find | grep name
Check what could be installed on OS or Spack:
$ apt-cache search name $ wcwc list # base spack stack # wcwc list -S wirecell # a dependent stack
- Contact the WCWC admin for assistance by providing package names and any version constraints. If software is not in OS or Spack or the available versions are not sufficient, there is some chance Spack packaging can be developed. User involvement in that packaging process is encouraged but not required.
Some software is out of scope to be provided by WCWC OS or Spack.
- Copies of software under development by the user (but see wcwc.html#devel).
- Proprietary software (at least in general, special cases may be considered).
3.2. Remote access
Remote access is when you the user are sitting at some local computer which may be internal to the wired BNL campus LAN or maybe external on a BNL wireless or non-BNL network and you wish to access a remote, internal WCWC account. To allow for transparent remote access across different local accounts, you are recommended to implement the guidance in this section.
As one example, place the following text at the top of ~/.ssh/config file and replace WCWCUSER with your WCWC user name:
Host *.phy
User WCWCUSER
ProxyJump WCWCUSER@ssh.phy.bnl.gov
Hostname %h.bnl.gov
ControlMaster auto
ControlPath ~/.ssh/%r@%h:%p
ControlPersist 10m
ForwardAgent yes
ForwardX11 yes
PreferredAuthentications publickey
ServerAliveInterval 60
Host *.phy.bnl.gov
User WCWCUSER
ProxyJump none
ControlMaster auto
ControlPath ~/.ssh/%r@%h:%p
ControlPersist 10m
ForwardAgent yes
ForwardX11 yes
PreferredAuthentications publickey
ServerAliveInterval 60
This configuration will work when used in both internal and external accounts. However, it relies on the you the user to use a host name like wcgpu1.phy when a jump through the SSH gateway is needed and wcgpu1.phy.bnl.gov when it isn’t. See below for more advanced configuration that allow for transparent access regardless of what network the local account resides.
Next, assure your local shell has access to an ssh-agent and that the agent has been loaded with SSH keys. A good desktop environment (DE) will have started an ssh-agent for you. You can check with:
$ ssh-add -l
If that responds with:
$ ssh-add -l Could not open a connection to your authentication agent.
then you must either fix your DE or manually start an agent at the shell level:
$ ssh-agent /bin/bash
If ssh-add -l responds with:
$ ssh-add -l The agent has no identities.
then you need to load your SSH private keys:
$ ssh-add Enter passphrase for /home/bv/.ssh/id_rsa: Identity added: /home/bv/.ssh/id_rsa (/home/bv/.ssh/id_rsa) $ ssh-add -l 4096 SHA256:m0bjsB5xmh0hHqs927+hvi94Vhsdi0e5DArl+36qa4s /home/bv/.ssh/id_rsa (RSA)
If your SSH key files have non-standard file names, simply list them on the ssh-add command line (they lack the .pub file name extension).
You can now make an unlimited chain of SSH connections starting from your local computer and on to any number of WCWC workstation accounts.
bv@local$ ssh hierocles.phy bviren@hierocles$ ssh wcgpu1.phy.bnl.gov bviren@wcgpu1$
We used the short alias wcgpu1.phy that matches the *.phy given in the first Host line of ~/.ssh/config in order to jump through the SSH gateway as hierocles is already internal. After that, real host names may be used. If the workstation has proper DNS configuration (search and domain) then the simple hostname may be used (eg ssh wcgpu1).
3.3. Advance remote access
The above ~/.ssh/config suffers from requiring the user to use different host names when the local computer is on different networks. There are a few ways to add network transparency to the configuration. One such way is the following:
Host *.bnl.gov
User WCWCUSER
ControlMaster auto
ControlPath ~/.ssh/%r@%h:%p
ControlPersist 10m
Hostname %h
ForwardAgent yes
ForwardX11 yes
PermitLocalCommand yes
PreferredAuthentications publickey
ServerAliveInterval 60
# Avoid loop, do not proxy jump when going to an SSH gateway
Match originalhost ssh.phy.bnl.gov
ProxyJump none
Match originalhost physsh.phy.bnl.gov
ProxyJump none
Match originalhost gateway.phy.bnl.gov
ProxyJump none
# But otherwise proxy jump. Note, ssh.phy.bnl.gov is only in external BNL DNS.
Match originalhost *.phy.bnl.gov !Exec "ip route get 1 | grep -q 130.199.22.24"
ProxyJump WCWCUSER@ssh.phy.bnl.gov
While the user is suggested to try this advanced configuration if they wish to have the same ~/.ssh/config in both internal and external accounts, the rest of this document will assume the configuration in section 3.2.
The Match keyword requires OpenSSH ca. version 6.5 and the “new style” ip command to work on the local computer.
3.4. Remote X11 access
It can be useful to run an X11 client on the remote workstation to display on your local X11 server. There are various ways to do this with different trade-offs.
3.4.1. Forward X11 through SSH
This is perhaps the simplest but is not very performant. Be sure to include ForwardX11 yes in ~/.ssh/config or use -X on the command line (or -Y to also forward “insecure” X11 which may be required for ROOT.
l$ echo $DISPLAY :0 l$ ssh bviren@hierocles.phy.bnl.gov r$ echo $DISPLAY localhost:12.0 r$ wcwc shell -l root/kl5ziao s$ root root [0] TBrowser b
Takes a few seconds and interacting with the GUI is rather slow but possible.
3.4.2. Xpra
The xpra command is installed on WC workstations. It provides a much faster protocol than standard X11, even when tunneled over SSH. It is cross platform so you can run the client on Linux as well as those others (Mac or Windows).
l$ xpra start --ssh=ssh ssh://hierocles.phy.bnl.gov/ --start=xterm ... Actual display used: :5
Note the display number printed here.
This will open an “xterm” (or whatever was given to --start).
r$ echo $DISPLAY :5 r$ wcwc shell -l root/kl5ziao s$ root root [0] TBrowser b
You will notice this is far faster than X11 forwarded over SSH.
The connection can be broken simply by hitting Ctrl-c in the local terminal:
l$ <Ctrl-c>
This will leave the remote Xpra running. You can reconnect and see the previous windows still running:
l$ xpra attach --ssh=ssh ssh://hierocles.phy.bnl.gov/5
To finally stop the remote Xpra you may either run this on your local account:
l$ xpra stop --ssh=ssh ssh://hierocles.phy.bnl.gov/5
Or, on the remote end:
r$ xpra list r$ xpra stop 5
Or, simply use ps and kill manually.
3.5. Troubles
3.5.1. I can’t log into the SSH gateway
Do you have an account? https://www.phy.bnl.gov/Accounts/
Do you have an ssh-agent started?
$ ssh-add -l Could not open a connection to your authentication agent $ ssh-agent /bin/bash
Do you have an ssh-agent but no keys?
$ ssh-add -l The agent has no identities. $ ssh-add Enter passphrase for /home/USER/.ssh/id_rsa: Identity added: /home/USER/.ssh/id_rsa (/home/USER/.ssh/id_rsa) $ ssh-add -l 4096 SHA256:<fingerprint> /home/USER/.ssh/id_rsa (RSA)
The key must be the one provided when getting the SSH gateway account.
3.5.2. I can not log into a WCWC workstation
Do you have an ssh-agent started and loaded with your keys (see above)?
Are you starting from an external account? Then, see above for ssh-agent info and be sure to forward it:
l$ ssh -A USER@ssh.phy.bnl.gov g$ ssh -A USER@workstation.phy.bnl.gov w$
Do you need to use a private key other than what is loaded into Physics dept IDM and you have access to the WCWC $HOME (eg you sit at a WCWC workstation and want to access another). You must authorize the key in your own account:
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
Do you need to use a private key other than what is loaded into Physics dept IDM and you do not have access to your WCWC $HOME (eg your use WCWC in “homeless” mode)?
Contact the WCWC admin.
3.6. Monitoring
3.6.1. Immediate
- top
- venerable CPU/mem process monitor
- btop
- better version of top
- nvtop
- monitor GPUs
3.6.2. Historical
t.b.d.
4. Developer
4.1. Become friends with GPUs
See what (nVidia) GPUs are available:
$ nvidia-smi
Probe with PyTorch
$ wcwc shell -l "py-torch+cuda cuda_arch=89" $ python -c 'import torch; print(torch.cuda.is_available())' True $ python -c 'import torch; print(torch.cuda.device_count())' 2
If you see error like:
UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero.
Tell your WCWC admin to check if nvidia_uvm kernel module has become unloaded.
5. Admin
This section describes specifics about administrating up WCWC workstations with a focus on initial setup into the BNL Physics department identity management (IDM) system (FreeIPA).
5.1. Workstation prerequisites
Prior to enrollment a potential WCWC workstation must be:
- running a supported and up-to-date operating system.
- available via a static IP address allocated from BNL networking.
- configured with the WCWC admin keys in
/root/.ssh/authorized_keys.
5.2. Hostname
The hostname command must resolve to a fully-qualified host name. This can be assured with:
# hostname <host>.phy.bnl.gov # echo <host>.phy.bnl.gov > /etc/hostname
5.3. Basic configuration
- The enrolling workstation shall be given a baseline configuration via Ansible.
The WCWC admin runs:
1$ cd ~/dev/wcwc 2$ emacs ansible/inventory.yml 3$ ansible workstations -m ping -l <hostname>.phy.bnl.gov 4$ ansible-playbook -l <hostname>.phy.bnl.gov' ansible/playbooks/wcwc-workstation.yml
- Enter the source providing Ansible and configured via
direnv - Add the fully qualified host name to the inventory
- Test it is accessible
- Apply the baseline configuration
5.4. Provide /wcwc
A few methods are supported to provide the /wcwc directory that contains all the software packages built by Spack. The “sync” method is recomended if a workstation has sufficient disk space for /wcwc to be a mirror. Otherwise, the workstation owner should choose autofs for robustness against loss of server or fstab for less robustness but to avoid automount delays. Details on these pros and cons and other issues for each choice are in the following sections.
5.4.1. NFS mount via autofs.
This method requires no client-side disk space. It is the most robust of the two NFS-based options for the case where connectivity to the NFS server is interrupted. However, there will be long delays (up to about one minute) each time an NFS mount is accessed any time after autofs has unmounted it. Because /wcwc is mounted on / these delays can manifest when issuing commands that are seemingly unrelated to /wcwc.
To make a client mount /wcwc via autofs they should be in the wcwc_nfs_automounters group in the Ansible inventory. This will apply configure a “direct” autofs entry:
$ cat /etc/auto.direct /wcwc -ro,defaults,_netdev lycastus.phy.bnl.gov:/wcwc
5.4.2. NFS mount via fstab.
This method requires no client-side disk space. After mounting at boot time /wcwc mount remains and there are not ongoing delays waiting for autofs to remount. However, if the server is ever not responsive then the accessing /wcwc will hang, including a boot time. To apply this option add a line to /etc/fstab like:
lycastus.phy.bnl.gov:/wcwc /wcwc nfs ro,defaults,_netdev 0 0
5.4.3. Mirror via lsyncd/rsync
This method avoids all the problems of NFS but requires substantial disk space and on going operations to keep the copy up to date. At time of writing, /wcwc holds 85 GB and it is still young. Options may be developed in the future to enable mirroring a subset.
The mirroring is performed by lsyncd on running on the lycastus server. It utilizes requires rsync over SSH to a special account wcwc-sync on each workstation in this category.
The configuration of the workstation, including creating the wcwc-sync user, installing the lsyncd SSH public key and including the workstation as an lsyncd target is handled by adding the host to wcwc_rsyncers and running:
ansible-playbook ansible/playbooks/wcwc-rsync.yml
5.5. User accounts
- A WCWC user must control a “distributed” BNL Physics Department SSH account from https://www.phy.bnl.gov/Accounts/.
- A WCWC user may control a “local” account on zero or more WCWC workstations.
Note, the distributed account system is provided by an instance of the FreeIPA IDM which also includes features to configure WCWC services. The text will refer to this service as “IDM”.
5.6. Migrating a pre-existing /home
Local account $HOME’s need adjusting if the workstation is a home provider or it will automount /home. Existing
- The directory
/nfs/homemust exist and be populated with any local$HOMEdirectories. - A preexisting direct-mount
/homemust have its/etc/fstabadjusted to mount the partition to/nfs/home - A preexisting non-direct-mount
/homemust be moved to/nfs/home. - The
/homemust exist and be empty prior to enrollment.
5.7. Add user to IDM
A users distributed identity must be added to the WCWC group in FreeIPA:
Identity->Users->usernameUser Groups,+Addand add towcwc.
A user’s $HOME must be added to the automount map:
Network services->Automount->wcwc-u->auto.u->+Add- The
Keyis the username - the
Mount informationis as follows, replacingWORKSTATIONwith the fully-qualified name of the workstation
-fstype=nfs4 -sync,rw,soft,sec=sys,proto=tcp,port=2049 WORKSTATION:/home/&
Internally, FreeIPA assigns large user (and group) ID numbers while we wish to keep the BNL “life” number used at the OS level. To allow this an “ID View” is needed for each user:
Identity->ID Views->wcwc->Users->+Add- Fill in the user name and matching user login, desired UID matching GID, shell and home directory (
/home/USER). - SAVE
5.8. Enroll workstation in the IDM
To enroll a workstation, the WCWC admin must issue these commands as user root.
# ipa-client-install \ --server=idm3.phy.bnl.gov \ --server=idm2.phy.bnl.gov \ --server=idm1.phy.bnl.gov \ --server=idm.phy.bnl.gov \ --domain=phy.bnl.gov # ipa-client-automount --server=idm.phy.bnl.gov --location=wcwc-u
Notes:
- Ignore message
conflicting time&date synchronization service 'ntp' will be disabled in favor of chronyd - Answer “yes” to
Proceed with fixed values and no DNS discovery? - Answer “yes” to
Do you want to configure chrony... - May skip entering of NTP sources.
- Enter authorized user credentials (WCWC admin)
5.9. Configure host to IDM
The WCWC admin must perform these settings via the FreeIPA IDM web interface:
Identity->Hosts->hostname.phy.bnl.gov.- Add primary owner’s real name and other general info about the workstation in description and Save.
- In the
Host Groupstab,+Add, add the host to the groupwcwc-workstations. Identity->ID Views->wcwc->Hosts->+Apply to host groups->wcwc-workstations
Note, this last command will try to re-add all previously added hosts. This throws harmless errors. The issue is FreeIPA does not support adding a host group per se but rather the Hosts list here is flat. One may instead +Apply to hosts and select individual hosts to add.
This may be required to finally get correct IDs:
# service sssd restart # getent passwd bviren bviren:*:22351:22351:Brett Viren:/home/bviren:/usr/bin/fish
5.10. UID on $HOME provider
- A WCWC user’s
$HOMEmust NFS-exported by a WCWC or non-WCWC “home provider” workstation. - A WCWC workstation shall have
/homeas an automount volume controlled by a map in IDM as detailed next. - The user ID number for
$HOMENFS-exported by any home provider (WCWC or non-WCWC) must be made unique according to the WCWC schema and the ownership and permission of user’s files modified to reflect that. - The ID for a user that is a BNL employee must be equal to the employee (aka “life”) number.
The ID for a user that is a BNL guest is formed by taking the 4 digits of the guest number and prepending it with
L - 'A' + 10where “L” is the prefix letter expressed in ASCII and “'A'” is the ASCII encoding for “A”.In order to migrate to a new user ID number the user must logout of the workstation and otherwise kill any running process. As user
root
# groupmod -g NEWGID GROUPNAME # usermod -g NEWGID -u NEWUID USERNAME
The usermod command will change permissions of files in the user’s $HOME but not elsewhere. To list top-level directories owned by the old user ID run:
find / -type d -user OLDUID -prune 2>/dev/null
One may then change ownership for each directory listed.
chown --from=CURRENT_OWNER:CURRENT_GROUP -R NEW_OWNER:NEW_GROUP DIRECTORY
5.11. $HOME for the homeless
A “homeless” user is one that does not have their “own” workstation to provide $HOME. They will be assigned a home provider by the WCWC admin. Likely this will be lycastus, the /wcwc NFS-server. This host is not a NFS-client thus effectively not available for user login.
Adding a “homeless” user requires variant of what is described in section 5.7.
- FreeIPA: Identity -> User -> username -> User Groups -> +Add -> add to “wcwc”.
- FreeIPA: Identity -> ID Views ->
wcwc-> Users -> +Add -> Add user ID override (user, login, uid, gid, shell, home) (see section #home-provider-uid for how to form UID from BNL Life or Guest identifier) - FreeIPA: Network Services -> wcwc-u -> auto.u -> +Add -> key: username, mount:
-fstype=nfs4 -sync,rw,soft,sec=sys,proto=tcp,port=2049 lycastus.phy.bnl.gov:/home/&
- Creating the physical (not actual) home:
# mkdir -p /nfs/home/USER # cp /etc/skel/.* /nfs/home/USER/ # chown -R USER:USER /nfs/home/USER/
- Check on NFS-client (not
lycastus):
$ getent passwd USER $ ls -la /u/USER $ ls -la /home/USER
5.12. Annexed $HOME provider
A workstation which is not yet a member of WCWC may be “annexed” to supply the $HOME for a user which is a member. This requires manual configuration of the annexed workstation some of which must be undone if/when the workstation later becomes a full WCWC member. Much of this configuration is in common with what must be done on a WCWC workstation except we will not automount WCWC $HOME and will leave the native /home in place.
Here are the steps to take, assuming a Debian-like system.
- Adjust UID as in section 5.10.
apt install nfs-kernel-servermkdir -p /nfs/home- Add
/home /nfs/home none bind 0 0to/etc/fstab - One time
mount /nfs/home - Configure NFS with following in
/etc/exports
/nfs 130.199.22.0/23(rw,sync,crossmnt,no_subtree_check,fsid=0) /nfs/home 130.199.22.0/23(rw,sync,no_subtree_check)
- One time
exportfs -avandshowmount -eto check. - FreeIPA: Identity -> User -> username -> User Groups -> +Add -> add to “wcwc”.
- FreeIPA: Identity -> ID Views ->
wcwc-> Users -> +Add -> Add user ID override (user, login, uid, gid, shell, home) - FreeIPA: Network Services -> wcwc-u -> auto.u -> +Add -> key: username, mount:
-fstype=nfs4 -sync,rw,soft,sec=sys,proto=tcp,port=2049 ANNEXED.phy.bnl.gov:/home/&
5.13. Autofs overview
5.14. Debugging autofs
The use of autofs to NFS-mount ties together several things.
- Adding a user’s
$HOME, see section 5.7. - Setting up a home provider, see section 5.10.
- Assuring the host is an
ipa-client-automount, see section 5.8. - Assuring
auto_home.shscript exists. (run ansible playbook).
On NFS-client, the basic check is:
ls -ld /u/USER /home/USER ### show correct ownership by name ls -ldn /u/USER /home/USER ### show ownership by "short" user ID
Both should exist on any WCWC workstation, except special case of the home provider for the “homeless” (users with no workstation of their own).
If they do not exist we go end to end starting at the NFS server:
On NFS-server, does the user account exist and with “short” user ID?
getent passwd USER
On NFS-server, does the the physical (not actual) home directory exists?
ls -la /nfs/home/USER
If it is missing then create it according to section 5.11.
On NFS-server, is /nfs/home exported?
# showmount -e
If not, check /etc/exports. Changing that requires:
# exportfs -ua # exportfs -av ### -v to check # showmount -e ### to double check
Assure /nfs/home is exported to the LAN subnet.
On NFS-client, check direct NFS-mount:
# mount -t nfs4 lycastus.phy.bnl.gov:/home /mnt # ls -l /mnt ### see any homes? # umount /mnt
If not, perhaps NFS-server or NFS-client has a local firewall interfering.
On NFS-client, check autofs is working. After any change to IDM related to autofs, services may need restarting:
# service sssd restart # service autofs restart
On NFS-client, check that autofs knows the maps
# automount --dumpmaps
Should see:
/wcwcinauto.directmap/homeinauto_home.shmap/uinauto.umap
If missing, then make corrections in FreeIPA.
On NFS-client, check that the “program” map script exists and works:
# /etc/auto_home.sh USER
This should return :/nfs/home/USER when run on the home-provider otherwise it should return /u/USER.
On NFS-client, check that the intermediate automount works:
# ls -ld /u/USER
And, finally, assure end result
# ls -ld /home/USER
5.15. Walk throughs
5.15.1. wcgpu1
This is one of 2 dual-GPU “workstations”. They are integrated as “normal” workstations except that they are not home providers. Here we walk through section 5.3.
$ ssh root@wcgpu1.bnl root@wcgpu1:~# hostname wcgpu1.phy.bnl.gov local$ ssh hierocles.bnl internal$ ansible workstations -m ping -l wcgpu1.phy.bnl.gov internal$ ansible-playbook -l 'wcgpu1.phy.bnl.gov' ansible/playbooks/wcwc-workstation.yml
Following section 5.6.
root@wcgpu1:~# mkdir -p /nfs/home root@wcgpu1:~# emacs -nw /etc/fstab # change /home to /nfs/home root@wcgpu1:~# mount /nfs/home
No new WCWC users associated with wcgpu* so skip section 5.7 and continue to section 5.8 performing both client install and automount commands and then section 5.9.
# service sssd restart # service autofs restart # getent passwd bviren bviren:*:22351:22351:Brett Viren:/home/bviren:/usr/bin/fish # ls -ld /home/bviren/ drwxr-xr-x 62 bviren bviren 4096 Aug 20 05:57 /home/bviren/ # ls -lnd /home/bviren/ drwxr-xr-x 62 22351 1601400019 4096 Aug 20 05:57 /home/bviren/ # ls -l /wcwc/spack/bin/spack -rwxr-xr-x 1 1001 1001 1765 Aug 16 11:52 /wcwc/spack/bin/spack $ ssh wcgpu1.bnl # full login check
Again, as this is not a home provider, the rest of the steps above are skipped and we are done.
Some todo items for wcgpu*
[ ]move static IP from DE config to/etc/network/interfaces[ ]purge DE packages
5.16. Internal data
TBD: how to share more than $HOME.
5.17. External data
TBD: how to access eg FNAL or SDCC storage.
5.18. Build host
Typically, package building is performed in an account (wcwc@wcgpu0) on a faster host than the /wcwc NFS provider (lycastus). As the /wcwc is mounted read-only, a writable copy is made in a non-canonical location, a state change is applied and the results returned.
$ rsync -av wcwc@lycastus.phy.bnl.gov:/wcwc/ ~/wcwc-dev/ $ wcwc --prefix=$HOME/wcwc-dev admin install <spec> $ rsync -av ~/wcwc-dev/ wcwc@lycastus.phy.bnl.gov:/wcwc/
5.19. External sync host
A home workstation or laptop or other computers that do not NFS-mount /wcwc may maintain a copy by periodically syncing with the central /wcwc.
wcwc@haiku:~$ rsync --del -a lycastus.bnl.phy.gov:/wcwc/ /wcwc/
The sync can be against any WCWC workstation. To avoid extra network traversal, this example goes to the horses mouth but any workstation NFSmounting /wcwc can also work. If the user wishes to extend the local copy beyond what is on WCWC, they should omit the --del option. Modifying the local copy entail commands as illustrated in section 5.18.
5.20. Misc checks
Desktop machines may be configured to suspend when they are idle and sitting at GDM login. To defeat this, edit /etc/gdm3/greeter.dconf-defaults to set these bits:
[org/gnome/settings-daemon/plugins/power] sleep-inactive-ac-type='nothing' sleep-inactive-battery-timeout=0
And
# systemctl restart gdm3 # systemctl restart gdm
5.21. Installing Spack packages
Spack state modifications are performed on a read-write copy of /wcwc and never directly on /wcwc To make the copy:
$ rsync -a wcwc@lycastus.phy.bnl.gov:/wcwc/ ~/wcwc-dev/
To return changes:
$ cat ~/bin/wcwc-sync rsync -av \ --exclude "*~" --exclude __pycache__ --exclude stage --exclude cache ~/wcwc-dev/ \ wcwc@lycastus.phy.bnl.gov:/wcwc/
This potentially opens a huge race condition if more than one copy is allowed. To avoid this, a single copy in wcwc@wcgpu0 is the working copy. A better workflow should be developed.
While it is possible to directly edit files under ~/wcwc-dev/ any installation and use of packages must be done inside a container:
u$ wcwc --image=yes --prefix=$HOME/wcwc-dev shell s$ ls /wcwc opt spack stacks stage wcwc.db wcwc.yaml s$ spack install [...]
Or, better use wcwc admin (which enforces the container):
$ wcwc --prefix=$HOME/wcwc-dev admin install [...]
The container must also be used to exercise a locally installed package:
$ wcwc --image=yes --prefix=$HOME/wcwc-dev admin shell -l <package> # or $ wcwc --image=yes --prefix=$HOME/wcwc-dev admin shell -S wirecell -l <package-in-wirecell>
When state changes applied to ~/wcwc-dev/ are complete, “commit” them by rsync’ing back:
$ ./bin/wcwc-sync
In the future, the sync parts of this workflow may become incorporated into wcwc.
5.22. Developing Spack packages
It is common that a Spack package recipe must be updated, if only to bump the supported version(s). Here is an example of that workflow, starting with entering the container (c$ prompt):
u$ wcwc --image=yes --prefix=$HOME/wcwc-dev shell c$ spack checksum <package>
When substantial fixes to the Spack recipe is needed, beyond a new version() line, it is good to PR it back to Spack. This is best done out-of-band of WCWC and in a personal fork of Spack:
$ git clone git@github.com:USER/spack.git ~/dev/spack-arrayfire $ cd ~/dev/spack-arrayfire $ git checkout -t -b bugfix-arrayfire origin/develop $ source spack/share/spack/setup-env.sh $ spack install arrayfire
Now, fix the build errors, push to a branch in the personal fork and from there make a PR to spack, proper.
5.23. Updating wire-cell-spack
General (non-wcwc) instructions are at wire-cell-spack. Before WCWC can use them, any updates need to be pushed there. To test that development, see Stand-alone Spack builds.
As wcwc@wcgpu0 do these steps:
rsync --archive --delete wcwc@lycastus.phy.bnl.gov:/wcwc/ ~/wcwc-dev/ wcwc --image=yes --prefix=$HOME/wcwc-dev shell
5.24. The /srv mount
The wcgpu machines have 2+2+4 TB SSDs (only 3 M.2 slots).
- nvme0
- 2TB system usage
- nvme1
- 2TB root and home
- nvme2
- 4TB data
Here is preparing nvme0
# fdisk -l # df -hl # fdisk /dev/nvme0n1 g n w # mke2fs /dev/nvme0n1p1 # blkid /dev/nvme0n1p1 # emacs /etc/fstab # mount as /srv
The factory partitioning of nvme1 what kept. It is not optimal and so nvme0 is used to back fill some small system areas. At some point (yeah, right) the system disk should be repartitioned to better allocate. Maybe if/when user storage needs upgrading.
5.25. CVMFS
With /srv mounted as per section 5.24,
# mkdir -p /srv/sys/var/lib/cvmfs /var/lib/cvmfs # emacs /etc/fstab # bind mount former on latter
wget https://ecsft.cern.ch/dist/cvmfs/cvmfs-release/cvmfs-release-latest_all.deb dpkg -i cvmfs-release-latest_all.deb rm cvmfs-release-latest_all.deb apt-get update apt-get install cvmfs cvmfs_config setup service autofs restart cat << EOF > /etc/cvmfs/default.local CVMFS_CLIENT_PROFILE=single CVMFS_REPOSITORIES=larsoft.opensciencegrid.org,uboone.opensciencegrid.org,dune.opensciencegrid.org,dune.osgstorage.org,oasis.opensciencegrid.org,singularity.opensciencegrid.org CVMFS_SERVER_URL="http://cvmfs.sdcc.bnl.gov:8000/cvmfs/@org@;http://cvmfs-s1fnal.opensciencegrid.org:8000/cvmfs/@fqrn@" CVMFS_HTTP_PROXY="http://cvmfs-cache.sdcc.bnl.gov:3128|http://cvmfs-cache.sdcc.bnl.gov:3128;DIRECT" EOF cvmfs_config probe