BNL WCWC

Table of Contents

1. Introduction

This document uses terms from and extends wcwc.html with BNL-specific information. In particular, review that document for definitions of local, remote, internal, external, etc. Unless stated otherwise, WCWC refers here to the BNL instance. The first section describes user and workstation enrollment in the BNL WCWC. Then sections with specific information intended for BNL WCWC users and admins, respectively. Specifications follow RFC 2119 language.

The WCWC project has all (meta) files in the brettviren/wcwc GitHub repo. Users of general WCWC features or the BNL WCWC instance are encouraged to report problems via that repository’s GitHub Issues. Any problems specific to non-BNL WCWC instances, should be reported elsewhere.

2. Enrollment

2.1. Workstation enrollment

Enrollment of a workstation into the BNL WCWC is initiated by its BNL custodian (“owner”) or a designee. These steps shall be followed:

  • Owner contacts the BNL WCWC admin to discuss any special needs for the workstation and or user(s) it brings along.
    • Remaining steps may be performed by the owner or with negotiation by the BNL WCWC admin.
  • A supported OS is installed (see section 2.3)
  • The OS is configured to use a static IP address that has been allocated from BNL ITD.
  • A copy of the BNL WCWC admin SSH public keys is placed in /root/.ssh/authorized_keys.

These requirements are for member workstations. WCWC users need not enroll a workstation in order to use the WCWC. The only requirement for non-member computers is to provide the user an SSH client.

2.2. User enrollment

Every user must perform these tasks to enroll in the BNL WCWC:

2.3. Supported operating systems

BNL WCWC currently supports the Debian version 12 distribution of GNU/Linux (aka “stable” aka “bookworm”). BNL WCWC workstations will be upgraded to Future releases of Debian in coordination with the workstations custodians and WCWC users.

3. User

This section provides guidance specifically for users of the BNL WCWC. See also wcwc.html#user.

3.1. Software packages

A goal of WCWC is to provide all software a user needs in a form that is consistent across all workstations in the cluster. Such software is provided either by OS packages or by Spack. If you need some software follow these guidelines to find/get it:

  • Check what is installed on OS or Spack:

    $ dpkg -l | grep name
    
    $ wcwc find | grep name
    
  • Check what could be installed on OS or Spack:

    $ apt-cache search name
    
    $ wcwc list              # base spack stack
    # wcwc list -S wirecell  # a dependent stack
    
  • Contact the WCWC admin for assistance by providing package names and any version constraints. If software is not in OS or Spack or the available versions are not sufficient, there is some chance Spack packaging can be developed. User involvement in that packaging process is encouraged but not required.

Some software is out of scope to be provided by WCWC OS or Spack.

  • Copies of software under development by the user (but see wcwc.html#devel).
  • Proprietary software (at least in general, special cases may be considered).

3.2. Remote access

Remote access is when you the user are sitting at some local computer which may be internal to the wired BNL campus LAN or maybe external on a BNL wireless or non-BNL network and you wish to access a remote, internal WCWC account. To allow for transparent remote access across different local accounts, you are recommended to implement the guidance in this section.

As one example, place the following text at the top of ~/.ssh/config file and replace WCWCUSER with your WCWC user name:

Host *.phy
	User WCWCUSER
	ProxyJump WCWCUSER@ssh.phy.bnl.gov
    Hostname %h.bnl.gov
	ControlMaster auto
	ControlPath ~/.ssh/%r@%h:%p
	ControlPersist 10m
	ForwardAgent yes
	ForwardX11 yes
	PreferredAuthentications publickey
	ServerAliveInterval 60

Host *.phy.bnl.gov
	User WCWCUSER
	ProxyJump none
	ControlMaster auto
	ControlPath ~/.ssh/%r@%h:%p
	ControlPersist 10m
	ForwardAgent yes
	ForwardX11 yes
	PreferredAuthentications publickey
	ServerAliveInterval 60

This configuration will work when used in both internal and external accounts. However, it relies on the you the user to use a host name like wcgpu1.phy when a jump through the SSH gateway is needed and wcgpu1.phy.bnl.gov when it isn’t. See below for more advanced configuration that allow for transparent access regardless of what network the local account resides.

Next, assure your local shell has access to an ssh-agent and that the agent has been loaded with SSH keys. A good desktop environment (DE) will have started an ssh-agent for you. You can check with:

$ ssh-add -l

If that responds with:

$ ssh-add -l
Could not open a connection to your authentication agent.

then you must either fix your DE or manually start an agent at the shell level:

$ ssh-agent /bin/bash

If ssh-add -l responds with:

$ ssh-add -l
The agent has no identities.

then you need to load your SSH private keys:

$ ssh-add
Enter passphrase for /home/bv/.ssh/id_rsa: 
Identity added: /home/bv/.ssh/id_rsa (/home/bv/.ssh/id_rsa)
$ ssh-add -l
4096 SHA256:m0bjsB5xmh0hHqs927+hvi94Vhsdi0e5DArl+36qa4s /home/bv/.ssh/id_rsa (RSA)

If your SSH key files have non-standard file names, simply list them on the ssh-add command line (they lack the .pub file name extension).

You can now make an unlimited chain of SSH connections starting from your local computer and on to any number of WCWC workstation accounts.

bv@local$ ssh hierocles.phy
bviren@hierocles$ ssh wcgpu1.phy.bnl.gov
bviren@wcgpu1$

We used the short alias wcgpu1.phy that matches the *.phy given in the first Host line of ~/.ssh/config in order to jump through the SSH gateway as hierocles is already internal. After that, real host names may be used. If the workstation has proper DNS configuration (search and domain) then the simple hostname may be used (eg ssh wcgpu1).

3.3. Advance remote access

The above ~/.ssh/config suffers from requiring the user to use different host names when the local computer is on different networks. There are a few ways to add network transparency to the configuration. One such way is the following:

Host *.bnl.gov
        User WCWCUSER
        ControlMaster auto
        ControlPath ~/.ssh/%r@%h:%p
        ControlPersist 10m
        Hostname %h
        ForwardAgent yes
        ForwardX11 yes
        PermitLocalCommand yes
        PreferredAuthentications publickey
        ServerAliveInterval 60

# Avoid loop, do not proxy jump when going to an SSH gateway
Match originalhost ssh.phy.bnl.gov 
        ProxyJump none
Match originalhost physsh.phy.bnl.gov 
        ProxyJump none
Match originalhost gateway.phy.bnl.gov 
        ProxyJump none
# But otherwise proxy jump.  Note, ssh.phy.bnl.gov is only in external BNL DNS.
Match originalhost *.phy.bnl.gov !Exec "ip route get 1 | grep -q 130.199.22.24"
        ProxyJump WCWCUSER@ssh.phy.bnl.gov

While the user is suggested to try this advanced configuration if they wish to have the same ~/.ssh/config in both internal and external accounts, the rest of this document will assume the configuration in section 3.2.

The Match keyword requires OpenSSH ca. version 6.5 and the “new style” ip command to work on the local computer.

3.4. Remote X11 access

It can be useful to run an X11 client on the remote workstation to display on your local X11 server. There are various ways to do this with different trade-offs.

3.4.1. Forward X11 through SSH

This is perhaps the simplest but is not very performant. Be sure to include ForwardX11 yes in ~/.ssh/config or use -X on the command line (or -Y to also forward “insecure” X11 which may be required for ROOT.

l$ echo $DISPLAY
:0

l$ ssh bviren@hierocles.phy.bnl.gov
r$ echo $DISPLAY
localhost:12.0

r$ wcwc shell -l root/kl5ziao
s$ root
root [0] TBrowser b

Takes a few seconds and interacting with the GUI is rather slow but possible.

3.4.2. Xpra

The xpra command is installed on WC workstations. It provides a much faster protocol than standard X11, even when tunneled over SSH. It is cross platform so you can run the client on Linux as well as those others (Mac or Windows).

l$ xpra start --ssh=ssh ssh://hierocles.phy.bnl.gov/ --start=xterm
...
Actual display used: :5

Note the display number printed here.

This will open an “xterm” (or whatever was given to --start).

r$ echo $DISPLAY
:5

r$ wcwc shell -l root/kl5ziao
s$ root
root [0] TBrowser b

You will notice this is far faster than X11 forwarded over SSH.

The connection can be broken simply by hitting Ctrl-c in the local terminal:

l$ <Ctrl-c>

This will leave the remote Xpra running. You can reconnect and see the previous windows still running:

l$ xpra attach --ssh=ssh ssh://hierocles.phy.bnl.gov/5

To finally stop the remote Xpra you may either run this on your local account:

l$ xpra stop --ssh=ssh ssh://hierocles.phy.bnl.gov/5

Or, on the remote end:

r$ xpra list
r$ xpra stop 5

Or, simply use ps and kill manually.

3.5. Troubles

3.5.1. I can’t log into the SSH gateway

Do you have an account? https://www.phy.bnl.gov/Accounts/

Do you have an ssh-agent started?

$ ssh-add -l
Could not open a connection to your authentication agent
$ ssh-agent /bin/bash

Do you have an ssh-agent but no keys?

$ ssh-add -l
The agent has no identities.
$ ssh-add
Enter passphrase for /home/USER/.ssh/id_rsa: 
Identity added: /home/USER/.ssh/id_rsa (/home/USER/.ssh/id_rsa)
$ ssh-add -l
4096 SHA256:<fingerprint> /home/USER/.ssh/id_rsa (RSA)

The key must be the one provided when getting the SSH gateway account.

3.5.2. I can not log into a WCWC workstation

Do you have an ssh-agent started and loaded with your keys (see above)?

Are you starting from an external account? Then, see above for ssh-agent info and be sure to forward it:

l$ ssh -A USER@ssh.phy.bnl.gov
g$ ssh -A USER@workstation.phy.bnl.gov
w$

Do you need to use a private key other than what is loaded into Physics dept IDM and you have access to the WCWC $HOME (eg you sit at a WCWC workstation and want to access another). You must authorize the key in your own account:

$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Do you need to use a private key other than what is loaded into Physics dept IDM and you do not have access to your WCWC $HOME (eg your use WCWC in “homeless” mode)? Contact the WCWC admin.

3.6. Monitoring

3.6.1. Immediate

top
venerable CPU/mem process monitor
btop
better version of top
nvtop
monitor GPUs

3.6.2. Historical

t.b.d.

4. Developer

4.1. Become friends with GPUs

See what (nVidia) GPUs are available:

$ nvidia-smi

Probe with PyTorch

$ wcwc shell -l "py-torch+cuda cuda_arch=89"
$ python -c 'import torch; print(torch.cuda.is_available())'
True
$ python -c 'import torch; print(torch.cuda.device_count())'
2

If you see error like:

UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero.

Tell your WCWC admin to check if nvidia_uvm kernel module has become unloaded.

5. Admin

This section describes specifics about administrating up WCWC workstations with a focus on initial setup into the BNL Physics department identity management (IDM) system (FreeIPA).

5.1. Workstation prerequisites

Prior to enrollment a potential WCWC workstation must be:

  • running a supported and up-to-date operating system.
  • available via a static IP address allocated from BNL networking.
  • configured with the WCWC admin keys in /root/.ssh/authorized_keys.

5.2. Hostname

The hostname command must resolve to a fully-qualified host name. This can be assured with:

# hostname <host>.phy.bnl.gov
# echo <host>.phy.bnl.gov > /etc/hostname

5.3. Basic configuration

  • The enrolling workstation shall be given a baseline configuration via Ansible.

The WCWC admin runs:

1$ cd ~/dev/wcwc
2$ emacs ansible/inventory.yml
3$ ansible workstations -m ping -l <hostname>.phy.bnl.gov
4$ ansible-playbook -l <hostname>.phy.bnl.gov' ansible/playbooks/wcwc-workstation.yml
  1. Enter the source providing Ansible and configured via direnv
  2. Add the fully qualified host name to the inventory
  3. Test it is accessible
  4. Apply the baseline configuration

5.4. Provide /wcwc

A few methods are supported to provide the /wcwc directory that contains all the software packages built by Spack. The “sync” method is recomended if a workstation has sufficient disk space for /wcwc to be a mirror. Otherwise, the workstation owner should choose autofs for robustness against loss of server or fstab for less robustness but to avoid automount delays. Details on these pros and cons and other issues for each choice are in the following sections.

5.4.1. NFS mount via autofs.

This method requires no client-side disk space. It is the most robust of the two NFS-based options for the case where connectivity to the NFS server is interrupted. However, there will be long delays (up to about one minute) each time an NFS mount is accessed any time after autofs has unmounted it. Because /wcwc is mounted on / these delays can manifest when issuing commands that are seemingly unrelated to /wcwc.

To make a client mount /wcwc via autofs they should be in the wcwc_nfs_automounters group in the Ansible inventory. This will apply configure a “direct” autofs entry:

$ cat /etc/auto.direct
/wcwc -ro,defaults,_netdev lycastus.phy.bnl.gov:/wcwc

5.4.2. NFS mount via fstab.

This method requires no client-side disk space. After mounting at boot time /wcwc mount remains and there are not ongoing delays waiting for autofs to remount. However, if the server is ever not responsive then the accessing /wcwc will hang, including a boot time. To apply this option add a line to /etc/fstab like:

lycastus.phy.bnl.gov:/wcwc /wcwc nfs ro,defaults,_netdev 0 0

5.4.3. Mirror via lsyncd/rsync

This method avoids all the problems of NFS but requires substantial disk space and on going operations to keep the copy up to date. At time of writing, /wcwc holds 85 GB and it is still young. Options may be developed in the future to enable mirroring a subset.

The mirroring is performed by lsyncd on running on the lycastus server. It utilizes requires rsync over SSH to a special account wcwc-sync on each workstation in this category.

The configuration of the workstation, including creating the wcwc-sync user, installing the lsyncd SSH public key and including the workstation as an lsyncd target is handled by adding the host to wcwc_rsyncers and running:

ansible-playbook  ansible/playbooks/wcwc-rsync.yml

5.5. User accounts

  • A WCWC user must control a “distributed” BNL Physics Department SSH account from https://www.phy.bnl.gov/Accounts/.
  • A WCWC user may control a “local” account on zero or more WCWC workstations.

Note, the distributed account system is provided by an instance of the FreeIPA IDM which also includes features to configure WCWC services. The text will refer to this service as “IDM”.

5.6. Migrating a pre-existing /home

Local account $HOME’s need adjusting if the workstation is a home provider or it will automount /home. Existing

  • The directory /nfs/home must exist and be populated with any local $HOME directories.
  • A preexisting direct-mount /home must have its /etc/fstab adjusted to mount the partition to /nfs/home
  • A preexisting non-direct-mount /home must be moved to /nfs/home.
  • The /home must exist and be empty prior to enrollment.

5.7. Add user to IDM

A users distributed identity must be added to the WCWC group in FreeIPA:

  1. Identity -> Users -> username
  2. User Groups, +Add and add to wcwc.

A user’s $HOME must be added to the automount map:

  1. Network services -> Automount -> wcwc-u -> auto.u -> +Add
  2. The Key is the username
  3. the Mount information is as follows, replacing WORKSTATION with the fully-qualified name of the workstation
-fstype=nfs4 -sync,rw,soft,sec=sys,proto=tcp,port=2049 WORKSTATION:/home/&

Internally, FreeIPA assigns large user (and group) ID numbers while we wish to keep the BNL “life” number used at the OS level. To allow this an “ID View” is needed for each user:

  1. Identity -> ID Views -> wcwc -> Users -> +Add
  2. Fill in the user name and matching user login, desired UID matching GID, shell and home directory (/home/USER).
  3. SAVE

5.8. Enroll workstation in the IDM

To enroll a workstation, the WCWC admin must issue these commands as user root.

# ipa-client-install \
  --server=idm3.phy.bnl.gov \
  --server=idm2.phy.bnl.gov \
  --server=idm1.phy.bnl.gov \
  --server=idm.phy.bnl.gov \
  --domain=phy.bnl.gov
# ipa-client-automount --server=idm.phy.bnl.gov --location=wcwc-u

Notes:

  • Ignore message conflicting time&date synchronization service 'ntp' will be disabled in favor of chronyd
  • Answer “yes” to Proceed with fixed values and no DNS discovery?
  • Answer “yes” to Do you want to configure chrony...
  • May skip entering of NTP sources.
  • Enter authorized user credentials (WCWC admin)

5.9. Configure host to IDM

The WCWC admin must perform these settings via the FreeIPA IDM web interface:

  1. Identity -> Hosts -> hostname.phy.bnl.gov.
  2. Add primary owner’s real name and other general info about the workstation in description and Save.
  3. In the Host Groups tab, +Add, add the host to the group wcwc-workstations.
  4. Identity -> ID Views -> wcwc -> Hosts -> +Apply to host groups -> wcwc-workstations

Note, this last command will try to re-add all previously added hosts. This throws harmless errors. The issue is FreeIPA does not support adding a host group per se but rather the Hosts list here is flat. One may instead +Apply to hosts and select individual hosts to add.

This may be required to finally get correct IDs:

# service sssd restart
# getent passwd bviren
bviren:*:22351:22351:Brett Viren:/home/bviren:/usr/bin/fish

5.10. UID on $HOME provider

  • A WCWC user’s $HOME must NFS-exported by a WCWC or non-WCWC “home provider” workstation.
  • A WCWC workstation shall have /home as an automount volume controlled by a map in IDM as detailed next.
  • The user ID number for $HOME NFS-exported by any home provider (WCWC or non-WCWC) must be made unique according to the WCWC schema and the ownership and permission of user’s files modified to reflect that.
  • The ID for a user that is a BNL employee must be equal to the employee (aka “life”) number.
  • The ID for a user that is a BNL guest is formed by taking the 4 digits of the guest number and prepending it with L - 'A' + 10 where “L” is the prefix letter expressed in ASCII and “'A'” is the ASCII encoding for “A”.

    In order to migrate to a new user ID number the user must logout of the workstation and otherwise kill any running process. As user root

# groupmod -g NEWGID GROUPNAME  
# usermod  -g NEWGID -u NEWUID USERNAME

The usermod command will change permissions of files in the user’s $HOME but not elsewhere. To list top-level directories owned by the old user ID run:

find / -type d -user OLDUID -prune 2>/dev/null

One may then change ownership for each directory listed.

chown --from=CURRENT_OWNER:CURRENT_GROUP -R NEW_OWNER:NEW_GROUP DIRECTORY

5.11. $HOME for the homeless

A “homeless” user is one that does not have their “own” workstation to provide $HOME. They will be assigned a home provider by the WCWC admin. Likely this will be lycastus, the /wcwc NFS-server. This host is not a NFS-client thus effectively not available for user login.

Adding a “homeless” user requires variant of what is described in section 5.7.

  1. FreeIPA: Identity -> User -> username -> User Groups -> +Add -> add to “wcwc”.
  2. FreeIPA: Identity -> ID Views -> wcwc -> Users -> +Add -> Add user ID override (user, login, uid, gid, shell, home) (see section #home-provider-uid for how to form UID from BNL Life or Guest identifier)
  3. FreeIPA: Network Services -> wcwc-u -> auto.u -> +Add -> key: username, mount:
-fstype=nfs4 -sync,rw,soft,sec=sys,proto=tcp,port=2049 lycastus.phy.bnl.gov:/home/&
  1. Creating the physical (not actual) home:
# mkdir -p /nfs/home/USER
# cp /etc/skel/.* /nfs/home/USER/
# chown -R USER:USER /nfs/home/USER/
  1. Check on NFS-client (not lycastus):
$ getent passwd USER
$ ls -la /u/USER
$ ls -la /home/USER 

5.12. Annexed $HOME provider

A workstation which is not yet a member of WCWC may be “annexed” to supply the $HOME for a user which is a member. This requires manual configuration of the annexed workstation some of which must be undone if/when the workstation later becomes a full WCWC member. Much of this configuration is in common with what must be done on a WCWC workstation except we will not automount WCWC $HOME and will leave the native /home in place.

Here are the steps to take, assuming a Debian-like system.

  1. Adjust UID as in section 5.10.
  2. apt install nfs-kernel-server
  3. mkdir -p /nfs/home
  4. Add /home /nfs/home none bind 0 0 to /etc/fstab
  5. One time mount /nfs/home
  6. Configure NFS with following in /etc/exports
/nfs		130.199.22.0/23(rw,sync,crossmnt,no_subtree_check,fsid=0)
/nfs/home	130.199.22.0/23(rw,sync,no_subtree_check)
  1. One time exportfs -av and showmount -e to check.
  2. FreeIPA: Identity -> User -> username -> User Groups -> +Add -> add to “wcwc”.
  3. FreeIPA: Identity -> ID Views -> wcwc -> Users -> +Add -> Add user ID override (user, login, uid, gid, shell, home)
  4. FreeIPA: Network Services -> wcwc-u -> auto.u -> +Add -> key: username, mount:
-fstype=nfs4 -sync,rw,soft,sec=sys,proto=tcp,port=2049 ANNEXED.phy.bnl.gov:/home/&

5.13. Autofs overview

5.14. Debugging autofs

The use of autofs to NFS-mount ties together several things.

  • Adding a user’s $HOME, see section 5.7.
  • Setting up a home provider, see section 5.10.
  • Assuring the host is an ipa-client-automount, see section 5.8.
  • Assuring auto_home.sh script exists. (run ansible playbook).

On NFS-client, the basic check is:

ls -ld /u/USER /home/USER  ### show correct ownership by name
ls -ldn /u/USER /home/USER ### show ownership by "short" user ID

Both should exist on any WCWC workstation, except special case of the home provider for the “homeless” (users with no workstation of their own).

If they do not exist we go end to end starting at the NFS server:

On NFS-server, does the user account exist and with “short” user ID?

getent passwd USER 

On NFS-server, does the the physical (not actual) home directory exists?

ls -la /nfs/home/USER

If it is missing then create it according to section 5.11.

On NFS-server, is /nfs/home exported?

# showmount -e

If not, check /etc/exports. Changing that requires:

# exportfs -ua
# exportfs -av   ### -v to check
# showmount -e   ### to double check

Assure /nfs/home is exported to the LAN subnet.

On NFS-client, check direct NFS-mount:

# mount -t nfs4 lycastus.phy.bnl.gov:/home /mnt
# ls -l /mnt     ### see any homes?
# umount /mnt

If not, perhaps NFS-server or NFS-client has a local firewall interfering.

On NFS-client, check autofs is working. After any change to IDM related to autofs, services may need restarting:

# service sssd restart
# service autofs restart

On NFS-client, check that autofs knows the maps

# automount --dumpmaps

Should see:

  • /wcwc in auto.direct map
  • /home in auto_home.sh map
  • /u in auto.u map

If missing, then make corrections in FreeIPA.

On NFS-client, check that the “program” map script exists and works:

# /etc/auto_home.sh USER

This should return :/nfs/home/USER when run on the home-provider otherwise it should return /u/USER.

On NFS-client, check that the intermediate automount works:

# ls -ld /u/USER

And, finally, assure end result

# ls -ld /home/USER

5.15. Walk throughs

5.15.1. wcgpu1

This is one of 2 dual-GPU “workstations”. They are integrated as “normal” workstations except that they are not home providers. Here we walk through section 5.3.

$ ssh root@wcgpu1.bnl
root@wcgpu1:~# hostname
wcgpu1.phy.bnl.gov

local$ ssh hierocles.bnl
internal$ ansible workstations -m ping -l wcgpu1.phy.bnl.gov
internal$ ansible-playbook -l 'wcgpu1.phy.bnl.gov' ansible/playbooks/wcwc-workstation.yml

Following section 5.6.

root@wcgpu1:~# mkdir -p /nfs/home
root@wcgpu1:~# emacs -nw /etc/fstab  # change /home to /nfs/home
root@wcgpu1:~# mount /nfs/home

No new WCWC users associated with wcgpu* so skip section 5.7 and continue to section 5.8 performing both client install and automount commands and then section 5.9.

# service sssd restart
# service autofs restart
# getent passwd bviren
bviren:*:22351:22351:Brett Viren:/home/bviren:/usr/bin/fish
# ls -ld /home/bviren/
drwxr-xr-x 62 bviren bviren 4096 Aug 20 05:57 /home/bviren/
# ls -lnd /home/bviren/
drwxr-xr-x 62 22351 1601400019 4096 Aug 20 05:57 /home/bviren/
# ls -l /wcwc/spack/bin/spack
-rwxr-xr-x 1 1001 1001 1765 Aug 16 11:52 /wcwc/spack/bin/spack

$ ssh wcgpu1.bnl # full login check

Again, as this is not a home provider, the rest of the steps above are skipped and we are done.

Some todo items for wcgpu*

  • [ ] move static IP from DE config to /etc/network/interfaces
  • [ ] purge DE packages

5.16. Internal data

TBD: how to share more than $HOME.

5.17. External data

TBD: how to access eg FNAL or SDCC storage.

5.18. Build host

Typically, package building is performed in an account (wcwc@wcgpu0) on a faster host than the /wcwc NFS provider (lycastus). As the /wcwc is mounted read-only, a writable copy is made in a non-canonical location, a state change is applied and the results returned.

$ rsync -av wcwc@lycastus.phy.bnl.gov:/wcwc/ ~/wcwc-dev/  
$ wcwc --prefix=$HOME/wcwc-dev admin install <spec>
$ rsync -av ~/wcwc-dev/  wcwc@lycastus.phy.bnl.gov:/wcwc/

5.19. External sync host

A home workstation or laptop or other computers that do not NFS-mount /wcwc may maintain a copy by periodically syncing with the central /wcwc.

wcwc@haiku:~$ rsync --del -a lycastus.bnl.phy.gov:/wcwc/ /wcwc/

The sync can be against any WCWC workstation. To avoid extra network traversal, this example goes to the horses mouth but any workstation NFSmounting /wcwc can also work. If the user wishes to extend the local copy beyond what is on WCWC, they should omit the --del option. Modifying the local copy entail commands as illustrated in section 5.18.

5.20. Misc checks

Desktop machines may be configured to suspend when they are idle and sitting at GDM login. To defeat this, edit /etc/gdm3/greeter.dconf-defaults to set these bits:

[org/gnome/settings-daemon/plugins/power]
sleep-inactive-ac-type='nothing'
sleep-inactive-battery-timeout=0

And

# systemctl restart gdm3
# systemctl restart gdm

5.21. Installing Spack packages

Spack state modifications are performed on a read-write copy of /wcwc and never directly on /wcwc To make the copy:

$ rsync -a wcwc@lycastus.phy.bnl.gov:/wcwc/ ~/wcwc-dev/

To return changes:

$ cat ~/bin/wcwc-sync 
rsync -av \
  --exclude "*~" --exclude __pycache__ --exclude stage --exclude cache ~/wcwc-dev/ \
  wcwc@lycastus.phy.bnl.gov:/wcwc/

This potentially opens a huge race condition if more than one copy is allowed. To avoid this, a single copy in wcwc@wcgpu0 is the working copy. A better workflow should be developed.

While it is possible to directly edit files under ~/wcwc-dev/ any installation and use of packages must be done inside a container:

u$ wcwc --image=yes --prefix=$HOME/wcwc-dev shell
s$ ls /wcwc
opt  spack  stacks  stage  wcwc.db  wcwc.yaml
s$ spack install [...]

Or, better use wcwc admin (which enforces the container):

$ wcwc --prefix=$HOME/wcwc-dev admin install [...]

The container must also be used to exercise a locally installed package:

$ wcwc --image=yes --prefix=$HOME/wcwc-dev admin shell -l <package>
# or
$ wcwc --image=yes --prefix=$HOME/wcwc-dev admin shell -S wirecell -l <package-in-wirecell>

When state changes applied to ~/wcwc-dev/ are complete, “commit” them by rsync’ing back:

$ ./bin/wcwc-sync

In the future, the sync parts of this workflow may become incorporated into wcwc.

5.22. Developing Spack packages

It is common that a Spack package recipe must be updated, if only to bump the supported version(s). Here is an example of that workflow, starting with entering the container (c$ prompt):

u$ wcwc --image=yes --prefix=$HOME/wcwc-dev shell
c$ spack checksum <package>

When substantial fixes to the Spack recipe is needed, beyond a new version() line, it is good to PR it back to Spack. This is best done out-of-band of WCWC and in a personal fork of Spack:

$ git clone git@github.com:USER/spack.git ~/dev/spack-arrayfire
$ cd ~/dev/spack-arrayfire
$ git checkout -t -b bugfix-arrayfire origin/develop
$ source spack/share/spack/setup-env.sh
$ spack install arrayfire

Now, fix the build errors, push to a branch in the personal fork and from there make a PR to spack, proper.

5.23. Updating wire-cell-spack

General (non-wcwc) instructions are at wire-cell-spack. Before WCWC can use them, any updates need to be pushed there. To test that development, see Stand-alone Spack builds.

As wcwc@wcgpu0 do these steps:

rsync --archive --delete wcwc@lycastus.phy.bnl.gov:/wcwc/ ~/wcwc-dev/
wcwc --image=yes --prefix=$HOME/wcwc-dev shell

5.24. The /srv mount

The wcgpu machines have 2+2+4 TB SSDs (only 3 M.2 slots).

nvme0
2TB system usage
nvme1
2TB root and home
nvme2
4TB data

Here is preparing nvme0

# fdisk -l
# df -hl
# fdisk /dev/nvme0n1
g
n 
w
# mke2fs /dev/nvme0n1p1
# blkid /dev/nvme0n1p1
# emacs /etc/fstab  # mount as /srv

The factory partitioning of nvme1 what kept. It is not optimal and so nvme0 is used to back fill some small system areas. At some point (yeah, right) the system disk should be repartitioned to better allocate. Maybe if/when user storage needs upgrading.

5.25. CVMFS

With /srv mounted as per section 5.24,

# mkdir -p /srv/sys/var/lib/cvmfs  /var/lib/cvmfs
# emacs /etc/fstab  # bind mount former on latter

Install

wget https://ecsft.cern.ch/dist/cvmfs/cvmfs-release/cvmfs-release-latest_all.deb
dpkg -i cvmfs-release-latest_all.deb 
rm cvmfs-release-latest_all.deb
apt-get update
apt-get install cvmfs
cvmfs_config setup
service autofs restart

cat << EOF > /etc/cvmfs/default.local
CVMFS_CLIENT_PROFILE=single
CVMFS_REPOSITORIES=larsoft.opensciencegrid.org,uboone.opensciencegrid.org,dune.opensciencegrid.org,dune.osgstorage.org,oasis.opensciencegrid.org,singularity.opensciencegrid.org
CVMFS_SERVER_URL="http://cvmfs.sdcc.bnl.gov:8000/cvmfs/@org@;http://cvmfs-s1fnal.opensciencegrid.org:8000/cvmfs/@fqrn@"
CVMFS_HTTP_PROXY="http://cvmfs-cache.sdcc.bnl.gov:3128|http://cvmfs-cache.sdcc.bnl.gov:3128;DIRECT"
EOF

cvmfs_config probe

Author: Brett Viren

Created: 2026-04-16 Thu 08:27

Validate