Table of Contents

0u62;52;c#+title: WCWC Packages

1. Overview

This note collects information about specific packages, be they provided by Spack or the host OS. It is intended for both WCWC admins and WCWC users.

2. Direnv

The recommended way to manage task-specific shell environment settings on WCWC (and in general) is with direnv.

In brief, direnv watches as you cd to different directories and if you enter a directory that has a special .envrc file or has a parent with this file then direnv automatically applies its settings to your shell environment. When you cd out of a .envrc controlled tree the settings are automatically undone.

2.1. Quick start

Here is a quick example:

$ mkdir ~/mywork/
$ echo 'export FOO=bar' > mywork/.envrc
$ cd ~/mywork

$ direnv allow

$ echo $FOO
bar

$ cd
$ echo $FOO

$ cd ~/mywork
$ echo $FOO
bar

A direnv allow is only required when the .envrc file is modified and direnv will alert the user when this authorization is required.

2.2. Hooking

For direnv to work it must be “hooked” into your shell. On BNL WCWC workstation users, use of direnv is automatic. Other users should read: https://direnv.net/docs/hook.html

2.3. The .envrc file

You may add lines to .envrc which set environment variables. Complex configuration can be generated by calling direnv functions and a standard library is provided (see the direnv-stdlib(1) man page).

There is no support to define shell functions or “aliases”. However, an environment variable may be set to hold a command line. Eg: export list='ls -l' and the user may run it with $list

For example, when wcwc view -d ... is used to create a “view” (see wcwc-howto.html#spack-view-wct for an example) a .envrc file is created with one line that generates setting for many PATH-like variables:

load_prefix viewdir

While .envrc file may look like bash syntax, it is not and no shell should ever be made to “source” it.

2.4. Python support

A common situation is a user wishes to install some Python packages with pip install but wishes to avoid having pip download and possibly compile large complex packages. This scenario is supported with direnv with one .envrc line:

# in .envrc
layout python3 --system-site-packages

After a cd to a directory controlled by this .envrc file the user might do:

$ pip install torch
Requirement already satisfied: torch in /usr/lib/python3/dist-packages (1.13.0a0+gitunknown)
$ python -c 'import torch; print(torch.cuda.is_available())'
False

Unfortunately, as shown, Debian’s python3-torch package does not include CUDA support.

3. Python

There are several options for using Python and Python packages on WCWC workstations. Unfortunately, there is not “one size fits all” answer on how to use Python on WCWC (or more broadly). Rough guidance is to use the first choice from the list below that matches your needs

Spack
If you will work in a Spack environment for other reasons, use Python and Python packages provided with Spack. When a Spack Python package is unavailable or not appropriate (too old, missing features) you may try to augment the Spack environment or view with Python packages from uv or venv (see below). Usage of Spack can help reduce redundant copies of large packages (eg, pytorch) by sharing the central install.
uv
You need to work with many different, semi-independent Python environments and/or are developing your own Python project. The uv tool is very powerful, flexible and highly recommended. It replaces many older tools like venv but it may be new to you and so requires some (simple) learning. Many packages will give examples on how to use venv and you will need to know to translate those instructions into uv usage. (see 3.1 section below). Use of uv will share packages between your different usages so can save storage space compared to use of venv.
venv
You don’t want to try new things.

With all methods, direnv can be very helpful to manage and tie environment settings to a directory. See section 2.

3.1. uv

The uv tool is provided as /usr/local/bin/uv on WCWC workstations. This is a fairly new tool on the Python scene but it already provides a wealth of benefits and replaces other tools commonly used. It can be used in many different modes to easily accomplish many Python related tasks:

  • Install various Python packages and of many versions without polluting global namespaces.
  • Install and make available various versions of Python itself.
  • Install Python packages so their main programs are available for your general use.
  • Assist in creating and managing Python projects.
  • Provide “better” versions of venv and pip.

Most simply, it can easily run packages, downloading as needed:

$ uv run --with ipython --with numpy --with matplotlib ipython --pylab
In [1]: plt.plot([0,1],[2,3])
Out[1]: [<matplotlib.lines.Line2D at 0x7f6a656dd890>]

The first uv run may install packages which may take some time while future runs will be quick. You can also expose these locally installed packages:

$ uv tool install git+https://github.com/brettviren/gegede
...
$ which gegede
/home/bv/.local/bin/gegede
$ gegede -h
...

3.2. Virtual environment

A Python virtual environment (venv) consists of two main elements

  • A directory patterned like the conventional /usr/local holding files from Python and various python packages.
  • An activate script to configure PATH-like environment variables to make use of these files.

A user needing to install Python packages should do so in a venv and not ever via --user option.

The recommended way to provide a Python venv is via direnv as described in section 2.4. With more effort, a user may instead create one manually as described here:

$ python -m venv myvenv
## or
$ uv venv [--python X.Y.Z] myvenv

To configure the shell environment:

$ source myvenv/bin/activate

There exists also activate.fish and activate.csh for some non-sh shells. To undo the environment settings:

$ deactivate

Similar to the direnv example, OS-installed Python packages can be leveraged by creating the venv with:

$ python -m venv myvenv --system-site-packages
## or
$ uv venv --system-site-packages myvenv

3.3. Mixing Spack, Debian and pip packages

Only certain combinations of Spack, Debian and pip packages can be expected to work. These are as described along with a direnv .envrc file to produce the venv.

Spack and pip but not OS

When the shell is configured for Spack Python packages (wcwc shell -l py-XXX) then a pip install in a Python venv will use those packages. The venv relies on the exact wcwc shell environment to be repeated for subsequent usage. This is irrespective of using --system-site-packages or not but in neither case will any OS packages be leveraged. Eg, wcwc shell -l py-numpy and with

layout python
No Spack but pip and OS

When the shell is configured for any Spack packages unrelated to Python (or no Spack at all), then the venv can be used without regards to Spack. Use of --system-site-packages will cause pip install to prefer use of an existing OS package to downloading/building it fresh.

layout python3 --system-site-packages
Spack and pip and OS with a hack

If Python is provided by Spack, system packages can be seen and used by pip install if PYTHONPATH is set to /usr/lib/python3/dist-packages.

layout python3
path_add PYTHONPATH /usr/lib/python3/dist-packages
Augment OS with Spack, no pip, no venv

Spack Python and packages can be mixed with OS Python.

$ wcwc shell -l 'py-torch+cuda cuda_arch=89'
$ jupyter notebook --no-browser  # from the OS

Visit the web and test with import torch; print(torch.cuda.device_count()). See section 10 for more info about use of Jupyter.

4. Wire-Cell Toolkit

5. CUDA

The main motivation for WCWC is to share workstation GPUs at BNL. Their GPUS are all currently from nVidia and so CUDA support is a must.

5.1. Versions

The nVidia hardware and CUDA software landscape is a little complex and described by several versions:

compute capability
(cc) a number like 9.0 which maps to a set of supported compute features
hardware family
eg “turing” is a name that implies one or more cc numbers (table of device cc, table of family cc)
driver version
a number like 525.147.05 which gives the version of the nVidia Linux kernel driver
toolkit version

a number like 11.8 or 12.0 which gives the version of CUDA user libraries and implies a minimum driver version.

Debian’s libcuda packages are given a driver version number and provide support for multiple toolkit version numbers. Debian 12’s libcuda1 package is at version 525.147.05 which provides various libcuda versions in the range 5.0 - 12.0.

A sampling of BNL GPUs, their amount of RAM and their cc, family and CUDA toolkit minimum version (all are supported up to and including CUDA 12).

Card RAM CC Family CUDA
GTX 750 Ti 2 GB 5.0 Maxwell 6.5
GTX 1050 Ti 4 GB 6.1 Pascal 8.0
GTX 1060 6 GB 6.1 Pascal  
GTX 1650 4 GB 7.4 Turing 10.0
GTX 1660 6 GB 7.5 Turing  
RTX 2080 Ti 11 GB 7.5 Turing  
RTX 3080 10 GB 8.6 Ampere 11.0
RTX 4070 Ti 12 GB 8.9 Lovelace 11.8
RTX 4090 24 GB 8.9 Lovelace  

Each workstation must have nVidia kernel driver installed and loaded. It is recommended to use Debian’s package:

apt install nvidia-kernel-dkms nvidia-smi

A reboot may be required. The nvidia-smi command is used to query your GPU device status. If it gives an error then the kernel driver is not working.

wcwc@haiku:~$ nvidia-smi  -L
GPU 0: NVIDIA GeForce GTX 750 Ti (UUID: GPU-b87cc150-ba6d-329e-02e4-7f8ddef6a581)

Receiving this error:

Cuda error in file '.../file.cu' in line XX : no kernel image is available for execution on the device.

Indicates CUDA code was not compiled to target the device (GPU card) compute capability version.

5.2. CUDA and Spack

Spack can compile CUDA from scratch with some caveats:

  • The Spack cuda package will only build a stubs/libcuda.so and not a “real” libcuda.so. The stub holds “dummy” implementations of the CUDA API and may be used to satisfy link-time dependencies when building in an environment that lacks a “real” libcuda.so. On WCWC we will rely on Debian’s libcuda.so.
  • Some Spack packages care about the CUDA “cc” number. Eg, they may have internal code switches that depend on the version. Of those that care, some (like kokkos) can be built with support for exactly one “cc”. The conventional way to communicate the “cc” to these packages is via cuda_arch variant parameter.

6. Torch

PyTorch is the primary DNN toolkit of WCWC users and WCT can make use of libtorch. Spack’s py-torch package builds the Python interface:

$ TMPDIR=$HOME/.cache/wcwc TEMP=$HOME/.cache/wcwc \
  wcwc install "py-torch+cuda cuda_arch=89"

FIXME: must set TMPDIR to avoid /tmp being filled. For safety, set this and TEMP in wcwc, but really, this is a bug in the py-torch/package.py which should honor the configured build_stage setting. Note, this is not required when building in a podman container (assuming podman has sufficient space in the user’s home).

Also, earlier builds failed due to OpenMP being detected in the OS but not sufficient to work. apt install libomp-dev was needed. The above was on a different Debian host. This also motivates the use of a container to build. In a minimal Debian 12 podman container, py-torch builds without problems (other than loooooong build time). The wcwc make-builder command creates the container.

ls /wcwc/opt/builtin/linux-debian12-x86_64_v4/gcc-12.2.0/py-torch-2.3.0-sq4psktodzbjq7v4bshow7rmcxwoa6r3/lib/python3.11/site-packages/torch/lib                                                                                                          
libc10_cuda.so	libcaffe2_nvrtc.so  libtorch_cpu.so	    libtorch_cuda.so	     libtorch_python.so
libc10.so	libshm.so	   libtorch_cuda_linalg.so  libtorch_global_deps.so  libtorch.so

Simple test:

$ python -c 'import torch; print(torch.cuda.is_available())'
True
$ python -c 'import torch; print(torch.cuda.device_count())'
2

Note, had to load a kernel module:

root@wcgpu0:~# modprobe nvidia_uvm
wcwc@wcgpu0:~$ wcwc admin install -S wirecell wire-cell-toolkit+torch+cuda cuda_arch=89

7. Ollama

The Ollama project provides the program ollama that exercises LLM models.

7.1. Versions

The official ollama/ollama supports a number of different LLM models. Some other models of interest require a modified ollama program. These variants may not support the same models as the official ollama. In general, one must only use models that are explicitly supported by a particular ollama variant.

WCWC provides official and variant ollama as different Spack packages. Currently:

ollama
From Spack builtins repo with CUDA support (spack install ollama+cuda, added in PR).
ollamabin
The official version. The “bin” implies the package consists of a binary release and to avoid conflict with the Spack ollama package.
ollamalux
A “hacked” version to run specifically aiden_lu/minicpm-v2.6. The “lux” portion of the name refers to it originating from luixiao0/ollama binary release.

Regardless of the package name, the user uses the program named ollama.

7.2. Server

The ollama server is started simply:

$ ollama serve

To stop the server, simply hit Ctrl-c.

7.3. Environment

To help keep the multiple ollama variants distinct, the WCWC ollamaXXX packages will set some environment variables (the ollama package from Spack, proper sets none) when a user “loads” the package (eg wcwc shell -l ollamaXXX).

OLLAMA_HOST
server IP address and port number.
OLLAMA_TMPDIR
set to $HOME/tmp to avoid filling /tmp
OLLAMA_MODELS
location to store files for models.

For other available environment variables, run ollama serve --help.

Multiple clients, possibly run by multiple users, may share the same server. The user that starts the server may also kill it which of course will impact all clients. Users are encouraged to coordinate any sharing amongst themselves. A user may attempt to isolate their server by setting OLLAMA_HOST.

Users should not leave ollama server running longer than needed as it consumes GPU memory.

For either server or command line client usage, start a shell like:

u$ wcwc shell -S wirecell -l ollamaXXX
s$ ollama [...]

Replace ollamaXXX with one of the package names in the list above.

7.4. Use GPUs

By default, ollama serve should auto detect GPU(s) as long as it finds the “real” libcuda.so provided by Debian. See section 5.2. If this seems to fail, running with the “debug” message level can help understand the cause:

$ OLLAMA_DEBUG=true ollama serve

Note, these warnings:

unable to locate gpu dependency libraries

are red-herrings and will be printed even when the GPU are found and usable.

By default ollama serve will use the GPUs it finds. To control which GPUs ollama may use you may set the CUDA_VISIBLE_DEVICES environment variable to a comma-separated list of GPU indices or IDs. Enumeration of GPU indices may not be stable so use IDs to list specific GPUs if stability is required. Some examples:

# use "first" device
$ CUDA_VISIBLE_DEVICES=0 ollama serve

# use "first" and "second" device
$ CUDA_VISIBLE_DEVICES=0,1 ollama serve

# use specific device(s)
$ nvidia-smi -L  # note the GPU IDs
$ CUDA_VISIBLE_DEVICES=GPU-7d6529b0-76ce-2e7f-d96e-2018b58d1f2b ollama serve

7.5. Client

The ollama program can also act as a command line client:

$ ollama run llama3.1

Chat is logged by the server. To exit, hit Ctrl-d.

The first time a model is run its files will be downloaded by the server.

7.6. Checks

You can run various checks to observe the state of the server. Assuming a subshell configured for the ollamaXXX package:

$ ollama ls  # list available models
NAME           	ID          	SIZE  	MODIFIED       
llama3.1:latest	f66fc8dc39ea	4.7 GB	42 seconds ago	

$ ollama ps  # list running models:
NAME           	ID          	SIZE  	PROCESSOR	UNTIL              
llama3.1:latest	f66fc8dc39ea	6.7 GB	100% GPU 	4 minutes from now	

$ nvidia-smi # check actual GPU memory usage
...
|    0   N/A  N/A    531064      C   ...unners/cuda_v12/ollama_llama_server       6142MiB |
...

7.7. Specific models

Above we showed examples with the official ollama program and supported models. This section collects guidance on variants. The models described below should not be expected to work with the official ollama nor should the ollama’s described her be expected to run the officially supported models.

7.7.1. lux / MiniCPM v2.6

The MiniCPM v2.6 model is claimed to be good for OCR. It requires a variant ollama which is provided by the ollamalux package. It may be used from two shells like:

$ wcwc shell -S wirecell -l ollamalux -c "ollama serve"

$ wcwc shell -S wirecell -l ollamalux -c "ollama run aiden_lu/minicpm-v2.6:Q4_K_M"

8. Open WebUI

Open WebUI provides a web browser user interface to Ollama and other LLM APIs. In the future, WCWC may provide a centrally-installed instance of Open WebUI but as of now each user must install/run it themselves. This is mostly due to two reasons: the project development is brisk and the specific version and usage pattern tends to be tied to individual user tasks.

This section describes ways for the user to run Open WebUI with a local 7 and remote APIs.

8.1. General running of Open WebUI

Open WebUI is a Python program and can be run in the “usual ways” as described in the section 3. The recommended method is with 3.1.

Use of open-webui may require a newer Python than available by default. We can tell uv to use a non-standard python by setting the environment variable: UV_PYTHON=3.12.3. uv will automatically install that version of Python as needed. When UV_PYTHON is set for a uv tool install ... command it need not be set when using the installed tools. We assume this usage below

To run the Open WebUI server command:

$ UV_PYTHON=3.12.3 uv run --with open-webui open-webui serve

Or, to install it as a “tool” so you do not need to give uv run ...:

$ UV_PYTHON=3.12.3 uv tool install open-webui
$ open-webui serve

8.2. Accessing Open WebUI web server

By default, Open WebUI serves on localhost port 8080. If you run a web browser on the same computer that runs Open WebUI simply visit:

http://127.0.0.1:8080/

On first run of open-webui serve command, Open WebUI will need some time to download various things. While that is ongoing, the server may not respond. Subsequent server starts will require less time.

On first access of a freshly started Open WebUI server you will be prompted to make an admin account.

8.3. Accessing remote Open WebUI

Open WebUI, as we use it on WCWC, should never be exposed to the Internet. However, it may be accessed remotely via SSH tunnels.

Assuming your SSH configuration defines the host wcgpu0 (see Remote access for examples) then a tunnel may be constructed by issuing this command to your local computer:

$ ssh -L 8080:localhost:8080 wcgpu0

And, from the your local computer, visit

http://127.0.0.1:8080/

8.4. Use with Ollama

While you can “chat” with Ollama directly from the terminal (ollama run ...), Open WebUI provides many additional conveniences and features. By default, Open WebUI can connect to a localhost Ollama server (running on same host as Open WebUI itself) if Ollama was started in the default manner. If Ollama server is configured to listen on a custom port, that port must be communicated to Open WebUI:

$ OLLAMA_BASE_URL=http://127.0.0.1:8054/ open-webui serve

8.5. Use with openai.azure remote APIs

While Open WebUI documentation currently (as of v0.6.5) mentions Azure OpenAI as being usable, it appears not to work on its own. There appear to be two ways to use a 3rd party tool to connect Open WebUI with Azure OpenAI:

8.5.1. Configure and run LiteLLM

We describe how to use the second. First, make a LiteLLM configuration file in YAML syntax that describes each model that you have been “deployed” on Azure OpenAI. An API URL and API key are needed as well as the deployment “name” for the model.

Multiple models can be deployed with the same API URL and key. This is assumed in the example. Adjust your LiteLM YAML configuration file if various values are relevant.

Here is an example.

model_list:
  - model_name: o4-mini
    litellm_params:
      model: azure/o4-mini
      api_base: "os.environ/AZURE_API_URL"
      api_key: "os.environ/AZURE_API_KEY"
      api_version: 2024-12-01-preview
  - model_name: gpt-4.1
    litellm_params:
      model: azure/gpt-4.1
      api_base: "os.environ/AZURE_API_URL"
      api_key: "os.environ/AZURE_API_KEY"
      api_version: 2024-12-01-preview

This configuration delegates to the user shell environment for the URL (api_base) and key.

Do not publicly disclose the API key unless you are willing to pay for the entire Internet using your account. In particular, keep API keys out of files that are inside git controlled directories or in general any directories exposed to a web server.

To run LiteLLM while providing these values:

$ AZURE_API_URL="https://THE-USER-NAME.openai.azure.com/" \
  AZURE_API_KEY=THE85CHARACTERAPIKEY \
  uv run --with 'litellm[proxy]' --with 'litellm[extra_proxy]' \
  litellm --config litellm.yml 

By default, this will listen on http://localhost:4000. You may visit this, expand the

GET /v1/models

section, click “Try it out” and then “execute”. Scroll down a bit and you should see the JSON response listing the models from the LiteLLM config.yaml file.

8.5.2. Connect Open WebUI to LiteLLM

Visit your Open WebUI page, click your avatar/initials, open Admin Panel, Settings, Connections and in Manage OpenAI API Connections, click “+”, enter for URL http://localhost:4000/v1. The API key and list of Model IDs can be left empty.

This use of LiteLLM with no API key or other auth is very insecure and should never be exposed outside of localhost.

Save and when starting a New Chat the models from the LiteLLM YAML configuration file should be listed and accessible.

9. Aider

The aider program gives a simple command line interface with a powerful set of LLM-assisted software development features. Installation is easy with uv

$ uv tool install [--force --upgrade] --python python3.12 aider-chat
$ aider --help
$ aider --model=MODEL --api-key PROVIDER=KEY

There are a lot of options. Here are some likely ones and examples are below

  • --no-auto-commits By default, aider will git commit after each round with the LLM. If you want to handle commits yourself, use this flag.

9.1. Aider + Gemini

Use with free Gemini

$ cd my-source-area/
$ aider --model gemini/gemini-flash-latest --api-key gemini=KEY [...]

You will need to get a KEY from Google. Sign in at https://aistudio.google.com and look for Get API Key and Create API Key, copy it and store it securely. Keep it out of any files managed by git. The key starts with the string AIza.

By default, each round of chat with the LLM will (may) change your code and will commit. If you would prefer to handle commits yourself add:

--no-auto-commits

9.2. Aider + Azure

Microsoft Azure provides access to many LLMs and aider knows how to use them.

$ export AZURE_API_BASE=https://RESOURCE.openai.azure.com/
$ export AZURE_API_VERSION=2025-09-01 
$ AZURE_API_KEY=KEY
$ aider  --model azure/DEPLOYMENT [...]

BNL provides licenses to use Microsoft Azure. The model name is actually the “deployment” name. You make a “deployment” through your Azure account.

10. Jupyter

This section describes the Jupyter family of applications.

The Spack packaging for jupyter is rather out of date. Until there is reason to address that, we rely on Debian packages.

10.1. Web user interface

Perhaps most users see Jupyter as something to use from a web browser. For purely local use, run:

$ jupyter notebook    # do not run on a remote account

A browser tab visiting the jupyter server should open. To control what browser to use, set BROWSER environment variable. Do not run this command on a remote workstation (see below for that). If it is run on BNL WCWC, the default browser will be opened which may be the terminal program lynx (hit q to quit that).

To jupyter notebook server tends to continue to run. If running in a terminal, the user may hit Ctrl-c and answer “y” to shutdown. If neglected, the user may run jupyter notebook stop PORT. To discover the port(s) of running server(s) run jupyter notebook list. Do not leave jupyter running longer than needed as it consumes memory and ports.

To run jupyter notebook a remote workstation an SSH tunnel must be established to forward local browser connections to the remote jupyter server.

$ ssh -L 8888:localhost:8888 wcgpu0.phy.bnl.gov jupyter-notebook --no-browser
[...] http://localhost:8888/?token=9b1e482113f4c807250429c905d0e4d04c8175ba3d1a43b5

You may then click that link (if your terminal supports URLs) or copy-paste the URL to a local browser.

The literal usage of the URL printed by jupyter notebook assumes you chose the same port number on both ends of the SSH tunnel and that “localhost” resolves correctly to your local workstation. If the local and remote ports differ then you must edit the URL to use the local port.

The above command establishes the tunnel assuming the default jupyter port will be available for use. If another user is using that port jupyter will select the next available higher port number and you will see a message like:

[...] The port 8888 is already in use, trying another port.
[...] http://localhost:8889/?token=9b1e482113f4c807250429c905d0e4d04c8175ba3d1a43b5

This change in expected port breaks the SSH tunnel. You must Ctrl-c and select “y” to kill the server and the SSH connection and then reconnect using a different port. In order to increase the chance to claim an unused port and keep consistent ports for the tunnel, we may specify a port of our choosing:

$ ssh -L 8998:localhost:8998 wcgpu0.phy.bnl.gov jupyter-notebook --no-browser --port 8998
[...] http://localhost:8998/?token=f33b43123890c20b169a7af168274f9e2998cd38f15a33be

10.2. Console

A command line alternative to the Jupyter web UI is the “console”. This can be run on the local or remote workstation without the need to establish an SSH tunnel or use a browser.

$ jupyter console
In [1]: import matplotlib.pyplot as plt

In [2]: plt.plot([1,2,3], [0,1,0])
Out[2]: [<matplotlib.lines.Line2D at 0x7f2907c68790>]

When showing plots or other graphics an X11 client window will be produced. When running on a remote workstation, this window will only be visible if X11 forwarding has been enabled. For SSH configuration guide see, for example, wcwc-bnl.html#remote-access with attention to the ForwardX11 configuration item.

10.3. Ipython

Jupyter originated from an older project called ipython, which is similar to jupyter console. ipython has some convenience features such as pre-loading numpy and matplotlib which makes it a useful alternative:

$ ipython --pylab
In [1]: plt.plot([1,2,3], [0,1,0])
Out[1]: [<matplotlib.lines.Line2D at 0x7f38adf35cd0>]

10.4. Python environment

Default use of jupyter (or ipython) will have available any Python packages installed via Debian packages. The user may require additional packages which can be provided via pip or Spack in the usual way, prior to running jupyter. See section 3.

To configure for such running, it is recommended to use direnv with a .envrc file like:

$ mkdir env-jupyter
$ echo 'layout python3 --system-site-packages' > env-jupyter/.envrc
$ cd env-jupyter
$ direnv allow
$ pip install jupyter
$ pip install spacy

This pip install jupyter should result in all or at least most packages reporting “Requirement already satisfied” which means the OS provides the actual package files. OTOH, the pip install spacy should bring in a mix of OS and fresh packages. This shows the --system-site-packages is operational.

The notebook server can then be started by hand:

$ python -m ipykernel install --user --name myenv
$ jupyter notebook --no-browser
...
http://localhost:8888/?token=32c26d0a705495b5a9e55bce6c60f4b7622c45a89fa3fafd

Based on the URL we may create a tunnel:

$ ssh -L 8888:localhost:8888 wcgpu0.phy.bnl.gov

Now visit the URL in the browser and locate myenv under New->Notebook: menu.

11. Geant4

Users / developers working inside an experiment stack will typically have Geant4 included in the dependency tree. Others may utilize those instances of Geant4. As usual spack find -lv geant4 will list what is available. Special-case builds of Geant4 will/can be made available.

11.1. Support for DUNE FD3

DUNE FD3 simulation uses geant4+qt+opengl+x11

$ wcwc spack find -lv geant4
-- linux-debian12-x86_64 / gcc@12.2.0 ---------------------------
wsqk5qa geant4@11.2.2~ipo~motif+opengl+qt~tbb+threads~timemory~vecgeom~vtk+x11 build_system=cmake build_type=Release cxxstd=17 generator=make

11.2. Opticks

A likely to-do item is to build Geant4 with support for Opticks.

12. DNNROI

Prepare:

$ wget --no-check-certificate -r --no-parent 'https://www.phy.bnl.gov/~hyu/dunefd/dnn-roi-pdvd
/Pytorch-UNet/data/'
$ mv www.phy.bnl.gov/~hyu/dunefd/dnn-roi-pdvd/Pytorch-UNet/data .
$ rm -rf www.phy.bnl.gov/

$ git clone https://github.com/HaiwangYu/Pytorch-UNet.git hyu-Pytorch-UNet
$ echo 'layout python > hyu-Pytorch-UNet/.envrc
$ cd hyu-Pytorch-UNet && direnv allow
$ ln -s ../data .

$ pip install torch torchvision torchaudio matplotlib h5py pillow pandas

$ pip list
Package                  Version
------------------------ -----------
contourpy                1.3.0
cycler                   0.12.1
filelock                 3.16.0
fonttools                4.53.1
fsspec                   2024.9.0
h5py                     3.11.0
Jinja2                   3.1.4
kiwisolver               1.4.7
MarkupSafe               2.1.5
matplotlib               3.9.2
mpmath                   1.3.0
networkx                 3.3
numpy                    2.1.1
nvidia-cublas-cu12       12.1.3.1
nvidia-cuda-cupti-cu12   12.1.105
nvidia-cuda-nvrtc-cu12   12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12        9.1.0.70
nvidia-cufft-cu12        11.0.2.54
nvidia-curand-cu12       10.3.2.106
nvidia-cusolver-cu12     11.4.5.107
nvidia-cusparse-cu12     12.1.0.106
nvidia-nccl-cu12         2.20.5
nvidia-nvjitlink-cu12    12.6.68
nvidia-nvtx-cu12         12.1.105
packaging                24.1
pandas                   2.2.2
pillow                   10.4.0
pip                      23.0.1
pyparsing                3.1.4
python-dateutil          2.9.0.post0
pytz                     2024.2
setuptools               66.1.1
six                      1.16.0
sympy                    1.13.2
torch                    2.4.1
torchaudio               2.4.1
torchvision              0.19.1
triton                   3.0.0
typing_extensions        4.12.2
tzdata                   2024.1

Run:

$ bash train3.sh

Epoch finished ! Loss: 0.032897
Checkpoint e0 saved !
Traceback (most recent call last):
  File "/home/wcwc/dev/dnnroi/hyu-Pytorch-UNet/train3.py", line 264, in <module>
    train_net(net=net,
  File "/home/wcwc/dev/dnnroi/hyu-Pytorch-UNet/train3.py", line 199, in train_net
    val_loss = eval_loss(net, criterion, val2, gpu)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wcwc/dev/dnnroi/hyu-Pytorch-UNet/eval_util.py", line 43, in eval_loss
    true_masks_flat = true_mask.view(-1)
                      ^^^^^^^^^^^^^^^^^^
RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.

real	0m28.260s
user	0m26.832s
sys	0m5.687s

Try SBND’s flavor

$ git clone https://github.com/wjdanswjddl/Pytorch-UNet.git moon-Pytorch-UNet

$ echo 'layout python > moon-Pytorch-UNet/.envrc
$ cd moon-Pytorch-UNet && direnv allow
$ ln -s ../data .

$ pip install torch torchvision torchaudio matplotlib h5py pillow pandas
$ pip install tqdm tensorboard

13. ArrayFire

c$ spack install arrayfire+cuda cuda_arch=89

Author: Brett Viren

Created: 2026-01-20 Tue 10:28

Validate