Table of Contents
0u62;52;c#+title: WCWC Packages
1. Overview
This note collects information about specific packages, be they provided by Spack or the host OS. It is intended for both WCWC admins and WCWC users.
2. Direnv
The recommended way to manage task-specific shell environment settings on WCWC (and in general) is with direnv.
In brief, direnv watches as you cd to different directories and if you enter a directory that has a special .envrc file or has a parent with this file then direnv automatically applies its settings to your shell environment. When you cd out of a .envrc controlled tree the settings are automatically undone.
2.1. Quick start
Here is a quick example:
$ mkdir ~/mywork/ $ echo 'export FOO=bar' > mywork/.envrc $ cd ~/mywork $ direnv allow $ echo $FOO bar $ cd $ echo $FOO $ cd ~/mywork $ echo $FOO bar
A direnv allow is only required when the .envrc file is modified and direnv will alert the user when this authorization is required.
2.2. Hooking
For direnv to work it must be “hooked” into your shell. On BNL WCWC workstation users, use of direnv is automatic. Other users should read: https://direnv.net/docs/hook.html
2.3. The .envrc file
You may add lines to .envrc which set environment variables. Complex configuration can be generated by calling direnv functions and a standard library is provided (see the direnv-stdlib(1) man page).
There is no support to define shell functions or “aliases”. However, an environment variable may be set to hold a command line. Eg: export list='ls -l' and the user may run it with $list
For example, when wcwc view -d ... is used to create a “view” (see wcwc-howto.html#spack-view-wct for an example) a .envrc file is created with one line that generates setting for many PATH-like variables:
load_prefix viewdir
While .envrc file may look like bash syntax, it is not and no shell should ever be made to “source” it.
2.4. Python support
A common situation is a user wishes to install some Python packages with pip install but wishes to avoid having pip download and possibly compile large complex packages. This scenario is supported with direnv with one .envrc line:
# in .envrc layout python3 --system-site-packages
After a cd to a directory controlled by this .envrc file the user might do:
$ pip install torch Requirement already satisfied: torch in /usr/lib/python3/dist-packages (1.13.0a0+gitunknown) $ python -c 'import torch; print(torch.cuda.is_available())' False
Unfortunately, as shown, Debian’s python3-torch package does not include CUDA support.
3. Python
There are several options for using Python and Python packages on WCWC workstations. Unfortunately, there is not “one size fits all” answer on how to use Python on WCWC (or more broadly). Rough guidance is to use the first choice from the list below that matches your needs
- Spack
- If you will work in a Spack environment for other reasons, use Python and Python packages provided with Spack. When a Spack Python package is unavailable or not appropriate (too old, missing features) you may try to augment the Spack environment or view with Python packages from uv or venv (see below). Usage of Spack can help reduce redundant copies of large packages (eg, pytorch) by sharing the central install.
- uv
- You need to work with many different, semi-independent Python environments and/or are developing your own Python project. The
uvtool is very powerful, flexible and highly recommended. It replaces many older tools likevenvbut it may be new to you and so requires some (simple) learning. Many packages will give examples on how to usevenvand you will need to know to translate those instructions intouvusage. (see 3.1 section below). Use ofuvwill share packages between your different usages so can save storage space compared to use ofvenv. - venv
- You don’t want to try new things.
With all methods, direnv can be very helpful to manage and tie environment settings to a directory. See section 2.
3.1. uv
The uv tool is provided as /usr/local/bin/uv on WCWC workstations. This is a fairly new tool on the Python scene but it already provides a wealth of benefits and replaces other tools commonly used. It can be used in many different modes to easily accomplish many Python related tasks:
- Install various Python packages and of many versions without polluting global namespaces.
- Install and make available various versions of Python itself.
- Install Python packages so their main programs are available for your general use.
- Assist in creating and managing Python projects.
- Provide “better” versions of venv and pip.
Most simply, it can easily run packages, downloading as needed:
$ uv run --with ipython --with numpy --with matplotlib ipython --pylab In [1]: plt.plot([0,1],[2,3]) Out[1]: [<matplotlib.lines.Line2D at 0x7f6a656dd890>]
The first uv run may install packages which may take some time while future runs will be quick. You can also expose these locally installed packages:
$ uv tool install git+https://github.com/brettviren/gegede ... $ which gegede /home/bv/.local/bin/gegede $ gegede -h ...
3.2. Virtual environment
A Python virtual environment (venv) consists of two main elements
- A directory patterned like the conventional
/usr/localholding files from Python and various python packages. - An
activatescript to configurePATH-like environment variables to make use of these files.
A user needing to install Python packages should do so in a venv and not ever via --user option.
The recommended way to provide a Python venv is via direnv as described in section 2.4. With more effort, a user may instead create one manually as described here:
$ python -m venv myvenv ## or $ uv venv [--python X.Y.Z] myvenv
To configure the shell environment:
$ source myvenv/bin/activate
There exists also activate.fish and activate.csh for some non-sh shells. To undo the environment settings:
$ deactivate
Similar to the direnv example, OS-installed Python packages can be leveraged by creating the venv with:
$ python -m venv myvenv --system-site-packages ## or $ uv venv --system-site-packages myvenv
3.3. Mixing Spack, Debian and pip packages
Only certain combinations of Spack, Debian and pip packages can be expected to work. These are as described along with a direnv .envrc file to produce the venv.
- Spack and pip but not OS
When the shell is configured for Spack Python packages (
wcwc shell -l py-XXX) then apip installin a Python venv will use those packages. The venv relies on the exactwcwc shellenvironment to be repeated for subsequent usage. This is irrespective of using--system-site-packagesor not but in neither case will any OS packages be leveraged. Eg,wcwc shell -l py-numpyand withlayout python
- No Spack but pip and OS
When the shell is configured for any Spack packages unrelated to Python (or no Spack at all), then the venv can be used without regards to Spack. Use of
--system-site-packageswill causepip installto prefer use of an existing OS package to downloading/building it fresh.layout python3 --system-site-packages
- Spack and pip and OS with a hack
If Python is provided by Spack, system packages can be seen and used by
pip installifPYTHONPATHis set to/usr/lib/python3/dist-packages.layout python3 path_add PYTHONPATH /usr/lib/python3/dist-packages
- Augment OS with Spack, no pip, no venv
Spack Python and packages can be mixed with OS Python.
$ wcwc shell -l 'py-torch+cuda cuda_arch=89' $ jupyter notebook --no-browser # from the OS
Visit the web and test with
import torch; print(torch.cuda.device_count()). See section 10 for more info about use of Jupyter.
4. Wire-Cell Toolkit
- For developing WCT against Spack packages see: wcwc-howto.html#spack-view-wct.
- See this section for general installation.
- See this section for BNL specifics on installing wcwc Spack packages and this section for specifics on updating wire-cell-spack.
5. CUDA
The main motivation for WCWC is to share workstation GPUs at BNL. Their GPUS are all currently from nVidia and so CUDA support is a must.
5.1. Versions
The nVidia hardware and CUDA software landscape is a little complex and described by several versions:
- compute capability
- (cc) a number like 9.0 which maps to a set of supported compute features
- hardware family
- eg “turing” is a name that implies one or more cc numbers (table of device cc, table of family cc)
- driver version
- a number like 525.147.05 which gives the version of the nVidia Linux kernel driver
- toolkit version
a number like 11.8 or 12.0 which gives the version of CUDA user libraries and implies a minimum driver version.
Debian’s
libcudapackages are given a driver version number and provide support for multiple toolkit version numbers. Debian 12’slibcuda1package is at version 525.147.05 which provides various libcuda versions in the range 5.0 - 12.0.
A sampling of BNL GPUs, their amount of RAM and their cc, family and CUDA toolkit minimum version (all are supported up to and including CUDA 12).
| Card | RAM | CC | Family | CUDA |
|---|---|---|---|---|
| GTX 750 Ti | 2 GB | 5.0 | Maxwell | 6.5 |
| GTX 1050 Ti | 4 GB | 6.1 | Pascal | 8.0 |
| GTX 1060 | 6 GB | 6.1 | Pascal | |
| GTX 1650 | 4 GB | 7.4 | Turing | 10.0 |
| GTX 1660 | 6 GB | 7.5 | Turing | |
| RTX 2080 Ti | 11 GB | 7.5 | Turing | |
| RTX 3080 | 10 GB | 8.6 | Ampere | 11.0 |
| RTX 4070 Ti | 12 GB | 8.9 | Lovelace | 11.8 |
| RTX 4090 | 24 GB | 8.9 | Lovelace |
Each workstation must have nVidia kernel driver installed and loaded. It is recommended to use Debian’s package:
apt install nvidia-kernel-dkms nvidia-smi
A reboot may be required. The nvidia-smi command is used to query your GPU device status. If it gives an error then the kernel driver is not working.
wcwc@haiku:~$ nvidia-smi -L GPU 0: NVIDIA GeForce GTX 750 Ti (UUID: GPU-b87cc150-ba6d-329e-02e4-7f8ddef6a581)
Receiving this error:
Cuda error in file '.../file.cu' in line XX : no kernel image is available for execution on the device.
Indicates CUDA code was not compiled to target the device (GPU card) compute capability version.
5.2. CUDA and Spack
Spack can compile CUDA from scratch with some caveats:
- The Spack
cudapackage will only build astubs/libcuda.soand not a “real”libcuda.so. The stub holds “dummy” implementations of the CUDA API and may be used to satisfy link-time dependencies when building in an environment that lacks a “real”libcuda.so. On WCWC we will rely on Debian’slibcuda.so. - Some Spack packages care about the CUDA “cc” number. Eg, they may have internal code switches that depend on the version. Of those that care, some (like
kokkos) can be built with support for exactly one “cc”. The conventional way to communicate the “cc” to these packages is viacuda_archvariant parameter.
6. Torch
PyTorch is the primary DNN toolkit of WCWC users and WCT can make use of libtorch. Spack’s py-torch package builds the Python interface:
$ TMPDIR=$HOME/.cache/wcwc TEMP=$HOME/.cache/wcwc \ wcwc install "py-torch+cuda cuda_arch=89"
FIXME: must set TMPDIR to avoid /tmp being filled. For safety, set this and TEMP in wcwc, but really, this is a bug in the py-torch/package.py which should honor the configured build_stage setting. Note, this is not required when building in a podman container (assuming podman has sufficient space in the user’s home).
Also, earlier builds failed due to OpenMP being detected in the OS but not sufficient to work.
apt install libomp-dev was needed. The above was on a different Debian host. This also motivates the use of a container to build. In a minimal Debian 12 podman container, py-torch builds without problems (other than loooooong build time). The wcwc make-builder command creates the container.
ls /wcwc/opt/builtin/linux-debian12-x86_64_v4/gcc-12.2.0/py-torch-2.3.0-sq4psktodzbjq7v4bshow7rmcxwoa6r3/lib/python3.11/site-packages/torch/lib libc10_cuda.so libcaffe2_nvrtc.so libtorch_cpu.so libtorch_cuda.so libtorch_python.so libc10.so libshm.so libtorch_cuda_linalg.so libtorch_global_deps.so libtorch.so
Simple test:
$ python -c 'import torch; print(torch.cuda.is_available())' True $ python -c 'import torch; print(torch.cuda.device_count())' 2
Note, had to load a kernel module:
root@wcgpu0:~# modprobe nvidia_uvm
wcwc@wcgpu0:~$ wcwc admin install -S wirecell wire-cell-toolkit+torch+cuda cuda_arch=89
7. Ollama
The Ollama project provides the program ollama that exercises LLM models.
7.1. Versions
The official ollama/ollama supports a number of different LLM models. Some other models of interest require a modified ollama program. These variants may not support the same models as the official ollama. In general, one must only use models that are explicitly supported by a particular ollama variant.
WCWC provides official and variant ollama as different Spack packages. Currently:
- ollama
- From Spack
builtinsrepo with CUDA support (spack install ollama+cuda, added in PR). - ollamabin
- The official version. The “bin” implies the package consists of a binary release and to avoid conflict with the Spack
ollamapackage. - ollamalux
- A “hacked” version to run specifically
aiden_lu/minicpm-v2.6. The “lux” portion of the name refers to it originating from luixiao0/ollama binary release.
Regardless of the package name, the user uses the program named ollama.
7.2. Server
The ollama server is started simply:
$ ollama serve
To stop the server, simply hit Ctrl-c.
7.3. Environment
To help keep the multiple ollama variants distinct, the WCWC ollamaXXX packages will set some environment variables (the ollama package from Spack, proper sets none) when a user “loads” the package (eg wcwc shell -l ollamaXXX).
OLLAMA_HOST- server IP address and port number.
OLLAMA_TMPDIR- set to
$HOME/tmpto avoid filling/tmp OLLAMA_MODELS- location to store files for models.
For other available environment variables, run ollama serve --help.
Multiple clients, possibly run by multiple users, may share the same server. The user that starts the server may also kill it which of course will impact all clients. Users are encouraged to coordinate any sharing amongst themselves. A user may attempt to isolate their server by setting OLLAMA_HOST.
Users should not leave ollama server running longer than needed as it consumes GPU memory.
For either server or command line client usage, start a shell like:
u$ wcwc shell -S wirecell -l ollamaXXX s$ ollama [...]
Replace ollamaXXX with one of the package names in the list above.
7.4. Use GPUs
By default, ollama serve should auto detect GPU(s) as long as it finds the “real” libcuda.so provided by Debian. See section 5.2. If this seems to fail, running with the “debug” message level can help understand the cause:
$ OLLAMA_DEBUG=true ollama serve
Note, these warnings:
unable to locate gpu dependency libraries
are red-herrings and will be printed even when the GPU are found and usable.
By default ollama serve will use the GPUs it finds. To control which GPUs ollama may use you may set the CUDA_VISIBLE_DEVICES environment variable to a comma-separated list of GPU indices or IDs. Enumeration of GPU indices may not be stable so use IDs to list specific GPUs if stability is required. Some examples:
# use "first" device $ CUDA_VISIBLE_DEVICES=0 ollama serve # use "first" and "second" device $ CUDA_VISIBLE_DEVICES=0,1 ollama serve # use specific device(s) $ nvidia-smi -L # note the GPU IDs $ CUDA_VISIBLE_DEVICES=GPU-7d6529b0-76ce-2e7f-d96e-2018b58d1f2b ollama serve
7.5. Client
The ollama program can also act as a command line client:
$ ollama run llama3.1
Chat is logged by the server. To exit, hit Ctrl-d.
The first time a model is run its files will be downloaded by the server.
7.6. Checks
You can run various checks to observe the state of the server. Assuming a subshell configured for the ollamaXXX package:
$ ollama ls # list available models NAME ID SIZE MODIFIED llama3.1:latest f66fc8dc39ea 4.7 GB 42 seconds ago $ ollama ps # list running models: NAME ID SIZE PROCESSOR UNTIL llama3.1:latest f66fc8dc39ea 6.7 GB 100% GPU 4 minutes from now $ nvidia-smi # check actual GPU memory usage ... | 0 N/A N/A 531064 C ...unners/cuda_v12/ollama_llama_server 6142MiB | ...
7.7. Specific models
Above we showed examples with the official ollama program and supported models. This section collects guidance on variants. The models described below should not be expected to work with the official ollama nor should the ollama’s described her be expected to run the officially supported models.
7.7.1. lux / MiniCPM v2.6
The MiniCPM v2.6 model is claimed to be good for OCR. It requires a variant ollama which is provided by the ollamalux package. It may be used from two shells like:
$ wcwc shell -S wirecell -l ollamalux -c "ollama serve" $ wcwc shell -S wirecell -l ollamalux -c "ollama run aiden_lu/minicpm-v2.6:Q4_K_M"
8. Open WebUI
Open WebUI provides a web browser user interface to Ollama and other LLM APIs. In the future, WCWC may provide a centrally-installed instance of Open WebUI but as of now each user must install/run it themselves. This is mostly due to two reasons: the project development is brisk and the specific version and usage pattern tends to be tied to individual user tasks.
This section describes ways for the user to run Open WebUI with a local 7 and remote APIs.
8.1. General running of Open WebUI
Open WebUI is a Python program and can be run in the “usual ways” as described in the section 3. The recommended method is with 3.1.
Use of open-webui may require a newer Python than available by default. We can tell uv to use a non-standard python by setting the environment variable: UV_PYTHON=3.12.3. uv will automatically install that version of Python as needed. When UV_PYTHON is set for a uv tool install ... command it need not be set when using the installed tools. We assume this usage below
To run the Open WebUI server command:
$ UV_PYTHON=3.12.3 uv run --with open-webui open-webui serve
Or, to install it as a “tool” so you do not need to give uv run ...:
$ UV_PYTHON=3.12.3 uv tool install open-webui $ open-webui serve
8.2. Accessing Open WebUI web server
By default, Open WebUI serves on localhost port 8080. If you run a web browser on the same computer that runs Open WebUI simply visit:
On first run of open-webui serve command, Open WebUI will need some time to download various things. While that is ongoing, the server may not respond. Subsequent server starts will require less time.
On first access of a freshly started Open WebUI server you will be prompted to make an admin account.
8.3. Accessing remote Open WebUI
Open WebUI, as we use it on WCWC, should never be exposed to the Internet. However, it may be accessed remotely via SSH tunnels.
Assuming your SSH configuration defines the host wcgpu0 (see Remote access for examples) then a tunnel may be constructed by issuing this command to your local computer:
$ ssh -L 8080:localhost:8080 wcgpu0
And, from the your local computer, visit
8.4. Use with Ollama
While you can “chat” with Ollama directly from the terminal (ollama run ...),
Open WebUI provides many additional conveniences and features. By default, Open
WebUI can connect to a localhost Ollama server (running on same host as Open
WebUI itself) if Ollama was started in the default manner. If Ollama server is
configured to listen on a custom port, that port must be communicated to Open
WebUI:
$ OLLAMA_BASE_URL=http://127.0.0.1:8054/ open-webui serve
8.5. Use with openai.azure remote APIs
While Open WebUI documentation currently (as of v0.6.5) mentions Azure OpenAI as being usable, it appears not to work on its own. There appear to be two ways to use a 3rd party tool to connect Open WebUI with Azure OpenAI:
- Install the azure AI “function” to your Open WebUI
- Use litellm proxy.
8.5.1. Configure and run LiteLLM
We describe how to use the second. First, make a LiteLLM configuration file in YAML syntax that describes each model that you have been “deployed” on Azure OpenAI. An API URL and API key are needed as well as the deployment “name” for the model.
Multiple models can be deployed with the same API URL and key. This is assumed in the example. Adjust your LiteLM YAML configuration file if various values are relevant.
Here is an example.
model_list:
- model_name: o4-mini
litellm_params:
model: azure/o4-mini
api_base: "os.environ/AZURE_API_URL"
api_key: "os.environ/AZURE_API_KEY"
api_version: 2024-12-01-preview
- model_name: gpt-4.1
litellm_params:
model: azure/gpt-4.1
api_base: "os.environ/AZURE_API_URL"
api_key: "os.environ/AZURE_API_KEY"
api_version: 2024-12-01-preview
This configuration delegates to the user shell environment for the URL (api_base) and key.
Do not publicly disclose the API key unless you are willing to pay for the entire Internet using your account. In particular, keep API keys out of files that are inside git controlled directories or in general any directories exposed to a web server.
To run LiteLLM while providing these values:
$ AZURE_API_URL="https://THE-USER-NAME.openai.azure.com/" \ AZURE_API_KEY=THE85CHARACTERAPIKEY \ uv run --with 'litellm[proxy]' --with 'litellm[extra_proxy]' \ litellm --config litellm.yml
By default, this will listen on http://localhost:4000. You may visit this, expand the
GET /v1/models
section, click “Try it out” and then “execute”. Scroll down a bit and you should see the JSON response listing the models from the LiteLLM config.yaml file.
8.5.2. Connect Open WebUI to LiteLLM
Visit your Open WebUI page, click your avatar/initials, open Admin Panel, Settings, Connections and in Manage OpenAI API Connections, click “+”, enter for URL http://localhost:4000/v1. The API key and list of Model IDs can be left empty.
This use of LiteLLM with no API key or other auth is very insecure and should never be exposed outside of localhost.
Save and when starting a New Chat the models from the LiteLLM YAML configuration file should be listed and accessible.
9. Aider
The aider program gives a simple command line interface with a powerful set of LLM-assisted software development features. Installation is easy with uv
$ uv tool install [--force --upgrade] --python python3.12 aider-chat $ aider --help $ aider --model=MODEL --api-key PROVIDER=KEY
There are a lot of options. Here are some likely ones and examples are below
--no-auto-commitsBy default, aider willgit commitafter each round with the LLM. If you want to handle commits yourself, use this flag.
9.1. Aider + Gemini
Use with free Gemini
$ cd my-source-area/ $ aider --model gemini/gemini-flash-latest --api-key gemini=KEY [...]
You will need to get a KEY from Google. Sign in at https://aistudio.google.com and look for Get API Key and Create API Key, copy it and store it securely. Keep it out of any files managed by git. The key starts with the string AIza.
By default, each round of chat with the LLM will (may) change your code and will commit. If you would prefer to handle commits yourself add:
--no-auto-commits
9.2. Aider + Azure
Microsoft Azure provides access to many LLMs and aider knows how to use them.
$ export AZURE_API_BASE=https://RESOURCE.openai.azure.com/ $ export AZURE_API_VERSION=2025-09-01 $ AZURE_API_KEY=KEY $ aider --model azure/DEPLOYMENT [...]
BNL provides licenses to use Microsoft Azure. The model name is actually the “deployment” name. You make a “deployment” through your Azure account.
10. Jupyter
This section describes the Jupyter family of applications.
The Spack packaging for jupyter is rather out of date. Until there is reason to address that, we rely on Debian packages.
10.1. Web user interface
Perhaps most users see Jupyter as something to use from a web browser. For purely local use, run:
$ jupyter notebook # do not run on a remote account
A browser tab visiting the jupyter server should open. To control what browser to use, set BROWSER environment variable. Do not run this command on a remote workstation (see below for that). If it is run on BNL WCWC, the default browser will be opened which may be the terminal program lynx (hit q to quit that).
To jupyter notebook server tends to continue to run. If running in a terminal, the user may hit Ctrl-c and answer “y” to shutdown. If neglected, the user may run jupyter notebook stop PORT. To discover the port(s) of running server(s) run jupyter notebook list. Do not leave jupyter running longer than needed as it consumes memory and ports.
To run jupyter notebook a remote workstation an SSH tunnel must be established to forward local browser connections to the remote jupyter server.
$ ssh -L 8888:localhost:8888 wcgpu0.phy.bnl.gov jupyter-notebook --no-browser [...] http://localhost:8888/?token=9b1e482113f4c807250429c905d0e4d04c8175ba3d1a43b5
You may then click that link (if your terminal supports URLs) or copy-paste the URL to a local browser.
The literal usage of the URL printed by jupyter notebook assumes you chose the same port number on both ends of the SSH tunnel and that “localhost” resolves correctly to your local workstation. If the local and remote ports differ then you must edit the URL to use the local port.
The above command establishes the tunnel assuming the default jupyter port will be available for use. If another user is using that port jupyter will select the next available higher port number and you will see a message like:
[...] The port 8888 is already in use, trying another port. [...] http://localhost:8889/?token=9b1e482113f4c807250429c905d0e4d04c8175ba3d1a43b5
This change in expected port breaks the SSH tunnel. You must Ctrl-c and select “y” to kill the server and the SSH connection and then reconnect using a different port. In order to increase the chance to claim an unused port and keep consistent ports for the tunnel, we may specify a port of our choosing:
$ ssh -L 8998:localhost:8998 wcgpu0.phy.bnl.gov jupyter-notebook --no-browser --port 8998 [...] http://localhost:8998/?token=f33b43123890c20b169a7af168274f9e2998cd38f15a33be
10.2. Console
A command line alternative to the Jupyter web UI is the “console”. This can be run on the local or remote workstation without the need to establish an SSH tunnel or use a browser.
$ jupyter console In [1]: import matplotlib.pyplot as plt In [2]: plt.plot([1,2,3], [0,1,0]) Out[2]: [<matplotlib.lines.Line2D at 0x7f2907c68790>]
When showing plots or other graphics an X11 client window will be produced. When running on a remote workstation, this window will only be visible if X11 forwarding has been enabled. For SSH configuration guide see, for example, wcwc-bnl.html#remote-access with attention to the ForwardX11 configuration item.
10.3. Ipython
Jupyter originated from an older project called ipython, which is similar to jupyter console. ipython has some convenience features such as pre-loading numpy and matplotlib which makes it a useful alternative:
$ ipython --pylab In [1]: plt.plot([1,2,3], [0,1,0]) Out[1]: [<matplotlib.lines.Line2D at 0x7f38adf35cd0>]
10.4. Python environment
Default use of jupyter (or ipython) will have available any Python packages installed via Debian packages. The user may require additional packages which can be provided via pip or Spack in the usual way, prior to running jupyter. See section 3.
To configure for such running, it is recommended to use direnv with a .envrc file like:
$ mkdir env-jupyter $ echo 'layout python3 --system-site-packages' > env-jupyter/.envrc $ cd env-jupyter $ direnv allow $ pip install jupyter $ pip install spacy
This pip install jupyter should result in all or at least most packages reporting “Requirement already satisfied” which means the OS provides the actual package files. OTOH, the pip install spacy should bring in a mix of OS and fresh packages. This shows the --system-site-packages is operational.
The notebook server can then be started by hand:
$ python -m ipykernel install --user --name myenv $ jupyter notebook --no-browser ... http://localhost:8888/?token=32c26d0a705495b5a9e55bce6c60f4b7622c45a89fa3fafd
Based on the URL we may create a tunnel:
$ ssh -L 8888:localhost:8888 wcgpu0.phy.bnl.gov
Now visit the URL in the browser and locate myenv under New->Notebook: menu.
11. Geant4
Users / developers working inside an experiment stack will typically have Geant4 included in the dependency tree. Others may utilize those instances of Geant4. As usual spack find -lv geant4 will list what is available. Special-case builds of Geant4 will/can be made available.
11.1. Support for DUNE FD3
DUNE FD3 simulation uses geant4+qt+opengl+x11
$ wcwc spack find -lv geant4 -- linux-debian12-x86_64 / gcc@12.2.0 --------------------------- wsqk5qa geant4@11.2.2~ipo~motif+opengl+qt~tbb+threads~timemory~vecgeom~vtk+x11 build_system=cmake build_type=Release cxxstd=17 generator=make
11.2. Opticks
A likely to-do item is to build Geant4 with support for Opticks.
12. DNNROI
Prepare:
$ wget --no-check-certificate -r --no-parent 'https://www.phy.bnl.gov/~hyu/dunefd/dnn-roi-pdvd /Pytorch-UNet/data/' $ mv www.phy.bnl.gov/~hyu/dunefd/dnn-roi-pdvd/Pytorch-UNet/data . $ rm -rf www.phy.bnl.gov/ $ git clone https://github.com/HaiwangYu/Pytorch-UNet.git hyu-Pytorch-UNet $ echo 'layout python > hyu-Pytorch-UNet/.envrc $ cd hyu-Pytorch-UNet && direnv allow $ ln -s ../data . $ pip install torch torchvision torchaudio matplotlib h5py pillow pandas $ pip list Package Version ------------------------ ----------- contourpy 1.3.0 cycler 0.12.1 filelock 3.16.0 fonttools 4.53.1 fsspec 2024.9.0 h5py 3.11.0 Jinja2 3.1.4 kiwisolver 1.4.7 MarkupSafe 2.1.5 matplotlib 3.9.2 mpmath 1.3.0 networkx 3.3 numpy 2.1.1 nvidia-cublas-cu12 12.1.3.1 nvidia-cuda-cupti-cu12 12.1.105 nvidia-cuda-nvrtc-cu12 12.1.105 nvidia-cuda-runtime-cu12 12.1.105 nvidia-cudnn-cu12 9.1.0.70 nvidia-cufft-cu12 11.0.2.54 nvidia-curand-cu12 10.3.2.106 nvidia-cusolver-cu12 11.4.5.107 nvidia-cusparse-cu12 12.1.0.106 nvidia-nccl-cu12 2.20.5 nvidia-nvjitlink-cu12 12.6.68 nvidia-nvtx-cu12 12.1.105 packaging 24.1 pandas 2.2.2 pillow 10.4.0 pip 23.0.1 pyparsing 3.1.4 python-dateutil 2.9.0.post0 pytz 2024.2 setuptools 66.1.1 six 1.16.0 sympy 1.13.2 torch 2.4.1 torchaudio 2.4.1 torchvision 0.19.1 triton 3.0.0 typing_extensions 4.12.2 tzdata 2024.1
Run:
$ bash train3.sh
Epoch finished ! Loss: 0.032897
Checkpoint e0 saved !
Traceback (most recent call last):
File "/home/wcwc/dev/dnnroi/hyu-Pytorch-UNet/train3.py", line 264, in <module>
train_net(net=net,
File "/home/wcwc/dev/dnnroi/hyu-Pytorch-UNet/train3.py", line 199, in train_net
val_loss = eval_loss(net, criterion, val2, gpu)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wcwc/dev/dnnroi/hyu-Pytorch-UNet/eval_util.py", line 43, in eval_loss
true_masks_flat = true_mask.view(-1)
^^^^^^^^^^^^^^^^^^
RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.
real 0m28.260s
user 0m26.832s
sys 0m5.687s
Try SBND’s flavor
$ git clone https://github.com/wjdanswjddl/Pytorch-UNet.git moon-Pytorch-UNet $ echo 'layout python > moon-Pytorch-UNet/.envrc $ cd moon-Pytorch-UNet && direnv allow $ ln -s ../data . $ pip install torch torchvision torchaudio matplotlib h5py pillow pandas $ pip install tqdm tensorboard
13. ArrayFire
c$ spack install arrayfire+cuda cuda_arch=89