Use of containers in WCWC

Table of Contents

1. Introduction

Containers are used with WCWC for a few purposes:

  • Packages are installed in a well defined, minimal container.
  • Images may be built from installed packages and defined Spack Environments.

2. Container technology

We consider here the use of podman for building containers.

3. Building packages

A container applies constraints on how spack installs packages beyond what the wcwc command supplies. To assure reproducible builds, it is required to fully specify the build environment because Spack packaging is not perfectly hermetic. Minor, seemingly unrelated details of the OS can radically determine if some package builds or not. For example, py-torch is sensitive to the existence of libgomp even when OpenMP is disabled. A container used for installing Spack packages can well specified and made minimal. Any special adjustments become documented in the Containerfile. A new WCWC site can take a proven container and have high expectation of success to rebuild an ecosystem. Building in a container enables flexibility in distributing installations across different hosts and/or accounts that may not have write access to the common /wcwc while still utilizing the expected prefix and not resorting to rpath rewriting. A Containerfile may also be used to capture history of installation. It may simply be continuously extended with novel wcwc install commands as new packages are required.

3.1. Images

Unless specified, every image is built from a Container.NAME like

cd wcwc/
podman build -t wcwc-NAME -f images/builder/Containerfile.NAME .

Some images build on others and thus it is required that NAME match as above.

Images allow for the container /wcwc to be provided as a host directory when run like:

podman run -v /host/path/wcwc:/wcwc:rw,U --userns=keep-id -it --rm NAME [COMMAND LINE]

Some images have a specific entry point, others allow running bash or other program.

The wcwc-builder image provides the minimal build OS, the wcwc user and command and the initial, empty /wcwc. This has no entry point and defaults to bash.

The spack-develop image is built upon the wcwc-builder image and simply adds spack with the develop branch checked out in /wcwc/spack. It has an entry point of wcwc. The intention is that it is used to consistently run install commands.

3.2. Bootstrapping

wcwc@wcgpu0:~$ wcwc make-builder
wcwc@wcgpu0:~$ podman run -v $HOME/wcwc-dev:/wcwc -it --rm wcwc-builder init --tag develop spack
wcwc@wcgpu0:~$ podman run -v $HOME/wcwc-dev:/wcwc -it --rm wcwc-builder install "py-torch+cuda cuda_arch=89"

112 packages. Most require a few second to a couple minutes and load is O(1) core. py-torch itself uses about 16 core (64 avail) and takes 20 minutes.

wcwc@wcgpu0:~$ podman run -v $HOME/wcwc-dev:/wcwc -it --rm wcwc-builder init wirecell
wcwc@wcgpu0:~$ podman run -v $HOME/wcwc-dev:/wcwc -it --rm wcwc-builder install -S wirecell wire-cell-prototype
wcwc@wcgpu0:~$ podman run -v $HOME/wcwc-dev:/wcwc -it --rm wcwc-builder install -S wirecell wire-cell-toolkit
wcwc@wcgpu0:~$ podman run -v $HOME/wcwc-dev:/wcwc -it --rm wcwc-builder install emacs
# see https://github.com/WireCell/wire-cell-toolkit/issues/330
# quickly add a +emacs variant
wcwc@wcgpu0:~$ podman run -v $HOME/wcwc-dev:/wcwc -it --rm wcwc-builder install -S wirecell wire-cell-toolkit+emacs
@wcgpu0:~$ podman run -v $HOME/wcwc-dev:/wcwc -it --rm wcwc-builder install -S wirecell wire-cell-toolkit+emacs+root+hdf+glpk

Note, wire-cell-toolkit+root does not concretize to the root from wire-cell-prototype. WCP needs a "bigger" root than WCT.

Next:

  • [ ] run-in-podman
  • [ ] update wire-cell-spack to depend on py-torch and py-torch+cuda
  • [ ] rsync to server.

Author: Brett Viren

Created: 2026-01-20 Tue 10:21

Validate