Pace

Pace is an implementation of the FV3GFS / SHiELD atmospheric model developed by NOAA/GFDL using the NDSL middleware in Python, itself based on GT4Py and DaCe. The model can be run on a laptop using Python-based backend or on thousands of heterogeneous compute nodes of a large supercomputer.

🚧 WARNING This repo is under active development - supported features and procedures can change rapidly and without notice. 🚧

The repository model code is split between pyFV3 for the dynamical core and pySHiELD for the physics parametrization. A full dependencies looks like the following:

flowchart TD
GT4Py.cartesian --> |Stencil DSL|NDSL
DaCe  --> |Full program opt|NDSL
NDSL --> pyFV3
NDSL --> pySHiELD
pyFV3 --> |Dynamics|Pace
pySHiELD --> |Physics|Pace

Quickstart - bare metal

Build

Pace requires:

• GCC > 9.2
• MPI
• Python 3.8.

For GPU backends CUDA and/or ROCm is required depending on the targeted hardware.

When cloning Pace you will need to update the repository's submodules as well:

git clone --recursive https://github.com/NOAA-GFDL/pace.git

or if you have already cloned the repository:

git submodule update --init --recursive

We recommend creating a python venv or conda environment specifically for Pace.

python -m venv .venv
source .venv/bin/activate

Inside of your pace venv or conda environment, install pace and its dependencies. For developers, we recommend an editable install with the [dev] extra:

pip install -e .[dev]

To install editable versions of the submodules of Pace, use the -e flag followed by the name of the package (if editing the local version) or path to edited version of the submodule after base installation of Pace:

pip install -e .
pip install -e NDSL
# pip install -e /path/to/edited/NDSL
pip install -e pyFV3
# pip install -e /path/to/edited/pyFV3
pip install -e pySHiELD
# pip install -e /path/to/edited/pySHiELD

For running tests, we recommend to install pace with the [test] extra (avoid pulling other dev dependencies):

pip install .[test]

For just running pace, you don't need any extra:

pip install .

Shell scripts to install Pace on specific machines such as Gaea can be found in examples/build_scripts/.

Run

Located in the directory examples/configs/ there are several example configurations to exhibit the current functionality of Pace. We suggest a new user start with the baroclinic_c12.yaml configuration.

mpirun -n 6 python3 -m pace.run examples/configs/baroclinic_c12.yaml

# or with oversubscribe if you do not have at least 6 cores
mpirun -n 6 --oversubscribe python3 -m pace.run examples/configs/baroclinic_c12.yaml

After the run completes, you will see an output directory output.zarr. An example to visualize the output is provided in examples/plot_output.py. See the driver example section for more details.

Environment variable configuration

• NDSL_CONSTANTS: Pace is bundled with various constants.
  • GFDL NOAA's FV3 dynamical core constants (original port)
  • GFS Constant as defined in NOAA GFS
  • GEOS Constant as defined in GEOS v13
• NDSL_LITERAL_PRECISION: default precision of the field & scalars in the numerics. Default to 64.
• NDSL_LOGLEVEL: logging level to display (DEBUG, INFO, WARNING, ERROR, CRITICAL). Default to INFO.

Quickstart - Docker

Build

While it is possible to install and build pace bare-metal, we can ensure all system libraries are installed with the correct versions by using a Docker container to test and develop pace.

First, you will need to update the git submodules so that any dependencies are cloned and at the correct version:

git submodule update --init --recursive

Then build the pace docker image at the top level.

make build

Run

make dev
mpirun --mca btl_vader_single_copy_mechanism none -n 6 python -m pace.run /examples/configs/baroclinic_c12.yaml

History

This repository was first developed at AI2 and the institute conserves an archived copy with the latest state before the NOAA took over.

Running pace in containers

Docker images exist in the Github Container Registry associated with the NOAA-GFDL organization.
These images are publicly accessible and can be used to run a Docker container to work with pace.
The following are directions on how to setup the pace conda environment interactively in a container.

The latest images can be pulled with the Docker as shown below or
with any other container management tools:

docker pull ghcr.io/noaa-gfdl/pace_mpich:3.8

for MPICH installation of MPI; and

docker pull ghcr.io/noaa-gfdl/pace_openmpi:3.8

for OpenMPI installation of MPI.

If permission issues arise during the pull, a Github personal token
may be required. The steps to create a personal token is found
here

Once the token has been generated, the image can be pulled for example with with:

docker login --username GITHUB_USERNAME --password TOKEN
docker pull ghcr.io/noaa-gfdl/pace_mpich:3.8

Any container management tools compatible with Docker images can be used
to run the container interactively from the pulled image.
With Docker, the following command runs the container interactively.

docker run -it pace_mpich:3.8

In the container, the default base conda environment is already activated.
The pace conda environment can be created by following the steps below:

git clone --recursive -b develop https://github.com/NOAA-GFDL/pace.git pace
cd pace
cp /home/scripts/setup_env.sh . && chmod +x setup_env.sh
source ./setup_env.sh