Data

Tools

Indexes

Applications

Experiments

Open Sustainable Technology

Recent Releases of Neural-LAM

Neural-LAM - v0.4.0

This release introduces a number of improvements to logging, multi-node training and variable rescaling, without making any major changes to the neural-lam structure.

Added

  • Add support for MLFlow logging and metrics tracking. #77
    @khintz

  • Add support for multi-node training.
    #103 @simonkamuk @sadamov

  • Add option to clamp output prediction using limits specified in config file #92 @SimonKamuk

  • Add publication of releases to pypi.org. #71 @leifdenby, @observingClouds

Fixed

  • Only print on rank 0 to avoid duplicates of all print statements.
    #103 @simonkamuk @sadamov

  • Fix MLFlow exception import introduced in #77.
    #111
    @observingClouds

  • Fix duplicate tensor copy to CPU #106 @observingClouds

  • Fix bug where the inverse_softplus used in clamping caused nans in the gradients #123 @SimonKamuk

  • Add standardization to state diff stats from mdp datastore #122 @SimonKamuk

  • Set ci/cd badges to refer to the new test matrix #130 @SimonKamuk

  • use correct split of data with the --eval val or --eval test cli arguments #139 @SimonKamuk

  • Fix step length calculation when dt >= 24h #141 @deinal

Maintenance

  • update ci/cd testing to use cuda 12.8 #140 @SimonKamuk

  • update ci/cd testing to use pre-commit v3.0.1 #140 @SimonKamuk

  • update AWS GPU ci/cd to use ami with larger (200GB) root volume and ensure
    nvme drive is used for pip venvn
    #126, @leifdenby

  • update ci/cd testing setup to install torch version compatible with neural-lam
    dependencies #115, @leifdenby

  • switch to new npyfiles MEPS and mdp DANRA test datasets which are coincident
    in time and space (on cropped ~100x100 grid-point domain)
    #110, @leifdenby

  • use dynamic versioning based on git tags and commit hashes
    #118, @observingClouds

  • add detect_anomaly=True to pl.Trainer in test_training.py #124, @SimonKamuk

Atmosphere - Meteorological Observation and Forecast - Python
Published by joeloskarsson 20 days ago

Neural-LAM - v0.3.0

This release introduces Datastores to represent input data from different sources (including zarr and numpy) while keeping graph generation within neural-lam.

Added

  • Introduce Datastores to represent input data from different sources, including zarr and numpy. #66 @leifdenby @sadamov

Fixed

  • Fix wandb environment variable disabling wandb during tests. Now correctly uses WANDB_MODE=disabled. #94 @joeloskarsson

  • Fix bugs introduced with datastores functionality relating visualation plots #91 @leifdenby

Atmosphere - Meteorological Observation and Forecast - Python
Published by leifdenby 5 months ago

Neural-LAM - v0.2.0

Highlights

This release focuses on setting up the neural-lam repository and codebase to enable collaboration.

Detailed Changes

Added

  • Added tests for loading dataset, creating graph, and training model based on reduced MEPS dataset stored on AWS S3, along with automatic running of tests on push/PR to GitHub, including push to main branch. Added caching of test data to speed up running tests.
    #38 #55
    @SimonKamuk

  • Replaced constants.py with data_config.yaml for data configuration management
    #31
    @sadamov

  • new metrics (nll and crps_gauss) and metrics submodule, stddiv output option
    c14b6b4
    @joeloskarsson

  • ability to "watch" metrics and log
    c14b6b4
    @joeloskarsson

  • pre-commit setup for linting and formatting
    #6, #8
    @sadamov, @joeloskarsson

  • added github pull-request template to ease contribution and review process
    #53, @leifdenby

  • ci/cd setup for running both CPU and GPU-based testing both with pdm and pip based installs #37, @khintz, @leifdenby

Changed

  • Clarify routine around requesting reviewer and assignee in PR template
    #74
    @joeloskarsson

  • Argument Parser updated to use action="store_true" instead of 0/1 for boolean arguments.
    (https://github.com/mllam/neural-lam/pull/72)
    @ErikLarssonDev

  • Optional multi-core/GPU support for statistics calculation in create_parameter_weights.py
    #22
    @sadamov

  • Robust restoration of optimizer and scheduler using ckpt_path
    #17
    @sadamov

  • Updated scripts and modules to use data_config.yaml instead of constants.py
    #31
    @sadamov

  • Added new flags in train_model.py for configuration previously in constants.py
    #31
    @sadamov

  • moved batch-static features ("water cover") into forcing component return by WeatherDataset
    #13
    @joeloskarsson

  • change validation metric from mae to rmse
    c14b6b4
    @joeloskarsson

  • change RMSE definition to compute sqrt after all averaging
    #10
    @joeloskarsson

Removed

  • WeatherDataset(torch.Dataset) no longer returns "batch-static" component of
    training item (only prev_state, target_state and forcing), the batch static features are
    instead included in forcing
    #13
    @joeloskarsson

Maintenance

  • simplify pre-commit setup by 1) reducing linting to only cover static
    analysis excluding imports from external dependencies (this will be handled
    in build/test cicd action introduced later), 2) pinning versions of linting
    tools in pre-commit config (and remove from requirements.txt) and 3) using
    github action to run pre-commit.
    #29
    @leifdenby

  • change copyright formulation in license to encompass all contributors
    #47
    @joeloskarsson

  • Fix incorrect ordering of x- and y-dimensions in comments describing tensor
    shapes for MEPS data
    #52
    @joeloskarsson

  • Cap numpy version to < 2.0.0 (this cap was removed in #37, see below)
    #68
    @joeloskarsson

  • Remove numpy < 2.0.0 version cap
    #37
    @leifdenby

  • turn neural-lam into a python package by moving all *.py-files into the
    neural_lam/ source directory and updating imports accordingly. This means
    all cli functions are now invoke through the package name, e.g. python -m neural_lam.train_model instead of python train_model.py (and can be done
    anywhere once the package has been installed).
    #32, @leifdenby

  • move from requirements.txt to pyproject.toml for defining package dependencies.
    #37, @leifdenby

  • Add slack and new publication info to readme
    #78
    @joeloskarsson

Compatibility

This version has been tested with Python 3.9-3.12.

Upgrade Steps

To upgrade to neural-lam v0.2.0:

  1. Update your local version:

    • git pull the latest changes from the repository and install locally pip install -e .
    • Note: This release is not yet available on PyPI. You will need to install it from the GitHub repository.

  2. Adapt any code relying on constants previously defined in neural_lam/constants.py to instead read them from the new YAML config file.

Links

Atmosphere - Meteorological Observation and Forecast - Python
Published by sadamov 8 months ago

Neural-LAM - v0.1.0

Initial version of Neural-LAM, reproduces the workshop paper.

Atmosphere - Meteorological Observation and Forecast - Python
Published by joeloskarsson over 1 year ago