CREDIT - v2025.2.0

Major Updates

• Documentation has been significantly updated.
• Rollouts on ERA5 and GFS have been significantly streamlined.
• Physics constraints as postblocks have been added.
• Vertical interpolation to pressure and height levels.
• Experimental ensemble capabilities.
• Many bug fixes and code cleanups.

What's Changed

• Addition of new multi-step dataset that allows batch size > 1 by @jsschreck in https://github.com/NCAR/miles-credit/pull/135
• Multistep training with batch_size >=1 per GPU by @jsschreck in https://github.com/NCAR/miles-credit/pull/139
• Fix train/valid split read in dataset by @kanz76 in https://github.com/NCAR/miles-credit/pull/140
• Gradscaler for grap clipping fix by @kanz76 in https://github.com/NCAR/miles-credit/pull/141
• Major update for the inference routine + QoL improvements by @yingkaisha in https://github.com/NCAR/miles-credit/pull/134
• Added valid_forecast_len to the result_dict by @kanz76 in https://github.com/NCAR/miles-credit/pull/143
• Refactored validate method in trainerERA5_multistep_grad_acum.py for new dataset by @kanz76 in https://github.com/NCAR/miles-credit/pull/145
• Minor bugfix on credit.parser by @yingkaisha in https://github.com/NCAR/miles-credit/pull/144
• Bug fix on the original multi-step dataset by @jsschreck in https://github.com/NCAR/miles-credit/pull/146
• Minor bugfix on credit.data by @yingkaisha in https://github.com/NCAR/miles-credit/pull/147
• bugfix: removed deprecated ERA5Dataset and Bridgescaler_dataset from train.py by @dkimpara in https://github.com/NCAR/miles-credit/pull/148
• Ensemble capability by @dkimpara in https://github.com/NCAR/miles-credit/pull/137
• Predict scripts support batch size >= 1 by @jsschreck in https://github.com/NCAR/miles-credit/pull/142
• rollouts enabled for ensemble, cftime, skebs. train_universal enabled for partial model loading. debugger model for dev support by @dkimpara in https://github.com/NCAR/miles-credit/pull/149
• Updates to grad clipping, seed, and checkpointing by @jsschreck in https://github.com/NCAR/miles-credit/pull/152
• CREDIT physcis for hybrid simga-pressure level configurations by @yingkaisha in https://github.com/NCAR/miles-credit/pull/129
• Fixed trainer classes to actually use DDP by @kanz76 in https://github.com/NCAR/miles-credit/pull/153
• Added std to printer log when using ensemble > 1 by @jsschreck in https://github.com/NCAR/miles-credit/pull/154
• Docs, requirements, and some code formatting by @djgagne in https://github.com/NCAR/miles-credit/pull/155
• Fix rollou_*_batcher.py by @kanz76 in https://github.com/NCAR/miles-credit/pull/156
• Fixed varnum_diag bug by @kanz76 in https://github.com/NCAR/miles-credit/pull/159
• FSDP to regular model or model weights by @jsschreck in https://github.com/NCAR/miles-credit/pull/157
• Fixes to interpolation to pressure levels by @djgagne in https://github.com/NCAR/miles-credit/pull/158
• Small fixes so option none works on casper using python and not torchrun by @jsschreck in https://github.com/NCAR/miles-credit/pull/161
• Cleaning up saving + multiprocessing in rollout_to_netcdf.py script by @jsschreck in https://github.com/NCAR/miles-credit/pull/162
• Updating packaging support by @djgagne in https://github.com/NCAR/miles-credit/pull/163
• Documentation improvement of example-v2025.2.0.yml by @yingkaisha in https://github.com/NCAR/miles-credit/pull/164
• Renaming scripts by @jsschreck in https://github.com/NCAR/miles-credit/pull/165
• Updates to data loader and solar by @djgagne in https://github.com/NCAR/miles-credit/pull/169
• Add regridding tools + minor bugfix on the inference dataset by @yingkaisha in https://github.com/NCAR/miles-credit/pull/166
• Updating config/ by @jsschreck in https://github.com/NCAR/miles-credit/pull/167
• Pulled out garbage collection from rollout_metrics by @jsschreck in https://github.com/NCAR/miles-credit/pull/172
• Fix unit testing and removed garbage collection by @djgagne in https://github.com/NCAR/miles-credit/pull/171
• Documentation updates by @djgagne in https://github.com/NCAR/miles-credit/pull/170
• configuration explaination to READtheDocs by @WillyChap in https://github.com/NCAR/miles-credit/pull/168
• Detached and re-assigned y_pred to discard computational graph by @kanz76 in https://github.com/NCAR/miles-credit/pull/174
• Small temporary update to pbs.py to fix newer cudnn settings on Derecho by @jsschreck in https://github.com/NCAR/miles-credit/pull/175
• Realtime Rollout and Predict + Interpolation Updates by @djgagne in https://github.com/NCAR/miles-credit/pull/176
• rollout bugfix + documentation by @yingkaisha in https://github.com/NCAR/miles-credit/pull/177
• Camulator v01.00 by @WillyChap in https://github.com/NCAR/miles-credit/pull/179
• bunch of small fixes, features, comments by @dkimpara in https://github.com/NCAR/miles-credit/pull/180
• Updates to bred vector implementation and noisy WXFormer by @jsschreck in https://github.com/NCAR/miles-credit/pull/178
• GFS initial conditions by @charlie-becker in https://github.com/NCAR/miles-credit/pull/173
• documentation updates by @ggantos in https://github.com/NCAR/miles-credit/pull/183
• Skebs v 1.0 by @dkimpara in https://github.com/NCAR/miles-credit/pull/182
• Trainer documentation by @jsschreck in https://github.com/NCAR/miles-credit/pull/181
• Parallel version of CRPS by @jsschreck in https://github.com/NCAR/miles-credit/pull/186
• Documentation hackathon by @djgagne in https://github.com/NCAR/miles-credit/pull/184

New Contributors

• @charlie-becker made their first contribution in https://github.com/NCAR/miles-credit/pull/173
• @ggantos made their first contribution in https://github.com/NCAR/miles-credit/pull/183

Full Changelog: https://github.com/NCAR/miles-credit/compare/v2024.1.0...v2025.2.0

Atmosphere - Meteorological Observation and Forecast - Jupyter Notebook
Published by djgagne 3 months ago

CREDIT - v2024.1.0

Our first public release of MILES CREDIT.

What's Changed

• add helper scripts to gather global data and scaling params by @WillyChap in https://github.com/NCAR/miles-credit/pull/1
• Updated dependencies and added scaler application by @djgagne in https://github.com/NCAR/miles-credit/pull/2
• Initial update of multi-step training by @jsschreck in https://github.com/NCAR/miles-credit/pull/3
• Adding PBC to crossformer model by @jsschreck in https://github.com/NCAR/miles-credit/pull/6
• Path fixing by @jsschreck in https://github.com/NCAR/miles-credit/pull/8
• reworked spectralLoss2D to be compatible with new models by @dkimpara in https://github.com/NCAR/miles-credit/pull/7
• fixed spectralLoss2D when using no lat weights, also removed deprecated code by @dkimpara in https://github.com/NCAR/miles-credit/pull/9
• Updating few bugs in model classes. Using units now in predict.py by @jsschreck in https://github.com/NCAR/miles-credit/pull/11
• zarrify.py script update by @sethmcg in https://github.com/NCAR/miles-credit/pull/4
• wrote PSDLoss, benchmarked values, added option fields to configs by @dkimpara in https://github.com/NCAR/miles-credit/pull/14
• Some quick improvments of draw_forecast within predict.py by @yingkaisha in https://github.com/NCAR/miles-credit/pull/16
• using os.path.expandvars in train.py,predict.py to enable generic save locs by @dkimpara in https://github.com/NCAR/miles-credit/pull/17
• Update predict.py, visualization_tools.py, and config file options by @yingkaisha in https://github.com/NCAR/miles-credit/pull/20
• Updated FSDP checkpointing by @jsschreck in https://github.com/NCAR/miles-credit/pull/21
• polar and laplacian diffusion filter module class by @WillyChap in https://github.com/NCAR/miles-credit/pull/23
• predict.py revamp - viz config options, async image generation, xarray creation, laplacian filtering, CPU-only runtime compatible by @dkimpara in https://github.com/NCAR/miles-credit/pull/27
• Modifications of predict.py by @jsschreck in https://github.com/NCAR/miles-credit/pull/28
• predict.py - reworked xr save format, using logger by @dkimpara in https://github.com/NCAR/miles-credit/pull/29
• Fixing FSDP revert; adding multi-step trajectory trainer by @jsschreck in https://github.com/NCAR/miles-credit/pull/31
• Quantile static by @WillyChap in https://github.com/NCAR/miles-credit/pull/32
• Updates to xformer model class and xformer configurations by @jsschreck in https://github.com/NCAR/miles-credit/pull/34
• Model base class by @dkimpara in https://github.com/NCAR/miles-credit/pull/35
• FuXi bug fix + Colorbar adjustments for visualization_tools by @yingkaisha in https://github.com/NCAR/miles-credit/pull/36
• TOA var and Quantile scaling, by @WillyChap in https://github.com/NCAR/miles-credit/pull/37
• Predict.py now supports FSDP by @jsschreck in https://github.com/NCAR/miles-credit/pull/38
• Adding static inputs by @jsschreck in https://github.com/NCAR/miles-credit/pull/39
• Conus404 data loader by @sethmcg in https://github.com/NCAR/miles-credit/pull/22
• adding transform/inverse for bscaler data by @WillyChap in https://github.com/NCAR/miles-credit/pull/40
• Addition of CONUS404 data, model, and training scripts by @jsschreck in https://github.com/NCAR/miles-credit/pull/41
• Updated FSDP-related bugs by @jsschreck in https://github.com/NCAR/miles-credit/pull/42
• edge case fix by @WillyChap in https://github.com/NCAR/miles-credit/pull/44
• Graph residual transformer with sparse edge calculation by @djgagne in https://github.com/NCAR/miles-credit/pull/26
• KE and spectrum visualization diagnostics + data conversions by @dkimpara in https://github.com/NCAR/miles-credit/pull/43
• Adding the replay buffer code as example by @jsschreck in https://github.com/NCAR/miles-credit/pull/45
• Adding Climate run capabilites by @WillyChap in https://github.com/NCAR/miles-credit/pull/46
• Fuxi update by @jsschreck in https://github.com/NCAR/miles-credit/pull/47
• Start of major refactoring of code base by @jsschreck in https://github.com/NCAR/miles-credit/pull/48
• TOA data type conversion and input tensor shape comment by @kanz76 in https://github.com/NCAR/miles-credit/pull/51
• FuXi model updates and data pipeline initial work by @yingkaisha in https://github.com/NCAR/miles-credit/pull/52
• Added Swin transformer model + new rollout script by @jsschreck in https://github.com/NCAR/miles-credit/pull/53
• The new data pipeline by @yingkaisha in https://github.com/NCAR/miles-credit/pull/55
• Data pipeline update with minor bugfix and documentation by @yingkaisha in https://github.com/NCAR/miles-credit/pull/58
• NetCDF metadata, docs, environment fixes by @djgagne in https://github.com/NCAR/miles-credit/pull/56
• Conus404 transforms by @sethmcg in https://github.com/NCAR/miles-credit/pull/54
• add 6hourly cached option by @WillyChap in https://github.com/NCAR/miles-credit/pull/57
• Emergency bug fix for the new roll-out and metadata loader by @yingkaisha in https://github.com/NCAR/miles-credit/pull/60
• update yaml to work by @WillyChap in https://github.com/NCAR/miles-credit/pull/61
• Add 6 hourly run configs with the new data pipeline + minor bugfix by @yingkaisha in https://github.com/NCAR/miles-credit/pull/62
• rollout_to_netcdf_new.py that works with the new data pipeline by @yingkaisha in https://github.com/NCAR/miles-credit/pull/64
• Updates to multi-step training a la Brenowitz scheme by @jsschreck in https://github.com/NCAR/miles-credit/pull/63
• Bugfix on rollout_to_netcdf_new.py + adding example.yml by @yingkaisha in https://github.com/NCAR/miles-credit/pull/67
• Memory optimization for one_shot=True + minor corrections for the example.yml by @yingkaisha in https://github.com/NCAR/miles-credit/pull/69
• Initialization of credit.trainers by @jsschreck in https://github.com/NCAR/miles-credit/pull/65
• Updates to the multi-step training code by @jsschreck in https://github.com/NCAR/miles-credit/pull/70
• new lat weighting by @WillyChap in https://github.com/NCAR/miles-credit/pull/66
• Add dynamic_forcing_variables into the data workflow by @yingkaisha in https://github.com/NCAR/miles-credit/pull/73
• Initial code for the Graph Residual Transformer + GRU model by @kanz76 in https://github.com/NCAR/miles-credit/pull/72
• Added reload_epoch option to the trainers by @jsschreck in https://github.com/NCAR/miles-credit/pull/76
• minor adjustments on credit.model + update configs by @yingkaisha in https://github.com/NCAR/miles-credit/pull/74
• Bugfix on model checkpointing and epoch update by @yingkaisha in https://github.com/NCAR/miles-credit/pull/77
• New solar processing and bridgescaler state scaler by @djgagne in https://github.com/NCAR/miles-credit/pull/78
• Updating the swin transformer by @jsschreck in https://github.com/NCAR/miles-credit/pull/80
• Adding parser and input data checks for config.yml + some minor fixes by @yingkaisha in https://github.com/NCAR/miles-credit/pull/79
• Model check-pointing and early-stopping fixes for base_trainer by @yingkaisha in https://github.com/NCAR/miles-credit/pull/81
• Bug fixes and few updates with variable weights by @jsschreck in https://github.com/NCAR/miles-credit/pull/82
• Small bugfix on parsing one_shot=False by @yingkaisha in https://github.com/NCAR/miles-credit/pull/88
• Updates to multi-step training by @jsschreck in https://github.com/NCAR/miles-credit/pull/87
• Adding arXiv paper production run configs + documentation by @yingkaisha in https://github.com/NCAR/miles-credit/pull/89
• [In Progress]: PBS Script and some minor improvements. by @negin513 in https://github.com/NCAR/miles-credit/pull/90
• few typos by @negin513 in https://github.com/NCAR/miles-credit/pull/92
• Fixes to new MPI launch; linting; reorganizing distributed by @jsschreck in https://github.com/NCAR/miles-credit/pull/93
• add lev functionality by @WillyChap in https://github.com/NCAR/miles-credit/pull/84
• Adding NVIDIA-makani scheme for multi-step training / rollout script for computing metrics on forecasts by @jsschreck in https://github.com/NCAR/miles-credit/pull/91
• multistep configurations + optimized cached dataset workflow + old script clean-up by @yingkaisha in https://github.com/NCAR/miles-credit/pull/94
• Updated learning rate scheduler and metrics handling by @jsschreck in https://github.com/NCAR/miles-credit/pull/95
• Notebooks and new github actions workflow by @djgagne in https://github.com/NCAR/miles-credit/pull/97
• Updated README with credit-derecho installation instructions added by @jsschreck in https://github.com/NCAR/miles-credit/pull/99
• Updated MPI support rollout_metrics by @jsschreck in https://github.com/NCAR/miles-credit/pull/100
• Emergency fix on rollout_to_netcdf.py and rollout_metrics by @yingkaisha in https://github.com/NCAR/miles-credit/pull/104
• Bugfix on trainerERA5_multistep_grad_accum.py by @yingkaisha in https://github.com/NCAR/miles-credit/pull/106
• Infrastructure for post-processing blocks (e.g. SKEBS, mass correction, laplacian filtering) by @dkimpara in https://github.com/NCAR/miles-credit/pull/105
• fixing environment_[device].yml specs by @dkimpara in https://github.com/NCAR/miles-credit/pull/108
• Major credit.postblock update + major credit.physics_core update + major bugfix + minor code optimization and cleaning by @yingkaisha in https://github.com/NCAR/miles-credit/pull/86
• Bugfix on the use of diagnostic variables in trainerERA5_multistep_grad_accum.py by @yingkaisha in https://github.com/NCAR/miles-credit/pull/111
• Fix NCLL_NET_GDR_LEVEL environment variable in credit.pbs.py by @kanz76 in https://github.com/NCAR/miles-credit/pull/115
• PHB does not work. Revert to PBH by @kanz76 in https://github.com/NCAR/miles-credit/pull/118
• credit.postblock major updates on GlobalMassFixer and GlobalEnergyFixer by @yingkaisha in https://github.com/NCAR/miles-credit/pull/110
• update unet init and call in models/init.py by @dkimpara in https://github.com/NCAR/miles-credit/pull/119
• updating the periodic padding function to roll then reflect lats by @WillyChap in https://github.com/NCAR/miles-credit/pull/117
• edits to pbs.py to improve usability by @dkimpara in https://github.com/NCAR/miles-credit/pull/114
• URGENT bugfix: removed cwd keyword from subprocess.popen by @dkimpara in https://github.com/NCAR/miles-credit/pull/124
• Add Python unit testing for credit.models.Fuxi and credit.boundary_padding by @yingkaisha in https://github.com/NCAR/miles-credit/pull/121
• Pressure interpolation and a bunch of formatting changes by @djgagne in https://github.com/NCAR/miles-credit/pull/125
• removed cwd from popen again. by @dkimpara in https://github.com/NCAR/miles-credit/pull/127
• Bugfix on the production of diagnostic variables in rollout_to_netcdf.py + credit.postblock update + other minor bugfix by @yingkaisha in https://github.com/NCAR/miles-credit/pull/120
• Add torch.clamp to trainers + add drop_path in FuXi + bugfix FuXi when use_spectral_norm: False + minor updates on credit.parser by @yingkaisha in https://github.com/NCAR/miles-credit/pull/130
• Fix mpi in rollout_to_netcdf.py by @kanz76 in https://github.com/NCAR/miles-credit/pull/132
• Bugifx on random seed inconsistencies and keyword grad_max_norm by @yingkaisha in https://github.com/NCAR/miles-credit/pull/133
• Version changes by @djgagne in https://github.com/NCAR/miles-credit/pull/136

New Contributors

• @WillyChap made their first contribution in https://github.com/NCAR/miles-credit/pull/1
• @djgagne made their first contribution in https://github.com/NCAR/miles-credit/pull/2
• @jsschreck made their first contribution in https://github.com/NCAR/miles-credit/pull/3
• @dkimpara made their first contribution in https://github.com/NCAR/miles-credit/pull/7
• @sethmcg made their first contribution in https://github.com/NCAR/miles-credit/pull/4
• @kanz76 made their first contribution in https://github.com/NCAR/miles-credit/pull/51
• @negin513 made their first contribution in https://github.com/NCAR/miles-credit/pull/90

Full Changelog: https://github.com/NCAR/miles-credit/commits/2024.1.0

Atmosphere - Meteorological Observation and Forecast - Jupyter Notebook
Published by djgagne 7 months ago