TopoPyScale - clustering fix

An error in remapping the pixels after clustering to their actual location was providing critical unwanted behavior. The latest commits fix this issue by specifying in the np.reshape() the type of order to reshape the vector using the Fortran type.

Line 452-457 in topoclass.py

With previous version, when using the clustering option, pixels would not be remapped properly. FIXED :)

Climate Change - Climate Downscaling - Python
Published by ArcticSnow 2 months ago

TopoPyScale - bug fix

update readme and documentation
fixed bug with mp.get_era5
started a class for fetching era5, that wraps all downloading functionnalities for era5 data

Climate Change - Climate Downscaling - Python
Published by ArcticSnow 3 months ago

TopoPyScale - V0.3.0 New parallelization scheme using Zarr store

This version is a big overall with 1) the parallelization procedure which now can use Zarr store to improve IO efficiency and 2) the migration and compilation of topocalc to Python 3.13. It also includes a number of small improvement and bug correction.

Large improvements

file topo_scall_zarr.py holds the new method to downscale from a Zarr store to netcdf (or zarr, in alpha test)
fetch_era5.py includes function to convert stack of netcdf to zarr
TopoPyScale is not running under Python 3.13

Other changes

usage of Path()
removing topo_da.py. Must remove gdal and osgeo dependency for a come back
migrate setup.py to pyproject.toml
topocalc is now forked and improved to a version that compiles under Python 3.13
added wrning for DEM projection
fixed bug with group clustering
added weights of cluster density in groups
fixed missing relative humidity in PLEV fetching from cds

Climate Change - Climate Downscaling - Python
Published by ArcticSnow 4 months ago

TopoPyScale - ERA5 from CDS working again

Big bug fix of the CDS API downloading capabilities with commit: https://github.com/ArcticSnow/TopoPyScale/commit/57d6a1c59854eb7491b972e2708540c6d7ee12af

now request are daily.
it automaticaaly deal with the weird ZIP files that ERA5 may deliver (in particular for surface data)
documentation is now part of the main repo under the folder doc/
function called 'process_SURF_file()was renamedunzip_file()`. it is run right after the download request. it is also more generic
dropped the need for the library cfgrib. Use the library cdo instead that has many useful and efficiently coded functionalities

I hope this is working robustly. There may still be some bugs as the logic has changed slightly.
Enjoy

Climate Change - Climate Downscaling - Python
Published by ArcticSnow 7 months ago

TopoPyScale - major bug on daily cycle of SW and minor other things

This release is triggered by the find by @mmazzolini on the SW orientation. Eastern and western slopes were unversed. I cannot believe this went unnoticed to all of us developers and users of TPS. So thank you again Marco for this find and your PR.

Then this new release code adds corrections to few minor typos, new bits of codes to fetch data from the Google Cloud Storage, improvements for mask and groups. For further detail you may refer to all the commits from Jan 29th 2025 (release of v0.2.6).

Climate Change - Climate Downscaling - Python
Published by ArcticSnow 8 months ago

TopoPyScale - New CDS API and more

This is a lone overdue release as new features and typos/bus have been corrected. This new release now supports the new CDS API for ERA5 data download.

WARNING: There may be some compatibility issues with workflows based on previous release.

New tool to fetch Copernicus DEM [19abe84] with paralleisation [1d834bd]
improvement of path directory code [72dcd3e]
debug of masking [3713f83]
function to export to FSM2oshd [1ed7b6d]
correction of a bug when elevation was lower than the lower pressure level [e133c6e]
adding compatibility with IFS data [134f7ce] and script to feth IFS data [69c9845]
implementation of an export function to FSM2 model [749a0e3]
improvements in the routine for forecast [cd75c5a]
added rioxarray to requirements [de8ae0c]
added export function for SURFEX model [f5ad251]
added explicit get_era5() function in workflow [7b35bcd]
fixed typos and bug in topo_sim.py [327fe4f] et al.
added dask chuncking to stop memory leaks [fd2adc4]
output format of the ERA5 data is now dynamic and not hard coded to netcdf [3cb0b49]. now support for
update of new variable keywords of CDS-beta [ef13238]
replaced raise warning ... by simple prints as it led to crash [f8da401]
new feature to download ERA5 from zarr repositories [3eb160d]
added possibilities to downscale at other timestep than 1h only [d77d988]

Climate Change - Climate Downscaling - Python
Published by ArcticSnow 11 months ago

TopoPyScale - addressing multiple typos

This release includes a working version for exporting to FSM2ohd, and a number of other correction in the code. These are minor typos that led to inconsistencies.

Climate Change - Climate Downscaling - Python
Published by ArcticSnow almost 2 years ago

TopoPyScale - FSM2oshd support & Arbitray domain mask

New release bringing latest development of new features into TopoPyScale:

Support to export to FSM2oshd format 3fe1e1a7bded57e282a8b35c0960881e5c6417a8
added support for masking out domain or split domain in independent regions 81298bfa8714172cf61cfbb33bdc4c456523009f
added support for custom outputs directory 951451e3e59c3cd633460ea923dd980ec803e338
rename files containign FSM simulation routines to sim_fsm.py
added support for C-implementation of solar position in pvlib library 074710a2e586d3bceadb78def108edca4e000dce
point_id can be anything from int, float to string 1a32a5d9a3003103928a243b89997613a53258b1

and a number of minor bug fixes

Climate Change - Climate Downscaling - Python
Published by ArcticSnow almost 2 years ago

TopoPyScale - v0.2.3 JOSS release.

This new version of the package is justified by the release of the JOSS publication following the review process. In the meantime a number of features have been added.

support for SLURM server type
realtime processing, downloading latest ERA5 data. For operational setup c0fb49fff0f106bb39601f77b6b5a120901c18f0
git commit hash and package version added to netcdf file metadata. This ensure to know which version of TopoPyScale was used to create data.
climate input can now be located outside of the project directory. This prevent to copy multiple times the same data. 77fd990500bfe6934ad6997d70acb106fd44b6d1
Outputs directory can be named independently of default
export FSM as netcdf
streamlined the environment.yml file 5af8c8dde5c0d8ca08c9c03de7dafa85a4c543e2
fix Dask indexing: 4b00cd60bffe6fff5fa16e8f16c95b822c5238a4
option to give specifc name to cluster points and not depend on the default point_id 7695450864e3b9f978c6e22c7c5fda800941d011
function to export griddedata 420f80a55110b31abd4edede7c69fb30e608375d
update xarray functions that were deprecated 3662908f635a6c7565a6c9217934b98a09b78b1d
possibility to run FSM in parallel mode 91c8528db69499f5041f9849f739b3253b7f1d25
added snow routine for steep slope 4d31fd1a2f9c628bd2a208e3862e449ab5af847a

Climate Change - Climate Downscaling - Python
Published by ArcticSnow over 2 years ago

TopoPyScale - Update to Python 3.9 and latest `xarray`

This release includes:

upgrade to Python 3.9 and Xarray 2.3
no more problem of memory byt using open_mfdataset
new testing script with github action
added units to default downscaled dataset
n_core option is less restrictive now.
solved a number of path issues
a number of improvements of the FSM simulation topo_sim.py routines
corrected projection issues when interpolating the climate dataset
to_cryogrid() now compatible with Cryogrid.jl
added an environment.yml file for version controlled install of VE
plotting function map_variable() has been generalized
and a number of bugs have been caught along the way.

Climate Change - Climate Downscaling - Python
Published by ArcticSnow over 2 years ago

TopoPyScale - Another release for less bugs

New release resolving a number of bugs:

a crucial bug occurring when storing datasets during downscalng in topo_scale.py. More details in issues: https://github.com/ArcticSnow/TopoPyScale/issues/60 and commit https://github.com/ArcticSnow/TopoPyScale/commit/a33e79f77bd32fd6952c4a269024a31a9a584dae
a bug resolved in to_crocus() https://github.com/ArcticSnow/TopoPyScale/commit/4e73489e85bc567d897752f6d8cc7db0eb6c3011
a bug when computing level elevation from geopotential https://github.com/ArcticSnow/TopoPyScale/commit/6a4c68e9a0e2f9f1c66427924e75d8c126fbd963
a bug in to_musa() https://github.com/ArcticSnow/TopoPyScale/commit/81e8978cb152a7b908e53a20eea90853b63aaa18

Climate Change - Climate Downscaling - Python
Published by ArcticSnow almost 3 years ago

TopoPyScale - Parallelization is up and running :)

TopoPyScale is improving and being tested by more users. New features, improvements and bug fixes justify a new release. The whole downscaling process gain speed by spreading the load on multicores. This was tested on laptops and servers.

WARNING: This new release requires Python >= 3.8. Otherwise, python runs into a memory leak and crashes.

New features:

multicore, paralleization of topo_scale
tools to explore optimal number of clusters in topo_sub
implementation of feature importance for clustering. New field in config file needed
added precipitation lapse-rate option, commit

Improvements:

simplification of path management using the library pathlib
remove all usage of open_mfdataset() or to_mfdataset that are not as efficient as expected
simplification of tmp files during downscaling. 85a4149137da173b392d5ceb336f6c18e4877ed9
added error message in case upper pressure level is not high enough

Bug fixes:

to_netcdf() default case bug fixed
plotting routines for variable mapping fixed
fixed bug in time-batch handling of files
bug in fetch_WMO() fixed

Climate Change - Climate Downscaling - Python
Published by ArcticSnow almost 3 years ago

TopoPyScale - pypi install

This release includes a new options in setup.py that the package can be installed in a single line using Pypi

pip install topopyscale