ecPoint-Calibrate - v1.0.1

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by dtip almost 2 years ago

ecPoint-Calibrate - v0.29.0

Changelog

• Refactor decision tree modes to use a dropdown.
• New and improved Menu layout. Notice how we now have categories inside the menu (import, export, navigation, etc).
• Rename some menu options with more concise names.
• Saving operation files is now simpler, with clear field descriptions. See #182 for more info.
• Update write directory for operational files. (#182)
• Fix headers in error rep CSV. (#182)
• Fix split level selection. (#186)
  • This should also fix some related errors in split-level being wrongly selected.
• Improvements to K-S test visualization:
  • Fix graph title to reflect the correct breakpoint.
  • Display the sample size for the weather type considered.

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by onyb almost 4 years ago

ecPoint-Calibrate - v0.28.1

Changelog

This is patch release fixing the regression in node splits introduced in v0.28.0. See #181 for more info.

As a reminder, when introducing splits, the new node is always added to the left of the node on which the user had clicked.

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by onyb almost 4 years ago

ecPoint-Calibrate - v0.28.0

Changelog

Allow removing predictors in postprocessing workflow (#178)

The last predictor in the sortable levels can now be removed by clicking on the cross button. The levels can be sorted by clicking on the hamburger icon and dragging it.

⚠️ Note that this is a destructive change, and cannot be undone. One can of course restart the postprocessing from scratch.

Additionally, the software is now able to save the PDT file with only the used predictors, in the "Save Operation" option in the Menu.

Add button to open computation logs file (#176)

A new button has been added to open the log file using the system's default text editor.

Display near real-time computations runtime status (#162)

The "Launch Computation" button (see the previous image) is now guaranteed to be disabled while computation is in progress, even if the user moves to a different tab.

Allow merging leaf nodes to the left (#174)

Introduced a new Decision Tree mode called "Merge leaf". When activated, clicking on a leaf node will merge it to the node on its left. In case a merge is not possible, an error is displayed.

Fixed bug when plotting the mapping function for a circular variable (#165)

Prevent single-child nodes in Decision Tree for circular predictors (#159)

⚠️ Note that the linked issue is not much relevant now, although it fixes a bug in the way circular predictor nodes were rendered in case of no breakpoints.

Fix bug preventing the merging of nodes immediately below the root (#155)

Users can now merge the Level 1 nodes in the decision tree from the breakpoints table, or directly from the tree itself (using the new "Merge leaf" mode).

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by onyb almost 4 years ago

ecPoint-Calibrate - v0.27.0

Changelog

Enable conditional verification maps in non-collapsible mode (#177)

Display units in the post-processing workflow (#163)

Allow dynamic update of Decision Tree levels (#160)

Let's consider the following view in the post-processing workflow.

When the order of predictors in A is changed, it is also reflected in B and C. However, the update made to C doesn't impact any predictor that didn't have its column index changed. The Decision Tree is updated in real-time and always reflects the state of C. To sync tables B and C (essentially, a reset operation), you can click on the "Generate WTs" button.

⚠️ Watch out for discrepancies in the contents of the breakpoints tables.

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by onyb about 4 years ago

ecPoint-Calibrate - v0.26.0

This release significantly improves the performance of rendering of the decision tree, which was observed to be really poor for a large number of tree nodes (see #168).

Summary of changes:

• The styling of the tree/nodes has been recreated using pure-SVG elements for faster performance, but there may be slight visual differences.
• Animations have been turned off, so changes to the tree will feel jaggy (no anti-aliasing).
• Click events on the nodes are now handled differently, but there's no observable difference for the user.
• Upgraded react-d3-tree to a newer lightweight version, that doesn't use d3.js as a peer dependency.

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by onyb about 4 years ago

ecPoint-Calibrate - v0.25.1

Small patch release fixing some bugs on v0.25.0.

Changelog

• Fix bug while merging children of Root node.
  • Possible fix for #173.
• Chunk predictor titles in groups of 6. (#170)

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by onyb about 4 years ago

ecPoint-Calibrate - v0.25.0

Changelog

This is a standalone release to allow merging non-leaf decision tree nodes with a single click. This was achieved by introducing a new mode called "Merge children" (see below).

It will work for arbitrarily nested trees and is a destructive operation (no way to undo). Once merge is finished, you must change the mode to "Simple" or "Non collapsible" in order to view the MFs.

⚠️ While this is a new mode, it impacts the creation of weather types in the other modes as well (i.e., "simple", and "Non collapsible"). Normally, the feature should work seamlessly, but please watch out for the following class of bugs just to be sure:

• Issues with the MFs, titles, codes of the weather types after the merge operation.
• Issues with the threshold breakpoints table after the merge.
• Merge operation affecting unrelated branches of the decision tree.

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by onyb about 4 years ago

ecPoint-Calibrate - v0.24.0

Changelog

K-S test improvements (#145)

• Display a scale above K-S test graph indicating min and max ranges.
  
• Display range of definitive breakpoint selected.
• Make the < and > buttons selectable.
  
• Make definitive breakpoints editable and add a delete button to each.
  
  Note: Double-click on the cell to edit.

Fix bug with weather-type code generation. (#161)

⚠️ This includes a large rework of the algorithm that constructs the decision tree and generates weather type codes. It impacts several parts of the post-processing workflow, so please lookout for bugs or broken features.

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by onyb about 4 years ago

ecPoint-Calibrate - v0.23.1

Changelog

This is a patch release fixing the following bugs:

• Make predictor selector dropdowns scrollable. (#157)
• Fix bug while breakpoints and WTs are saved. (#156)

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by onyb about 4 years ago

ecPoint-Calibrate - v0.23.0

Changelog

Fix bias correction factor for FE error type

Closes #150.

Improvements to K-S test

Part of #145 and #153.

• Implement switcher for computing breakpoints. (18bffc3)
• Improve layout of primary stats in K-S Test modal. (12eb821)
• Improve layout of definitive breakpoints section. (1dc61e0)
• Display minor ticks in the K-S test graph. (f7dbf18)
• Fix labels in K-S test graph. (9087abe)
• Use wide fullscreen modal for K-S test mode. (f423bb0)
• Eliminate first and last breakpoint indexes in K-S test. (b40b23b)
• Support ranged queries on definitive breakpoints in K-S test. (b0dc6b4)

Misc

• Refactor toast error handler into a common lib.
• Security updates to NodeJS dependencies.

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by onyb about 4 years ago

ecPoint-Calibrate - v0.22.0

Changelog

New and improved Kolmogorov–Smirnov test for breakpoint suggestions

Please refer to #145 for more information on how the new K-S test works.

Improve core Dockerfile for multi-stage builds

This has no impact on the user but prevents issues like #151 from reoccurring in future. See commit 459b7c1 for more information.

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by onyb about 4 years ago

ecPoint-Calibrate - v0.21.0

Changelog

• Read geopoint observations metadata and autofill in the GUI. (#90)
• Improve toast errors to provide more context. (#148)

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by onyb about 4 years ago

ecPoint-Calibrate - v0.20.0

Changelog

• Fix removal of computation from an arbitrary index. (#149)

• Start computation step for maximum / minimum fields, from the sampling interval.

• Allow loading/saving workflows from the menu.
  

• Introduce functional end-to-end tests for testing the processing with a real dataset.

• Introduce Git LFS for storing large GRIB datasets.

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by onyb about 4 years ago

ecPoint-Calibrate - v0.19.0

Changelog

Periodic predictors

You can now specify the period of each predictor in the post-processing page. You only need to enter the range (low/high) of the predictor values, and the software will figure out the period from that. This was done to allow for greater flexibility and better validation of predictor thresholds.

Validation of predictor thresholds (#139)

The software now performs an exhaustive set of checks on the predictor threshold sequences. It works for both circular and linear predictors. In case of an error (see example below), it points out the specific predictor that has an invalid sequence.

The validation algorithm is based on a stack-based state machine to prevent the GUI from stalling in case of invalid inputs that cannot be understood. The implementation also comes with extensive unit-tests that are automatically run on every commit on GitHub. Here's the list of test cases if anyone's curious.

Contextual errors

Errors originating on the backend now display more context in the toast cards (see example below).

Viewing weather types of non-leaf nodes (#134)

We now have 3 different modes for interacting with the decision tree:

• Simple ⇨ the default mode - clicking on the leaf nodes pops up the weather type; clicking on non-leaf nodes collapses the children.
• Edit ⇨ if enabled, clicking on the leaf nodes opens a modal to run the KS test and introduce splits.
• Non-collapsible ⇨ if enabled, none of the nodes can be collapsed; you can view the weather types on all nodes.

It works by figuring out the rows in the breakpoints matrix based on the selected non-leaf node and simulates a "merge" from the relevant leaf nodes backtracking to the selected node. It's as if you manually merged the children, without affecting the state of the breakpoints matrix.

⚠️ The algorithm required a rework of the code that renders the decision tree by making it recursive. While I have tested it extensively, there may be bugs, especially in the non-leaf nodes. Please watch out!

Framework to compare Weather Types (#105)

The original idea was to split the GUI window into different workspaces such that they don't affect the states of one another. One can use this feature to visually compare two histograms, for example.

The above was achieved by introducing a --fork command-line argument that can be used to spawn as many copies of the GUI as the user wants while sharing the same backend (Docker containers) in order to be lightweight.

⚠️ The first GUI window should be launched as normal. Subsequent GUI windows must be launched using the--fork argument.

Here's how it looks on my laptop. Note that the alignment is not great since my laptop screen is quite small, but it shouldn't be a problem on a desktop monitor.

Miscellaneous

• Display error message for name collisions with predictor variables. (#121)
• Implement more unit-tests for both the backend and electron code.
• Setup electron automated tests on the CI using GitHub Actions.
• Refactored some API routes in the backend to be more modular.
• Some refactorings were done in the excel-like sheet to input breakpoint values.

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by onyb over 4 years ago

ecPoint-Calibrate - v0.18.0

Changelog

⚠️ This release includes a large non-functional update to the backend, so launching the GUI for the first time will require a slightly longer download. If you have any issues after the GUI has launched, you can try closing the window and launching it again.

Continuous deployment

This is the first release that uses a fully automated continuous-deployment pipeline, including generation of the final GUI that's available for download. This completes #101.

Starting this release, DMG images of ecPoint-Calibrate are available for MacOS. 🍎 This vastly simplifies installation and launching of the software for MacOS users. Simply open the .dmg file and drag-and-drop the icon to the Applications folder. Screenshot below:

Running automated tests

On every push to GitHub, we now run automated unit-tests on the backend code. The coverage of the unit-tests will be improved further in future. Live status of the latest build can be seen in the README (see screenshot below).

Detailed summary of the entire CI/CD pipeline can be viewed here.

Miscellaneous

• Allow negative values in manual splitting. (#140)
• Allow negative numbers like -.5 throughout the software (including binning values - see #136). The format -0.5 works too.
• Fixed broken unit tests in the core backend.

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by onyb over 4 years ago

ecPoint-Calibrate - v0.17.0

Changelog

• Fix overflow bug in the datasheet to edit threshold split. (#129)
• Fix decision tree evaluation for periodic predictors, for example, Local Solar Time (LST). (#94)
• Add WT bias in mapping functions. (#104)

• Add new menu item to save summary of WT biases. (#106)

• Disable immutable default computation for instantaneous predictor. (#111)

  • includes all the fixes discussed in the GitHub issue.

• Display point data table statistics for all predictors. (#133)

  • ⚠️ The stats for FER/FE values in the "Binning" section has subsequently been removed (see image below).

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by onyb over 4 years ago

ecPoint-Calibrate - v0.16.0

Changelog

• Disable navigation while computation is in progress. Closes #20.
• Allow overriding predictor metadata, for example, to customize units. Closes #128.
  • Screenshots explaining the usage are available here.

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by onyb over 4 years ago

ecPoint-Calibrate - v0.15.0

(If you got an email about a release with the title v0, that was a mistake. Please ignore.)

Summary of changes

This release allows users to save/upload files directly from the menu. This closes #115 and #122.

New items in the menu

The items are greyed out if the user is not on the second page of the postprocessing workflow.

Saving all operation files

The Menu contains a "Save Operation" action that opens the following modal. For more information on how operation files are stored, please refer to #115.

Saving individual files

This was already possible with the software, but this release makes them available as menu items. Here's an example of how a user would save the breakpoints as a CSV file.

Upload Breakpoints CSV

See #122

Note on backwards-compatibility / stability

The features in this release involved refactoring a large part of the code. In order to reduce the risk of introducing regressions, the existing buttons to save/upload files have not been removed. They will be deprecated in a future release once the users have tested the new system.

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by onyb over 4 years ago

ecPoint-Calibrate - v0.14.0

Summary of changes

Introduce begin/middle/end variants for Instantaneous Field

In case of accumulated predictand, we make available three variants of the Instantaneous Field with the following names:

• Instantaneous Field - Begin
• Instantaneous Field - Middle
• Instantaneous Field - End

This resolves #95, which is the root cause for confusion regarding which step data is chosen for instantaneous fields in case of accumulated predictands.

Display metadata of GRIB files in Computations step

We now fetch the name of the fieldset, in addition to the units, and display this information in the Computations step as in the example below. This resolves #97.

Prevent error when clicking on non-leaf decision-tree nodes

Clicking on non-leaf nodes of a decision tree now always collapses its children. This resolves the bug reported in #127.

Improvements to conditional verification maps

In the image below, you can see the difference between the old version (left) and the new one (right). The image on the right is rendering the fonts correctly, and the quality is significantly higher. With the new map viewer, users can also zoom-in, zoom-out, save and print the map. This fixes #125.

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by onyb over 4 years ago

ecPoint-Calibrate - v0.13.0

Changelog

PDT Metadata Viewer

Allow users to view metadata of Point Data Tables, directly on the GUI.

Once the PDT is uploaded, you'll see a button called "View metadata" as shown below.

Clicking on the button will display the metadata inside a modal. See example below:

This feature works for both ASCII as well as Parquet formats. Note that it only displays the metadata and not the full contents of the PDT file. In case of ASCII, it can be viewed directly in a text editor or with the following command for large ASCII files:

$ head -n 100 <path to ascii file>  # replace 100 with the number of lines to read

⚠️ In the case of Parquet files, viewing the full contents is not possible since Parquet uses a columnar data storage format.

Miscellaneous

• Fix some more wording issues, as outlined in #93. The previous fix was not enough.
• Fix bugs in Local Solar Time, as detailed in #94.
• Various code improvements and cleanups in the PDT loader interface.

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by onyb over 4 years ago

ecPoint-Calibrate - v0.12.1

Patch release fixing some issues, notably #126.

Changelog

• Wording change for computation add/multiply factors. (part of #121)
• Save errors as {FER/FE}.csv depending upon the error type. (part of #121)
• Fix bug in level selector for Kolmogorov-Smirnov test. (#126)

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by onyb over 4 years ago

ecPoint-Calibrate - v0.12.0

Introducing Cheaper mode

This release introduces a new mode called Cheaper, for efficiently accessing large point-data tables. There's no functional difference if this mode is enabled. It works for both ASCII files as well as the recently introduced Parquet format. It can be toggled from the GUI in the post-processing workflow (disabled by default).

When Cheaper is activated, the point-data loaders use a modified decision-tree algorithm that lazy-loads only the required columns, as opposed to holding the entire DataFrame in memory. This is particularly useful for very large point-data files that may not fit in the user's RAM. Note that compared to ASCII files, the performance of Parquet should be much better since it is designed for columnar storage.

⚠️ The modified algorithm has been carefully reviewed for correctness, although there may be small bugs due to the magnitude of the changes.

Performance

Evaluating a decision-tree is slower when the cheaper mode is activated. This is due to frequent disk I/O which is essential for keeping the memory usage minimal.

Memory usgae

In my tests with an ASCII point-data table of size 584 MB on disk, I got a significant reduction in the consumption of memory by the backend process.

Cheaper disabled	Cheaper enabled
1.8 GB	380 MB

Summary of changes

• Implement cheaper algorithm to evaluate decision tree. (9d9a9c6, fixes #112 and #116)
• Interpret CSV as ASCII file. (81b1260; fixes #120)
• Fix sanitization of Docker paths. (maybe fixes #119)
• Add /tmp, /var/tmp, and /scratch to default volume bindings.
  • Users can now read/write in these directories.
• Fix a bug with displaying of computation logs in the GUI. (d3554b2)

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by onyb over 4 years ago

ecPoint-Calibrate - v0.11.2

Changelog

• Enable Cross-Origin Resource Sharing (CORS) on the backend server. (6f313f5)
• Display error on the console when Docker daemon is not running. (5d0b883)
• Replace inf with 9999 while saving breakpoints CSV. (f9eed3d)
• Display total number of observations in histogram, during postprocessing. (e43dfc0)
• Filter predictor inputs for 24h Solar Radiation computation. (73fac53)
• Connect to localhost instead of 0.0.0.0. (63d71a6)

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by onyb over 4 years ago

ecPoint-Calibrate - v0.11.1

Summary

This is a patch release replacing HTTP client in the GUI with Axios. This introduces better handling of errors, which were previously not caught. It'll be useful in pinpointing the issues with accessing volumes (see #118).

This release also introduces toast messages in the GUI. For example, if there's an error while invoking the backend server, you'll be able to see toast-style notifications like this:

The notifications are dismissed automatically after 10 seconds or can be dismissed manually with the x button. If there are multiple notifications, they'll be stacked together. Different colours are available for indicating the severity of the message.

Changelog

• Replace request with axios and add error handling. (b577b67)
• Add toasts framework for displaying floating global messages. (b85888c)

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by onyb over 4 years ago

ecPoint-Calibrate - v0.11.0

Changelog

This release primarily addresses #112 and #116 by providing the user with an option to choose a format for PDT that's highly optimized and more likely to fit in the memory than ASCII files. This has been achieved using a combination of Parquet data format, lazy evaluation, and Categorical Types.

The release also includes fixes for some bug in Milestone 1.

User-facing updates

• Date fields in PDT are formatted as YYYY-MM-DD (previously YYYYMMDD), to prevent automatic type-cast to int64.
  (5c1580e)
• Implement a high-performance loader based on Apache Parquet. (fcc7ee8)
• Allow selecting between ASCII and Parquet containers for storing PDT, on the GUI. (91193c1)
• Support reading from Parquet files in postprocessing. The container type will be automatically inferred. (c8c519c)
• Improve memory utilization of PDT preloader during postprocessing, using lazy evaluation. (1d569b9)
• Fix metadata in PDT for Forecast Error (#93, 06c8ae7)
• Add loader in postprocessing workflow for expensive actions. (edfa101)
• Make FE(R) CSV dumps more legible. (#96, d9d8af4)
• Allow negative addition factor in computations. (#83, 0cad6b8)

Technical improvements

• Significantly reduce Docker build context for the core service.
• Implement an interface for PDT loaders. (3cfccf9)
• Allow selecting specific columns from a PDT, without loading the whole file in memory. (3cfccf9)
• Implement iterator protocol for point data table loaders. This will allow us to perform lazy operations on PDT in future. (a0179fa)
• Cache dataframe and column properties in PDT loaders.
• Other typing improvements to loader classes.

Note: PDT = Point Data Table

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by onyb over 4 years ago

ecPoint-Calibrate - v0.10.0

Changelog

• Automatically stop stale containers on startup.
• Implement generic host bindings for Docker Volumes.
  • /vol is now supported by the software.
• Pin Docker image tag to always use the package.json version.

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by onyb over 4 years ago

ecPoint-Calibrate - v0.9.0

This release marks the deprecation of Dockerized GUI for ecPoint-Calibrate.

Context

Today, the GUI doesn't actually run on the user's machine, but rather inside a Docker container which streams the display back to the host OS. It's just like looking at the world through your phone's camera, which is completely unnecessary and comes with a lot of overhead.

This was done at the time mainly for lack of a better way to avoid installing hundreds of dependencies on the user's machine. Fortunately, this is no longer a hard problem in 2020.

This change was also necessary for continuing to support the software on some of the newer Linux based operating systems, which were not compatible with the legacy approach.

What it means for you

• It is no longer necessary to clone the git repository for running the software. Just download the latest artefact from the Releases page, and execute it. Usage instructions have been updated in the README.md file.
• No requirement on Docker Compose.
• Any file in the user's home directory or external volumes (/media on Linux, and /Volumes on MacOS), are automatically available for use in the software without any configuration.
• Lower memory footprint, and therefore faster user experience on the GUI.

Backwards compatibility

The old way of running the GUI with Docker Compose and ./go.sh still works and has been kept alive to allow existing users to migrate to the new version at their convenience. However, please note that this version of the software will not receive any updates in future, and will eventually be removed.

Please remember to do a git pull if you wish to continue using the legacy version of the software.

Atmosphere - Meteorological Observation and Forecast - JavaScript
Published by onyb over 4 years ago