OasisLMF - 1.19.0

OasisLMF Changelog - 1.19.0

#856 - improve memory usage of fmpy
#883 - Fix ri file detection
#877 - Feature/873 automated issues tests
#832 - Consolidate Arch2020 work

OasisLMF Notes

Improved memory usage for fmpy - (PR #880)

In Fmpy, losses were store in arrays where the index in the array correspond to the sidx id.
with the help of numpy and numba this storage is very efficient for lots of operation needed in the financial module in particular if lots of sidx are present (if the density is high).
However, it present a weakness towards memory consumption when number of sample increase (O(N) complexity)

As in general model requiring high number of sample have loss array of small density one way to decrease memory usage and also increase performances is to use sparse array to store the loss.

this PR introduce a new mode for fmpy, where sparse array are used for the computation. It is activated when the number of sidx is bigger that 16 meaning when the max sidx value is 14 ( + the -3 and -1 sidx makes 16 values)

benchmark result performed on a complex 1000 location portfolio with a model generating loss of 3% density

performance are calculated as a number of time faster than the gul

In those condition that are representative of most high sample model,

the memory consumption go from around 40MO per 10K samples to 3MO per 10K sample, dividing it by an order of magnitude.
This is roughly inline with the 3% sparsity (3% of 40 is 1.2) of the loss.
for high Sample count, the performances of fmpy are improved by a factor of around 3.

Resolved issue test cases added to regression tests - (PR #877)

Regression tests now include a set of OED test cases linked to specific git issues which have been found and fixed.

Refactor genbash and LossGeneration to chunk events separately - (PR #879)

CLI Example:

NUM_CHUNKS=10
oasislmf model generate-oasis-files --oasis-files-dir  ~/ram/run/input

# run each chunk of events 
for (( i=1; i<=$NUM_CHUNKS; i++ )); do
oasislmf model generate-losses-chunk --max-process-id $NUM_CHUNKS --process-number $i --model-run-dir ~/ram/run --oasis-files-dir  ~/ram/run/input
    echo $i
done

# run output reports 
oasislmf model generate-losses-output --model-run-dir ~/ram/run/ --oasis-files-dir ~/ram/run/input/

This creates individual scripts to run each event chunk plus a single output report script

├── 1.run_analysis.sh
├── 2.run_analysis.sh
├── 3.run_analysis.sh
├── 4.run_analysis.sh
├── 5.run_analysis.sh
├── 6.run_analysis.sh
├── 7.run_analysis.sh
├── 8.run_analysis.sh
├── 9.run_analysis.sh
├── 10.run_analysis.sh
├── run_outputs.sh
  ..

Climate Change - Natural Hazard and Storm - Python
Published by awsbuild over 3 years ago

OasisLMF - 1.15.14

#856 - improve memory usage of fmpy

OasisLMF Notes

In Fmpy, losses were store in arrays where the index in the array correspond to the sidx id.
with the help of numpy and numba this storage is very efficient for lots of operation needed in the financial module in particular if lots of sidx are present (if the density is high).
However, it present a weakness towards memory consumption when number of sample increase (O(N) complexity)

As in general model requiring high number of sample have loss array of small density one way to decrease memory usage and also increase performances is to use sparse array to store the loss.

this PR introduce a new mode for fmpy, where sparse array are used for the computation. It is activated when the number of sidx is bigger that 16 meaning when the max sidx value is 14 ( + the -3 and -1 sidx makes 16 values)

benchmark result performed on a complex 1000 location portfolio with a model generating loss of 3% density

performance are calculated as a number of time faster than the gul

In those condition that are representative of most high sample model,

the memory consumption go from around 40MO per 10K samples to 3MO per 10K sample, dividing it by an order of magnitude.
This is roughly inline with the 3% sparsity (3% of 40 is 1.2) of the loss.
for high Sample count, the performances of fmpy are improved by a factor of around 3.

Climate Change - Natural Hazard and Storm - Python
Published by awsbuild over 3 years ago

OasisLMF - 1.19.0rc1

OasisLMF Changelog - 1.19.0rc1

#856 - improve memory usage of fmpy
#883 - Fix ri file detection
#877 - Feature/873 automated issues tests
#832 - Consolidate Arch2020 work

OasisLMF Notes

In Fmpy, losses were store in arrays where the index in the array correspond to the sidx id.
with the help of numpy and numba this storage is very efficient for lots of operation needed in the financial module in particular if lots of sidx are present (if the density is high).
However, it present a weakness towards memory consumption when number of sample increase (O(N) complexity)

As in general model requiring high number of sample have loss array of small density one way to decrease memory usage and also increase performances is to use sparse array to store the loss.

this PR introduce a new mode for fmpy, where sparse array are used for the computation. It is activated when the number of sidx is bigger that 16 meaning when the max sidx value is 14 ( + the -3 and -1 sidx makes 16 values)

benchmark result performed on a complex 1000 location portfolio with a model generating loss of 3% density

performance are calculated as a number of time faster than the gul

In those condition that are representative of most high sample model,

the memory consumption go from around 40MO per 10K samples to 3MO per 10K sample, dividing it by an order of magnitude.
This is roughly inline with the 3% sparsity (3% of 40 is 1.2) of the loss.
for high Sample count, the performances of fmpy are improved by a factor of around 3.

Resolved issue test cases added to regression tests - (PR #877)

Regression tests now include a set of OED test cases linked to specific git issues which have been found and fixed.

Refactor genbash and LossGeneration to chunk events separately - (PR #879)

CLI Example:

NUM_CHUNKS=10
oasislmf model generate-oasis-files --oasis-files-dir  ~/ram/run/input

# run each chunk of events 
for (( i=1; i<=$NUM_CHUNKS; i++ )); do
oasislmf model generate-losses-chunk --max-process-id $NUM_CHUNKS --process-number $i --model-run-dir ~/ram/run --oasis-files-dir  ~/ram/run/input
    echo $i
done

# run output reports 
oasislmf model generate-losses-output --model-run-dir ~/ram/run/ --oasis-files-dir ~/ram/run/input/

This creates individual scripts to run each event chunk plus a single output report script

├── 1.run_analysis.sh
├── 2.run_analysis.sh
├── 3.run_analysis.sh
├── 4.run_analysis.sh
├── 5.run_analysis.sh
├── 6.run_analysis.sh
├── 7.run_analysis.sh
├── 8.run_analysis.sh
├── 9.run_analysis.sh
├── 10.run_analysis.sh
├── run_outputs.sh
  ..

Climate Change - Natural Hazard and Storm - Python
Published by awsbuild over 3 years ago

OasisLMF - 1.18.0

OasisLMF Changelog - 1.18.0

#865 - correction for PolDed6All fields
#861 - Add PALT to genbash
#820 - Pol Fac Contracts
#837 - added new doc
#869 - Feature/761 fm tests
#862 - oasislmf test fm doesn't work and there is no documentation
#828 - RI FM File Generation issue when locations have zero TIV
#850 - Improve exit handler's process cleanup
#874 - Check and set minimum versions for optional packages
#846 - Update package dependencies for flexible lookup code
#852 - Issue with custom lookup_module_path relative path
#826 - Automate Change logs and release notes
#860 - slight interface changes for key server functions

OasisLMF Notes

FM supported fields documentation correction - (PR #865)

PolDed6All, PolDedType6All, PolMinDed6All, PolMaxDed6All marked as supported in documentation

Enabled the PALT ORD outputs - (PR #866)

Updated Bash script to create palt.csv ORD files when selected in the analysis_settings file.

Example analysis_settings

    "gul_summaries": [
        {   
            "id": 1,
            "ord_output": {
                "alt_period": true
            }   
        }   
    ],

New reports

$ tree output/
output/
├── analysis_settings.json
├── gul_S1_palt.csv
└── gul_S1_summary-info.csv

Introduce Vectorisation in generation of Reinsurance FM Files - (PR #868)

The introduction of vectorisation, and resultant dropping of the tree structure, should lead to an improvement in performance in the generation of reinsurance FM files.

Added documentation on supported OED financial fields - (PR #837)

Some new documentation on what OED financial fields are supported and an overview of how they are applied in the Oasis kernel

Maintenance upgrade of fm validation tests - (PR #869)

Added scripts to manage test cases and align file formats in preparation for OED schema update

Fixed run acceptance test CLI command - (PR #871)

Fixed command to work run from the 'current working directory' if its a valid fm test or holds sub directories of fm test cases.
Update help text with examples

$ oasislmf test fm --help
usage: oasislmf test fm [-h] [-V] [-C CONFIG] [-c TEST_CASE_NAME] [-l] [-t TEST_CASE_DIR] [-r RUN_DIR] [-p NUM_SUBPERILS] [--test-tolerance TEST_TOLERANCE] [--fmpy [FMPY]] [--fmpy-low-memory [FMPY_LOW_MEMORY]] [--fmpy-sort-output [FMPY_SORT_OUTPUT]]

Run FM acceptance tests from "oasisLMF/validation/*"

    Example use: 
    1. Run all test cases: "oasisLMF/validation$ oasislmf test fm"
        Stating oasislmf command - RunFmTest
        RUNNING: oasislmf.manager.interface
        Running: 4 Tests from '/home/sam/repos/core/oasisLMF/validation'
        Test names: ['insurance', 'insurance_step', 'reinsurance1', 'reinsurance2']
         ...

    2. Run Directly from a acceptance test Directory: "oasisLMF/validation/reinsurance1$ oasislmf test fm"
        Stating oasislmf command - RunFmTest
        RUNNING: oasislmf.manager.interface
         ... 

    3. Select test case sub-directory: "oasisLMF/validation$ oasislmf test fm --test-case-name reinsurance1"
        Stating oasislmf command - RunFmTest
        RUNNING: oasislmf.manager.interface
         ...

Fixed issue when running reinsurance with a location file that has zero TIV rows - (PR #872)

Add loc_idx column directly after loading in the location.csv file and before any rows get dropped

Updated bash exit handler - (PR #870)

Kill errant processes in reverse order
Added inherit_errexit option to run_ktools scripts

Add optional package dependencies for an oasislmf installation - (PR #847)

Added a new dependency file optional-package.in which lists packages needed for the builtin lookup class and extra utility commands. These won't install by default so if the host system is missing the C++ library libspatialindex the oasislmf package will still install, but without the builtin lookup functionality.

To install oasislmf with all the extra dependencies, use pip install oasislmf[extra]

Release notes feature title - (PR #854)

fix the relative path reference for lookup_module_path from execution path to model config path
allow lookup class to be specify using the python environment
using 'lookup_module' will load the module and use the class named f'{self.config['model']['model_id']}KeysLookup'
using 'lookup_class' will load and return the class from the env.
"lookup_class": "jba_fly_oasis.lookup.MySpecialLookup" will do like "from jba_fly_oasis.lookup import MySpecialLookup"

Automated changelogs and release notes - (PR #826)

The build system will now automatically create changelogs and release notes

Climate Change - Natural Hazard and Storm - Python
Published by awsbuild almost 4 years ago

OasisLMF - 1.15.13

OasisLMF Changelog - 1.15.13

#828 - RI FM File Generation issue when locations have zero TIV
#850 - Improve exit handler's process cleanup
#820 - Pol Fac Contracts

OasisLMF Notes

Fixed issue when running reinsurance with a location file that has zero TIV rows - (PR #872)

Add loc_idx column directly after loading in the location.csv file and before any rows get dropped

Updated bash exit handler - (PR #870)

Kill errant processes in reverse order
Added inherit_errexit option to run_ktools scripts

Introduce Vectorisation in generation of Reinsurance FM Files - (PR #868)

The introduction of vectorisation, and resultant dropping of the tree structure, should lead to an improvement in performance in the generation of reinsurance FM files.

Climate Change - Natural Hazard and Storm - Python
Published by awsbuild almost 4 years ago

OasisLMF - 1.17.0

OasisLMF Changelog - 1.17.0

#833 - Minor issue - error parsing fm CLI help text
#831 - Add timestamps to logger
#844 - Built-in Lookup revamp
#840 - Fix flaky tests in model_preparation/test_lookup.py
#843 - Fix CI pipeline error, double tagging release causes script to fail
#842 - Produce summary index files by default to reduce memory use of ktools output components
#816 - Issue in coverage_type_id grouping of ground up loss results for multi-peril
#809 - Error handling for invalid oasislmf.json config files
#849 - Fix CVE-2021-33503
#821 - Add missing items to data settings
#824 - Inputs directory preparation issue when ORD is enabled
#826 - Automate Change logs and release notes
#822 - Ktools exit handler killing off bash logging on exit

OasisLMF Notes

Added function timestamps to 'info' log level - (PR #835)

Log execution times If a function takes longer than 0.01 sec to complete

Example Output

COMPLETED: oasislmf.execution.bin.csv_to_bin in 0.02s
RUNNING: oasislmf.execution.bin.csv_to_bin
COMPLETED: oasislmf.execution.bin.csv_to_bin in 0.01s
RUNNING: oasislmf.execution.bin.prepare_run_inputs
RUNNING: oasislmf.execution.runner.run
[OK] eve
[OK] getmodel
[OK] gulcalc
[OK] summarycalc
[OK] eltcalc
[OK] aalcalc
[OK] leccalc
Run Completed
Run Completed

COMPLETED: oasislmf.execution.runner.run in 4.25s

Release notes feature title - (PR #836)

Built-in Lookup class that implement the OasisLookupInterface
provide a data driven lookup capability that will be both flexible (sub-classing capability) and efficient.
lookup function provided include:
    - Location preparation (set default min and max value)
    - 2 geospacial option (direct computation or using rtree)
    - merge to map column keys to an id
    - split_loc_perils_covered to properly use data from LocPerilsCovered
    - simple_pivot allow to pivot columns of the locations dataframe into multiple rows
example of config provided in Piwind lookup_new_key_server.json

Set summarycalc option to output summary index files in genbash - (PR #845)

The ktools binary summarycalc can now produce index files, which leads to a reduction in memory use by the downstream output components leccalc and ordleccalc. To take advantage of this, the option to output these summary index files is set as the default in genbash.

Fixed ordering of item_id column for gulsummaryxref - (PR #817)

The issue was introduced in 1.15.2 from https://github.com/OasisLMF/OasisLMF/pull/774

Before this change, the 'gul' summary groups were created using 'gul_summary_map.csv', but its now using 'fm_summary_map.csv' to include the IL fields.

The problem happens because the summary group code expects that the column id_set_index is sorted before running the grouping, however these are different columns between, gul and fm and fm_summary_map defaults to sorting by output_id

id_set_index columns

gul == item_id
fm == output_id

fm_summary_map

   acc_idx accnumber  agg_id  coverage_id  coverage_type_id  layer_id  loc_id  loc_idx locnumber peril_id polnumber portnumber         tiv  output_id  item_id
0        0      PPT3       1            1                 1         1       1        0      PPT3      WTC      PPT3  T20100_C1  10000000.0          1        1   
1        0      PPT3       3            1                 1         1       1        0      PPT3      WSS      PPT3  T20100_C1  10000000.0          2        3   
2        0      PPT3       2            2                 3         1       1        0      PPT3      WTC      PPT3  T20100_C1  20000000.0          3        2   
3        0      PPT3       4            2                 3         1       1        0      PPT3      WSS      PPT3  T20100_C1  20000000.0          4        4

Updated model settings schema to extend data settings - (PR #821)

docs_version and test_files_version: as a way to track metadata for model specific documentation and test files versions
keys_data_version: some model have their keys_data included in model_data, in which case item could be empty
additional_assets.deploy: needed to target specific deployments (default value for all is "always")

Fixed issue creating occurrence.bin when using ord options in analysis settings. - (PR #825)

Using only ord output options skipped occurrence file creation because the binary preparation code was keyed to check for
"lec_output": true, which is not needed for ord settings. The following JSON now correctly prepares an occurrence.bin file. See issue #824 for details

"gul_summaries": [
    {
        "id": 1,
        "ord_output": {
            "ept_full_uncertainty_aep": true,
            "return_period_file": true
        }
    }
],

Automated changelogs and release notes - (PR #826)

The build system will now automatically create changelogs and release notes

Updated bash exit handler to ignore logging commands - (PR #827)

Fix for issue #822, with this update the exit handler will not kill the bash logging command on exit.

Example of issue

/data/output/run/run_ktools.sh: line 13: 73690 Killed                  tee -ia log/bash.log
/data/output/run/run_ktools.sh: line 14: 73692 Killed                  tee -ia log/bash.log 1>&2

After fix

$cat log/subprocess_list 
    PID TTY      STAT   TIME COMMAND
  19325 pts/4    Ss     0:00 bash
  35622 pts/4    Sl     0:00  \_ /usr/bin/python -O /usr/bin/ranger
  35688 pts/4    S      0:00      \_ /bin/bash
  40382 pts/4    S+     0:00          \_ /bin/bash ./run_ktools.sh
  40387 pts/4    S+     0:00              \_ /bin/bash ./run_ktools.sh
  40389 pts/4    S+     0:00              |   \_ tee -ia log/bash.log
  40388 pts/4    S+     0:00              \_ /bin/bash ./run_ktools.sh
  40390 pts/4    S+     0:00              |   \_ tee -ia log/bash.log
  41134 pts/4    R+     0:00              \_ ps f -g 19325
  35792 pts/4    S      0:00 /bin/bash ./run_ktools.sh
  35791 pts/4    S      0:00 /bin/bash ./run_ktools.sh
  35781 pts/4    S      0:00 /bin/bash ./run_ktools.sh

$cat log/subprocess_list | egrep -v *\\.log$ |  egrep -v *\\.sh$
    PID TTY      STAT   TIME COMMAND
  19325 pts/4    Ss     0:00 bash
  35622 pts/4    Sl     0:00  \_ /usr/bin/python -O /usr/bin/ranger
  35688 pts/4    S      0:00      \_ /bin/bash
  41134 pts/4    R+     0:00              \_ ps f -g 19325

Climate Change - Natural Hazard and Storm - Python
Published by awsbuild almost 4 years ago

OasisLMF - 1.15.10

OasisLMF Changelog - 1.15.10

#822 - Ktools exit handler killing off bash logging on exit
#843 - Fix CI pipeline error, double tagging release causes script to fail

OasisLMF Notes

Updated bash exit handler to ignore logging commands - (PR #827)

Fix for issue #822, with this update the exit handler will not kill the bash logging command on exit.

Example of issue

/data/output/run/run_ktools.sh: line 13: 73690 Killed                  tee -ia log/bash.log
/data/output/run/run_ktools.sh: line 14: 73692 Killed                  tee -ia log/bash.log 1>&2

After fix

$cat log/subprocess_list 
    PID TTY      STAT   TIME COMMAND
  19325 pts/4    Ss     0:00 bash
  35622 pts/4    Sl     0:00  \_ /usr/bin/python -O /usr/bin/ranger
  35688 pts/4    S      0:00      \_ /bin/bash
  40382 pts/4    S+     0:00          \_ /bin/bash ./run_ktools.sh
  40387 pts/4    S+     0:00              \_ /bin/bash ./run_ktools.sh
  40389 pts/4    S+     0:00              |   \_ tee -ia log/bash.log
  40388 pts/4    S+     0:00              \_ /bin/bash ./run_ktools.sh
  40390 pts/4    S+     0:00              |   \_ tee -ia log/bash.log
  41134 pts/4    R+     0:00              \_ ps f -g 19325
  35792 pts/4    S      0:00 /bin/bash ./run_ktools.sh
  35791 pts/4    S      0:00 /bin/bash ./run_ktools.sh
  35781 pts/4    S      0:00 /bin/bash ./run_ktools.sh

$cat log/subprocess_list | egrep -v *\\.log$ |  egrep -v *\\.sh$
    PID TTY      STAT   TIME COMMAND
  19325 pts/4    Ss     0:00 bash
  35622 pts/4    Sl     0:00  \_ /usr/bin/python -O /usr/bin/ranger
  35688 pts/4    S      0:00      \_ /bin/bash
  41134 pts/4    R+     0:00              \_ ps f -g 19325

Climate Change - Natural Hazard and Storm - Python
Published by awsbuild almost 4 years ago

OasisLMF - 1.17.0rc1

Changelog - 1.17.0rc1

#833 - Minor issue - error parsing fm CLI help text
#831 - Add timestamps to logger
#844 - Built-in Lookup revamp
#840 - Fix flaky tests in model_preparation/test_lookup.py
#843 - Fix CI pipeline error, double tagging release causes script to fail
#842 - Produce summary index files by default to reduce memory use of ktools output components
#816 - Issue in coverage_type_id grouping of ground up loss results for multi-peril
#809 - Error handling for invalid oasislmf.json config files
#821 - Add missing items to data settings
#824 - Inputs directory preparation issue when ORD is enabled
#826 - Automate Change logs and release notes
#822 - Ktools exit handler killing off bash logging on exit

OasisLMF Notes

Added function timestamps to 'info' log level

Log execution times If a function takes longer than 0.01 sec to complete

Example Output

COMPLETED: oasislmf.execution.bin.csv_to_bin in 0.02s
RUNNING: oasislmf.execution.bin.csv_to_bin
COMPLETED: oasislmf.execution.bin.csv_to_bin in 0.01s
RUNNING: oasislmf.execution.bin.prepare_run_inputs
RUNNING: oasislmf.execution.runner.run
[OK] eve
[OK] getmodel
[OK] gulcalc
[OK] summarycalc
[OK] eltcalc
[OK] aalcalc
[OK] leccalc
Run Completed
Run Completed

COMPLETED: oasislmf.execution.runner.run in 4.25s

Release notes feature title

Built-in Lookup class that implement the OasisLookupInterface
provide a data driven lookup capability that will be both flexible (sub-classing capability) and efficient.
lookup function provided include:
    - Location preparation (set default min and max value)
    - 2 geospacial option (direct computation or using rtree)
    - merge to map column keys to an id
    - split_loc_perils_covered to properly use data from LocPerilsCovered
    - simple_pivot allow to pivot columns of the locations dataframe into multiple rows
example of config provided in Piwind lookup_new_key_server.json

Set summarycalc option to output summary index files in genbash

The ktools binary summarycalc can now produce index files, which leads to a reduction in memory use by the downstream output components leccalc and ordleccalc. To take advantage of this, the option to output these summary index files is set as the default in genbash.

Fixed ordering of item_id column for gulsummaryxref

The issue was introduced in 1.15.2 from https://github.com/OasisLMF/OasisLMF/pull/774

Before this change, the 'gul' summary groups were created using 'gul_summary_map.csv', but its now using 'fm_summary_map.csv' to include the IL fields.

The problem happens because the summary group code expects that the column id_set_index is sorted before running the grouping, however these are different columns between, gul and fm and fm_summary_map defaults to sorting by output_id

id_set_index columns

gul == item_id
fm == output_id

fm_summary_map

   acc_idx accnumber  agg_id  coverage_id  coverage_type_id  layer_id  loc_id  loc_idx locnumber peril_id polnumber portnumber         tiv  output_id  item_id
0        0      PPT3       1            1                 1         1       1        0      PPT3      WTC      PPT3  T20100_C1  10000000.0          1        1   
1        0      PPT3       3            1                 1         1       1        0      PPT3      WSS      PPT3  T20100_C1  10000000.0          2        3   
2        0      PPT3       2            2                 3         1       1        0      PPT3      WTC      PPT3  T20100_C1  20000000.0          3        2   
3        0      PPT3       4            2                 3         1       1        0      PPT3      WSS      PPT3  T20100_C1  20000000.0          4        4

Updated model settings schema to extend data settings

docs_version and test_files_version: as a way to track metadata for model specific documentation and test files versions
keys_data_version: some model have their keys_data included in model_data, in which case item could be empty
additional_assets.deploy: needed to target specific deployments (default value for all is "always")

Fixed issue creating occurrence.bin when using ord options in analysis settings.

Using only ord output options skipped occurrence file creation because the binary preparation code was keyed to check for
"lec_output": true, which is not needed for ord settings. The following JSON now correctly prepares an occurrence.bin file. See issue #824 for details

"gul_summaries": [
    {
        "id": 1,
        "ord_output": {
            "ept_full_uncertainty_aep": true,
            "return_period_file": true
        }
    }
],

Automated changelogs and release notes

The build system will now automatically create changelogs and release notes

Updated bash exit handler to ignore logging commands

Fix for issue #822, with this update the exit handler will not kill the bash logging command on exit.

Example of issue

/data/output/run/run_ktools.sh: line 13: 73690 Killed                  tee -ia log/bash.log
/data/output/run/run_ktools.sh: line 14: 73692 Killed                  tee -ia log/bash.log 1>&2

After fix

$cat log/subprocess_list 
    PID TTY      STAT   TIME COMMAND
  19325 pts/4    Ss     0:00 bash
  35622 pts/4    Sl     0:00  \_ /usr/bin/python -O /usr/bin/ranger
  35688 pts/4    S      0:00      \_ /bin/bash
  40382 pts/4    S+     0:00          \_ /bin/bash ./run_ktools.sh
  40387 pts/4    S+     0:00              \_ /bin/bash ./run_ktools.sh
  40389 pts/4    S+     0:00              |   \_ tee -ia log/bash.log
  40388 pts/4    S+     0:00              \_ /bin/bash ./run_ktools.sh
  40390 pts/4    S+     0:00              |   \_ tee -ia log/bash.log
  41134 pts/4    R+     0:00              \_ ps f -g 19325
  35792 pts/4    S      0:00 /bin/bash ./run_ktools.sh
  35791 pts/4    S      0:00 /bin/bash ./run_ktools.sh
  35781 pts/4    S      0:00 /bin/bash ./run_ktools.sh

$cat log/subprocess_list | egrep -v *\\.log$ |  egrep -v *\\.sh$
    PID TTY      STAT   TIME COMMAND
  19325 pts/4    Ss     0:00 bash
  35622 pts/4    Sl     0:00  \_ /usr/bin/python -O /usr/bin/ranger
  35688 pts/4    S      0:00      \_ /bin/bash
  41134 pts/4    R+     0:00              \_ ps f -g 19325

Climate Change - Natural Hazard and Storm - Python
Published by awsbuild almost 4 years ago

OasisLMF - 1.15.9

Changelog - 1.15.9

#833 - Minor issue - error parsing fm CLI help text
#831 - Add timestamps to logger
#840 - Fix flaky tests in model_preparation/test_lookup.py
#821 - Add missing items to data settings
#826 - Automate Change logs and release notes

OasisLMF Notes

Added function timestamps to 'info' log level

Log execution times If a function takes longer than 0.01 sec to complete

Example Output

COMPLETED: oasislmf.execution.bin.csv_to_bin in 0.02s
RUNNING: oasislmf.execution.bin.csv_to_bin
COMPLETED: oasislmf.execution.bin.csv_to_bin in 0.01s
RUNNING: oasislmf.execution.bin.prepare_run_inputs
RUNNING: oasislmf.execution.runner.run
[OK] eve
[OK] getmodel
[OK] gulcalc
[OK] summarycalc
[OK] eltcalc
[OK] aalcalc
[OK] leccalc
Run Completed
Run Completed

COMPLETED: oasislmf.execution.runner.run in 4.25s

Updated model settings schema to extend data settings

docs_version and test_files_version: as a way to track metadata for model specific documentation and test files versions
keys_data_version: some model have their keys_data included in model_data, in which case item could be empty
additional_assets.deploy: needed to target specific deployments (default value for all is "always")

Automated changelogs and release notes

The build system will now automatically create changelogs and release notes

Climate Change - Natural Hazard and Storm - Python
Published by sambles almost 4 years ago