{"id":303033,"name":"cbers-2-stac","description":"Create and keep up-to-date a STAC static catalog and API for CBERS-4/4A and Amazonia-1 images on AWS.","url":"https://github.com/fredliporace/cbers-2-stac","last_synced_at":"2026-05-23T18:00:39.652Z","repository":{"id":43209421,"uuid":"126712813","full_name":"fredliporace/cbers-2-stac","owner":"fredliporace","description":"STAC static and API for CBERS-4/4A and Amazonia-1 on AWS","archived":false,"fork":false,"pushed_at":"2025-03-12T14:39:05.000Z","size":1623,"stargazers_count":14,"open_issues_count":14,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2026-05-15T14:04:41.216Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fredliporace.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["fredliporace"]}},"created_at":"2018-03-25T15:41:35.000Z","updated_at":"2025-03-12T14:39:09.000Z","dependencies_parsed_at":"2024-11-08T03:00:40.236Z","dependency_job_id":"c3c54c87-1f15-41a3-8028-b443cd76fb54","html_url":"https://github.com/fredliporace/cbers-2-stac","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/fredliporace/cbers-2-stac","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fredliporace%2Fcbers-2-stac","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fredliporace%2Fcbers-2-stac/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fredliporace%2Fcbers-2-stac/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fredliporace%2Fcbers-2-stac/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fredliporace","download_url":"https://codeload.github.com/fredliporace/cbers-2-stac/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fredliporace%2Fcbers-2-stac/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33406444,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-23T04:15:53.637Z","status":"ssl_error","status_checked_at":"2026-05-23T04:15:53.242Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"owner":{"login":"fredliporace","name":"Frederico Liporace","uuid":"28077347","kind":"user","description":"Remote sensing, GPU, CBERS","email":"","website":null,"location":"Rio de Janeiro, Brazil","twitter":"liporace","company":"Scitekno","icon_url":"https://avatars.githubusercontent.com/u/28077347?u=36fd8d54710e0ffc20aae41c8932d990ac622ed0\u0026v=4","repositories_count":33,"last_synced_at":"2026-05-13T19:33:41.572Z","metadata":{"has_sponsors_listing":true},"html_url":"https://github.com/fredliporace","funding_links":["https://github.com/sponsors/fredliporace"],"total_stars":49,"followers":36,"following":23,"created_at":"2023-05-22T23:25:13.508Z","updated_at":"2026-05-13T19:33:41.572Z","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fredliporace","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fredliporace/repositories"},"packages":[],"commits":{"id":3925840,"full_name":"fredliporace/cbers-2-stac","default_branch":"master","total_commits":353,"total_committers":1,"total_bot_commits":0,"total_bot_committers":0,"mean_commits":353.0,"dds":0.0,"past_year_total_commits":0,"past_year_total_committers":0,"past_year_total_bot_commits":0,"past_year_total_bot_committers":0,"past_year_mean_commits":0.0,"past_year_dds":0.0,"last_synced_at":"2026-05-21T17:00:42.881Z","last_synced_commit":"eb3fb41dffeaa497a8046cb57633b76b55a3c2f7","created_at":"2024-11-01T00:19:02.539Z","updated_at":"2026-05-21T17:00:42.820Z","committers":[{"name":"Frederico Liporace","email":"liporace@amskepler.com","login":"fredliporace","count":353}],"past_year_committers":[],"commits_url":"https://commits.ecosyste.ms/api/v1/hosts/GitHub/repositories/fredliporace%2Fcbers-2-stac/commits","host":{"name":"GitHub","url":"https://github.com","kind":"github","last_synced_at":"2026-05-23T00:00:39.628Z","repositories_count":6239390,"commits_count":884352778,"contributors_count":34897942,"owners_count":1156481,"icon_url":"https://github.com/github.png","host_url":"https://commits.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://commits.ecosyste.ms/api/v1/hosts/GitHub/repositories"}},"issues_stats":{"full_name":"fredliporace/cbers-2-stac","html_url":"https://github.com/fredliporace/cbers-2-stac","last_synced_at":"2026-05-17T15:00:49.103Z","status":"error","issues_count":87,"pull_requests_count":17,"avg_time_to_close_issue":6225333.849315069,"avg_time_to_close_pull_request":5386.588235294118,"issues_closed_count":73,"pull_requests_closed_count":17,"pull_request_authors_count":1,"issue_authors_count":4,"avg_comments_per_issue":0.7471264367816092,"avg_comments_per_pull_request":0.058823529411764705,"merged_pull_requests_count":17,"bot_issues_count":0,"bot_pull_requests_count":0,"past_year_issues_count":3,"past_year_pull_requests_count":1,"past_year_avg_time_to_close_issue":4041.0,"past_year_avg_time_to_close_pull_request":901.0,"past_year_issues_closed_count":1,"past_year_pull_requests_closed_count":1,"past_year_pull_request_authors_count":1,"past_year_issue_authors_count":1,"past_year_avg_comments_per_issue":0.0,"past_year_avg_comments_per_pull_request":0.0,"past_year_bot_issues_count":0,"past_year_bot_pull_requests_count":0,"past_year_merged_pull_requests_count":1,"created_at":"2024-11-01T00:19:02.906Z","updated_at":"2026-05-17T15:00:49.103Z","repository_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/repositories/fredliporace%2Fcbers-2-stac","issues_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/repositories/fredliporace%2Fcbers-2-stac/issues","issue_labels_count":{},"pull_request_labels_count":{},"issue_author_associations_count":{"OWNER":83,"NONE":6},"pull_request_author_associations_count":{"OWNER":20},"issue_authors":{"fredliporace":83,"matthewhanson":4,"keithpeck":1,"jsscgrnt":1},"pull_request_authors":{"fredliporace":20},"host":{"name":"GitHub","url":"https://github.com","kind":"github","last_synced_at":"2026-05-19T00:00:20.226Z","repositories_count":14638575,"issues_count":34119126,"pull_requests_count":111723532,"authors_count":11266386,"icon_url":"https://github.com/github.png","host_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/repositories","owners_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/owners","authors_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/authors"},"past_year_issue_labels_count":{},"past_year_pull_request_labels_count":{},"past_year_issue_author_associations_count":{},"past_year_pull_request_author_associations_count":{},"past_year_issue_authors":{},"past_year_pull_request_authors":{},"maintainers":[{"login":"fredliporace","count":103,"url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/authors/fredliporace"}],"active_maintainers":[]},"events":{"total":{"PullRequestEvent":2,"IssuesEvent":3,"WatchEvent":2,"PushEvent":6},"last_year":{}},"keywords":[],"dependencies":[{"ecosystem":"pypi","filepath":"cbers2stac/elasticsearch/requirements.txt","sha":null,"kind":"manifest","created_at":"2022-09-15T03:31:26.162Z","updated_at":"2022-09-15T03:31:26.162Z","repository_link":"https://github.com/fredliporace/cbers-2-stac/blob/master/cbers2stac/elasticsearch/requirements.txt","dependencies":[{"id":4887760684,"package_name":"elasticsearch","ecosystem":"pypi","requirements":"==7.13.4","direct":true,"kind":"runtime","optional":false},{"id":4887760685,"package_name":"elasticsearch-dsl","ecosystem":"pypi","requirements":"==7.4.0","direct":true,"kind":"runtime","optional":false},{"id":4887760686,"package_name":"aws-requests-auth","ecosystem":"pypi","requirements":"==0.4.3","direct":true,"kind":"runtime","optional":false}]},{"ecosystem":"pypi","filepath":"cbers2stac/process_new_scene_queue/requirements.txt","sha":null,"kind":"manifest","created_at":"2022-09-15T03:31:26.316Z","updated_at":"2022-09-15T03:31:26.316Z","repository_link":"https://github.com/fredliporace/cbers-2-stac/blob/master/cbers2stac/process_new_scene_queue/requirements.txt","dependencies":[{"id":4887760752,"package_name":"utm","ecosystem":"pypi","requirements":"==0.7.0","direct":true,"kind":"runtime","optional":false}]},{"ecosystem":"actions","filepath":".github/workflows/ci.yml","sha":null,"kind":"manifest","created_at":"2024-02-08T22:53:18.990Z","updated_at":"2024-02-08T22:53:18.990Z","repository_link":"https://github.com/fredliporace/cbers-2-stac/blob/master/.github/workflows/ci.yml","dependencies":[{"id":16121559417,"package_name":"hmarr/debug-action","ecosystem":"actions","requirements":"v2","direct":true,"kind":"composite","optional":false},{"id":16121560708,"package_name":"actions/checkout","ecosystem":"actions","requirements":"v2","direct":true,"kind":"composite","optional":false},{"id":16121560709,"package_name":"actions/setup-python","ecosystem":"actions","requirements":"v2","direct":true,"kind":"composite","optional":false},{"id":16121560710,"package_name":"actions/cache","ecosystem":"actions","requirements":"v2","direct":true,"kind":"composite","optional":false}]},{"ecosystem":"docker","filepath":"test/docker-compose.yml","sha":null,"kind":"manifest","created_at":"2024-02-08T22:53:20.315Z","updated_at":"2024-02-08T22:53:20.315Z","repository_link":"https://github.com/fredliporace/cbers-2-stac/blob/master/test/docker-compose.yml","dependencies":[{"id":16121560711,"package_name":"localstack/localstack-full","ecosystem":"docker","requirements":"1.0.3","direct":true,"kind":"runtime","optional":false}]},{"ecosystem":"pypi","filepath":"setup.py","sha":null,"kind":"manifest","created_at":"2024-02-08T22:53:20.499Z","updated_at":"2024-02-08T22:53:20.499Z","repository_link":"https://github.com/fredliporace/cbers-2-stac/blob/master/setup.py","dependencies":[]}],"score":3.332204510175204,"created_at":"2024-11-01T00:18:52.681Z","updated_at":"2026-05-23T18:00:39.654Z","avatar_url":"https://github.com/fredliporace.png","language":"Python","category":"Sustainable Development","sub_category":"Environmental Satellites","monthly_downloads":0,"total_dependent_repos":0,"total_dependent_packages":0,"readme":"# cbers-2-stac\n\n![CI](https://github.com/fredliporace/cbers-2-stac/actions/workflows/ci.yml/badge.svg?branch=master)\n\nCreate and keep up-to-date a [STAC](https://github.com/radiantearth/stac-spec) static catalog and API for [CBERS-4/4A](https://registry.opendata.aws/cbers/) and [Amazonia-1](https://registry.opendata.aws/amazonia/) images on AWS.\n\n# STAC version and extensions\n\nImplements STAC version 1.0.0 and STAC-API version 1.0.0-beta.1.\n\n# Live version\n\nA live version of the stack is deployed to AWS and serve its contents in:\n\n  * `arn:aws:s3:::br-eo-stac-1-0-0` S3 bucket with static STAC content, including its [root catalog](https://br-eo-stac-1-0-0.s3.amazonaws.com/catalog.json).\n  * STAC API [endpoint](https://stac.scitekno.com.br/v100).\n  * SNS topic for new scenes: `arn:aws:sns:us-west-2:769537946825:br-eo-stac-prod-stacitemtopic4BCE3141-Z8he7LYjqXFe`. This topic receive the complete STAC item for each ingested scene.\n\n# Deployment to AWS\n\n## Install\n\n```bash\n$ git clone git@github.com:fredliporace/cbers-2-stac.git\n$ cd cbers-2-stac\n$ pip install -e .[dev,test,deploy]\n```\n\nSome lambdas require extra pip packages to be installed in the lambda directory before deployment. To install these packages execute:\n\n```bash\n./pip-on-lambdas.sh\n```\n\n## CDK bootstrap\n\nDeployment uses AWS CDK2.\n\nRequirements:\n* node: Use [nvm](https://heynode.com/tutorial/install-nodejs-locally-nvm/) to make sure a supported node is being used, tested with 18.0.0\n* AWS credentials configured\n\nTo install and check AWS CDK (tested with CDK 2.129.0):\n```bash\n$ npm install -g aws-cdk\n$ cdk --version\n\n$ cdk bootstrap # Deploys the CDK toolkit stack into an AWS environment\n\n# in specific region\n$ cdk bootstrap aws://${AWS_ACCOUNT_ID}/eu-central-1\n```\n\n## Configuration\n\nCreate a ```.env``` file in the project root directory and configure your application. You should use ```.env_example``` as a guide, this file contains the documentation for all parameters.\n\nIf `STAC_ENABLE_API` is set you should review the Elasticsearch domain configuration, for which some parameters are [hard-coded](https://github.com/fredliporace/cbers-2-stac/blob/dev/stack/app.py#L536-L552), see this [issue](https://github.com/fredliporace/cbers-2-stac/issues/61).\n\nAfter the configuration is completed check if CDK shows the configured stack:\n\n```bash\n$ cdk list\n```\n\n## Deployment\n\nDeploy the stack, replacing `cbers2stac-dev` with your configured stack name:\n\n```bash\n$ cdk deploy cbers2stac-dev\n```\n\nThe e-mail configured in `STACK_OPERATOR_EMAIL` receives execution alarms and while the first deploy is made it should receive a message requesting confirmation for the alarm topic subscription. Accept the request to receive the alarms.\n\nThe stack output shows the SNS topic for new scenes and the API endpoint, if configured:\n```bash\n ✅  cbers2stac-prod\n\nOutputs:\ncbers2stac-prod.stacapiEndpointBED73CCA = https://....\ncbers2stac-prod.stacitemtopicoutput = arn:aws:sns:us-east-1:...:...\n\n```\n\n## Static catalogs and collections\n\nEmpty static stac catalogs and collections are created when the stack is deployed. Note that when these files are updated and a new deploy is executed to an already populated stac bucket the [deployment may fail](https://github.com/fredliporace/cbers-2-stac/issues/88).\n\nIn order to overcome that you may set `STACK_DEPLOY_STATIC_CATALOG_STRUCTURE` to `false` in `.env` and manually copy the static files. From a prompt in the `stack/static_catalog_structure` dir:\n\n```bash\n$ aws s3 cp ./ s3://[REPLACE_WITH_STAC_BUCKET_NAME] --recursive\n```\n\nReplace `cbers-stac-1-0-0` with the given stac bucket name. The collections initially do not contain the links to the children, this is updated when a new scene is inserted at any level.\n\n## Creating the Elasticsearch index\n\nIf ```STACK_ENABLE_API``` is set in the configuration you should now create the Elasticsearch index. This needs to be executed only once, right after the first deploy that enables the API. The index is created by the lambda ```create_elastic_index_lambda```, which may be executed from the AWS console or awscli. The function requires no parameters.\n\nIt is recommended to change the cluster configuration to disable the automatic creation of indices. AFAIK this can't be done through CDK options, you need to directly access the domain configuration endpoint, see [example](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#index-creation).\n\n## Populate the catalog\n\nOnce created the stack automatically listen to the SNS topics that publish new quicklooks. The stac item is automatically created as soon as the message is received.\n\n### Reconciliation from INPE's original metadata\n\nThe lambda `populate_reconcile_queue_lambda` may be used to reconcile the STAC catalog with the original CBERS metadata catalog. This is typically used to populate a new STAC instance with data from the open datasets.\n\nThe lambda payload is a bucket name and a prefix, all directories under the prefix are queued. Each directory is then scanned separately by a distinct lambda instance, all XMLs are converted to static STAC documents and indexed again, if the API is configured.\n\nSome examples are shown below.\n\nTo index all CBERS-4 MUX scenes with path 102 and row 83.\n```json\n{\n  \"bucket\": \"cbers-pds\",\n  \"prefix\": \"CBERS4/MUX/102/083/\"\n}\n```\n\nTo index all CBERS-4A MUX scenes:\n```json\n{\n  \"bucket\": \"cbers-pds\",\n  \"prefix\": \"CBERS4A/MUX/\"\n}\n```\n\nTo index all CBERS-4A MUX scenes with path 120:\n```json\n{\n  \"bucket\": \"cbers-pds\",\n  \"prefix\": \"CBERS4A/MUX/120/\"\n}\n```\n\nTo index all Amazonia-1 WFI scenes:\n```json\n{\n    \"bucket\": \"amazonia-pds\",\n    \"prefix\": \"AMAZONIA1/WFI/\"\n}\n```\n\nThe indexed documents are immediately available through the STAC API. The static catalogs are updated every 30 minutes. To update the static catalogs before that you may execute the ```generate_catalog_levels_to_be_updated_lambda``` lambda.\n\n### Reconciliation from STAC static catalog\n\nThe lambda ```reindex_stac_items_lambda``` may be used to reconcile the STAC API service with the static catalog. The lambda payload are the parameters to be passed to `list_objects_v2`, all STAC items under the prefix are queued and re-indexed. Some examples are shown below.\n\nTo index all CBERS-4 AWFI scenes with path 1 and row 27:\n```json\n{\n  \"prefix\": \"CBERS4/AWFI/001/\"\n}\n```\n\n## Operation\n\n### SQS-Lambda and dead letter queues (DLQs)\n\nThe system makes extensive use of the SQS-lambda integration pattern. DLQs are defined to store messages representing failed jobs:\n\n * `reconcile_queue`: jobs representing the S3 prefixes that will be reconciled are queued here. Consumed by `consume_reconcile_queue_lambda`. Failed jobs are sent to `consume_reconcile_queue_dlq`.\n * `new_scenes_queue`: jobs representing a key for a scene to be converted to STAC and indexed. Consumed by `process_new_scene_lambda`. Failed jobs are sent to `process_new_scenes_queue_dlq`.\n * `insert_into_elasticsearch_queue`: jobs representing a STAC item. This queue subscribes to `stac_item_topic` and `reconcile_stac_item_topic`, receiving the STAC itemas as notifications. Consumed by `insert_into_elastic_lambda`. Failed jobs (for now) are sent to `dead_letter_queue`.\n\nFailed lambda executions from other queues are sent to the general `dead_letter_queue`.\n\nA tool is provided to move messages from SQS queues, this may be used to re-queue failed jobs:\n```bash\ncb2stac-redrive-sqs --src-url=https://... --dst-url=https://... --messages-no=100\n```\n\nThe jobs may also be re-queued using the new `Start DLQ redrive` now available from the AWS console.\n\n### Recovering from ElasticSearch (ES) cluster failures\n\n#### Restore from ES snapshot\n\nSee [AWS documentation](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-snapshots.html#managedomains-snapshot-restore).\n\n#### Re-ingest items from backup queue\n\nThe system keeps `backup_insert_into_elasticsearch_queue` with the STAC items ingested in the last days, see `STACK_BACKUP_QUEUE_RETENTION_DAYS` configuration parameter. The items may be re-ingested after a restore to make sure that the archive includes the items processed between the restored backup date and the failure.\n\nUse `cb2stac-redrive-sqs` (or `Start DLQ redrive` from AWS console) to transfer messages from DLQ to `insert_into_elasticsearch_queue`.\n\n# Development\n\n## Git hooks\n\nThis repo is set to use `pre-commit` to run *isort*, *pylint*, *pydocstring*, *black* (\"uncompromising Python code formatter\") and mypy when committing new code.\n\n```bash\n$ pre-commit install\n```\n\n## Testing\n\nRequires localstack up to execute tests:\n\nCheck before if `/tmp/localstack/es_backup` directory exists, this is required to start the ES service.\n\n```bash\n$ cd test \u0026\u0026 docker compose up # Starts localstack\n$ pytest\n```\n\n## Check CI integration testing before pushing\n\n[act](https://github.com/nektos/act) may be used to test github actions locally. At the project's root directory:\n\n```bash\n$ act --env-file \u003c /dev/null -j tests\n$ act --env-filr \u003c /dev/null -r -j tests # To keep docker container's state\n```\n\nThe `env-file` option is used to make sure that the `.env` file is used only for CDK and not docker.\n\n# References\n\nThe mechanism to include new items into the archive as soon as they are ingested is described in [this AWS blog post](https://aws.amazon.com/blogs/publicsector/keeping-a-spatiotemporal-asset-catalog-stac-up-to-date-with-sns-sqs/).\n\n# Acknowledgments\n\n[Radiant Earth Foundation](https://www.radiant.earth/) supported the migration to STAC 1.0.0 final.\n","funding_links":["https://github.com/sponsors/fredliporace"],"readme_doi_urls":[],"works":{},"citation_counts":{},"total_citations":0,"keywords_from_contributors":[],"project_url":"https://ost.ecosyste.ms/api/v1/projects/303033","html_url":"https://ost.ecosyste.ms/projects/303033"}