{"id":118365,"name":"biotrade","description":"Analyses international trade of bio-based products with a focus on the agriculture and forestry sectors.","url":"https://gitlab.com/bioeconomy/forobs/biotrade","last_synced_at":"2026-05-25T13:05:07.572Z","repository":{"id":157079957,"uuid":"29422866","full_name":"bioeconomy/forobs/biotrade","owner":"bioeconomy","description":"This package loads trade data on bio based commodities and products from the UN Comtrade\r\nand from FAOSTAT.","archived":false,"fork":false,"pushed_at":null,"size":null,"stargazers_count":2,"open_issues_count":44,"forks_count":0,"subscribers_count":null,"default_branch":"main","last_synced_at":"2025-12-30T21:23:13.628Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":null,"metadata":{},"created_at":"2021-09-07T10:31:14.718Z","updated_at":"2025-12-23T13:53:22.734Z","dependencies_parsed_at":"2023-05-09T15:23:45.988Z","dependency_job_id":null,"html_url":"https://gitlab.com/bioeconomy/forobs/biotrade","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"purl":"pkg:gitlab/bioeconomy/biotrade","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com/repositories/bioeconomy%2Fforobs%2Fbiotrade","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com/repositories/bioeconomy%2Fforobs%2Fbiotrade/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com/repositories/bioeconomy%2Fforobs%2Fbiotrade/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com/repositories/bioeconomy%2Fforobs%2Fbiotrade/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com/owners/bioeconomy","download_url":"https://gitlab.com/bioeconomy/forobs/biotrade/-/archive/main/biotrade-main.zip","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com/repositories/bioeconomy%2Fforobs%2Fbiotrade/sbom","scorecard":null,"host":{"name":"gitlab.com","url":"https://gitlab.com","kind":"gitlab","repositories_count":4521539,"owners_count":7348,"icon_url":"https://github.com/gitlab.png","version":null,"created_at":"2022-05-30T11:31:42.605Z","updated_at":"2026-01-12T22:45:04.389Z","status":"online","status_checked_at":"2026-05-14T02:00:07.252Z","response_time":287,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.382Z","robots_txt_url":"https://gitlab.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com/owners"}},"owner":{"login":"bioeconomy","name":"Bioeconomy","uuid":"organization-4897061","kind":"organization","description":"This group gathers repositories relating to the provision of data and analysis on biomass/bioeconomy supply and demand on a long-term basis in the context of the European Commission's in-house science service, DG Joint Research Center.","email":null,"website":null,"location":null,"twitter":null,"company":null,"icon_url":"https://gitlab.com/uploads/-/system/group/avatar/4897061/bioeconomy_graphic_full.jpg","repositories_count":7,"last_synced_at":"2023-04-27T10:38:57.438Z","metadata":{},"html_url":"https://gitlab.com/bioeconomy","funding_links":[],"total_stars":null,"followers":null,"following":null,"created_at":"2023-04-27T10:38:57.439Z","updated_at":"2023-04-27T10:38:57.439Z","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com/owners/bioeconomy","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com/owners/bioeconomy/repositories"},"packages":[],"commits":{"id":1638834,"full_name":"bioeconomy/forobs/biotrade","default_branch":"main","total_commits":995,"total_committers":6,"total_bot_commits":0,"total_bot_committers":0,"mean_commits":165.83333333333334,"dds":0.4663316582914573,"past_year_total_commits":51,"past_year_total_committers":2,"past_year_total_bot_commits":0,"past_year_total_bot_committers":0,"past_year_mean_commits":25.5,"past_year_dds":0.196078431372549,"last_synced_at":"2026-05-21T11:05:05.829Z","last_synced_commit":"1b9bd83f149d1adf3e35f4556d1ddc52229971d8","created_at":"2024-07-24T20:00:43.651Z","updated_at":"2026-05-21T11:05:05.511Z","committers":[{"name":"Paul Rougieux","email":"paul.rougieux@gmail.com","login":null,"count":531},{"name":"selenepatani91","email":"selene.patani@ext.ec.europa.eu","login":null,"count":398},{"name":"selenepatani","email":"selene.patani@gmail.com","login":null,"count":55},{"name":"Mirco Migliavacca","email":"mirco.migliavacca@gmail.com","login":null,"count":6},{"name":"Selene Patani","email":"patasel@ies.jrc.it","login":null,"count":4},{"name":"TBras","email":"bras.teresa@gmail.com","login":null,"count":1}],"past_year_committers":[{"name":"selenepatani91","email":"selene.patani@ext.ec.europa.eu","login":null,"count":41},{"name":"Paul Rougieux","email":"paul.rougieux@gmail.com","login":null,"count":10}],"commits_url":"https://commits.ecosyste.ms/api/v1/hosts/gitlab.com/repositories/bioeconomy%2Fforobs%2Fbiotrade/commits","host":{"name":"gitlab.com","url":"https://gitlab.com","kind":"gitlab","last_synced_at":"2026-05-22T00:00:12.949Z","repositories_count":24059,"commits_count":4205449,"contributors_count":115667,"owners_count":12392,"icon_url":"https://github.com/gitlab.png","host_url":"https://commits.ecosyste.ms/api/v1/hosts/gitlab.com","repositories_url":"https://commits.ecosyste.ms/api/v1/hosts/gitlab.com/repositories"}},"issues_stats":{"full_name":"bioeconomy/forobs/biotrade","html_url":"https://gitlab.com/bioeconomy/forobs/biotrade","last_synced_at":"2026-05-21T07:47:36.235Z","status":"error","issues_count":null,"pull_requests_count":null,"avg_time_to_close_issue":null,"avg_time_to_close_pull_request":null,"issues_closed_count":null,"pull_requests_closed_count":null,"pull_request_authors_count":null,"issue_authors_count":null,"avg_comments_per_issue":null,"avg_comments_per_pull_request":null,"merged_pull_requests_count":null,"bot_issues_count":null,"bot_pull_requests_count":null,"past_year_issues_count":null,"past_year_pull_requests_count":null,"past_year_avg_time_to_close_issue":null,"past_year_avg_time_to_close_pull_request":null,"past_year_issues_closed_count":null,"past_year_pull_requests_closed_count":null,"past_year_pull_request_authors_count":null,"past_year_issue_authors_count":null,"past_year_avg_comments_per_issue":null,"past_year_avg_comments_per_pull_request":null,"past_year_bot_issues_count":null,"past_year_bot_pull_requests_count":null,"past_year_merged_pull_requests_count":null,"created_at":"2024-08-14T06:01:05.250Z","updated_at":"2026-05-21T07:47:36.236Z","repository_url":"https://issues.ecosyste.ms/api/v1/hosts/gitlab.com/repositories/bioeconomy%2Fforobs%2Fbiotrade","issues_url":"https://issues.ecosyste.ms/api/v1/hosts/gitlab.com/repositories/bioeconomy%2Fforobs%2Fbiotrade/issues","issue_labels_count":{"qaqc":10,"wontfix":3,"2":2,"1":1,"bug":1},"pull_request_labels_count":{},"issue_author_associations_count":{},"pull_request_author_associations_count":{},"issue_authors":{"paulrougieux":67,"selenepatani91":26,"giovanni.bausano":1,"oliver.gardi":1},"pull_request_authors":{},"host":{"name":"gitlab.com","url":"https://gitlab.com","kind":"gitlab","last_synced_at":"2026-05-21T00:00:41.699Z","repositories_count":0,"issues_count":0,"pull_requests_count":0,"authors_count":61892,"icon_url":"https://github.com/gitlab.png","host_url":"https://issues.ecosyste.ms/api/v1/hosts/gitlab.com","repositories_url":"https://issues.ecosyste.ms/api/v1/hosts/gitlab.com/repositories","owners_url":"https://issues.ecosyste.ms/api/v1/hosts/gitlab.com/owners","authors_url":"https://issues.ecosyste.ms/api/v1/hosts/gitlab.com/authors"},"past_year_issue_labels_count":{},"past_year_pull_request_labels_count":{},"past_year_issue_author_associations_count":{},"past_year_pull_request_author_associations_count":{},"past_year_issue_authors":{},"past_year_pull_request_authors":{},"maintainers":[],"active_maintainers":[]},"events":null,"keywords":[],"dependencies":[{"ecosystem":"pypi","filepath":"requirements.txt","sha":null,"kind":"manifest","created_at":"2023-05-09T15:23:45.723Z","updated_at":"2023-05-09T15:23:45.723Z","repository_link":"https://gitlab.com/bioeconomy/forobs/biotrade/blob/main/requirements.txt","dependencies":[{"id":9487192900,"package_name":"pre-commit","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":9487192901,"package_name":"pandas","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":9487192902,"package_name":"sqlalchemy","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":9487192903,"package_name":"psycopg2","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":9487192904,"package_name":"sqlalchemy-utils","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":9487192905,"package_name":"scipy","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":9487192906,"package_name":"pymannkendall","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":9487192907,"package_name":"openpyxl","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":9487192908,"package_name":"requests","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false}]},{"ecosystem":"pypi","filepath":"setup.py","sha":null,"kind":"manifest","created_at":"2023-05-09T15:23:45.783Z","updated_at":"2023-05-09T15:23:45.783Z","repository_link":"https://gitlab.com/bioeconomy/forobs/biotrade/blob/main/setup.py","dependencies":[{"id":9487194159,"package_name":"pandas","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":9487194160,"package_name":"sqlalchemy","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":9487194161,"package_name":"sqlalchemy_utils","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":9487194162,"package_name":"psycopg2","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":9487194163,"package_name":"requests","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false}]}],"score":5.62040086571715,"created_at":"2023-10-18T00:00:24.799Z","updated_at":"2026-05-25T13:05:07.574Z","avatar_url":null,"language":null,"category":"Consumption","sub_category":"Agriculture and Nutrition","monthly_downloads":0,"total_dependent_repos":0,"total_dependent_packages":0,"readme":"The `biotrade` package analyses international trade of bio-based products. It focuses on\nthe agriculture and forestry sectors, from primary production to secondary products\ntransformation. It loads bilateral trade data from UN Comtrade, production and trade\ndata from FAOSTAT and socio-economic indicators from the World Bank.\n\n\n# Documentation\n\nThe documentation is available at:\nhttps://bioeconomy.gitlab.io/forobs/biotrade/\n\n\n# Installation\n\n## Base installation\n\nInstall the biotrade package from the python package index with pip:\n\n    python -m pip install biotrade\n\nUpgrade the package to the latest version with:\n\n    python -m pip install --upgrade biotrade\n\nTo install the latest development version, use the `--upgrade` parameter and install\nfrom the main branch of the gitlab repository:\n\n    python -m pip install --upgrade --force-reinstall https://gitlab.com/bioeconomy/forobs/biotrade/-/archive/main/biotrade-main.tar.gz\n\n\n## Installation for contributors\n\nIf you plan to contribute to the development of the biotrade package, clone the biotrade\nrepository at https://gitlab.com/bioeconomy/forobs/biotrade/. You need to tell python\nwhere the package is located by adding it to your PYTHONPATH. You can do this by\nchanging the environment variables or by adding the following line to your shell\nconfiguration file such as `.bash_aliases`:\n\n    export PYTHONPATH=\"$HOME/repos/biotrade/\":$PYTHONPATH\n\nSpecify where you want to store the data by adding the following environment variable:\n\n    export BIOTRADE_DATA=\"$HOME/repos/biotrade_data/\"\n\nDependencies are listed in the `install_requires` argument of [setup.py](setup.py).\n\n\n# Usage\n\nThe biotrade package can download data from FAOSTAT and UN Comtrade and store it inside\na database. By default it will use an SQLite database stored locally in the folder\ndefined by the environment variable `BIOTRADE_DATA`. Alternatively, a PostGRESQL\ndatabase can be used if a connection string is defined in the environment variable\n`BIOTRADE_DATABASE_URL`, for example by adding the following to your .bash_aliases or\n.bash_rc:\n\n    export BIOTRADE_DATABASE_URL=\"postgresql://user@localhost/biotrade\"\n\nNote that database queries are abstracted with [SQL\nAlchemy](https://www.sqlalchemy.org/) which is what makes it possible to use different\ndatabase engines. SQLite is convenient for data analysis on laptops. PostGreSQL can be\nused on servers.\n\n\n## FAOSTAT\n\nFaostat provides agriculture and forestry data on their website https://www.fao.org/faostat/en/#data/\n\nThe biotrade package has a `faostat.pump` object that loads a selection of datasets. The\nlist of downloaded datasets is visible in `faostat.pump.datasets`. Column names and\nproduct descriptions are reformatted to snake case in a way that is convenient for\nanalysis. The data is then stored into an SQLite database (or PostgreSQL). The following\ncommands download and transfer the given datasets to the database:\n\n    \u003e\u003e\u003e from biotrade.faostat import faostat\n    \u003e\u003e\u003e faostat.pump.update([\"crop_production\", \"forestry_production\"])\n    \u003e\u003e\u003e # Loading trade data can take a long time on slow connections\n    \u003e\u003e\u003e faostat.pump.update([\"crop_trade\", \"forestry_trade\"])\n    \u003e\u003e\u003e faostat.pump.update([\"food_balance\"])\n    \u003e\u003e\u003e faostat.pump.update([\"land_use\", \"land_cover\", \"forest_land\"])\n\nList available datasets and metadata links:\n\n    \u003e\u003e\u003e faostat.pump.datasets\n    \u003e\u003e\u003e faostat.pump.metadata_link\n\nOnce the data has been loaded into the database, you can query it. For example select\ncrop production data for 2 countries\n\n    \u003e\u003e\u003e from biotrade.faostat import faostat\n    \u003e\u003e\u003e db = faostat.db_sqlite\n    \u003e\u003e\u003e cp2 = db.select(table=\"crop_production\",\n    \u003e\u003e\u003e                 reporter=[\"Portugal\", \"Estonia\"])\n\nSelect forestry trade flows data reported by all countries, with\nAustria as a partner country:\n\n    \u003e\u003e\u003e ft_aut = db.select(table=\"forestry_trade\",\n    \u003e\u003e\u003e                    partner=[\"Austria\"])\n\nSelect crop trade flows reported by the Netherlands where Brazil was a\npartner\n\n    \u003e\u003e\u003e ct_nel_bra = db.select(table=\"crop_trade\",\n    \u003e\u003e\u003e                        reporter=\"Netherlands\",\n    \u003e\u003e\u003e                        partner=\"Brazil\")\n\nSelect the mirror flows reported by Brazil, where the Netherlands was a partner\n\n    \u003e\u003e\u003e ct_bra_bel = db.select(table=\"crop_trade\",\n    \u003e\u003e\u003e                        reporter=\"Brazil\",\n    \u003e\u003e\u003e                        partner=\"Netherlands\")\n\nSelect land use and land cover data\n\n    \u003e\u003e\u003e lu = faostat.db.select(\"land_use\")\n    \u003e\u003e\u003e lc = faostat.db.select(\"land_cover\")\n\n\n## Comtrade\n\nSee the documentation of the various methods. As an example  here is how to download\ntrade data from the Comtrade API and return a data frame, for debugging purposes:\n\n    \u003e\u003e\u003e from biotrade.comtrade import comtrade\n    \u003e\u003e\u003e # Other sawnwood\n    \u003e\u003e\u003e swd99 = comtrade.pump.download(cc = \"440799\")\n    \u003e\u003e\u003e # Soy\n    \u003e\u003e\u003e soy = comtrade.pump.download(cc = \"120190\")\n\nDisplay information on column names used for renaming\nand dropping less important columns:\n\n    \u003e\u003e\u003e comtrade.column_names\n\nGet the list of products from the Comtrade API\n\n    \u003e\u003e\u003e hs = comtrade.pump.get_parameter_list(\"classificationHS.json\")\n\nGet the list of reporter and partner countries\n\n    \u003e\u003e\u003e comtrade.pump.get_parameter_list(\"reporterAreas.json\")\n    \u003e\u003e\u003e comtrade.pump.get_parameter_list(\"partnerAreas.json\")\n\n\n\n# Metadata and configuration data\n\n\n## Release dates\n\nFAOSTAT release dates are available at :\nhttps://fenixservices.fao.org/faostat/static/releasecalendar/Default.aspx\n\n\n## Variable definitions and harmonization\n\nColumn names and product descriptions are reformatted to snake case in a way that is\nconvenient for analysis. See example below.\n\n- Variables are defined and compared between the data sources in a notebook called\n  [definitions_and_harmonization](notebooks/definitions_and_harmonization.md)\n\n- Variable names are harmonized between the different sources using a mapping table\n  defined in\n  [biotrade/config_data/column_names.csv](https://gitlab.com/bioeconomy/biotrade/-/blob/main/biotrade/config_data/column_names.csv)\n  See for example how the `product_code` column is called  `PRODUCT_NC` in Eurostat Comext,\n  `commodity_code` or `cmdcode` in UN Comtrade and `item_code` in FAOSTAT.\n\n- `snake_case` is the preferred way of naming files and variables in the code. This\n  follows the R [tidyverse style guide for object\n  names](https://style.tidyverse.org/syntax.html) and the python [PEP\n  8](https://www.python.org/dev/peps/pep-0008/#function-and-variable-names) style guide\n  for function and variable names.\n\nTo illustrate the advantage of using snake case for data exploration, compare the use of\ncolumn names with space which have to be quoted. Python\n\n    \u003e\u003e\u003e df[\"Product Code\"]\n    \u003e\u003e\u003e df.product_code\n\nR\n\n    R\u003e df[\"Product Code\"]\n    R\u003e df$`Product Code`\n    R\u003e df$product_code\n\n\n## Configuration data\n\nThe `biotrade` package stores a small amount of configuration data such as country and\nproduct mapping tables and conversion coefficients in the `biotrade/config_data` folder.\n\n\n\n# Licence\n\nThis software is licenced under the MIT licence.\nSee the [LICENCE.md](LICENCE.md) file.\n\n\n# Similar projects\n\n- The R packages FAOSTAT and WDI\n\n\n# Tests\n\nThis package uses pytest for unit testing. Run the test suite with\n\n    pytest\n\nRun pytest with code coverage\n\n    cd python_project_dir\n    coverage run --source=. -m pytest\n\nFollowed by\n\n    coverage html\n\nTo generate a report.\nThese tests are run as part of the Continuous Integration.\n\n\n# Acknowledgements\n\nThe authors would like to acknowledge ideas and feedback received from the following\npersons: Giovanni Bausano, Noemi Cazzaniga, Mirco Migliavacca, Roberto Pilli, Lucas\nSinclair.\n","funding_links":[],"readme_doi_urls":[],"works":{},"citation_counts":{},"total_citations":0,"keywords_from_contributors":[],"project_url":"https://ost.ecosyste.ms/api/v1/projects/118365","html_url":"https://ost.ecosyste.ms/projects/118365"}