https://github.com/alasdairforsythe/tokenmonster
Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript
https://github.com/alasdairforsythe/tokenmonster
Keywords
text-tokenization tokenisation tokenization tokenize tokenizer tokenizing vocabulary vocabulary-builder vocabulary-generator
Last synced: 11 months ago
JSON representation
Acceptance Criteria
- Revelant topics? true
- External users? true
- Open source license? true
- Active? true
- Fork? false
Repository metadata
Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript
- Host: GitHub
- URL: https://github.com/alasdairforsythe/tokenmonster
- Owner: alasdairforsythe
- License: mit
- Created: 2023-05-12T04:58:39.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-01-28T20:55:34.000Z (over 1 year ago)
- Last Synced: 2024-06-03T17:29:04.562Z (11 months ago)
- Topics: text-tokenization, tokenisation, tokenization, tokenize, tokenizer, tokenizing, vocabulary, vocabulary-builder, vocabulary-generator
- Language: Go
- Homepage:
- Size: 734 KB
- Stars: 515
- Watchers: 10
- Forks: 20
- Open Issues: 11
- Releases: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Owner metadata
- Name:
- Login: alasdairforsythe
- Email:
- Kind: user
- Description:
- Website:
- Location:
- Twitter:
- Company:
- Icon url: https://avatars.githubusercontent.com/u/77910352?v=4
- Repositories: 1
- Last ynced at: 2023-05-11T10:29:01.526Z
- Profile URL: https://github.com/alasdairforsythe
GitHub Events
Total
- Create event: 2
- Commit comment event: 1
- Issues event: 41
- Watch event: 496
- Issue comment event: 42
- Push event: 152
- Pull request event: 4
- Fork event: 17
Last Year
- Commit comment event: 1
- Fork event: 15
- Issue comment event: 32
- Issues event: 32
- Pull request event: 4
- Push event: 99
- Watch event: 336
Committers metadata
Last synced: 11 months ago
Total Commits: 168
Total Committers: 1
Avg Commits per committer: 168.0
Development Distribution Score (DDS): 0.0
Commits in past year: 117
Committers in past year: 1
Avg Commits per committer in past year: 117.0
Development Distribution Score (DDS) in past year: 0.0
| Name | Commits | |
|---|---|---|
| Alasdair | 7****e | 168 |
Committer domains:
Issue and Pull Request metadata
Last synced: 11 months ago
Total issues: 25
Total pull requests: 3
Average time to close issues: 19 days
Average time to close pull requests: about 3 hours
Total issue authors: 20
Total pull request authors: 3
Average comments per issue: 1.8
Average comments per pull request: 0.33
Merged pull request: 0
Bot issues: 0
Bot pull requests: 0
Past year issues: 22
Past year pull requests: 3
Past year average time to close issues: 12 days
Past year average time to close pull requests: about 3 hours
Past year issue authors: 18
Past year pull request authors: 3
Past year average comments per issue: 1.95
Past year average comments per pull request: 0.33
Past year merged pull request: 0
Past year bot issues: 0
Past year bot pull requests: 0
Top Issue Authors
- kerighan (3)
- kyegomez (2)
- Calvinnncy97 (2)
- ianderrington (2)
- worstpractice (1)
- Maxscha (1)
- kosiakk (1)
- konstantinjdobler (1)
- JorgeCepeda (1)
- gautierdag (1)
- enpassanty (1)
- ElleLeonne (1)
- dsdanielpark (1)
- botsbreeder (1)
- BlinkDL (1)
Top Pull Request Authors
- codinglover0111 (1)
- vovw (1)
- amazingvince (1)
Top Issue Labels
Top Pull Request Labels
Package metadata
- Total packages: 2
-
Total downloads:
- pypi: 885 last-month
- Total dependent packages: 2 (may contain duplicates)
- Total dependent repositories: 1 (may contain duplicates)
- Total versions: 18
- Total maintainers: 1
proxy.golang.org: github.com/alasdairforsythe/tokenmonster
- Homepage: https://github.com/alasdairforsythe/tokenmonster
- Documentation: https://pkg.go.dev/github.com/alasdairforsythe/tokenmonster#section-documentation
- Licenses: MIT
- Latest release: v0.0.0-20231115032503-8d32435658a8 (published over 1 year ago)
- Last Synced: 2024-06-03T17:38:47.035Z (11 months ago)
- Versions: 3
- Dependent Packages: 0
- Dependent Repositories: 0
-
Rankings:
- Stargazers count: 3.084%
- Forks count: 6.465%
- Average: 7.254%
- Dependent packages count: 8.899%
- Dependent repos count: 10.567%
pypi.org: tokenmonster
Tokenize and decode text with TokenMonster vocabularies.
- Homepage: https://github.com/alasdairforsythe/tokenmonster
- Documentation: https://tokenmonster.readthedocs.io/
- Licenses: MIT
- Latest release: 1.1.12 (published over 1 year ago)
- Last Synced: 2024-06-03T17:38:44.660Z (11 months ago)
- Versions: 15
- Dependent Packages: 2
- Dependent Repositories: 1
- Downloads: 885 Last month
-
Rankings:
- Stargazers count: 3.228%
- Dependent packages count: 4.736%
- Forks count: 9.566%
- Downloads: 9.589%
- Average: 9.755%
- Dependent repos count: 21.657%
- Maintainers (1)
Score: 13.055398448251616