Implementation of today's most used tokenizers in Rust
159.30 KB
built 2026-10-21 11:40:41 GMT
Description
Provides an implementation of today's most used tokenizers, with a focus
on performances and versatility.
This package contains library source intended for building other packages which
use the "tokenizers" crate.
Dependencies
38 totalRequires35
- (crate(ahash/default) >= 0.8.11 with crate(ahash/default) < 0.9.0~)
- (crate(ahash/serde) >= 0.8.11 with crate(ahash/serde) < 0.9.0~)
- (crate(compact_str/default) >= 0.9.0 with crate(compact_str/default) < 0.10.0~)
- (crate(compact_str/serde) >= 0.9.0 with crate(compact_str/serde) < 0.10.0~)
- (crate(daachorse/default) >= 1.0.1 with crate(daachorse/default) < 2.0.0~)
- (crate(dary_heap/default) >= 0.3.6 with crate(dary_heap/default) < 0.4.0~)
- (crate(dary_heap/serde) >= 0.3.6 with crate(dary_heap/serde) < 0.4.0~)
- (crate(derive_builder/default) >= 0.20.0 with crate(derive_builder/default) < 0.21.0~)
- (crate(esaxx-rs) >= 0.1.10 with crate(esaxx-rs) < 0.2.0~)
- (crate(getrandom/default) >= 0.3.0 with crate(getrandom/default) < 0.4.0~)
- (crate(itertools/default) >= 0.14.0 with crate(itertools/default) < 0.15.0~)
- (crate(log/default) >= 0.4.0 with crate(log/default) < 0.5.0~)
- (crate(macro_rules_attribute/default) >= 0.2.0 with crate(macro_rules_attribute/default) < 0.3.0~)
- (crate(monostate/default) >= 1.0.0 with crate(monostate/default) < 2.0.0~)
- (crate(paste/default) >= 1.0.14 with crate(paste/default) < 2.0.0~)
- (crate(rand/default) >= 0.9.0 with crate(rand/default) < 0.10.0~)
- (crate(rayon-cond/default) >= 0.4.0 with crate(rayon-cond/default) < 0.5.0~)
- (crate(rayon/default) >= 1.10.0 with crate(rayon/default) < 2.0.0~)
- (crate(regex-syntax/default) >= 0.8.0 with crate(regex-syntax/default) < 0.9.0~)
- (crate(regex/default) >= 1.10.0 with crate(regex/default) < 2.0.0~)
- (crate(serde/default) >= 1.0.0 with crate(serde/default) < 2.0.0~)
- (crate(serde/derive) >= 1.0.0 with crate(serde/derive) < 2.0.0~)
- (crate(serde_json/default) >= 1.0.0 with crate(serde_json/default) < 2.0.0~)
- (crate(spm_precompiled/default) >= 0.1.3 with crate(spm_precompiled/default) < 0.2.0~)
- (crate(thiserror/default) >= 2.0.0 with crate(thiserror/default) < 3.0.0~)
- (crate(unicode-normalization-alignments/default) >= 0.1.0 with crate(unicode-normalization-alignments/default) < 0.2.0~)
- (crate(unicode-segmentation/default) >= 1.11.0 with crate(unicode-segmentation/default) < 2.0.0~)
- (crate(unicode_categories/default) >= 0.1.0 with crate(unicode_categories/default) < 0.2.0~)
- cargo
- rpmlib(CompressedFileNames) <= 3.0.4-1
- rpmlib(FileDigests) <= 4.6.0-1
- rpmlib(PayloadFilesHavePrefix) <= 4.0-1
- rpmlib(PayloadIsZstd) <= 5.4.18-1
- rpmlib(RichDependencies) <= 4.12.0-1
- rpmlib(TildeInVersions) <= 4.10.0-1
Provides3
- crate(tokenizers) = 0.23.1
- purl(pkg:cargo/tokenizers@0.23.1)
- rust-tokenizers-devel = 0.23.1-1.fc45
ObsoletesNo Obsoletes
ConflictsNo Conflicts
RecommendsNo Recommends
SuggestsNo Suggests
SupplementsNo Supplements
EnhancesNo Enhances
Files
118
Files 1–50 of 118
Next →
Component of
buildroots that installed this RPMNo Buildroots
