Wed, 18 Nov 2026 18:48:59 UTC Log in
Implementation of today's most used tokenizers in Rust 219.40 KB built 2026-10-21 11:40:41 GMT

Description

Provides an implementation of today's most used tokenizers, with a focus on performances and versatility.

Dependencies

52 total
Requires42
  • (crate(ahash/default) >= 0.8.11 with crate(ahash/default) < 0.9.0~)
  • (crate(ahash/serde) >= 0.8.11 with crate(ahash/serde) < 0.9.0~)
  • (crate(assert_approx_eq/default) >= 1.1.0 with crate(assert_approx_eq/default) < 2.0.0~)
  • (crate(compact_str/default) >= 0.9.0 with crate(compact_str/default) < 0.10.0~)
  • (crate(compact_str/serde) >= 0.9.0 with crate(compact_str/serde) < 0.10.0~)
  • (crate(daachorse/default) >= 1.0.1 with crate(daachorse/default) < 2.0.0~)
  • (crate(dary_heap/default) >= 0.3.6 with crate(dary_heap/default) < 0.4.0~)
  • (crate(dary_heap/serde) >= 0.3.6 with crate(dary_heap/serde) < 0.4.0~)
  • (crate(derive_builder/default) >= 0.20.0 with crate(derive_builder/default) < 0.21.0~)
  • (crate(esaxx-rs) >= 0.1.10 with crate(esaxx-rs) < 0.2.0~)
  • (crate(esaxx-rs/cpp) >= 0.1.10 with crate(esaxx-rs/cpp) < 0.2.0~)
  • (crate(getrandom/default) >= 0.3.0 with crate(getrandom/default) < 0.4.0~)
  • (crate(indicatif/default) >= 0.18.0 with crate(indicatif/default) < 0.19.0~)
  • (crate(itertools/default) >= 0.14.0 with crate(itertools/default) < 0.15.0~)
  • (crate(log/default) >= 0.4.0 with crate(log/default) < 0.5.0~)
  • (crate(macro_rules_attribute/default) >= 0.2.0 with crate(macro_rules_attribute/default) < 0.3.0~)
  • (crate(monostate/default) >= 1.0.0 with crate(monostate/default) < 2.0.0~)
  • (crate(onig) >= 6.5.1 with crate(onig) < 7.0.0~)
  • (crate(paste/default) >= 1.0.14 with crate(paste/default) < 2.0.0~)
  • (crate(rand/default) >= 0.9.0 with crate(rand/default) < 0.10.0~)
  • (crate(rayon-cond/default) >= 0.4.0 with crate(rayon-cond/default) < 0.5.0~)
  • (crate(rayon/default) >= 1.10.0 with crate(rayon/default) < 2.0.0~)
  • (crate(regex-syntax/default) >= 0.8.0 with crate(regex-syntax/default) < 0.9.0~)
  • (crate(regex/default) >= 1.10.0 with crate(regex/default) < 2.0.0~)
  • (crate(serde/default) >= 1.0.0 with crate(serde/default) < 2.0.0~)
  • (crate(serde/derive) >= 1.0.0 with crate(serde/derive) < 2.0.0~)
  • (crate(serde_json/default) >= 1.0.0 with crate(serde_json/default) < 2.0.0~)
  • (crate(spm_precompiled/default) >= 0.1.3 with crate(spm_precompiled/default) < 0.2.0~)
  • (crate(tempfile/default) >= 3.10.0 with crate(tempfile/default) < 4.0.0~)
  • (crate(thiserror/default) >= 2.0.0 with crate(thiserror/default) < 3.0.0~)
  • (crate(tracing-subscriber/default) >= 0.3.18 with crate(tracing-subscriber/default) < 0.4.0~)
  • (crate(tracing/default) >= 0.1.0 with crate(tracing/default) < 0.2.0~)
  • (crate(unicode-normalization-alignments/default) >= 0.1.0 with crate(unicode-normalization-alignments/default) < 0.2.0~)
  • (crate(unicode-segmentation/default) >= 1.11.0 with crate(unicode-segmentation/default) < 2.0.0~)
  • (crate(unicode_categories/default) >= 0.1.0 with crate(unicode_categories/default) < 0.2.0~)
  • cargo-rpm-macros >= 24
  • rpmlib(CompressedFileNames) <= 3.0.4-1
  • rpmlib(DynamicBuildRequires) <= 4.15.0-1
  • rpmlib(FileDigests) <= 4.6.0-1
  • rpmlib(RichDependencies) <= 4.12.0-1
  • rpmlib(TildeInVersions) <= 4.10.0-1
  • rust >= 1.31
Provides10
  • rust-tokenizers+default-devel = 0.23.1-1.fc45
  • rust-tokenizers+esaxx_fast-devel = 0.23.1-1.fc45
  • rust-tokenizers+fancy-regex-devel = 0.23.1-1.fc45
  • rust-tokenizers+hf-hub-devel = 0.23.1-1.fc45
  • rust-tokenizers+http-devel = 0.23.1-1.fc45
  • rust-tokenizers+indicatif-devel = 0.23.1-1.fc45
  • rust-tokenizers+onig-devel = 0.23.1-1.fc45
  • rust-tokenizers+progressbar-devel = 0.23.1-1.fc45
  • rust-tokenizers+rustls-tls-devel = 0.23.1-1.fc45
  • rust-tokenizers-devel = 0.23.1-1.fc45
ObsoletesNo Obsoletes
ConflictsNo Conflicts
RecommendsNo Recommends
SuggestsNo Suggests
SupplementsNo Supplements
EnhancesNo Enhances

Files

4
Files 1–4 of 4

Component of

buildroots that installed this RPM

No Buildroots