Implementation of today's most used tokenizers
built by guoguo
completed Thu, 29 Oct 2026 23:33:20 UTC
Description
Provides an implementation of today's most used tokenizers,
with a focus on performance and versatility.
Bindings over the rust-tokenizers implementation.
RPMs
No RPMs
Changelog
* Sat Jul 25 2026 Benjamin A. Beasley <code@musicinmybrain.net> - 0.23.1-1 - Update to 0.23.1 and patch for PyO3 0.29 - PyO3 0.29 fixes RUSTSEC-2026-0176 and RUSTSEC-2026-0177. * Wed Jul 22 2026 Python Maint <python-maint@redhat.com> - 0.22.2-5 - Rebuilt for Python 3.15.0b4 ABI change * Thu Jul 16 2026 Fedora Release Engineering <releng@fedoraproject.org> - 0.22.2-4 - Rebuilt for https://fedoraproject.org/wiki/Fedora_45_Mass_Rebuild * Mon Jun 08 2026 Python Maint <python-maint@redhat.com> - 0.22.2-3 - Rebuilt for Python 3.15 * Thu Apr 09 2026 Benjamin A. Beasley <code@musicinmybrain.net> - 0.22.2-2 - Conditionalize datasets test dependency for F42/F43 * Thu Feb 12 2026 Alexander F. Lent <lx@xanderlent.com> - 0.22.2-1 - Initial import of 0.22.2, fixes rhbz#2388154