Wed, 18 Nov 2026 19:45:34 UTC Log in
An unsupervised text tokenizer for Neural Network-based text generation 8.60 MB built 2026-10-17 22:51:41 GMT

Description

The SentencePiece is an unsupervised text tokenizer for Neural Network-based text generation. It is an unsupervised text tokenizer and detokenizer mainly for Neural Network-based text generation systems where the vocabulary size is predetermined prior to the neural model training. SentencePiece implements subword units and unigram language model with the extension of direct training from raw sentences. SentencePiece allows us to make a purely end-to-end system that does not depend on language-specific pre/post-processing.

Dependencies

15 total
Requires11
  • cmake
  • gcc-c++
  • gperftools-devel
  • ninja-build
  • pkgconfig
  • protobuf-lite-devel
  • python3-devel
  • python3-setuptools
  • python3dist(pip)
  • rpmlib(CompressedFileNames) <= 3.0.4-1
  • rpmlib(FileDigests) <= 4.6.0-1
Provides4
  • python3-sentencepiece = 0.2.1-4.fc45
  • sentencepiece-devel = 0.2.1-4.fc45
  • sentencepiece-libs = 0.2.1-4.fc45
  • sentencepiece-tools = 0.2.1-4.fc45
ObsoletesNo Obsoletes
ConflictsNo Conflicts
RecommendsNo Recommends
SuggestsNo Suggests
SupplementsNo Supplements
EnhancesNo Enhances

Files

2
Files 1–2 of 2

Component of

buildroots that installed this RPM

No Buildroots