An unsupervised text tokenizer for Neural Network-based text generation
8.60 MB
built 2026-10-17 22:51:41 GMT
Description
The SentencePiece is an unsupervised text tokenizer for Neural Network-based
text generation.
It is an unsupervised text tokenizer and detokenizer mainly for
Neural Network-based text generation systems where the vocabulary size is
predetermined prior to the neural model training.
SentencePiece implements subword units and unigram language model with the
extension of direct training from raw sentences.
SentencePiece allows us to make a purely end-to-end system that does not
depend on language-specific pre/post-processing.
Dependencies
15 totalRequires11
- cmake
- gcc-c++
- gperftools-devel
- ninja-build
- pkgconfig
- protobuf-lite-devel
- python3-devel
- python3-setuptools
- python3dist(pip)
- rpmlib(CompressedFileNames) <= 3.0.4-1
- rpmlib(FileDigests) <= 4.6.0-1
Provides4
- python3-sentencepiece = 0.2.1-4.fc45
- sentencepiece-devel = 0.2.1-4.fc45
- sentencepiece-libs = 0.2.1-4.fc45
- sentencepiece-tools = 0.2.1-4.fc45
ObsoletesNo Obsoletes
ConflictsNo Conflicts
RecommendsNo Recommends
SuggestsNo Suggests
SupplementsNo Supplements
EnhancesNo Enhances
Files
2Name ![]() |
Size |
|---|---|
| sentencepiece-0.2.1.tar.gz | 12.86 MB |
| sentencepiece.spec | 7.20 KB |
Files 1–2 of 2
Component of
buildroots that installed this RPMNo Buildroots
