Module: Tokenizers
- Extended by:
- FromPretrained
- Defined in:
- lib/tokenizers.rb,
lib/tokenizers/version.rb,
lib/tokenizers/encoding.rb,
lib/tokenizers/tokenizer.rb,
lib/tokenizers/from_pretrained.rb,
lib/tokenizers/char_bpe_tokenizer.rb
Defined Under Namespace
Modules: FromPretrained Classes: CharBPETokenizer, Encoding, Error, Tokenizer
Constant Summary collapse
- VERSION =
"0.2.3"
Constants included from FromPretrained
FromPretrained::TOKENIZERS_VERSION