Module: Tokenizers

Extended by:

FromPretrained

Defined in:

lib/tokenizers.rb,
lib/tokenizers/version.rb,
lib/tokenizers/from_pretrained.rb,
lib/tokenizers/char_bpe_tokenizer.rb

Defined Under Namespace

Modules: FromPretrained Classes: CharBPETokenizer, Error

Constant Summary

VERSION =

"0.2.2"

Constants included from FromPretrained

FromPretrained::TOKENIZERS_VERSION

Method Summary

Methods included from FromPretrained

from_pretrained