Module: Tokenizers

Defined in:: lib/tokenizers.rb,
lib/tokenizers/version.rb,
lib/tokenizers/char_bpe_tokenizer.rb

Defined Under Namespace

Classes: CharBPETokenizer, Error

Constant Summary collapse

VERSION =

"0.1.2"

Class Method Summary collapse

.from_pretrained(identifier, revision: "main", auth_token: nil) ⇒ Object

Class Method Details

.from_pretrained(identifier, revision: "main", auth_token: nil) ⇒ `Object`



11
12
13

# File 'lib/tokenizers.rb', line 11

def self.from_pretrained(identifier, revision: "main", auth_token: nil)
  _from_pretrained(identifier, revision, auth_token)
end