Class: Informers::BertTokenizer

Inherits:

PreTrainedTokenizer

• Object
• PreTrainedTokenizer
• Informers::BertTokenizer

Defined in:

lib/informers/tokenizers.rb

Instance Attribute Summary

Attributes inherited from PreTrainedTokenizer

#sep_token_id

Method Summary

Methods inherited from PreTrainedTokenizer

#call, #convert_tokens_to_string, #decode, #initialize

Constructor Details

This class inherits a constructor from Informers::PreTrainedTokenizer