Class: Informers::DebertaV2Tokenizer

Inherits:

PreTrainedTokenizer

• Object
• PreTrainedTokenizer
• Informers::DebertaV2Tokenizer

Defined in:

lib/informers/tokenizers.rb

Instance Attribute Summary

Attributes inherited from PreTrainedTokenizer

#sep_token_id

Method Summary

Methods inherited from PreTrainedTokenizer

#call, #convert_tokens_to_string, #decode, #initialize

Constructor Details

This class inherits a constructor from Informers::PreTrainedTokenizer