Class: DiscourseAi::Tokenizer::OpenAiCl100kTokenizer

Inherits:

OpenAiTokenizer

Object
BasicTokenizer
OpenAiTokenizer
DiscourseAi::Tokenizer::OpenAiCl100kTokenizer

show all

Defined in:: lib/discourse_ai/tokenizer/open_ai_cl100k_tokenizer.rb

Overview

Wrapper for OpenAI tokenizer library for compatibility with Discourse AI API

Class Method Summary collapse

.tokenizer ⇒ Object

Methods inherited from OpenAiTokenizer

below_limit?, decode, encode, tokenize, truncate

Methods inherited from BasicTokenizer

available_llm_tokenizers, below_limit?, decode, encode, size, tokenize, truncate

Class Method Details

.tokenizer ⇒ `Object`



8
9
10

# File 'lib/discourse_ai/tokenizer/open_ai_cl100k_tokenizer.rb', line 8

def tokenizer
  @tokenizer ||= Tiktoken.get_encoding("cl100k_base")
end