Module: RubyLLM::Protocols::Cohere::Tokenization

Defined in:
lib/ruby_llm/protocols/cohere/tokenization.rb

Overview

Cohere's plain-text tokenizer is served by the v1 API.

Instance Method Summary collapse

Instance Method Details

#parse_tokenization_response(response, model:) ⇒ Object 



16
17
18
 
# File 'lib/ruby_llm/protocols/cohere/tokenization.rb', line 16

def parse_tokenization_response(response, model:)
  RubyLLM::Tokenization.new(ids: response.body.fetch('tokens'), model:, raw: response.body)
end

#render_tokenization_payload(text, model:) ⇒ Object 



12
13
14
 
# File 'lib/ruby_llm/protocols/cohere/tokenization.rb', line 12

def render_tokenization_payload(text, model:)
  { model:, text: }
end

#tokenization_url ⇒ Object 



8
9
10
 
# File 'lib/ruby_llm/protocols/cohere/tokenization.rb', line 8

def tokenization_url
  'v1/tokenize'
end