Module: RubyLLM::Protocols::XAI::Tokenization

Included in:
RubyLLM::Providers::XAI::ChatCompletions, RubyLLM::Providers::XAI::Responses
Defined in:
lib/ruby_llm/protocols/xai/tokenization.rb

Overview

xAI's plain-text tokenizer.

Instance Method Summary collapse

Instance Method Details

#parse_tokenization_response(response, model:) ⇒ Object 



16
17
18
19
 
# File 'lib/ruby_llm/protocols/xai/tokenization.rb', line 16

def parse_tokenization_response(response, model:)
  ids = response.body.fetch('token_ids').map { |token| token.fetch('token_id') }
  RubyLLM::Tokenization.new(ids:, model:, raw: response.body)
end

#render_tokenization_payload(text, model:) ⇒ Object 



12
13
14
 
# File 'lib/ruby_llm/protocols/xai/tokenization.rb', line 12

def render_tokenization_payload(text, model:)
  { model:, text: }
end

#tokenization_url ⇒ Object 



8
9
10
 
# File 'lib/ruby_llm/protocols/xai/tokenization.rb', line 8

def tokenization_url
  'tokenize-text'
end