Class: RubyLLM::Tokenization

Inherits:
Object
  • Object
show all
Includes:
Support::Inspectable
Defined in:
lib/ruby_llm/tokenization.rb

Overview

The token IDs produced by a model's tokenizer for plain text. These exclude chat formatting, tools, and media, and do not measure billable generation usage.

Constant Summary

Constants included from Support::Inspectable

Support::Inspectable::TRUNCATE_AT

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from Support::Inspectable

#full_inspect, #inspect, #pretty_print

Constructor Details

#initialize(ids:, model:, raw: nil) ⇒ Tokenization

:nodoc:



44
45
46
47
48
 
# File 'lib/ruby_llm/tokenization.rb', line 44

def initialize(ids:, model:, raw: nil) # :nodoc:
  @ids = ids.dup.freeze
  @model = model
  @raw = raw
end

Instance Attribute Details

#ids ⇒ Object (readonly)

The integer token IDs in text order.



11
12
13
 
# File 'lib/ruby_llm/tokenization.rb', line 11

def ids
  @ids
end

#model ⇒ Object (readonly)

The id of the model whose tokenizer was used.



14
15
16
 
# File 'lib/ruby_llm/tokenization.rb', line 14

def model
  @model
end

#raw ⇒ Object (readonly)

The provider's unmodified response, including token strings when available.



17
18
19
 
# File 'lib/ruby_llm/tokenization.rb', line 17

def raw
  @raw
end

Class Method Details

.tokenize(text, model: nil, provider: nil, assume_model_exists: false, context: nil) ⇒ Object

Tokenizes text and returns a Tokenization. Most code calls RubyLLM.tokenize. model: defaults to the configured chat model; provider: selects its provider. assume_model_exists: skips registry lookup, and context: supplies an isolated configuration.

result = RubyLLM.tokenize("Hello Ruby", model: "grok-4.3", provider: :xai)
result.ids
result.count

Raises ArgumentError for non-string input and RubyLLM::Error when the provider does not expose a tokenizer.

Raises:

  • (ArgumentError) 


30
31
32
33
34
35
36
37
38
39
40
41
42
 
# File 'lib/ruby_llm/tokenization.rb', line 30

def self.tokenize(text, model: nil, provider: nil, assume_model_exists: false, context: nil)
  raise ArgumentError, 'text must be a String' unless text.is_a?(String)

  config = context&.config || RubyLLM.config
  model ||= config.default_model
  model, provider_instance = Models.resolve(model, provider:, assume_model_exists:, config:)
  payload = { model: model.id, provider: provider_instance.slug }
  RubyLLM.instrument('tokenization.ruby_llm', payload, config:) do |event|
    result = provider_instance.tokenize(text, model:)
    event[:result] = result
    result
  end
end

Instance Method Details

#count ⇒ Object

Returns the number of tokens in the text.



51
52
53
 
# File 'lib/ruby_llm/tokenization.rb', line 51

def count
  ids.length
end

#inspect_attributes ⇒ Object

:nodoc:



55
56
57
 
# File 'lib/ruby_llm/tokenization.rb', line 55

def inspect_attributes # :nodoc:
  { model:, count: }
end