Class: SwarmSDK::ContextCompactor::TokenCounter

Inherits:
Object
  • Object
show all
Defined in:
lib/swarm_sdk/context_compactor/token_counter.rb

Overview

TokenCounter provides token estimation for messages

This uses a simple heuristic approach:

  • ~4 characters per token for English prose
  • ~3.5 characters per token for code

For production use with OpenAI models, consider using the tiktoken gem for accurate token counting. For Claude models, use Claude's token API.

Usage

tokens = TokenCounter.estimate_message(message)
total_tokens = TokenCounter.estimate_messages(messages)

Class Method Summary collapse

Class Method Details

.estimate_content(content) ⇒ Integer

Estimate tokens for content string

Uses heuristic to detect code vs prose and adjust accordingly.

Parameters:

  • content (String, RubyLLM::Content, nil) —

    Content to estimate

Returns:

  • (Integer) —

    Estimated token count



60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
 
# File 'lib/swarm_sdk/context_compactor/token_counter.rb', line 60

def estimate_content(content)
  return 0 if content.nil?

  # Handle RubyLLM::Content objects
  text = if content.respond_to?(:to_s)
    content.to_s
  else
    content
  end

  return 0 if text.empty?

  # Detect if content is mostly code
  code_ratio = detect_code_ratio(text)

  # Choose characters per token based on content type
  chars_per_token = if code_ratio > 0.1
    SwarmSDK.config.chars_per_token_code # Code
  else
    SwarmSDK.config.chars_per_token_prose # Prose
  end

  (text.length / chars_per_token).ceil
end

.estimate_message(message) ⇒ Integer

Estimate tokens for a single message

Parameters:

  • message (RubyLLM::Message) —

    Message to estimate

Returns:

  • (Integer) —

    Estimated token count



25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
 
# File 'lib/swarm_sdk/context_compactor/token_counter.rb', line 25

def estimate_message(message)
  case message.role
  when :user, :assistant
    estimate_content(message.content)
  when :system
    estimate_content(message.content)
  when :tool
    # Tool results typically have overhead
    base_overhead = 50
    content_tokens = estimate_content(message.content)
    base_overhead + content_tokens
  else
    # Unknown message type
    begin
      estimate_content(message.content)
    rescue
      0
    end
  end
end

.estimate_messages(messages) ⇒ Integer

Estimate tokens for multiple messages

Parameters:

  • messages (Array<RubyLLM::Message>) —

    Messages to estimate

Returns:

  • (Integer) —

    Total estimated token count



50
51
52
 
# File 'lib/swarm_sdk/context_compactor/token_counter.rb', line 50

def estimate_messages(messages)
  messages.sum { |msg| estimate_message(msg) }
end