Module: SwarmSDK::Defaults::TokenEstimation

Defined in:

lib/swarm_sdk/defaults.rb

Overview

Token estimation factors

Used for approximate token counting when precise counts aren't available.

Constant Summary

CHARS_PER_TOKEN_PROSE =

Characters per token for prose text

Average of ~4 characters per token for natural language text. Based on empirical analysis of tokenization patterns.

4.0

CHARS_PER_TOKEN_CODE =

Characters per token for code

Code tends to have shorter tokens due to symbols and operators. ~3.5 characters per token accounts for this density.

3.5