Class: Kotoshu::Server::App

Inherits:
Sinatra::Base
  • Object
show all
Defined in:
lib/kotoshu/server/app.rb

Constant Summary collapse

VERSION =

The release version, from lib/kotoshu/server/version.rb — the only source of truth (the release workflow bumps it).

Kotoshu::Server::VERSION
MODEL_MIN_KOTOSHU =

Semantic models (tiers, registry, confidence cascade) ship in kotoshu 0.7.0. The gemspec keeps its kotoshu ~> 0.6 constraint (dependency floors are the owner's call), so the server validates the installed gem at boot instead.

Gem::Version.new("0.7.0")
DETECT_MIN_KOTOSHU =

Native language identification (plan 106) ships in kotoshu 0.10.0: Kotoshu.detect_language returning a Language::Detection backed by the lid-176 model through the native extension. Same policy as MODEL_MIN_KOTOSHU — the gemspec floor stays kotoshu ~> 0.6 (older gems keep the heuristic), so the installed gem is checked per request.

Gem::Version.new("0.10.0")

Class Method Summary collapse

Class Method Details

.configured_languages ⇒ Array<String>

Languages to set up at boot, from KOTOSHU_SERVER_LANGUAGES (space separated). Single source of truth for the env var: the boot pre-warm and /v1/health both read it here.

Returns:

  • Language codes to pre-warm



38
39
40
 
# File 'lib/kotoshu/server/app.rb', line 38

def self.configured_languages
  ENV.fetch("KOTOSHU_SERVER_LANGUAGES", "en").split
end

.detect_language_with_engine(text) ⇒ Array(String, Float, String)

Detect for /v1/detect: the language code, its score, and the engine that served — "lid-176" or "heuristic".

Engine selection, in order:

  1. KOTOSHU_DETECT=heuristic — the heuristic, pinned.
  2. kotoshu < 0.10.0 — the heuristic, as in 0.1.1; the server keeps working on old gems.
  3. Otherwise the gem's own selection: lid-176 through the native extension when it is built, the backend is not explicitly ruby, and the artifact pair is cached; the heuristic otherwise (KOTOSHU_BACKEND=ruby, pure-Ruby install, missing model).

The lid model is set up lazily on the first detect reaching case 3 — one download, off the boot path, honoring KOTOSHU_OFFLINE through the gem; a failed setup (offline with no cache, no registry entry, checksum mismatch) logs once and detection answers from whatever the gem can serve.

Parameters:

  • the text to analyze

Returns:

  • code (nil when the heuristic is uncertain), score in [0, 1], engine name



206
207
208
209
210
211
212
 
# File 'lib/kotoshu/server/app.rb', line 206

def self.detect_language_with_engine(text)
  return heuristic_detection(text) if heuristic_detect? || !lid_supported?

  ensure_lid_setup!
  detection = Kotoshu.detect_language(text)
  [detection.code, detection.score, lid_engine]
end

.ensure_lid_setup! ⇒ void

This method returns an undefined value.

Run Kotoshu.setup_lid once per process, on the first detect. Idempotent in the gem; the once-guard keeps concurrent requests from racing the download. Any failure is logged and swallowed — detect then falls back inside the gem to whatever is cached (typically the heuristic).



241
242
243
244
245
246
247
248
249
250
251
252
 
# File 'lib/kotoshu/server/app.rb', line 241

def self.ensure_lid_setup!
  @lid_setup_mutex.synchronize do
    return if @lid_setup_attempted

    @lid_setup_attempted = true
    Kotoshu.setup_lid
  end
rescue StandardError => e
  Logger.new($stderr).warn(
    "lid model setup failed, falling back to the heuristic: #{e.class}: #{e.message}"
  )
end

.heuristic_detect? ⇒ Boolean

Whether KOTOSHU_DETECT=heuristic pins /v1/detect to the 7-language heuristic, bypassing the gem engine selection (the 0.1.1 behavior, e.g. to reproduce earlier results).

Returns:



180
181
182
 
# File 'lib/kotoshu/server/app.rb', line 180

def self.heuristic_detect?
  ENV.fetch("KOTOSHU_DETECT", nil) == "heuristic"
end

.heuristic_detection(text) ⇒ Array(String, Float, String)

The heuristic detection (Language::Detector, the same engine /v1/detect served in 0.1.1, unchanged across gem versions).

Parameters:

Returns:



219
220
221
222
 
# File 'lib/kotoshu/server/app.rb', line 219

def self.heuristic_detection(text)
  code, confidence = Kotoshu::Language::Detector.detect_with_confidence(text)
  [code, confidence, "heuristic"]
end

.lid_engine ⇒ String

Which engine the gem serves detect from: lid-176 when the native model is loadable, the heuristic otherwise. Asked after a detect, so it always names the engine behind the returned code.

Returns:



230
231
232
 
# File 'lib/kotoshu/server/app.rb', line 230

def self.lid_engine
  Kotoshu::Language::LidDetector.available? ? "lid-176" : "heuristic"
end

.lid_supported? ⇒ Boolean

Whether the installed kotoshu gem carries the lid-176 detection surface (0.10.0+): Kotoshu.detect_language returning a Language::Detection, plus setup_lid / LidDetector.available?.

Returns:



171
172
173
 
# File 'lib/kotoshu/server/app.rb', line 171

def self.lid_supported?
  Gem::Version.new(Kotoshu::VERSION) >= DETECT_MIN_KOTOSHU
end

.model_available?(language) ⇒ Boolean

Whether a semantic model is actually set up server-side for a language (cache-only lookup; false on gems without model support, so the /v1/check default flag never turns models on underneath an old install).

Parameters:

  • Language code

Returns:



76
77
78
79
80
81
82
 
# File 'lib/kotoshu/server/app.rb', line 76

def self.model_available?(language)
  return false unless semantic_models_supported?

  Kotoshu.setup?(language.to_sym, :model)
rescue StandardError
  false
end

.model_languages ⇒ Array<String>

Languages to set up with spelling + semantic model at boot, from KOTOSHU_SERVER_MODEL_LANGS (space separated, e.g. "en de"). Empty when unset — boot-time opt-in only, never an implicit download (the gem's two-stage promise).

Returns:

  • Language codes to set up with a model



48
49
50
 
# File 'lib/kotoshu/server/app.rb', line 48

def self.model_languages
  ENV.fetch("KOTOSHU_SERVER_MODEL_LANGS", "").split
end

.model_tier ⇒ String

Model tier used for setup and resolution, from KOTOSHU_SERVER_MODEL_TIER. Defaults to "fluency", the ecosystem default (owner decision 2026-09-04).

Returns:

  • "full", "fluency", or "mini"



57
58
59
 
# File 'lib/kotoshu/server/app.rb', line 57

def self.model_tier
  ENV.fetch("KOTOSHU_SERVER_MODEL_TIER", "fluency")
end

.prewarm!(languages) ⇒ void

This method returns an undefined value.

Synchronously set up the given languages (downloads on a cold or expired cache). Runs on the pre-warm thread; also usable directly by embedders that want a blocking warm-up.

Languages listed in KOTOSHU_SERVER_MODEL_LANGS are set up with spelling + model (at KOTOSHU_SERVER_MODEL_TIER) in one setup call; the rest get spelling only, as before. Both lists are unioned, and setup is idempotent, so a language in both is set up once with the model.

Parameters:

  • Language codes



123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
 
# File 'lib/kotoshu/server/app.rb', line 123

def self.prewarm!(languages)
  logger = Logger.new($stderr)
  model_langs = model_languages
  tier = model_tier
  (languages | model_langs).each do |lang|
    with_model = model_langs.include?(lang)
    logger.info("pre-warming #{lang}#{with_model ? " (spelling + model, #{tier} tier)" : ""}")
    begin
      if with_model
        Kotoshu.setup(lang.to_sym, want: %i[spelling model], tier: tier)
      else
        Kotoshu.setup(lang.to_sym)
      end
      logger.info("pre-warm #{lang} complete")
    rescue StandardError => e
      logger.warn("pre-warm #{lang} failed: #{e.message}")
    end
  end
end

.prewarm_async!(languages) ⇒ Thread

Start pre-warming in a detached background thread and return immediately. The server must bind and serve within seconds of boot, while resource setup can block on the network (download retries, and DNS resolution that no Net::HTTP timeout bounds) — so setup never runs on the boot path. Progress and completion are logged; /v1/health reports per-language readiness while it runs.

Parameters:

  • Language codes

Returns:

  • the detached pre-warm thread



153
154
155
156
157
158
159
 
# File 'lib/kotoshu/server/app.rb', line 153

def self.prewarm_async!(languages)
  validate_model_config! # fail fast, on the caller's thread
  Thread.new do
    Thread.current.name = "kotoshu-server-prewarm"
    prewarm!(languages)
  end
end

.semantic_analyzer_for(language, model_info) ⇒ Kotoshu::Analyzers::SemanticAnalyzer

A memoized Analyzers::SemanticAnalyzer for the model file the ResourceManager resolved for language. Loading a model (vocab + ONNX session) is expensive; one analyzer per language is kept for the process lifetime. A changed model file (different path) loads a fresh analyzer.

Parameters:

  • Language code

  • bundle.model from ResourceManager (needs :model_path)

Returns:



269
270
271
272
273
274
275
276
277
 
# File 'lib/kotoshu/server/app.rb', line 269

def self.semantic_analyzer_for(language, model_info)
  path = model_info[:model_path]
  @semantic_analyzers_mutex.synchronize do
    @semantic_analyzers[[language.to_s, path]] ||= begin
      model = Kotoshu::Models::OnnxModel.from_file(path, language_code: language.to_s)
      Kotoshu::Analyzers::SemanticAnalyzer.new(model)
    end
  end
end

.semantic_models_supported? ⇒ Boolean

Whether the installed kotoshu gem carries the semantic-model surface the server wires against (0.7.0+).

Returns:



65
66
67
 
# File 'lib/kotoshu/server/app.rb', line 65

def self.semantic_models_supported?
  Gem::Version.new(Kotoshu::VERSION) >= MODEL_MIN_KOTOSHU
end

.validate_model_config! ⇒ void

This method returns an undefined value.

Validate the semantic-model environment before anything is set up. Raises ModelConfigError with an actionable message when KOTOSHU_SERVER_MODEL_LANGS is set but the installed kotoshu gem predates model support, or the configured tier is unknown. A no-op when the vars are unset — boot behavior is then exactly today's.

Raises:

  • on an unsatisfiable configuration



93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
 
# File 'lib/kotoshu/server/app.rb', line 93

def self.validate_model_config!
  return if model_languages.empty?

  unless semantic_models_supported?
    raise ModelConfigError,
          "KOTOSHU_SERVER_MODEL_LANGS requires kotoshu >= 0.7.0 " \
          "(installed: #{Kotoshu::VERSION}). Semantic models ship in " \
          "kotoshu 0.7.0; unset KOTOSHU_SERVER_MODEL_LANGS or upgrade " \
          "the kotoshu gem."
  end

  begin
    Kotoshu::Cache::ModelCache.normalize_tier(model_tier)
  rescue ArgumentError => e
    raise ModelConfigError, "KOTOSHU_SERVER_MODEL_TIER: #{e.message}"
  end
end