Module: Aireview::ConfigJev

Included in:
Config
Defined in:
lib/aireview/config_jev.rb

Overview

The critique engine and the Jev (TypeSafe) settings. llm.critique.engine picks who checks the candidates: an LLM (model, the default) or Jev, a fast classifier that decides keep/reject but cannot refine a finding. llm.jev.fallback says what happens when Jev cannot decide: the LLM Critique takes over (model) or the run fails (fail). llm.jev.shadow runs Jev next to the LLM Critique for the log only.

Constant Summary collapse

CRITIQUE_ENGINES = 
%w[model jev].freeze
JEV_FALLBACKS = 
%w[model fail].freeze
DEFAULT_JEV_MODEL =

A pinned version, not an alias: thresholds are tuned against one version, and an alias moves when a release ships.

'jev-1.13.0'
JEV_ALIASES = 
%w[jev-latest jev-preview].freeze
DEFAULT_JEV_TIMEOUT = 
10
JEV_THRESHOLD_DEFAULTS =

Provisional values until real thresholds are chosen from the shadow logs; the logs carry the raw probabilities for that.

{
  'keep_above' => 0.5,
  'enough_context' => 0.5,
  'version_claim' => 0.5,
  'duplicate' => 0.5
}.freeze

Instance Method Summary collapse

Instance Method Details

#critique_disabled? ⇒ Boolean

Returns:

  • (Boolean) 


54
55
56
 
# File 'lib/aireview/config_jev.rb', line 54

def critique_disabled?
  @data['critique_disabled'] == true
end

#critique_engine ⇒ Object 



30
31
32
 
# File 'lib/aireview/config_jev.rb', line 30

def critique_engine
  one_of!(dig('llm', 'critique', 'engine') || 'model', CRITIQUE_ENGINES, 'llm.critique.engine')
end

#jev_api_key ⇒ Object 



70
71
72
 
# File 'lib/aireview/config_jev.rb', line 70

def jev_api_key
  @data['jev_api_key']
end

#jev_critique?(critique: true) ⇒ Boolean

Jev decides in this run: the engine is jev and the run has a critique at all (--no-critique switches off every engine, as the critique: argument or as a CLI override of the config).

Returns:

  • (Boolean) 


41
42
43
 
# File 'lib/aireview/config_jev.rb', line 41

def jev_critique?(critique: true)
  critique && !critique_disabled? && critique_engine == 'jev'
end

#jev_fallback ⇒ Object 



34
35
36
 
# File 'lib/aireview/config_jev.rb', line 34

def jev_fallback
  one_of!(dig('llm', 'jev', 'fallback') || 'model', JEV_FALLBACKS, 'llm.jev.fallback')
end

#jev_model ⇒ Object 



66
67
68
 
# File 'lib/aireview/config_jev.rb', line 66

def jev_model
  Aireview::Utils.presence(dig('llm', 'jev', 'model')) || DEFAULT_JEV_MODEL
end

#jev_shadow? ⇒ Boolean

Returns:

  • (Boolean)

Raises:



58
59
60
61
62
63
64
 
# File 'lib/aireview/config_jev.rb', line 58

def jev_shadow?
  value = dig('llm', 'jev', 'shadow')
  return false if value.nil?
  return value if [true, false].include?(value)

  raise ConfigError, "llm.jev.shadow must be true or false, got #{value.inspect}"
end

#jev_signature ⇒ Object

Everything besides the questions that changes a Jev decision; nil when Jev does not decide. The fallback is part of it: it decides what happens to the candidates Jev could not judge.



88
89
90
91
92
 
# File 'lib/aireview/config_jev.rb', line 88

def jev_signature
  return nil unless critique_engine == 'jev'

  ['jev', jev_model, *jev_thresholds.values, jev_fallback]
end

#jev_thresholds ⇒ Object 



78
79
80
81
82
83
 
# File 'lib/aireview/config_jev.rb', line 78

def jev_thresholds
  JEV_THRESHOLD_DEFAULTS.to_h do |name, default|
    value = dig('llm', 'jev', name)
    [name.to_sym, value.nil? ? default : probability!(value, "llm.jev.#{name}")]
  end
end

#jev_timeout ⇒ Object 



74
75
76
 
# File 'lib/aireview/config_jev.rb', line 74

def jev_timeout
  positive_integer!(dig('llm', 'jev', 'timeout') || DEFAULT_JEV_TIMEOUT, 'llm.jev.timeout')
end

#jev_warnings ⇒ Object 



106
107
108
109
110
111
112
113
114
115
116
117
118
119
 
# File 'lib/aireview/config_jev.rb', line 106

def jev_warnings
  return [] unless jev_shadow?
  return ['llm.jev.shadow is ignored: Jev already decides as the critique engine'] if critique_engine == 'jev'

  warnings = []
  if Aireview::Utils.blank?(jev_api_key)
    warnings << 'llm.jev.shadow is on, but JEV_API_KEY is not set: Jev is skipped'
  end
  if JEV_ALIASES.include?(jev_model)
    warnings << "llm.jev.model #{jev_model.inspect} is an alias: its answers change when TypeSafe ships " \
                'a release; pin a version such as jev-1.13.0'
  end
  warnings
end

#llm_stages(critique: true) ⇒ Object

The stages that need an LLM in a run. Jev with fallback: model still needs a ready LLM Critique; with fallback: fail it needs none.



47
48
49
50
51
52
 
# File 'lib/aireview/config_jev.rb', line 47

def llm_stages(critique: true)
  return ['generate'] unless critique && !critique_disabled?
  return ['generate'] if jev_critique? && jev_fallback == 'fail'

  STAGES
end

#require_jev!(critique: true) ⇒ Object

Jev as the critic needs a key and a pinned version: the thresholds are tuned against one version, and falling back to the LLM silently would hide a broken setup.

Raises:



97
98
99
100
101
102
103
104
 
# File 'lib/aireview/config_jev.rb', line 97

def require_jev!(critique: true)
  return unless jev_critique?(critique: critique)
  raise ConfigError, 'llm.critique.engine is jev, but JEV_API_KEY is not set' if Aireview::Utils.blank?(jev_api_key)
  return unless JEV_ALIASES.include?(jev_model)

  raise ConfigError, "llm.jev.model #{jev_model.inspect} is an alias; with llm.critique.engine: jev " \
                     'pin a version such as jev-1.13.0'
end