Class: Aireview::JevShadow

Inherits:
Object
  • Object
show all
Defined in:
lib/aireview/jev_shadow.rb

Overview

llm.jev.shadow: after the LLM Critique the same candidates go to Jev, and its decisions are logged next to the Critique verdicts. The review result and the review key do not depend on it, and a Jev failure is a warning: this is data for choosing thresholds, not a check.

Instance Method Summary collapse

Constructor Details

#initialize(config:, scrub:, logger: Logger.new($stderr), critic: nil) ⇒ JevShadow

Returns a new instance of JevShadow.



14
15
16
17
18
19
 
# File 'lib/aireview/jev_shadow.rb', line 14

def initialize(config:, scrub:, logger: Logger.new($stderr), critic: nil)
  @config = config
  @scrub = scrub
  @logger = logger
  @critic = critic
end

Instance Method Details

#run(context:, candidates:, accepted:) ⇒ Object

accepted — what the LLM Critique kept; only the ids are compared.



22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
 
# File 'lib/aireview/jev_shadow.rb', line 22

def run(context:, candidates:, accepted:)
  critic = self.critic
  return @logger.warn('Jev shadow skipped: JEV_API_KEY is not set') unless critic

  started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
  result = critic.assess(context: context, candidates: candidates)
  seconds = Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_at
  log(result, accepted.map { |candidate| (candidate['id'] || candidate[:id]).to_s }, seconds)
rescue JevError => e
  @logger.warn("Jev shadow failed, the review is not affected: #{e.message}")
# The shadow is an experiment running in real review jobs: a bug in it
# must not cost a review either, but it must stay visible as a bug.
rescue StandardError => e
  @logger.warn("Jev shadow crashed, the review is not affected: #{e.class}: #{e.message} " \
               "(#{e.backtrace&.first})")
end