Class: Aireview::ModelChecker

Inherits:
Object
  • Object
show all
Defined in:
lib/aireview/model_checker.rb

Overview

aireview models check: every model of both stage chains gets one request with the production Generate schema and one with the Critique schema on a tiny synthetic MR. The answer goes through the same validation as in a run: the provider's catalog is not consulted, "the model is listed" does not mean "our request with the schema passes on it". No reserves, quarantine or walking — this is a check, not a review. Only the stages that need an LLM are checked (Jev with fallback: fail needs no LLM Critique); Jev gets a probe of its own when it is the critique engine, and in shadow mode too, but then its result is shown without counting: the shadow never blocks a release.

Defined Under Namespace

Classes: Result

Constant Summary collapse

PROBE_MERGE_REQUEST = 
{
  'title' => 'Fix order total',
  'description' => 'The total must include the quantity.',
  'source_branch' => 'fix/order-total',
  'target_branch' => 'master',
  'author' => {'name' => 'aireview'}
}.freeze
PROBE_CHANGES = 
[
  {
    'old_path' => 'app/models/order.rb',
    'new_path' => 'app/models/order.rb',
    'diff' => "@@ -1,5 +1,5 @@\n class Order\n   def total\n-    price * quantity\n+    price\n   end\n"
  }
].freeze
PROBE_CANDIDATE_IDS = 
['C1'].freeze
JEV_PROBE_STATE = 
'The order total must include the quantity.'
JEV_PROBE_QUESTIONS = 
{
  'mentions_quantity' => {'type' => 'noul', 'instructions' => 'Does the text mention a quantity?'}
}.freeze
PASSING =

ok and skipped do not block a release, everything else does. In strict mode (a runner where Ollama must be running) skipped fails too.

i[ok skipped].freeze
STRICT_PASSING = 
i[ok].freeze

Instance Method Summary collapse

Constructor Details

#initialize(config:, out:, strict: false, logger: Logger.new($stderr), **dependencies) ⇒ ModelChecker

strict — skipped counts as a failure (a runner where Ollama must be running).



58
59
60
61
62
63
64
65
66
67
68
69
70
 
# File 'lib/aireview/model_checker.rb', line 58

def initialize(config:, out:, strict: false, logger: Logger.new($stderr), **dependencies)
  @config = config
  @out = out
  @strict = strict
  client = dependencies[:client]
  sleeper = dependencies[:sleeper]
  @jev_client = dependencies[:jev_client]
  @logger = logger
  @client = client || LlmClient.new(config: config, logger: logger)
  @sleeper = sleeper || ->(seconds) { sleep(seconds) }
  @pipeline = ReviewPipeline.new(config: config, logger: logger)
  @parser = ResultParser.new
end

Instance Method Details

#run ⇒ Object

Exit code: 0 — every model answered by the schema (or was skipped), 1 — at least one did not.



74
75
76
77
78
79
80
81
82
83
84
 
# File 'lib/aireview/model_checker.rb', line 74

def run
  @config.require_llm_configuration!
  stages = @config.llm_stages
  candidates = stages.flat_map { |stage| @config.stage_chain(stage) }.uniq(&:to_s)
  prompts = probe_prompts
  @out.puts("Checking #{candidates.size} model(s) with the #{stages.join(' and ')} schemas")
  results = candidates.flat_map do |candidate|
    stages.map { |stage| check(candidate, stage, prompts).tap { |result| @out.puts(result) } }
  end
  summary(results + jev_results)
end