Class: Leva::Optimizers::Base Abstract

Inherits:
Object
  • Object
show all
Defined in:
app/services/leva/optimizers/base.rb

Overview

This class is abstract.

Subclass and override #compile to implement a strategy

Base class for optimization strategies.

Each optimizer implements a different approach to finding optimal prompt instructions and few-shot examples.

Direct Known Subclasses

Bootstrap, GepaOptimizer, Miprov2Optimizer

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(model:, metric:, mode:, progress_callback: nil) ⇒ Base

Returns a new instance of Base.

Parameters:

  • model (String) —

    The model to use for optimization

  • metric (Proc) —

    The evaluation metric

  • mode (Symbol) —

    Optimization intensity (:light, :medium, :heavy)

  • progress_callback (Proc, nil) (defaults to: nil) —

    Callback for progress updates



28
29
30
31
32
33
34
 
# File 'app/services/leva/optimizers/base.rb', line 28

def initialize(model:, metric:, mode:, progress_callback: nil)
  @model = model
  @metric = metric
  @mode = mode
  @progress_callback = progress_callback
  @last_progress = nil
end

Instance Attribute Details

#metric ⇒ Proc (readonly)

Returns The evaluation metric.

Returns:

  • (Proc) —

    The evaluation metric



16
17
18
 
# File 'app/services/leva/optimizers/base.rb', line 16

def metric
  @metric
end

#mode ⇒ Symbol (readonly)

Returns The optimization mode.

Returns:

  • (Symbol) —

    The optimization mode



22
23
24
 
# File 'app/services/leva/optimizers/base.rb', line 22

def mode
  @mode
end

#model ⇒ String (readonly)

Returns The model identifier.

Returns:

  • (String) —

    The model identifier



13
14
15
 
# File 'app/services/leva/optimizers/base.rb', line 13

def model
  @model
end

#progress_callback ⇒ Proc? (readonly)

Returns Progress callback.

Returns:

  • (Proc, nil) —

    Progress callback



19
20
21
 
# File 'app/services/leva/optimizers/base.rb', line 19

def progress_callback
  @progress_callback
end

Instance Method Details

#optimize(splits, signature) ⇒ Hash

Runs the optimization and returns results.

Parameters:

  • splits (Hash) —

    The train/val/test splits

  • signature (Class) —

    The generated DSPy signature class

Returns:

  • (Hash) —

    Hash with :instruction, :few_shot_examples, :score

Raises:



42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
 
# File 'app/services/leva/optimizers/base.rb', line 42

def optimize(splits, signature)
  train_examples = splits[:train]
  val_examples = splits[:val]

  report_progress(step: step_name, progress: 30, examples_processed: 0, total: train_examples.size)

  predictor = DSPy::Predict.new(signature)
  predictor.config.lm = create_lm

  result = compile(predictor, train_examples, val_examples, signature)

  report_progress(step: "evaluating", progress: 85)

  instruction = result[:instruction_override] || extract_instruction(result[:optimized], signature)
  score = evaluate(result[:optimized] || predictor, val_examples)

  report_progress(step: "building_result", progress: 95)

  {
    instruction: instruction,
    few_shot_examples: result[:few_shot_examples] || [],
    score: score
  }
rescue StandardError => e
  Rails.logger.error "[Leva::Optimizers::#{self.class.name.demodulize}] Optimization failed: #{e.message}"
  Rails.logger.error e.backtrace.first(5).join("\n")
  raise Leva::OptimizationError, "#{optimizer_name} optimization failed: #{e.message}"
end

#optimizer_name ⇒ String

Human-readable optimizer name.

Returns:

  • (String)

Raises:

  • (NotImplementedError) 


79
80
81
 
# File 'app/services/leva/optimizers/base.rb', line 79

def optimizer_name
  raise NotImplementedError, "Subclasses must implement #optimizer_name"
end

#optimizer_type ⇒ Symbol

Returns The optimizer type symbol.

Returns:

  • (Symbol) —

    The optimizer type symbol

Raises:

  • (NotImplementedError) 


84
85
86
 
# File 'app/services/leva/optimizers/base.rb', line 84

def optimizer_type
  raise NotImplementedError, "Subclasses must implement #optimizer_type"
end

#step_name ⇒ String

The name used in progress reporting.

Returns:

  • (String)

Raises:

  • (NotImplementedError) 


73
74
75
 
# File 'app/services/leva/optimizers/base.rb', line 73

def step_name
  raise NotImplementedError, "Subclasses must implement #step_name"
end