Class: Leva::PromptOptimizer

Inherits:
Object
  • Object
show all
Defined in:
app/services/leva/prompt_optimizer.rb

Overview

Optimizes prompts using DSPy.rb optimizers.

This service coordinates the optimization process, delegating the actual optimization work to strategy classes.

Examples:

Optimize a prompt for a dataset

optimizer = Leva::PromptOptimizer.new(dataset: dataset, mode: :medium)
result = optimizer.optimize
# => { system_prompt: "...", user_prompt: "...", metadata: {...} }

With GEPA optimizer

optimizer = Leva::PromptOptimizer.new(dataset: dataset, optimizer: :gepa, mode: :medium)
result = optimizer.optimize

Constant Summary collapse

MINIMUM_EXAMPLES =

Minimum number of examples required for optimization

10
OPTIMIZERS =

Available optimizers with their strategy classes

{
  bootstrap: {
    name: "Bootstrap",
    strategy_class: Leva::Optimizers::Bootstrap,
    gem: nil,
    description: "Fast and simple. Automatically selects optimal few-shot examples from your dataset. " \
                 "Best for quick iteration and when you have limited data (10-50 examples). " \
                 "Does not modify instructions, only adds demonstrations."
  },
  gepa: {
    name: "GEPA",
    strategy_class: Leva::Optimizers::GepaOptimizer,
    gem: "dspy-gepa",
    description: "State-of-the-art optimizer using reflective prompt evolution. Uses LLM reflection " \
                 "to identify what works and propose improvements. Outperforms MIPROv2 by 10-14% " \
                 "while being more sample efficient. Best choice for maximum quality."
  },
  miprov2: {
    name: "MIPROv2",
    strategy_class: Leva::Optimizers::Miprov2Optimizer,
    gem: "dspy-miprov2",
    description: "Uses Bayesian optimization to search for optimal instructions and few-shot examples. " \
                 "Good for larger datasets (200+ examples). More computationally demanding but thorough. " \
                 "Can overfit on small datasets."
  }
}.freeze
DEFAULT_OPTIMIZER =

Default optimizer

:bootstrap
MODES =

Optimization modes with their approximate durations

{
  light: { description: "Fast optimization (~5 min)", trials: 5 },
  medium: { description: "Balanced optimization (~15 min)", trials: 15 },
  heavy: { description: "Thorough optimization (~30 min)", trials: 30 }
}.freeze
DEFAULT_MODEL =

Default model if none specified (fast and cheap)

"gemini-2.5-flash"

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(dataset:, metric: nil, mode: :light, model: nil, optimizer: nil, progress_callback: nil) ⇒ PromptOptimizer

Returns a new instance of PromptOptimizer.

Parameters:

  • dataset (Leva::Dataset) —

    The dataset to optimize for

  • metric (Proc, nil) (defaults to: nil) —

    Custom evaluation metric (default: exact string match)

  • mode (Symbol) (defaults to: :light) —

    Optimization intensity (:light, :medium, :heavy)

  • model (String, nil) (defaults to: nil) —

    The model to use (default: DEFAULT_MODEL)

  • optimizer (Symbol, String) (defaults to: nil) —

    The optimizer to use (default: :bootstrap)

  • progress_callback (Proc, nil) (defaults to: nil) —

    Callback for progress updates



100
101
102
103
104
105
106
107
108
 
# File 'app/services/leva/prompt_optimizer.rb', line 100

def initialize(dataset:, metric: nil, mode: :light, model: nil, optimizer: nil, progress_callback: nil)
  @dataset = dataset
  @metric = metric || default_metric
  @mode = mode.to_sym
  @model = model.presence || DEFAULT_MODEL
  @optimizer = (optimizer.presence || DEFAULT_OPTIMIZER).to_sym
  @progress_callback = progress_callback
  @last_progress = nil
end

Instance Attribute Details

#dataset ⇒ Leva::Dataset (readonly)

Returns The dataset being optimized.

Returns:



83
84
85
 
# File 'app/services/leva/prompt_optimizer.rb', line 83

def dataset
  @dataset
end

#mode ⇒ Symbol (readonly)

Returns The optimization mode (:light, :medium, :heavy).

Returns:

  • (Symbol) —

    The optimization mode (:light, :medium, :heavy)



86
87
88
 
# File 'app/services/leva/prompt_optimizer.rb', line 86

def mode
  @mode
end

#model ⇒ String (readonly)

Returns The model to use for optimization.

Returns:

  • (String) —

    The model to use for optimization



89
90
91
 
# File 'app/services/leva/prompt_optimizer.rb', line 89

def model
  @model
end

#optimizer ⇒ Symbol (readonly)

Returns The optimizer to use (:bootstrap, :gepa, :miprov2).

Returns:

  • (Symbol) —

    The optimizer to use (:bootstrap, :gepa, :miprov2)



92
93
94
 
# File 'app/services/leva/prompt_optimizer.rb', line 92

def optimizer
  @optimizer
end

Class Method Details

.available_models ⇒ Array<RubyLLM::Model>

Returns available models from RubyLLM. Results are cached for 5 minutes to avoid repeated expensive calls.

Returns:

  • (Array<RubyLLM::Model>) —

    All available chat models



66
67
68
69
70
 
# File 'app/services/leva/prompt_optimizer.rb', line 66

def self.available_models
  Rails.cache.fetch("leva/available_models", expires_in: 5.minutes) do
    RubyLLM.models.chat_models
  end
end

.find_model(model_id) ⇒ RubyLLM::Model?

Finds a model by ID.

Parameters:

  • model_id (String) —

    The model ID to find

Returns:

  • (RubyLLM::Model, nil) —

    The model or nil if not found



76
77
78
79
80
 
# File 'app/services/leva/prompt_optimizer.rb', line 76

def self.find_model(model_id)
  RubyLLM.models.find(model_id)
rescue RubyLLM::ModelNotFoundError
  nil
end

.optimizer_available?(optimizer_type) ⇒ Boolean

Checks if a specific optimizer is available.

Parameters:

  • optimizer_type (Symbol) —

    The optimizer to check

Returns:

  • (Boolean) —

    True if the optimizer is available



154
155
156
157
158
159
160
161
162
163
164
165
166
 
# File 'app/services/leva/prompt_optimizer.rb', line 154

def self.optimizer_available?(optimizer_type)
  optimizer_type = optimizer_type.to_sym
  return true if optimizer_type == :bootstrap

  case optimizer_type
  when :gepa
    !!defined?(DSPy::Teleprompt::GEPA)
  when :miprov2
    !!defined?(DSPy::Teleprompt::MIPROv2)
  else
    false
  end
end

Instance Method Details

#can_optimize? ⇒ Boolean

Checks if the dataset is ready for optimization.

Returns:

  • (Boolean) —

    True if the dataset can be optimized



139
140
141
 
# File 'app/services/leva/prompt_optimizer.rb', line 139

def can_optimize?
  @dataset.dataset_records.count >= MINIMUM_EXAMPLES
end

#optimize ⇒ Hash

Runs the optimization process.

Returns:

  • (Hash) —

    Hash containing :system_prompt, :user_prompt, and :metadata

Raises:



115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
 
# File 'app/services/leva/prompt_optimizer.rb', line 115

def optimize
  report_progress(step: "validating", progress: 0)
  validate_dataset!
  validate_dspy_configuration!
  validate_optimizer!

  report_progress(step: "splitting_data", progress: 10)
  splits = DatasetConverter.new(@dataset).split

  report_progress(step: "generating_signature", progress: 20)
  signature = SignatureGenerator.new(@dataset).generate

  # Delegate to optimizer strategy
  strategy = build_optimizer_strategy
  result = strategy.optimize(splits, signature)

  report_progress(step: "complete", progress: 100)

  build_final_result(result, splits, strategy.optimizer_type)
end

#records_needed ⇒ Integer

Returns the number of additional records needed for optimization.

Returns:

  • (Integer) —

    Number of records still needed (0 if ready)



146
147
148
 
# File 'app/services/leva/prompt_optimizer.rb', line 146

def records_needed
  [ MINIMUM_EXAMPLES - @dataset.dataset_records.count, 0 ].max
end