Class: RubyLLM::Moderation

Inherits:
Object
  • Object
show all
Includes:
Accounting::Usage::Result, Support::Inspectable
Defined in:
lib/ruby_llm/moderation.rb

Overview

A Moderation holds the result of screening text or images for potentially harmful content. Most code obtains one through RubyLLM.moderate.

result = RubyLLM.moderate("This is a safe message about Ruby programming")
result.flagged?  # => false

RubyLLM.moderate(with: "profile.png").flagged?

Defined Under Namespace

Classes: Result

Constant Summary

Constants included from Support::Inspectable

Support::Inspectable::TRUNCATE_AT

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from Accounting::Usage::Result

#ruby_llm_usage_entries, #ruby_llm_usage_entries=

Methods included from Support::Inspectable

#full_inspect, #inspect, #pretty_print

Constructor Details

#initialize(id:, model:, results:, raw: nil) ⇒ Moderation

:nodoc:



72
73
74
75
76
77
 
# File 'lib/ruby_llm/moderation.rb', line 72

def initialize(id:, model:, results:, raw: nil) # :nodoc:
  @id = id
  @model = model
  @results = results
  @raw = raw
end

Instance Attribute Details

#id ⇒ Object (readonly)

The provider-assigned identifier of the moderation request.



58
59
60
 
# File 'lib/ruby_llm/moderation.rb', line 58

def id
  @id
end

#model ⇒ Object (readonly)

The id of the model that performed the moderation, or nil for an operation that does not select a model.



62
63
64
 
# File 'lib/ruby_llm/moderation.rb', line 62

def model
  @model
end

#raw ⇒ Object (readonly)

The original provider response, or an array of responses when each input requires a separate request.



70
71
72
 
# File 'lib/ruby_llm/moderation.rb', line 70

def raw
  @raw
end

#results ⇒ Object (readonly)

The per-input verdicts, as an array of Result objects, one per moderated input.



66
67
68
 
# File 'lib/ruby_llm/moderation.rb', line 66

def results
  @results
end

Class Method Details

.moderate(input = nil, model: nil, with: nil, provider: nil, assume_model_exists: false, context: nil, provider_options: {}, metadata: nil) ⇒ Object

Screens input and optional image attachments and returns a Moderation with the provider's verdict. Uses the configured default moderation model when model is not given. Pass provider: and assume_model_exists: true to use a model that is not in the registry. An explicitly selected provider may instead use a configured resource without a model.

RubyLLM.moderate("User message")
RubyLLM.moderate(["First comment", "Second comment"]).results
RubyLLM.moderate("Caption", with: "screenshot.png")

Raises:



89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
 
# File 'lib/ruby_llm/moderation.rb', line 89

def self.moderate(input = nil,
                  model: nil,
                  with: nil,
                  provider: nil,
                  assume_model_exists: false,
                  context: nil,
                  provider_options: {},
                  metadata: nil)
  attachments = Attachment.wrap(with)
  raise ArgumentError, 'must provide input text, image attachment, or both' if input.nil? && attachments.empty?

  config = context&.config || RubyLLM.config
  model, provider_instance = Models.resolve(model, provider: provider, assume_model_exists: assume_model_exists,
                                                   config: config, operation: :moderate,
                                                   default_model: config.default_moderation_model)
  empty_tokens = Tokens.new
  payload = {
    provider: provider_instance.slug,
    provider_class: provider_instance.name,
    model: model&.id,
    model_info: model,
    input: input,
    attachment_count: attachments.size,
    provider_options: provider_options,
    metadata: metadata,
    tokens: empty_tokens,
    cost: Cost.new(tokens: empty_tokens, model:)
  }

  RubyLLM.instrument('moderation.ruby_llm', payload, config: config) do |event|
    result = provider_instance.moderate(input, model:, with: attachments, provider_options:)
    event[:result] = result
    event[:flagged] = result.flagged?
    event[:tokens] = result.tokens
    event[:cost] = result.cost
    result
  end
end

Instance Method Details

#category_scores ⇒ Object

Returns the confidence scores across all results, as a hash of category name to a score between 0.0 and 1.0. Keeps the highest score per category when there are multiple results.

result.category_scores["violence"]  # => 0.0001


159
160
161
162
163
 
# File 'lib/ruby_llm/moderation.rb', line 159

def category_scores
  results.map(&:category_scores).reduce({}) do |merged, scores|
    merged.merge(scores) { |_category, left, right| [left, right].max }
  end
end

#cost ⇒ Object

Returns the moderation cost across every provider attempt.



141
142
143
 
# File 'lib/ruby_llm/moderation.rb', line 141

def cost
  ruby_llm_usage_cost
end

#flagged? ⇒ Boolean

Returns true if any input was flagged as potentially harmful, false otherwise.

Returns:



130
131
132
 
# File 'lib/ruby_llm/moderation.rb', line 130

def flagged?
  results.any?(&:flagged?)
end

#flagged_categories ⇒ Object

Returns the unique names of the categories flagged across all results.

result.flagged_categories  # => ["harassment", "violence"]


149
150
151
 
# File 'lib/ruby_llm/moderation.rb', line 149

def flagged_categories
  results.flat_map(&:categories).uniq
end

#inspect_attributes ⇒ Object

:nodoc:



165
166
167
 
# File 'lib/ruby_llm/moderation.rb', line 165

def inspect_attributes # :nodoc:
  { id: id, model: model, flagged: flagged? }
end

#tokens ⇒ Object

Returns provider-reported usage across every attempt. Its fields are nil when the provider did not report any.



136
137
138
 
# File 'lib/ruby_llm/moderation.rb', line 136

def tokens
  ruby_llm_usage_tokens
end