Class: RubyLLM::Batch

Inherits:
Object
  • Object
show all
Includes:
Support::Inspectable
Defined in:
lib/ruby_llm/batch.rb

Overview

A Batch is a provider-side batch of requests: chats awaiting a response (or texts awaiting embeddings) go in together, answers come back at batch prices, typically within hours. Persist the id, pick the batch back up from any process, and collect the results once processing ends.

chats = documents.map do |doc|
RubyLLM.chat(model: "claude-haiku-4-5").ask_later(doc.text)
end
batch = RubyLLM.batch(chats)
batch.id                # => "msgbatch_01EhcDuvb5XfWqcdJArbsfNX"
batch.refresh.complete? # => false, check back later
batch.messages          # the responses, in submission order

Defined Under Namespace

Modules: Helpers

Constant Summary collapse

AWAITING_ROLES =

:nodoc:

%i[user tool].freeze

Constants included from Support::Inspectable

Support::Inspectable::TRUNCATE_AT

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from Support::Inspectable

#full_inspect, #inspect, #pretty_print

Constructor Details

#initialize(provider:, chats: nil, requests: nil, batch_protocol: nil, store: nil, **attributes) ⇒ Batch

:nodoc:



170
171
172
173
174
175
176
177
178
179
 
# File 'lib/ruby_llm/batch.rb', line 170

def initialize(provider:, chats: nil, requests: nil, batch_protocol: nil, store: nil, **attributes) # :nodoc:
  @provider = provider
  @chats = chats
  @requests = requests
  @batch_protocol = protocol_name(batch_protocol)
  @store = store
  @delivered = {}
  @statuses = []
  apply(attributes)
end

Instance Attribute Details

#batch_protocol ⇒ Object (readonly)

:nodoc:



186
187
188
 
# File 'lib/ruby_llm/batch.rb', line 186

def batch_protocol
  @batch_protocol
end

#chats ⇒ Object (readonly)

The submitted Chat objects in order, or nil when the batch was loaded by id via ::find or holds embedding requests.



41
42
43
 
# File 'lib/ruby_llm/batch.rb', line 41

def chats
  @chats
end

#id ⇒ Object (readonly)

The provider's batch id. Persist it to load the batch again later from any process with ::find.



25
26
27
 
# File 'lib/ruby_llm/batch.rb', line 25

def id
  @id
end

#raw_status ⇒ Object (readonly)

The provider-reported status string, such as "in_progress". Refreshed by #refresh.



33
34
35
 
# File 'lib/ruby_llm/batch.rb', line 33

def raw_status
  @raw_status
end

#reported_cost ⇒ Object (readonly)

:nodoc:



186
187
188
 
# File 'lib/ruby_llm/batch.rb', line 186

def reported_cost
  @reported_cost
end

#request_counts ⇒ Object (readonly)

The provider-reported request tallies by state, or nil when the provider does not report them.



37
38
39
 
# File 'lib/ruby_llm/batch.rb', line 37

def request_counts
  @request_counts
end

#requests ⇒ Object (readonly)

The submitted EmbeddingRequest objects in order, or nil when the batch was loaded by id via ::find or holds chats.



45
46
47
 
# File 'lib/ruby_llm/batch.rb', line 45

def requests
  @requests
end

#status ⇒ Object (readonly)

The provider-neutral lifecycle status: :pending, :succeeded, :failed, or :cancelled. Refreshed by #refresh.



29
30
31
 
# File 'lib/ruby_llm/batch.rb', line 29

def status
  @status
end

#statuses ⇒ Object (readonly)

The normalized outcome of each collected request, in submission order. Values are :succeeded, :failed, or :cancelled.



49
50
51
 
# File 'lib/ruby_llm/batch.rb', line 49

def statuses
  @statuses
end

Class Method Details

.find(id, provider: nil, context: nil) ⇒ Object

Returns a Batch reflecting the provider's current state for id. Use it to pick a batch back up from any process.

batch = RubyLLM::Batch.find("msgbatch_01EhcDuvb5XfWqcdJArbsfNX",
                          provider: :anthropic)
batch.complete? # => true

Pass context: to use a Context in place of the global configuration. Raises ArgumentError if provider is not given.

Raises:



83
84
85
86
87
88
89
90
91
92
93
94
95
96
 
# File 'lib/ruby_llm/batch.rb', line 83

def find(id, provider: nil, context: nil)
  config = context&.config || RubyLLM.config
  persisted = config.batch_store&.fetch(id, provider:, context:)
  return persisted if persisted

  unless provider
    raise ArgumentError, 'Provider must be specified to find a batch that is not persisted by RubyLLM'
  end

  provider = Provider.resolve!(provider).new(config)
  raise Error, "#{provider.slug} doesn't support batch requests" unless provider.batches?

  new(provider:, store: config.batch_store, **provider.find_batch(id))
end

.submit(chats) ⇒ Object

Submits chats or embedding requests to their shared provider as a batch and returns a new Batch. Accepts a single Chat or an array. Every chat must be awaiting the model (see Chat#ask_later), and all requests must use the same provider.

chats = tickets.map do |ticket|
RubyLLM.chat(model: "claude-haiku-4-5").ask_later(ticket.body)
end
batch = RubyLLM::Batch.submit(chats)
batch.status     # => :pending
batch.raw_status # => "in_progress"

Raises ArgumentError if the batch is empty, mixes providers, mixes chats with embedding requests, or includes a chat that is not awaiting the model.



67
68
69
70
71
72
 
# File 'lib/ruby_llm/batch.rb', line 67

def submit(chats)
  records = wrap_records(chats)
  return submit_embeddings(records) if records.any?(EmbeddingRequest)

  submit_chats(records)
end

Instance Method Details

#cancel ⇒ Object

Asks the provider to cancel the batch and applies the new state. Requests already processed still return results. Returns self.



223
224
225
226
227
 
# File 'lib/ruby_llm/batch.rb', line 223

def cancel
  apply(@provider.cancel_batch(id))
  persist_state
  self
end

#cancelled? ⇒ Boolean

Returns whether the provider cancelled the batch.

Returns:

  • (Boolean) 


209
210
211
 
# File 'lib/ruby_llm/batch.rb', line 209

def cancelled?
  status == :cancelled
end

#complete? ⇒ Boolean

Returns whether the batch has finished processing, as of the last state fetched from the provider. Never contacts the provider; poll with #refresh.

sleep 60 until batch.refresh.complete?

Returns:

  • (Boolean) 


194
195
196
 
# File 'lib/ruby_llm/batch.rb', line 194

def complete?
  @completed
end

#cost ⇒ Object

Returns a Cost for the batch. Uses the provider's reported total when available, otherwise aggregates collected response costs at batch rates. The total is nil until the batch ends or when pricing is unknown.



258
259
260
261
262
263
 
# File 'lib/ruby_llm/batch.rb', line 258

def cost
  return Cost.aggregate([reported_cost], complete: complete?) if reported_cost
  return Cost.aggregate([], complete: false) unless complete?

  Cost.aggregate(messages.compact.map(&:cost))
end

#failed? ⇒ Boolean

Returns whether the provider failed or expired the batch.

Returns:

  • (Boolean) 


204
205
206
 
# File 'lib/ruby_llm/batch.rb', line 204

def failed?
  status == :failed
end

#messages ⇒ Object Also known as: results

Returns the answers in submission order, nil where a request failed. In a chat batch the answers are Messages, each also appended to its chat; in an embeddings batch they are Embeddings, each also hydrated into its request's EmbeddingRequest#result. Fetches results from the provider; cached once #complete? is true, so collecting early keeps reading fresh.

batch.messages.each do |message|
puts message.content
end


240
241
242
243
244
245
246
 
# File 'lib/ruby_llm/batch.rb', line 240

def messages
  return @messages if @messages

  collected = collect_results
  @messages = collected if @completed
  collected
end

#provider ⇒ Object

The slug of the provider running the batch, as a String.



182
183
184
 
# File 'lib/ruby_llm/batch.rb', line 182

def provider
  @provider.slug
end

#refresh ⇒ Object

Re-fetches the batch from the provider, updating #status, #raw_status, #request_counts, and #complete?. Returns self.



215
216
217
218
219
 
# File 'lib/ruby_llm/batch.rb', line 215

def refresh
  apply(@provider.find_batch(id))
  persist_state
  self
end

#succeeded? ⇒ Boolean

Returns whether the provider completed the batch successfully.

Returns:

  • (Boolean) 


199
200
201
 
# File 'lib/ruby_llm/batch.rb', line 199

def succeeded?
  status == :succeeded
end

#tokens ⇒ Object

Returns token usage aggregated across the batch's collected responses.



251
252
253
 
# File 'lib/ruby_llm/batch.rb', line 251

def tokens
  Tokens.aggregate(messages.compact.map(&:tokens))
end