Class: RubyLLM::Protocol

Inherits:
Object
  • Object
show all
Includes:
BinaryStreaming, Streaming
Defined in:
lib/ruby_llm/protocol.rb,
lib/ruby_llm/protocol/streaming.rb,
lib/ruby_llm/protocol/binary_streaming.rb,
lib/ruby_llm/protocol/stream_accumulator.rb

Overview

A Protocol knows how to talk to a family of provider APIs: rendering request payloads, parsing responses, streaming chunks, and naming the endpoints involved. Its counterpart, Provider, knows where to talk and who it is. The protocols that ship with the gem live under RubyLLM::Protocols.

Subclass Protocol, or a shipped subclass such as RubyLLM::Protocols::ChatCompletions, to support a new wire format. Each operation (chat, embeddings, moderation, image generation, video generation, speech, transcription, OCR, reranking, token counting, and model listing) is served by three kinds of seam method you override:

  • render_* serializes a RubyLLM request into the wire payload, such as render_payload for chat or render_embedding_payload.
  • *_url names the endpoint, such as completion_url or embedding_url.
  • parse_* turns the wire response back into RubyLLM objects, such as parse_completion_body or parse_embedding_response.

Override the seams for the operations you support; the rest raise NotImplementedError. For example:

class ChatCompletions < RubyLLM::Protocols::ChatCompletions
def completion_url
  'v2/chat'
end
end

A protocol instance is constructed by its Provider and borrows the provider's Connection, so subclasses never build HTTP clients themselves.

Defined Under Namespace

Modules: BinaryStreaming, Streaming Classes: StreamAccumulator

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from BinaryStreaming

#stream_binary

Methods included from Streaming

build_on_data_handler, build_stream_error_response, error_chunk?, failed_http_status, faraday_1?, handle_data, handle_error_chunk, handle_error_event, handle_failed_response, handle_json_error_chunk, handle_sse, handle_stream, json_error_payload?, parse_error_from_json, parse_streaming_error, process_stream_chunk, raise_stream_error, stream_events, stream_response, stream_state

Constructor Details

#initialize(provider, model = nil) ⇒ Protocol

Returns a new instance of Protocol.



85
86
87
88
89
90
 
# File 'lib/ruby_llm/protocol.rb', line 85

def initialize(provider, model = nil)
  @provider = provider
  @config = provider.config
  @connection = provider.connection
  @model = model
end

Instance Attribute Details

#config ⇒ Object (readonly)

The provider's Configuration.



45
46
47
 
# File 'lib/ruby_llm/protocol.rb', line 45

def config
  @config
end

#connection ⇒ Object (readonly)

The provider's HTTP connection. Subclasses use it to reach their endpoints.



49
50
51
 
# File 'lib/ruby_llm/protocol.rb', line 49

def connection
  @connection
end

#model ⇒ Object (readonly)

The Model this instance targets, or nil for model-less operations such as listing models.



53
54
55
 
# File 'lib/ruby_llm/protocol.rb', line 53

def model
  @model
end

#provider ⇒ Object (readonly)

The Provider this protocol talks through.



42
43
44
 
# File 'lib/ruby_llm/protocol.rb', line 42

def provider
  @provider
end

Class Method Details

.abstract(*names) ⇒ Object

Declares seam methods that raise NotImplementedError until a subclass overrides them: render_* serializes a request to wire form, url names an endpoint, and parse reads a wire response back into RubyLLM objects.



60
61
62
63
64
65
66
 
# File 'lib/ruby_llm/protocol.rb', line 60

def self.abstract(*names)
  names.each do |name|
    define_method(name) do |*_args, **_opts|
      raise NotImplementedError, "#{self.class} must implement ##{name}"
    end
  end
end

Instance Method Details

#animate_later(prompt, model:, with: nil, extend: nil, provider_options: {}) ⇒ Object

Video generation is asynchronous on every provider: this submits the job and returns a VideoJob, whose #refresh and #video poll and download through this protocol instance.



300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
 
# File 'lib/ruby_llm/protocol.rb', line 300

def animate_later(prompt, model:, with: nil, extend: nil, provider_options: {})
  raise ArgumentError, 'with: and extend: cannot be combined' if with && extend

  if extend
    payload = render_video_extension_payload(prompt, model:, extend:, provider_options:)
    url = video_extension_url
  else
    attachments = Attachment.wrap(with, config: @config)
    validate_animate_inputs!(with: attachments)
    payload = render_video_payload(prompt, model:, with: attachments, provider_options:)
    url = video_request_url(payload)
  end
  response = post_video(url, payload)
  parse_video_job(response, model:)
rescue NotImplementedError
  raise Error, "#{@provider.name} doesn't support video generation"
end

#apply_compaction(payload, compaction) ⇒ Object

Writes the chat's context compaction options into the request payload and returns it. compaction is the provider-neutral Hash from Chat#with_compaction, whose keys are Chat::COMPACTION_OPTIONS. The default drops it with a debug log, so providers that manage context themselves simply ignore the request. Protocols whose API compacts server-side override this, mapping the options they support and logging the ones they do not.

def apply_compaction(payload, compaction)
payload.merge(context_management: [{ type: 'compaction' }])
end


176
177
178
179
180
181
 
# File 'lib/ruby_llm/protocol.rb', line 176

def apply_compaction(payload, compaction)
  RubyLLM.logger.debug do
    "#{@provider.name} has no context compaction parameter, dropping #{compaction.inspect}"
  end
  payload
end

#apply_compaction_headers(headers, _compaction) ⇒ Object

Returns the completion request's HTTP headers with anything context compaction requires added. The default changes nothing; protocols that gate compaction behind a beta header override this.



186
187
188
 
# File 'lib/ruby_llm/protocol.rb', line 186

def apply_compaction_headers(headers, _compaction)
  headers
end

#apply_end_user(payload, identifier) ⇒ Object

Writes the chat's safety identifier into the request payload and returns it. The default drops the value with a debug log, so providers with no equivalent field simply omit it. Protocols whose API accepts one override this.

def apply_end_user(payload, identifier)
payload.merge(safety_identifier: identifier)
end


157
158
159
160
161
162
 
# File 'lib/ruby_llm/protocol.rb', line 157

def apply_end_user(payload, identifier)
  RubyLLM.logger.debug do
    "#{@provider.name} has no safety identifier parameter, dropping #{identifier}"
  end
  payload
end

#cache_content(content, model:, ttl: nil, instructions: nil, with: nil) ⇒ Object 



439
440
441
442
443
444
445
446
 
# File 'lib/ruby_llm/protocol.rb', line 439

def cache_content(content, model:, ttl: nil, instructions: nil, with: nil)
  payload = render_cache_payload(content, model:, ttl:, instructions:,
                                          attachments: Attachment.wrap(with, config: @config))
  response = @connection.post caches_url, payload, idempotent: false
  parse_cache_response(response.body)
rescue NotImplementedError
  raise Error, "#{@provider.name} doesn't support explicit content caching"
end

#compact(messages, headers: {}, before_request: [], usage_recorder: nil) ⇒ Object 



226
227
228
229
230
231
232
233
234
235
236
 
# File 'lib/ruby_llm/protocol.rb', line 226

def compact(messages, headers: {}, before_request: [], usage_recorder: nil)
  payload = apply_before_request_hooks(render_compaction_payload(messages), before_request)
  track_usage(:chat, on_finish: usage_recorder) do
    response = @connection.post compaction_url, payload, usage: @usage_tracker do |request|
      request.headers = headers.merge(request.headers) unless headers.empty?
    end
    parse_compaction_response(response)
  end
rescue NotImplementedError
  raise Error, "#{@provider.name} doesn't support manual compaction"
end

#complete(messages, tools:, temperature:, provider_options: {}, headers: {}, schema: nil, thinking: nil, max_output_tokens: nil, citations: false, caching: nil, tool_prefs: nil, before_request: [], usage_recorder: nil, provider_tools: [], compaction: nil, end_user: nil) ⇒ Object 



97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
 
# File 'lib/ruby_llm/protocol.rb', line 97

def complete(messages, tools:, temperature:, provider_options: {}, headers: {}, schema: nil, thinking: nil,
             max_output_tokens: nil, citations: false, caching: nil, tool_prefs: nil, before_request: [],
             usage_recorder: nil, provider_tools: [], compaction: nil, end_user: nil, &)
  resolution = resolve_provider_tools_for_request(provider_tools)
  headers = resolution.headers.merge(headers) if resolution
  headers = apply_compaction_headers(headers, compaction) if compaction
  payload = render(
    messages, tools:, tool_prefs:, temperature:, max_output_tokens:, provider_options:, schema:, thinking:,
              citations:, caching:, compaction:, end_user:, before_request:, provider_tools:,
              stream: block_given?
  )

  track_usage(:chat, on_finish: usage_recorder) do
    if block_given?
      stream_response(payload, headers) do |chunk|
        @usage_tracker.observe(chunk)
        yield chunk
      end
    else
      sync_response payload, headers
    end
  end
rescue NotImplementedError
  raise Error, "#{@provider.name} doesn't support chat"
end

#count_tokens(messages, tools:, tool_prefs: nil, thinking: nil, schema: nil, citations: false, caching: nil) ⇒ Object 



197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
 
# File 'lib/ruby_llm/protocol.rb', line 197

def count_tokens(messages, tools:, tool_prefs: nil, thinking: nil, schema: nil, citations: false, caching: nil)
  payload = render_count_tokens_payload(
    messages,
    tools: tools,
    tool_prefs: tool_prefs,
    model: model,
    schema: schema,
    thinking: thinking,
    citations: citations,
    caching: caching
  )
  parse_count_tokens_response post_count_tokens(payload)
rescue NotImplementedError
  raise Error, "#{@provider.name} doesn't support token counting"
end

#delete_cache(name) ⇒ Object 



455
456
457
458
459
460
 
# File 'lib/ruby_llm/protocol.rb', line 455

def delete_cache(name)
  @connection.delete cache_url(name)
  true
rescue NotImplementedError
  raise Error, "#{@provider.name} doesn't support explicit content caching"
end

#embed(text, model:, dimensions:, task_type: nil, title: nil, with: nil, provider_options: {}) ⇒ Object 



238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
 
# File 'lib/ruby_llm/protocol.rb', line 238

def embed(text, model:, dimensions:, task_type: nil, title: nil, with: nil, provider_options: {})
  attachments = Attachment.wrap(with, config: @config)
  raise UnsupportedAttachmentError, attachments.first.mime_type if attachments.any? && !supports_embedding_media?

  track_usage(:embedding) do
    payload = if attachments.any?
                render_embedding_payload(text, model:, dimensions:, task_type:, title:, with: attachments,
                                               provider_options:)
              else
                render_embedding_payload(text, model:, dimensions:, task_type:, title:, provider_options:)
              end
    response = @connection.post(embedding_url(model:), payload, usage: @usage_tracker)
    parse_embedding_response(response, model:, text:)
  end
rescue NotImplementedError
  raise Error, "#{@provider.name} doesn't support embeddings"
end

#extend_cache(name, ttl:) ⇒ Object 



462
463
464
465
466
467
 
# File 'lib/ruby_llm/protocol.rb', line 462

def extend_cache(name, ttl:)
  response = @connection.patch cache_url(name), render_cache_update_payload(ttl:)
  parse_cache_response(response.body)
rescue NotImplementedError
  raise Error, "#{@provider.name} doesn't support explicit content caching"
end

#find_cache(name) ⇒ Object 



448
449
450
451
452
453
 
# File 'lib/ruby_llm/protocol.rb', line 448

def find_cache(name)
  response = @connection.get cache_url(name)
  parse_cache_response(response.body)
rescue NotImplementedError
  raise Error, "#{@provider.name} doesn't support explicit content caching"
end

#list_models ⇒ Object 



213
214
215
216
 
# File 'lib/ruby_llm/protocol.rb', line 213

def list_models
  response = @connection.get models_url
  parse_list_models_response response, @provider.slug
end

#moderate(input, model:, with: [], provider_options: {}) ⇒ Object 



260
261
262
263
264
265
266
267
268
269
 
# File 'lib/ruby_llm/protocol.rb', line 260

def moderate(input, model:, with: [], provider_options: {})
  track_usage(:moderation) do
    payload = render_moderation_payload(input, model:, with: Attachment.wrap(with, config: @config),
                                               provider_options:)
    response = @connection.post moderation_url, payload, usage: @usage_tracker
    parse_moderation_response(response, model:)
  end
rescue NotImplementedError
  raise Error, "#{@provider.name} doesn't support moderation"
end

#ocr(file, model:, pages: nil, provider_options: {}) ⇒ Object 



419
420
421
422
423
424
425
426
427
 
# File 'lib/ruby_llm/protocol.rb', line 419

def ocr(file, model:, pages: nil, provider_options: {})
  track_usage(:ocr) do
    payload = render_ocr_payload(file, model:, pages:, provider_options:)
    response = @connection.post ocr_url, payload, usage: @usage_tracker
    parse_ocr_response(response, model:)
  end
rescue NotImplementedError
  raise Error, "#{@provider.name} doesn't support OCR"
end

#paint(prompt, model:, size:, count: nil, with: nil, mask: nil, provider_options: {}) ⇒ Object 



271
272
273
274
275
276
277
278
279
280
281
282
 
# File 'lib/ruby_llm/protocol.rb', line 271

def paint(prompt, model:, size:, count: nil, with: nil, mask: nil, provider_options: {})
  track_usage(:image) do
    validate_paint_inputs!(with:, mask:)
    payload = render_image_payload(prompt, model:, size:, count:, with:, mask:, provider_options:)
    response = post_image(payload, with:, mask:)
    images = parse_image_responses(response, model:)
    images.each { |image| image.config = @config }
    images.size <= 1 ? images.first : images
  end
rescue NotImplementedError
  raise Error, "#{@provider.name} doesn't support image generation"
end

#parse_error(response) ⇒ Object 



469
470
471
 
# File 'lib/ruby_llm/protocol.rb', line 469

def parse_error(response)
  @provider.parse_error(response)
end

#parse_image_responses(response, model:) ⇒ Object

Returns every Image in an image generation response, as an Array. The default asks for the one image parse_image_response reads; protocols whose API can return several images per request override it. Only the first image carries the call's usage, so summing across the array gives the cost of the call.



293
294
295
 
# File 'lib/ruby_llm/protocol.rb', line 293

def parse_image_responses(response, model:)
  Array(parse_image_response(response, model:))
end

#post_image(payload, with:, mask:) ⇒ Object 



284
285
286
 
# File 'lib/ruby_llm/protocol.rb', line 284

def post_image(payload, with:, mask:)
  @connection.post images_url(with:, mask:), payload, usage: @usage_tracker
end

#post_video(url, payload) ⇒ Object 



318
319
320
 
# File 'lib/ruby_llm/protocol.rb', line 318

def post_video(url, payload)
  @connection.post url, payload, idempotent: false
end

#preprocess_message(message) ⇒ Object 



473
474
475
476
477
478
479
480
481
482
483
 
# File 'lib/ruby_llm/protocol.rb', line 473

def preprocess_message(message)
  return message.without_thinking if foreign_thinking?(message)
  return message unless auto_upload_large_files?
  return message unless message.role == :user
  return message if message.attachments.empty?

  uploaded = message.attachments.map { |attachment| preprocess_attachment(attachment) }
  return message if uploaded == message.attachments

  message.with_attachments(uploaded)
end

#raise_transcription_streaming_unsupported ⇒ Object

:nodoc:

Raises:



408
409
410
 
# File 'lib/ruby_llm/protocol.rb', line 408

def raise_transcription_streaming_unsupported # :nodoc:
  raise Error, "#{@provider.name} doesn't support streaming transcription"
end

#refresh_video_job(job) ⇒ Object 



349
350
351
 
# File 'lib/ruby_llm/protocol.rb', line 349

def refresh_video_job(job)
  parse_video_job_status @connection.get(video_job_url(job)), job: job
end

#render(messages, tools:, temperature:, provider_options: {}, schema: nil, thinking: nil, max_output_tokens: nil, citations: false, caching: nil, tool_prefs: nil, before_request: [], stream: false, provider_tools: [], compaction: nil, end_user: nil) ⇒ Object 



123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
 
# File 'lib/ruby_llm/protocol.rb', line 123

def render(messages, tools:, temperature:, provider_options: {}, schema: nil, thinking: nil,
           max_output_tokens: nil, citations: false, caching: nil, tool_prefs: nil, before_request: [],
           stream: false, provider_tools: [], compaction: nil, end_user: nil)
  payload = render_payload(
    messages,
    tools: tools,
    tool_prefs: tool_prefs,
    temperature: temperature,
    max_output_tokens: max_output_tokens,
    model: model,
    stream: stream,
    schema: schema,
    thinking: thinking,
    citations: citations,
    caching: caching
  )
  payload = apply_end_user(payload, end_user) if end_user
  payload = apply_compaction(payload, compaction) if compaction
  payload = Support::Utils.deep_merge(payload, provider_options)
  payload = apply_provider_tools(payload, provider_tools)
  apply_before_request_hooks(payload, before_request)
rescue NotImplementedError
  raise Error, "#{@provider.name} doesn't support chat"
end

#render_embedding(text, model:, dimensions: nil, task_type: nil, title: nil, provider_options: {}) ⇒ Object

:nodoc:



256
257
258
 
# File 'lib/ruby_llm/protocol.rb', line 256

def render_embedding(text, model:, dimensions: nil, task_type: nil, title: nil, provider_options: {}) # :nodoc:
  render_embedding_payload(text, model:, dimensions:, task_type:, title:, provider_options:)
end

#render_transcription_options(timestamps:) ⇒ Object

Raises:

  • (ArgumentError) 


378
379
380
381
382
 
# File 'lib/ruby_llm/protocol.rb', line 378

def render_transcription_options(timestamps:, **)
  return {} if timestamps.nil?

  raise ArgumentError, 'This transcription protocol does not support timestamps'
end

#render_video_extension_payload ⇒ Object

Raises:



330
331
332
 
# File 'lib/ruby_llm/protocol.rb', line 330

def render_video_extension_payload(*)
  raise Error, "#{@provider.name} doesn't support video extension"
end

#rerank(query, documents, model:, top_n: nil, provider_options: {}) ⇒ Object 



429
430
431
432
433
434
435
436
437
 
# File 'lib/ruby_llm/protocol.rb', line 429

def rerank(query, documents, model:, top_n: nil, provider_options: {})
  track_usage(:rerank) do
    payload = render_rerank_payload(query, documents, model:, top_n:, provider_options:)
    response = @connection.post rerank_url, payload, usage: @usage_tracker
    parse_rerank_response(response, model:, documents:)
  end
rescue NotImplementedError
  raise Error, "#{@provider.name} doesn't support reranking"
end

#server_tool_aliases ⇒ Object

The alias table mapping portable server tool names to this protocol's wire format. Protocols with provider-tool support override this; nil means the protocol has no provider-tool support at all.



193
194
195
 
# File 'lib/ruby_llm/protocol.rb', line 193

def server_tool_aliases
  nil
end

#speak(input, model:, voice:, format:, provider_options: {}, &block) ⇒ Object 



353
354
355
356
357
358
359
360
361
362
363
 
# File 'lib/ruby_llm/protocol.rb', line 353

def speak(input, model:, voice:, format:, provider_options: {}, &block)
  track_usage(:speech) do
    payload = render_speech_payload(input, model:, voice:, format:, provider_options:)
    next stream_speech(payload, model:, voice:, format:, &block) if block

    response = @connection.post speech_url(model:), payload, usage: @usage_tracker
    parse_speech_response(response, model:, voice:, format:)
  end
rescue NotImplementedError
  raise Error, "#{@provider.name} doesn't support speech generation"
end

#stream_speech ⇒ Object

Raises:



365
366
367
 
# File 'lib/ruby_llm/protocol.rb', line 365

def stream_speech(*, **, &)
  raise Error, "#{@provider.name} doesn't support streaming speech with this protocol"
end

#stream_speech_response(url, payload, model:, voice:, format:) ⇒ Object 



369
370
371
372
373
374
375
376
 
# File 'lib/ruby_llm/protocol.rb', line 369

def stream_speech_response(url, payload, model:, voice:, format:)
  empty_response = Faraday::Response.new(body: '')
  audio = parse_speech_response(empty_response, model:, voice:, format:)
  response = stream_binary(url, payload) do |data|
    yield SpeechChunk.new(data:, format: audio.format, mime_type: audio.mime_type)
  end
  parse_speech_response(response, model:, voice:, format:)
end

#stream_transcription ⇒ Object

Streams a transcription, yielding TranscriptionChunk objects and returning the final Transcription. Protocols whose provider streams transcriptions override this.



404
405
406
 
# File 'lib/ruby_llm/protocol.rb', line 404

def stream_transcription(*, **, &)
  raise_transcription_streaming_unsupported
end

#supports_embedding_media? ⇒ Boolean

Whether the protocol can embed media attachments alongside text. Protocols that support multimodal embeddings override this and accept a with: array of Attachments in render_embedding_payload.

Returns:

  • (Boolean) 


415
416
417
 
# File 'lib/ruby_llm/protocol.rb', line 415

def supports_embedding_media?
  false
end

#tokenize(text, model:) ⇒ Object 



218
219
220
221
222
223
224
 
# File 'lib/ruby_llm/protocol.rb', line 218

def tokenize(text, model:)
  payload = render_tokenization_payload(text, model:)
  response = @connection.post tokenization_url, payload
  parse_tokenization_response(response, model:)
rescue NotImplementedError
  raise Error, "#{@provider.name} doesn't support text tokenization"
end

#tool_approval_response(tool_call, approved:) ⇒ Object 



92
93
94
95
 
# File 'lib/ruby_llm/protocol.rb', line 92

def tool_approval_response(tool_call, approved:)
  Message.new(role: :tool, content: approved ? 'Approved' : 'Denied', tool_call_id: tool_call.id,
              raw_content: render_tool_approval_response(tool_call, approved:))
end

#transcribe(audio_file, model:, language:, format: nil, speaker_names: nil, speaker_references: nil, provider_options: {}, prompt: nil, temperature: nil, &block) ⇒ Object 



384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
 
# File 'lib/ruby_llm/protocol.rb', line 384

def transcribe(audio_file, model:, language:, format: nil, speaker_names: nil,
               speaker_references: nil, provider_options: {}, prompt: nil, temperature: nil, &block)
  streaming = block_given?
  track_usage(:transcription) do
    file_part = build_audio_file_part(audio_file)
    payload = render_transcription_payload(file_part, model:, language:, format:, speaker_names:,
                                                      speaker_references:, provider_options:, prompt:,
                                                      temperature:)
    next stream_transcription(payload, model:, &block) if streaming

    response = @connection.post transcription_url, payload, usage: @usage_tracker
    parse_transcription_response(response, model:)
  end
rescue NotImplementedError
  raise Error, "#{@provider.name} doesn't support transcription"
end

#video_extension_attachment(source) ⇒ Object

Raises:

  • (ArgumentError) 


334
335
336
337
338
339
340
341
342
343
344
345
346
347
 
# File 'lib/ruby_llm/protocol.rb', line 334

def video_extension_attachment(source)
  source = source.url || StringIO.new(source.to_blob) if source.is_a?(Video)
  attachments = if source.respond_to?(:read)
                  [Attachment.new(source, filename: 'video.mp4', config: @config)]
                else
                  Attachment.wrap(source, config: @config)
                end
  raise ArgumentError, 'extend: takes exactly one video' unless attachments.one?

  attachment = attachments.first
  raise UnsupportedAttachmentError, attachment.mime_type unless attachment.video?

  attachment
end

#video_extension_url ⇒ Object 



326
327
328
 
# File 'lib/ruby_llm/protocol.rb', line 326

def video_extension_url
  video_url
end

#video_request_url(_payload) ⇒ Object 



322
323
324
 
# File 'lib/ruby_llm/protocol.rb', line 322

def video_request_url(_payload)
  video_url
end