Module: Raix::ChatCompletion

Extended by:
ActiveSupport::Concern
Included in:
Predicate
Defined in:
lib/raix/chat_completion.rb

Overview

The ‘ChatCompletion` module is a Rails concern that provides a way to interact with the OpenRouter Chat Completion API via its client. The module includes a few methods that allow you to build a transcript of messages and then send them to the API for completion. The API will return a response that you can use however you see fit.

When the AI responds with tool function calls instead of a text message, this module automatically:

  1. Executes the requested tool functions

  2. Adds the function results to the conversation transcript

  3. Sends the updated transcript back to the AI for another completion

  4. Repeats this process until the AI responds with a regular text message

This automatic continuation ensures that tool calls are seamlessly integrated into the conversation flow. The AI can use tool results to formulate its final response to the user. You can limit the number of tool calls using the ‘max_tool_calls` parameter to prevent excessive function invocations.

Tool functions must be defined on the class that includes this module. The ‘FunctionDispatch` module provides a Rails-like DSL for declaring these functions at the class level, which is cleaner than implementing them as instance methods.

Note that some AI models can make multiple tool function calls in a single response. When that happens, the module executes all requested functions before continuing the conversation.

Instance Attribute Summary collapse

Instance Method Summary collapse

Instance Attribute Details

#available_tools ⇒ Object

Returns the value of attribute available_tools.



43
44
45
 
# File 'lib/raix/chat_completion.rb', line 43

def available_tools
  @available_tools
end

#cache_at ⇒ Object

Returns the value of attribute cache_at.



43
44
45
 
# File 'lib/raix/chat_completion.rb', line 43

def cache_at
  @cache_at
end

#frequency_penalty ⇒ Object

Returns the value of attribute frequency_penalty.



43
44
45
 
# File 'lib/raix/chat_completion.rb', line 43

def frequency_penalty
  @frequency_penalty
end

#logit_bias ⇒ Object

Returns the value of attribute logit_bias.



43
44
45
 
# File 'lib/raix/chat_completion.rb', line 43

def logit_bias
  @logit_bias
end

#logprobs ⇒ Object

Returns the value of attribute logprobs.



43
44
45
 
# File 'lib/raix/chat_completion.rb', line 43

def logprobs
  @logprobs
end

#loop ⇒ Object

Returns the value of attribute loop.



43
44
45
 
# File 'lib/raix/chat_completion.rb', line 43

def loop
  @loop
end

#max_completion_tokens ⇒ Object

Returns the value of attribute max_completion_tokens.



43
44
45
 
# File 'lib/raix/chat_completion.rb', line 43

def max_completion_tokens
  @max_completion_tokens
end

#max_tokens ⇒ Object

Returns the value of attribute max_tokens.



43
44
45
 
# File 'lib/raix/chat_completion.rb', line 43

def max_tokens
  @max_tokens
end

#max_tool_calls ⇒ Object

Returns the value of attribute max_tool_calls.



43
44
45
 
# File 'lib/raix/chat_completion.rb', line 43

def max_tool_calls
  @max_tool_calls
end

#min_p ⇒ Object

Returns the value of attribute min_p.



43
44
45
 
# File 'lib/raix/chat_completion.rb', line 43

def min_p
  @min_p
end

#model ⇒ Object

Returns the value of attribute model.



43
44
45
 
# File 'lib/raix/chat_completion.rb', line 43

def model
  @model
end

#prediction ⇒ Object

Returns the value of attribute prediction.



43
44
45
 
# File 'lib/raix/chat_completion.rb', line 43

def prediction
  @prediction
end

#presence_penalty ⇒ Object

Returns the value of attribute presence_penalty.



43
44
45
 
# File 'lib/raix/chat_completion.rb', line 43

def presence_penalty
  @presence_penalty
end

#provider ⇒ Object

Returns the value of attribute provider.



43
44
45
 
# File 'lib/raix/chat_completion.rb', line 43

def provider
  @provider
end

#repetition_penalty ⇒ Object

Returns the value of attribute repetition_penalty.



43
44
45
 
# File 'lib/raix/chat_completion.rb', line 43

def repetition_penalty
  @repetition_penalty
end

#response_format ⇒ Object

Returns the value of attribute response_format.



43
44
45
 
# File 'lib/raix/chat_completion.rb', line 43

def response_format
  @response_format
end

#seed ⇒ Object

Returns the value of attribute seed.



43
44
45
 
# File 'lib/raix/chat_completion.rb', line 43

def seed
  @seed
end

#stop ⇒ Object

Returns the value of attribute stop.



43
44
45
 
# File 'lib/raix/chat_completion.rb', line 43

def stop
  @stop
end

#stop_tool_calls_and_respond ⇒ Object

Returns the value of attribute stop_tool_calls_and_respond.



43
44
45
 
# File 'lib/raix/chat_completion.rb', line 43

def stop_tool_calls_and_respond
  @stop_tool_calls_and_respond
end

#stream ⇒ Object

Returns the value of attribute stream.



43
44
45
 
# File 'lib/raix/chat_completion.rb', line 43

def stream
  @stream
end

#temperature ⇒ Object

Returns the value of attribute temperature.



43
44
45
 
# File 'lib/raix/chat_completion.rb', line 43

def temperature
  @temperature
end

#tool_choice ⇒ Object

Returns the value of attribute tool_choice.



43
44
45
 
# File 'lib/raix/chat_completion.rb', line 43

def tool_choice
  @tool_choice
end

#tools ⇒ Object

Returns the value of attribute tools.



43
44
45
 
# File 'lib/raix/chat_completion.rb', line 43

def tools
  @tools
end

#top_a ⇒ Object

Returns the value of attribute top_a.



43
44
45
 
# File 'lib/raix/chat_completion.rb', line 43

def top_a
  @top_a
end

#top_k ⇒ Object

Returns the value of attribute top_k.



43
44
45
 
# File 'lib/raix/chat_completion.rb', line 43

def top_k
  @top_k
end

#top_logprobs ⇒ Object

Returns the value of attribute top_logprobs.



43
44
45
 
# File 'lib/raix/chat_completion.rb', line 43

def top_logprobs
  @top_logprobs
end

#top_p ⇒ Object

Returns the value of attribute top_p.



43
44
45
 
# File 'lib/raix/chat_completion.rb', line 43

def top_p
  @top_p
end

Instance Method Details

#chat_completion(params: {}, loop: false, json: false, raw: false, openai: nil, save_response: true, messages: nil, available_tools: nil, max_tool_calls: nil) ⇒ String|Hash

This method performs chat completion based on the provided transcript and parameters.

Parameters:

  • params (Hash) (defaults to: {}) —

    The parameters for chat completion.

  • loop (Hash) (defaults to: false) —

    a customizable set of options

  • tools (Hash) —

    a customizable set of options

  • max_tool_calls (Hash) (defaults to: nil) —

    a customizable set of options

Options Hash (params:):

  • :json (Boolean) — default: false —

    Whether to return the parse the response as a JSON object. Will search for <json> tags in the response first, then fall back to the default JSON parsing of the entire response.

  • :openai (String) — default: nil —

    If non-nil, use OpenAI with the model specified in this param.

  • :raw (Boolean) — default: false —

    Whether to return the raw response or dig the text content.

  • :messages (Array) — default: nil —

    An array of messages to use instead of the transcript.

Options Hash (loop:):

  • :loop (Boolean) — default: false —

    DEPRECATED - The system now automatically continues after tool calls.

Options Hash (max_tool_calls:):

  • :max_tool_calls (Integer) —

    Maximum number of tool calls before forcing a text response. Defaults to the configured value.

Returns:

  • (String|Hash) —

    The completed chat response.



76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
 
# File 'lib/raix/chat_completion.rb', line 76

def chat_completion(params: {}, loop: false, json: false, raw: false, openai: nil, save_response: true, messages: nil, available_tools: nil, max_tool_calls: nil)
  # set params to default values if not provided
  params[:cache_at] ||= cache_at.presence
  params[:frequency_penalty] ||= frequency_penalty.presence
  params[:logit_bias] ||= logit_bias.presence
  params[:logprobs] ||= logprobs.presence
  params[:max_completion_tokens] ||= max_completion_tokens.presence || configuration.max_completion_tokens
  params[:max_tokens] ||= max_tokens.presence || configuration.max_tokens
  params[:min_p] ||= min_p.presence
  params[:prediction] = { type: "content", content: params[:prediction] || prediction } if params[:prediction] || prediction.present?
  params[:presence_penalty] ||= presence_penalty.presence
  params[:provider] ||= provider.presence
  params[:repetition_penalty] ||= repetition_penalty.presence
  params[:response_format] ||= response_format.presence
  params[:seed] ||= seed.presence
  params[:stop] ||= stop.presence
  params[:temperature] ||= temperature.presence || configuration.temperature
  params[:tool_choice] ||= tool_choice.presence
  params[:tools] = if available_tools == false
                     nil
                   elsif available_tools.is_a?(Array)
                     filtered_tools(available_tools)
                   else
                     tools.presence
                   end
  params[:top_a] ||= top_a.presence
  params[:top_k] ||= top_k.presence
  params[:top_logprobs] ||= top_logprobs.presence
  params[:top_p] ||= top_p.presence

  json = true if params[:response_format].is_a?(Raix::ResponseFormat)

  if json
    unless openai
      params[:provider] ||= {}
      params[:provider][:require_parameters] = true
    end
    if params[:response_format].blank?
      params[:response_format] ||= {}
      params[:response_format][:type] = "json_object"
    end
  end

  # Deprecation warning for loop parameter
  if loop
    warn "\n\nWARNING: The 'loop' parameter is DEPRECATED and will be ignored.\nChat completions now automatically continue after tool calls until the AI provides a text response.\nUse 'max_tool_calls' to limit the number of tool calls (default: #{configuration.max_tool_calls}).\n\n"
  end

  # Set max_tool_calls from parameter or configuration default
  self.max_tool_calls = max_tool_calls || configuration.max_tool_calls

  # Reset stop_tool_calls_and_respond flag
  @stop_tool_calls_and_respond = false

  # Track tool call count
  tool_call_count = 0

  # set the model to the default if not provided
  self.model ||= configuration.model

  adapter = MessageAdapters::Base.new(self)

  # duplicate the transcript to avoid race conditions in situations where
  # chat_completion is called multiple times in parallel
  # TODO: Defensive programming, ensure messages is an array
  messages ||= transcript.flatten.compact
  messages = messages.map { |msg| adapter.transform(msg) }.dup
  raise "Can't complete an empty transcript" if messages.blank?

  begin
    response = if openai
                 openai_request(params:, model: openai, messages:)
               else
                 openrouter_request(params:, model:, messages:)
               end
    retry_count = 0
    content = nil

    # no need for additional processing if streaming
    return if stream && response.blank?

    # tuck the full response into a thread local in case needed
    Thread.current[:chat_completion_response] = response.with_indifferent_access

    # TODO: add a standardized callback hook for usage events
    # broadcast(:usage_event, usage_subject, self.class.name.to_s, response, premium?)

    tool_calls = response.dig("choices", 0, "message", "tool_calls") || []
    if tool_calls.any?
      tool_call_count += tool_calls.size

      # Check if we've exceeded max_tool_calls
      if tool_call_count > self.max_tool_calls
        # Add system message about hitting the limit
        messages << { role: "system", content: "Maximum tool calls (#{self.max_tool_calls}) exceeded. Please provide a final response to the user without calling any more tools." }

        # Force a final response without tools
        params[:tools] = nil
        response = if openai
                     openai_request(params:, model: openai, messages:)
                   else
                     openrouter_request(params:, model:, messages:)
                   end

        # Process the final response
        content = response.dig("choices", 0, "message", "content")
        transcript << { assistant: content } if save_response
        return raw ? response : content.strip
      end

      # Dispatch tool calls
      tool_calls.each do |tool_call| # TODO: parallelize this?
        # dispatch the called function
        function_name = tool_call["function"]["name"]
        arguments = JSON.parse(tool_call["function"]["arguments"].presence || "{}")
        raise "Unauthorized function call: #{function_name}" unless self.class.functions.map { |f| f[:name].to_sym }.include?(function_name.to_sym)

        dispatch_tool_function(function_name, arguments.with_indifferent_access)
      end

      # After executing tool calls, we need to continue the conversation
      # to let the AI process the results and provide a text response.
      # We continue until the AI responds with a regular assistant message
      # (not another tool call request), unless stop_tool_calls_and_respond! was called.

      # Use the updated transcript for the next call, not the original messages
      updated_messages = transcript.flatten.compact
      last_message = updated_messages.last

      if !@stop_tool_calls_and_respond && (last_message[:role] != "assistant" || last_message[:tool_calls].present?)
        # Send the updated transcript back to the AI
        return chat_completion(
          params:,
          json:,
          raw:,
          openai:,
          save_response:,
          messages: nil, # Use transcript instead
          available_tools:,
          max_tool_calls: self.max_tool_calls - tool_call_count
        )
      elsif @stop_tool_calls_and_respond
        # If stop_tool_calls_and_respond was set, force a final response without tools
        params[:tools] = nil
        response = if openai
                     openai_request(params:, model: openai, messages:)
                   else
                     openrouter_request(params:, model:, messages:)
                   end

        content = response.dig("choices", 0, "message", "content")
        transcript << { assistant: content } if save_response
        return raw ? response : content.strip
      end
    end

    response.tap do |res|
      content = res.dig("choices", 0, "message", "content")

      transcript << { assistant: content } if save_response
      content = content.strip

      if json
        # Make automatic JSON parsing available to non-OpenAI providers that don't support the response_format parameter
        content = content.match(%r{<json>(.*?)</json>}m)[1] if content.include?("<json>")

        return JSON.parse(content)
      end

      return content unless raw
    end
  rescue JSON::ParserError => e
    if e.message.include?("not a valid") # blank JSON
      warn "Retrying blank JSON response... (#{retry_count} attempts) #{e.message}"
      retry_count += 1
      sleep 1 * retry_count # backoff
      retry if retry_count < 3

      raise e # just fail if we can't get content after 3 attempts
    end

    warn "Bad JSON received!!!!!!: #{content}"
    raise e
  rescue Faraday::BadRequestError => e
    # make sure we see the actual error message on console or Honeybadger
    warn "Chat completion failed!!!!!!!!!!!!!!!!: #{e.response[:body]}"
    raise e
  end
end

#configuration ⇒ Object

Instance level access to the class-level configuration.



61
62
63
 
# File 'lib/raix/chat_completion.rb', line 61

def configuration
  self.class.configuration
end

#dispatch_tool_function(function_name, arguments, cache: nil) ⇒ Object

Dispatches a tool function call with the given function name and arguments. This method can be overridden in subclasses to customize how function calls are handled.

Parameters:

  • function_name (String) —

    The name of the function to call

  • arguments (Hash) —

    The arguments to pass to the function

  • cache (ActiveSupport::Cache) (defaults to: nil) —

    Optional cache object

Returns:

  • (Object) —

    The result of the function call



292
293
294
 
# File 'lib/raix/chat_completion.rb', line 292

def dispatch_tool_function(function_name, arguments, cache: nil)
  public_send(function_name, arguments, cache)
end

#transcript ⇒ Array

This method returns the transcript array. Manually add your messages to it in the following abbreviated format before calling ‘chat_completion`.

{ system: “You are a pumpkin” }, { user: “Hey what time is it?” }, { assistant: “Sorry, pumpkins do not wear watches” }

to add a function call use the following format: { function: { name: ‘fancy_pants_function’, arguments: { param: ‘value’ } } }

to add a function result use the following format: { function: result, name: ‘fancy_pants_function’ }

Returns:

  • (Array) —

    The transcript array.



281
282
283
 
# File 'lib/raix/chat_completion.rb', line 281

def transcript
  @transcript ||= []
end