Class: SharedTools::Tools::ComputerTool

Inherits:
RubyLLM::Tool
  • Object
show all
Defined in:
lib/shared_tools/tools/computer_tool.rb

Overview

A tool for interacting with a computer. Be careful with using as it can perform actions on your computer!

Examples:

computer = SharedTools::Tools::ComputerTool.new
computer.execute(action: 'mouse_position')
computer.execute(action: 'type', text: 'Hello')

Defined Under Namespace

Modules: Action, MouseButton, ScrollDirection

Constant Summary collapse

ACTIONS = 
[
  Action::KEY,
  Action::HOLD_KEY,
  Action::MOUSE_POSITION,
  Action::MOUSE_MOVE,
  Action::MOUSE_CLICK,
  Action::MOUSE_DOWN,
  Action::MOUSE_DRAG,
  Action::MOUSE_UP,
  Action::TYPE,
  Action::SCROLL,
  Action::WAIT,
].freeze
MOUSE_BUTTON_OPTIONS = 
[
  MouseButton::LEFT,
  MouseButton::MIDDLE,
  MouseButton::RIGHT,
].freeze
SCROLL_DIRECTION_OPTIONS = 
[
  ScrollDirection::UP,
  ScrollDirection::DOWN,
  ScrollDirection::LEFT,
  ScrollDirection::RIGHT,
].freeze

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(driver: nil, logger: nil) ⇒ ComputerTool

Returns a new instance of ComputerTool.

Parameters:

  • driver (Computer::BaseDriver) (defaults to: nil) —

    optional, will attempt to create platform-specific driver when execute is called

  • logger (Logger) (defaults to: nil) —

    optional logger



144
145
146
147
 
# File 'lib/shared_tools/tools/computer_tool.rb', line 144

def initialize(driver: nil, logger: nil)
  @logger = logger || RubyLLM.logger
  @driver = driver  # Defer default_driver to execute time to support RubyLLM tool discovery
end

Instance Attribute Details

#driver=(value) ⇒ Object (writeonly)

Set driver after instantiation (useful when tool is discovered by RubyLLM)



150
151
152
 
# File 'lib/shared_tools/tools/computer_tool.rb', line 150

def driver=(value)
  @driver = value
end

Class Method Details

.name ⇒ Object 



14
 
# File 'lib/shared_tools/tools/computer_tool.rb', line 14

def self.name = 'computer_tool'

Instance Method Details

#execute(action:, coordinate: nil, text: nil, duration: nil, mouse_button: nil, scroll_direction: nil, scroll_amount: nil) ⇒ Object

Parameters:

  • action (String)
  • coordinate (Hash<{ width: Integer, height: Integer }>) (defaults to: nil) —

    the (x,y) coordinate

  • text (String) (defaults to: nil)
  • duration (Integer) (defaults to: nil) —

    the duration in seconds

  • mouse_button (String) (defaults to: nil) —

    e.g. “left”, “middle”, “right”

  • scroll_direction (String) (defaults to: nil) —

    e.g. “up”, “down”, “left”, “right”

  • scroll_amount (Integer) (defaults to: nil) —

    the amount of clicks to scroll



159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
 
# File 'lib/shared_tools/tools/computer_tool.rb', line 159

def execute(
  action:,
  coordinate: nil,
  text: nil,
  duration: nil,
  mouse_button: nil,
  scroll_direction: nil,
  scroll_amount: nil
)
  # Lazily resolve driver at execute time
  @driver ||= default_driver

  @logger.info({
    action:,
    coordinate:,
    text:,
    duration:,
    mouse_button:,
    scroll_direction:,
    scroll_amount:,
  }.compact.map { |key, value| "#{key}=#{value.inspect}" }.join(" "))

  # Convert coordinate hash keys to symbols if coordinate is provided
  coordinate = coordinate&.transform_keys(&:to_sym)

  case action
  when Action::KEY then @driver.key(text:)
  when Action::HOLD_KEY then @driver.hold_key(text:, duration:)
  when Action::MOUSE_POSITION then @driver.mouse_position
  when Action::MOUSE_MOVE then @driver.mouse_move(coordinate:)
  when Action::MOUSE_CLICK then @driver.mouse_click(coordinate:, button: mouse_button)
  when Action::MOUSE_DOUBLE_CLICK then @driver.mouse_double_click(coordinate:, button: mouse_button)
  when Action::MOUSE_TRIPLE_CLICK then @driver.mouse_triple_click(coordinate:, button: mouse_button)
  when Action::MOUSE_DOWN then @driver.mouse_down(coordinate:, button: mouse_button)
  when Action::MOUSE_UP then @driver.mouse_up(coordinate:, button: mouse_button)
  when Action::MOUSE_DRAG then @driver.mouse_drag(coordinate:, button: mouse_button)
  when Action::TYPE then @driver.type(text:)
  when Action::SCROLL then @driver.scroll(amount: scroll_amount, direction: scroll_direction)
  when Action::WAIT then @driver.wait(duration:)
  end
end