Class: LLM::LlamaCpp

Inherits:
OpenAI show all
Defined in:
lib/llm/shell/internal/llm.rb/lib/llm/providers/llamacpp.rb

Overview

The LlamaCpp class implements a provider for [llama.cpp](github.com/ggml-org/llama.cpp) through the OpenAI-compatible API provided by the llama-server binary. Similar to the ollama provider, this provider supports a wide range of models and is straightforward to run on your own hardware.

Examples:

#!/usr/bin/env ruby
require "llm"

llm = LLM.llamacpp(key: nil)
bot = LLM::Bot.new(llm)
bot.chat ["Tell me about this photo", File.open("/images/frog.jpg", "rb")]
bot.messages.select(&:assistant?).each { print "[#{_1.role}]", _1.content, "\n" }

Constant Summary

Constants inherited from OpenAI

OpenAI::HOST

Instance Method Summary collapse

Methods inherited from OpenAI

#assistant_role, #complete, #embed, #models, #server_tools, #web_search

Methods included from OpenAI::Format

#format

Methods inherited from Provider

#assistant_role, #chat, clients, #complete, #embed, #inspect, #models, #respond, #schema, #server_tool, #server_tools, #web_search, #with

Constructor Details

#initialize(host: "localhost", port: 8080, ssl: false) ⇒ LLM::LlamaCpp

Parameters:

  • key (String, nil) —

    The secret key for authentication

  • host (String) (defaults to: "localhost") —

    The host address of the LLM provider

  • port (Integer) (defaults to: 8080) —

    The port number

  • timeout (Integer) —

    The number of seconds to wait for a response

  • ssl (Boolean) (defaults to: false) —

    Whether to use SSL for the connection

  • persistent (Boolean) —

    Whether to use a persistent connection. Requires the net-http-persistent gem.



26
27
28
 
# File 'lib/llm/shell/internal/llm.rb/lib/llm/providers/llamacpp.rb', line 26

def initialize(host: "localhost", port: 8080, ssl: false, **)
  super
end

Instance Method Details

#audio ⇒ Object

Raises:

  • (NotImplementedError) 


44
45
46
 
# File 'lib/llm/shell/internal/llm.rb/lib/llm/providers/llamacpp.rb', line 44

def audio
  raise NotImplementedError
end

#default_model ⇒ String

Returns the default model for chat completions

Returns:

  • (String)

See Also:



70
71
72
 
# File 'lib/llm/shell/internal/llm.rb/lib/llm/providers/llamacpp.rb', line 70

def default_model
  "qwen3"
end

#files ⇒ Object

Raises:

  • (NotImplementedError) 


32
33
34
 
# File 'lib/llm/shell/internal/llm.rb/lib/llm/providers/llamacpp.rb', line 32

def files
  raise NotImplementedError
end

#images ⇒ Object

Raises:

  • (NotImplementedError) 


38
39
40
 
# File 'lib/llm/shell/internal/llm.rb/lib/llm/providers/llamacpp.rb', line 38

def images
  raise NotImplementedError
end

#moderations ⇒ Object

Raises:

  • (NotImplementedError) 


50
51
52
 
# File 'lib/llm/shell/internal/llm.rb/lib/llm/providers/llamacpp.rb', line 50

def moderations
  raise NotImplementedError
end

#responses ⇒ Object

Raises:

  • (NotImplementedError) 


56
57
58
 
# File 'lib/llm/shell/internal/llm.rb/lib/llm/providers/llamacpp.rb', line 56

def responses
  raise NotImplementedError
end

#vector_stores ⇒ Object

Raises:

  • (NotImplementedError) 


62
63
64
 
# File 'lib/llm/shell/internal/llm.rb/lib/llm/providers/llamacpp.rb', line 62

def vector_stores
  raise NotImplementedError
end