Class: SwarmMemory::Core::SemanticIndex

Inherits:
Object
  • Object
show all
Defined in:
lib/swarm_memory/core/semantic_index.rb

Overview

Semantic search abstraction layer

Provides embedding computation and semantic search operations that work with any storage adapter. Easily replaceable with vector database implementations (Qdrant, Milvus, Chroma, etc.)

Uses hybrid search: combines semantic similarity with keyword matching for better recall accuracy.

Examples:

index = SemanticIndex.new(adapter: adapter, embedder: embedder)
results = index.search(query: "how to debug", top_k: 5, threshold: 0.7)

Constant Summary collapse

DEFAULT_SEMANTIC_WEIGHT =

Default weights for hybrid scoring (optimal: 50/50 discovered via systematic evaluation) Configurable via ENV vars: SWARM_MEMORY_SEMANTIC_WEIGHT, SWARM_MEMORY_KEYWORD_WEIGHT

(ENV["SWARM_MEMORY_SEMANTIC_WEIGHT"] || "0.5").to_f
DEFAULT_KEYWORD_WEIGHT = 
(ENV["SWARM_MEMORY_KEYWORD_WEIGHT"] || "0.5").to_f

Instance Method Summary collapse

Constructor Details

#initialize(adapter:, embedder:, semantic_weight: DEFAULT_SEMANTIC_WEIGHT, keyword_weight: DEFAULT_KEYWORD_WEIGHT) ⇒ SemanticIndex

Returns a new instance of SemanticIndex.

Parameters:

  • adapter (Adapters::Base) —

    Storage adapter

  • embedder (Embeddings::Embedder) —

    Embedding model

  • semantic_weight (Float) (defaults to: DEFAULT_SEMANTIC_WEIGHT) —

    Weight for semantic similarity (0.0-1.0)

  • keyword_weight (Float) (defaults to: DEFAULT_KEYWORD_WEIGHT) —

    Weight for keyword matching (0.0-1.0)



27
28
29
30
31
32
 
# File 'lib/swarm_memory/core/semantic_index.rb', line 27

def initialize(adapter:, embedder:, semantic_weight: DEFAULT_SEMANTIC_WEIGHT, keyword_weight: DEFAULT_KEYWORD_WEIGHT)
  @adapter = adapter
  @embedder = embedder
  @semantic_weight = semantic_weight
  @keyword_weight = keyword_weight
end

Instance Method Details

#compute_embedding(text) ⇒ Array<Float>

Compute embedding for text

Parameters:

  • text (String) —

    Text to embed

Returns:

  • (Array<Float>) —

    Embedding vector



38
39
40
 
# File 'lib/swarm_memory/core/semantic_index.rb', line 38

def compute_embedding(text)
  @embedder.embed(text)
end

#find_similar(embedding:, top_k: 10, threshold: 0.0, filter: nil) ⇒ Array<Hash>

Find similar entries by embedding vector

Parameters:

  • embedding (Array<Float>) —

    Embedding vector

  • top_k (Integer) (defaults to: 10) —

    Number of results to return

  • threshold (Float) (defaults to: 0.0) —

    Minimum similarity score (0.0-1.0)

  • filter (Hash, nil) (defaults to: nil) —

    Optional metadata filters

Returns:

  • (Array<Hash>) —

    Similar entries sorted by similarity descending



97
98
99
100
101
102
103
104
105
106
107
108
109
 
# File 'lib/swarm_memory/core/semantic_index.rb', line 97

def find_similar(embedding:, top_k: 10, threshold: 0.0, filter: nil)
  results = @adapter.semantic_search(
    embedding: embedding,
    top_k: top_k * 2,
    threshold: threshold,
  )

  # Apply metadata filters if provided
  results = apply_filters(results, filter) if filter

  # Return top K after filtering
  results.take(top_k)
end

#search(query:, top_k: 10, threshold: 0.0, filter: nil) ⇒ Array<Hash>

Semantic search by text query

Examples:

results = index.search(
  query: "how to create a swarm",
  top_k: 3,
  threshold: 0.65,
  filter: { "type" => "skill" }
)

results.each do |result|
  puts "#{result[:path]} (#{result[:similarity]})"
  puts result[:title]
end

Parameters:

  • query (String) —

    Search query

  • top_k (Integer) (defaults to: 10) —

    Number of results to return

  • threshold (Float) (defaults to: 0.0) —

    Minimum similarity score (0.0-1.0)

  • filter (Hash, nil) (defaults to: nil) —

    Optional metadata filters (e.g., { "type" => "skill" })

Returns:

  • (Array<Hash>) —

    Results with similarity scores, sorted by similarity descending



62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
 
# File 'lib/swarm_memory/core/semantic_index.rb', line 62

def search(query:, top_k: 10, threshold: 0.0, filter: nil)
  # Extract keywords from query for keyword matching
  query_keywords = extract_keywords(query)

  # Compute query embedding
  query_embedding = compute_embedding(query)

  # Delegate to adapter-specific search (gets semantic similarity only)
  # Use threshold of 0.0 to get all results, we'll filter after hybrid scoring
  results = @adapter.semantic_search(
    embedding: query_embedding,
    top_k: top_k * 3, # Get extra for reranking
    threshold: 0.0,   # No threshold yet - will apply after hybrid scoring
  )

  # Calculate hybrid scores (semantic + keyword)
  results = calculate_hybrid_scores(results, query_keywords)

  # Apply metadata filters if provided
  results = apply_filters(results, filter) if filter

  # Filter by threshold on hybrid score
  results = results.select { |r| r[:similarity] >= threshold }

  # Return top K after filtering and reranking
  results.take(top_k)
end