Class: EvalRuby::RetrievalResult

Inherits:
Object
  • Object
show all
Defined in:
lib/eval_ruby/evaluator.rb

Overview

Holds retrieval evaluation results with IR metric accessors.

Examples:

result = EvalRuby.evaluate_retrieval(question: "...", retrieved: [...], relevant: [...])
result.precision_at_k(5) # => 0.6
result.mrr               # => 1.0

Instance Method Summary collapse

Constructor Details

#initialize(retrieved:, relevant:) ⇒ RetrievalResult

Returns a new instance of RetrievalResult.

Parameters:

  • retrieved (Array<String>) —

    retrieved document IDs in ranked order

  • relevant (Array<String>) —

    ground-truth relevant document IDs



86
87
88
89
 
# File 'lib/eval_ruby/evaluator.rb', line 86

def initialize(retrieved:, relevant:)
  @retrieved = retrieved
  @relevant = relevant
end

Instance Method Details

#hit_rate ⇒ Float

Returns 1.0 if any relevant doc is retrieved, 0.0 otherwise.

Returns:

  • (Float) —

    1.0 if any relevant doc is retrieved, 0.0 otherwise



115
116
117
 
# File 'lib/eval_ruby/evaluator.rb', line 115

def hit_rate
  @retrieved.any? { |doc| @relevant.include?(doc) } ? 1.0 : 0.0
end

#mrr ⇒ Float

Returns mean reciprocal rank.

Returns:

  • (Float) —

    mean reciprocal rank



104
105
106
 
# File 'lib/eval_ruby/evaluator.rb', line 104

def mrr
  Metrics::MRR.new.call(retrieved: @retrieved, relevant: @relevant)
end

#ndcg(k: nil) ⇒ Float

Returns normalized discounted cumulative gain.

Parameters:

  • k (Integer, nil) (defaults to: nil) —

    number of top results (nil for all)

Returns:

  • (Float) —

    normalized discounted cumulative gain



110
111
112
 
# File 'lib/eval_ruby/evaluator.rb', line 110

def ndcg(k: nil)
  Metrics::NDCG.new.call(retrieved: @retrieved, relevant: @relevant, k: k)
end

#precision_at_k(k) ⇒ Float

Returns precision at k.

Parameters:

  • k (Integer) —

    number of top results to consider

Returns:

  • (Float) —

    precision at k



93
94
95
 
# File 'lib/eval_ruby/evaluator.rb', line 93

def precision_at_k(k)
  Metrics::PrecisionAtK.new.call(retrieved: @retrieved, relevant: @relevant, k: k)
end

#recall_at_k(k) ⇒ Float

Returns recall at k.

Parameters:

  • k (Integer) —

    number of top results to consider

Returns:

  • (Float) —

    recall at k



99
100
101
 
# File 'lib/eval_ruby/evaluator.rb', line 99

def recall_at_k(k)
  Metrics::RecallAtK.new.call(retrieved: @retrieved, relevant: @relevant, k: k)
end

#to_h ⇒ Hash{Symbol => Float}

Returns all retrieval metrics.

Returns:

  • (Hash{Symbol => Float}) —

    all retrieval metrics



120
121
122
123
124
125
126
127
128
 
# File 'lib/eval_ruby/evaluator.rb', line 120

def to_h
  {
    precision_at_k: precision_at_k(@retrieved.length),
    recall_at_k: recall_at_k(@retrieved.length),
    mrr: mrr,
    ndcg: ndcg,
    hit_rate: hit_rate
  }
end