Class: OpenTox::Validation

Inherits:
Object
  • Object
show all
Includes:
OpenTox
Defined in:
lib/validation.rb

Instance Attribute Summary

Attributes included from OpenTox

#metadata, #uri

Class Method Summary collapse

Instance Method Summary collapse

Methods included from OpenTox

#add_metadata, all, #delete, #initialize, sign_in, text_to_html, #to_rdfxml

Class Method Details

.create_bootstrapping_validation(params, subjectid = nil, waiting_task = nil) ⇒ OpenTox::Validation

creates a bootstrapping validation, waits until it finishes, may take some time

Parameters:

  • params (Hash) —

    (required:algorithm_uri,dataset_uri,prediction_feature, optional:algorithm_params,random_seed(1))

  • subjectid (String, optional) (defaults to: nil)
  • waiting_task (OpenTox::Task, optional) (defaults to: nil) —

    (can be a OpenTox::Subtask as well), progress is updated accordingly

Returns:



58
59
60
61
62
63
 
# File 'lib/validation.rb', line 58

def self.create_bootstrapping_validation( params, subjectid=nil, waiting_task=nil )
  params[:subjectid] = subjectid if subjectid
  uri = OpenTox::RestClientWrapper.post( File.join(CONFIG[:services]["opentox-validation"],"bootstrapping"),
    params,{:content_type => "text/uri-list"},waiting_task )
  Validation.new(uri)
end

.create_training_test_split(params, subjectid = nil, waiting_task = nil) ⇒ OpenTox::Validation

creates a training test split validation, waits until it finishes, may take some time

Parameters:

  • params (Hash) —

    (required:algorithm_uri,dataset_uri,prediction_feature, optional:algorithm_params,split_ratio(0.67),random_seed(1))

  • subjectid (String, optional) (defaults to: nil)
  • waiting_task (OpenTox::Task, optional) (defaults to: nil) —

    (can be a OpenTox::Subtask as well), progress is updated accordingly

Returns:



34
35
36
37
38
39
 
# File 'lib/validation.rb', line 34

def self.create_training_test_split( params, subjectid=nil, waiting_task=nil )
  params[:subjectid] = subjectid if subjectid
  uri = OpenTox::RestClientWrapper.post( File.join(CONFIG[:services]["opentox-validation"],"training_test_split"),
    params,{:content_type => "text/uri-list"},waiting_task )
  Validation.new(uri)
end

.create_training_test_validation(params, subjectid = nil, waiting_task = nil) ⇒ OpenTox::Validation

creates a training test validation, waits until it finishes, may take some time

Parameters:

  • params (Hash) —

    (required:algorithm_uri,training_dataset_uri,prediction_feature,test_dataset_uri,optional:algorithm_params)

  • subjectid (String, optional) (defaults to: nil)
  • waiting_task (OpenTox::Task, optional) (defaults to: nil) —

    (can be a OpenTox::Subtask as well), progress is updated accordingly

Returns:



46
47
48
49
50
51
 
# File 'lib/validation.rb', line 46

def self.create_training_test_validation( params, subjectid=nil, waiting_task=nil )
  params[:subjectid] = subjectid if subjectid
  uri = OpenTox::RestClientWrapper.post( File.join(CONFIG[:services]["opentox-validation"],"training_test_validation"),
    params,{:content_type => "text/uri-list"},waiting_task )
  Validation.new(uri)
end

.find(uri, subjectid = nil) ⇒ OpenTox::Validation

find validation, raises error if not found

Parameters:

  • uri (String)
  • subjectid (String, optional) (defaults to: nil)

Returns:



10
11
12
13
14
 
# File 'lib/validation.rb', line 10

def self.find( uri, subjectid=nil )
  val = Validation.new(uri)
  val.load_metadata( subjectid )
  val
end

.from_cv_statistics(crossvalidation_uri, subjectid = nil) ⇒ OpenTox::Validation

creates a validation object from crossvaldiation statistics, raise error if not found (as crossvaldiation statistics are returned as an average valdidation over all folds)

Parameters:

  • crossvalidation (String) —

    uri

  • subjectid (String, optional) (defaults to: nil)

Returns:



80
81
82
 
# File 'lib/validation.rb', line 80

def self.from_cv_statistics( crossvalidation_uri, subjectid=nil )
  find( File.join(crossvalidation_uri, 'statistics'),subjectid )
end

.list(params = {}) ⇒ Array

returns a filtered list of validation uris

Parameters:

  • params, (Hash, optional) —

    validation-params to filter the uris (could be model, training_dataset, ..)

Returns:



19
20
21
22
23
24
25
26
27
 
# File 'lib/validation.rb', line 19

def self.list( params={} )
  filter_string = ""
  params.each do |k,v|
    filter_string += (filter_string.length==0 ? "?" : "&")
    v = v.to_s.gsub(/;/, "%3b") if v.to_s =~ /;/
    filter_string += k.to_s+"="+v.to_s
  end
  (OpenTox::RestClientWrapper.get(CONFIG[:services]["opentox-validation"]+filter_string).split("\n"))
end

Instance Method Details

#confusion_matrix ⇒ Object

returns confusion matrix as array, predicted values are in rows example:

[nil,“active”,“moderate”,“inactive”],,[“moderate”,4,2,8],

-> 99 inactive compounds have been predicted as active



94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
 
# File 'lib/validation.rb', line 94

def confusion_matrix
  raise "no classification statistics, probably a regression valdiation" unless @metadata[OT.classificationStatistics]
  matrix =  @metadata[OT.classificationStatistics][OT.confusionMatrix][OT.confusionMatrixCell]
  values = matrix.collect{|cell| cell[OT.confusionMatrixPredicted]}.uniq
  table = [[nil]+values]
  values.each do |c|
    table << [c]
    values.each do |r|
      matrix.each do |cell|
        if cell[OT.confusionMatrixPredicted]==c and cell[OT.confusionMatrixActual]==r
          table[-1] << cell[OT.confusionMatrixValue].to_f
          break
        end
      end
    end
  end
  table
end

#find_or_create_report(subjectid = nil, waiting_task = nil) ⇒ String

looks for report for this validation, creates a report if no report is found

Parameters:

  • subjectid (String, optional) (defaults to: nil)
  • waiting_task (OpenTox::Task, optional) (defaults to: nil) —

    (can be a OpenTox::Subtask as well), progress is updated accordingly

Returns:



69
70
71
72
73
 
# File 'lib/validation.rb', line 69

def find_or_create_report( subjectid=nil, waiting_task=nil )
  @report = ValidationReport.find_for_validation(@uri, subjectid) unless @report
  @report = ValidationReport.create(@uri, {}, subjectid, waiting_task) unless @report
  @report.uri
end

#load_metadata(subjectid = nil) ⇒ Object

loads metadata via yaml from validation object fields (like for example the validated model) can be acces via validation.metadata



86
87
88
 
# File 'lib/validation.rb', line 86

def load_metadata( subjectid=nil )
  @metadata = YAML.load(OpenTox::RestClientWrapper.get(uri,{:subjectid => subjectid, :accept => "application/x-yaml"}))
end

#probabilities(confidence, prediction, subjectid = nil) ⇒ Hash

returns probability-distribution for a given prediction it takes all predictions into account that have a confidence value that is >= confidence and that have the same predicted value (minimum 12 predictions with the hightest confidence are selected (even if the confidence is lower than the given param)

Example 1: validation.probabilities(0.3,“active”) -> :num_predictions=>20, :probs=>{“active”=>0.7, “moderate”=>0.25 “inactive”=>0.05} there have been 20 “active” predictions with confidence >= 0.3, 70 percent of them beeing correct

Example 2: validation.probabilities(0.8,“active”) -> :num_predictions=>12, :probs=>{“active”=>0.9, “moderate”=>0.1 “inactive”=>0} the given confidence value was to high (i.e. <12 predictions with confidence value >= 0.8) the top 12 “active” predictions have a min_confidence of 0.45, 90 percent of them beeing correct

Parameters:

  • confidence (Float) —

    value (between 0 and 1)

  • predicted (String) —

    value

  • subjectid (String, optional) (defaults to: nil)

Returns:

  • (Hash) —

    see example



133
134
135
136
 
# File 'lib/validation.rb', line 133

def probabilities( confidence, prediction, subjectid=nil )
  YAML.load(OpenTox::RestClientWrapper.get(@uri+"/probabilities?prediction="+prediction.to_s+"&confidence="+confidence.to_s,
    {:subjectid => subjectid, :accept => "application/x-yaml"}))
end