Class: SVMKit::LinearModel::LogisticRegression

Inherits:
SGDLinearEstimator show all
Includes:
Base::Classifier
Defined in:
lib/svmkit/linear_model/logistic_regression.rb

Overview

LogisticRegression is a class that implements Logistic Regression with mini-batch stochastic gradient descent optimization. For multiclass classification problem, it uses one-vs-the-rest strategy.

Reference

    1. Shalev-Shwartz, Y. Singer, N. Srebro, and A. Cotter, “Pegasos: Primal Estimated sub-GrAdient SOlver for SVM,” Mathematical Programming, vol. 127 (1), pp. 3–30, 2011.

Examples:

estimator =
  SVMKit::LinearModel::LogisticRegression.new(reg_param: 1.0, max_iter: 1000, batch_size: 20, random_seed: 1)
estimator.fit(training_samples, traininig_labels)
results = estimator.predict(testing_samples)

Instance Attribute Summary collapse

Attributes included from Base::BaseEstimator

#params

Instance Method Summary collapse

Methods included from Base::Classifier

#score

Constructor Details

#initialize(reg_param: 1.0, fit_bias: false, bias_scale: 1.0, max_iter: 1000, batch_size: 20, optimizer: nil, random_seed: nil) ⇒ LogisticRegression

Create a new classifier with Logisitc Regression by the SGD optimization.

Parameters:

  • reg_param (Float) (defaults to: 1.0) —

    The regularization parameter.

  • fit_bias (Boolean) (defaults to: false) —

    The flag indicating whether to fit the bias term.

  • bias_scale (Float) (defaults to: 1.0) —

    The scale of the bias term. If fit_bias is true, the feature vector v becoms [v; bias_scale].

  • max_iter (Integer) (defaults to: 1000) —

    The maximum number of iterations.

  • batch_size (Integer) (defaults to: 20) —

    The size of the mini batches.

  • optimizer (Optimizer) (defaults to: nil) —

    The optimizer to calculate adaptive learning rate. If nil is given, Nadam is used.

  • random_seed (Integer) (defaults to: nil) —

    The seed value using to initialize the random generator.



52
53
54
55
56
57
58
59
60
61
 
# File 'lib/svmkit/linear_model/logistic_regression.rb', line 52

def initialize(reg_param: 1.0, fit_bias: false, bias_scale: 1.0,
               max_iter: 1000, batch_size: 20, optimizer: nil, random_seed: nil)
  check_params_float(reg_param: reg_param, bias_scale: bias_scale)
  check_params_integer(max_iter: max_iter, batch_size: batch_size)
  check_params_boolean(fit_bias: fit_bias)
  check_params_type_or_nil(Integer, random_seed: random_seed)
  check_params_positive(reg_param: reg_param, bias_scale: bias_scale, max_iter: max_iter, batch_size: batch_size)
  super
  @classes = nil
end

Instance Attribute Details

#bias_term ⇒ Numo::DFloat (readonly)

Return the bias term (a.k.a. intercept) for Logistic Regression.

Returns:

  • (Numo::DFloat) —

    (shape: [n_classes])



31
32
33
 
# File 'lib/svmkit/linear_model/logistic_regression.rb', line 31

def bias_term
  @bias_term
end

#classes ⇒ Numo::Int32 (readonly)

Return the class labels.

Returns:

  • (Numo::Int32) —

    (shape: [n_classes])



35
36
37
 
# File 'lib/svmkit/linear_model/logistic_regression.rb', line 35

def classes
  @classes
end

#rng ⇒ Random (readonly)

Return the random generator for performing random sampling.

Returns:

  • (Random) 


39
40
41
 
# File 'lib/svmkit/linear_model/logistic_regression.rb', line 39

def rng
  @rng
end

#weight_vec ⇒ Numo::DFloat (readonly)

Return the weight vector for Logistic Regression.

Returns:

  • (Numo::DFloat) —

    (shape: [n_classes, n_features])



27
28
29
 
# File 'lib/svmkit/linear_model/logistic_regression.rb', line 27

def weight_vec
  @weight_vec
end

Instance Method Details

#decision_function(x) ⇒ Numo::DFloat

Calculate confidence scores for samples.

Parameters:

  • x (Numo::DFloat) —

    (shape: [n_samples, n_features]) The samples to compute the scores.

Returns:

  • (Numo::DFloat) —

    (shape: [n_samples, n_classes]) Confidence score per sample.



97
98
99
100
 
# File 'lib/svmkit/linear_model/logistic_regression.rb', line 97

def decision_function(x)
  check_sample_array(x)
  x.dot(@weight_vec.transpose) + @bias_term
end

#fit(x, y) ⇒ LogisticRegression

Fit the model with given training data.

Parameters:

  • x (Numo::DFloat) —

    (shape: [n_samples, n_features]) The training data to be used for fitting the model.

  • y (Numo::Int32) —

    (shape: [n_samples]) The labels to be used for fitting the model.

Returns:



68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
 
# File 'lib/svmkit/linear_model/logistic_regression.rb', line 68

def fit(x, y)
  check_sample_array(x)
  check_label_array(y)
  check_sample_label_size(x, y)

  @classes = Numo::Int32[*y.to_a.uniq.sort]
  n_classes = @classes.size
  n_features = x.shape[1]

  if n_classes > 2
    @weight_vec = Numo::DFloat.zeros(n_classes, n_features)
    @bias_term = Numo::DFloat.zeros(n_classes)
    n_classes.times do |n|
      bin_y = Numo::Int32.cast(y.eq(@classes[n])) * 2 - 1
      @weight_vec[n, true], @bias_term[n] = partial_fit(x, bin_y)
    end
  else
    negative_label = y.to_a.uniq.min
    bin_y = Numo::Int32.cast(y.ne(negative_label)) * 2 - 1
    @weight_vec, @bias_term = partial_fit(x, bin_y)
  end

  self
end

#marshal_dump ⇒ Hash

Dump marshal data.

Returns:

  • (Hash) —

    The marshal data about LogisticRegression.



135
136
137
138
139
140
141
 
# File 'lib/svmkit/linear_model/logistic_regression.rb', line 135

def marshal_dump
  { params: @params,
    weight_vec: @weight_vec,
    bias_term: @bias_term,
    classes: @classes,
    rng: @rng }
end

#marshal_load(obj) ⇒ nil

Load marshal data.

Returns:

  • (nil) 


145
146
147
148
149
150
151
152
 
# File 'lib/svmkit/linear_model/logistic_regression.rb', line 145

def marshal_load(obj)
  @params = obj[:params]
  @weight_vec = obj[:weight_vec]
  @bias_term = obj[:bias_term]
  @classes = obj[:classes]
  @rng = obj[:rng]
  nil
end

#predict(x) ⇒ Numo::Int32

Predict class labels for samples.

Parameters:

  • x (Numo::DFloat) —

    (shape: [n_samples, n_features]) The samples to predict the labels.

Returns:

  • (Numo::Int32) —

    (shape: [n_samples]) Predicted class label per sample.



106
107
108
109
110
111
112
113
114
 
# File 'lib/svmkit/linear_model/logistic_regression.rb', line 106

def predict(x)
  check_sample_array(x)

  return Numo::Int32.cast(predict_proba(x)[true, 1].ge(0.5)) * 2 - 1 if @classes.size <= 2

  n_samples, = x.shape
  decision_values = predict_proba(x)
  Numo::Int32.asarray(Array.new(n_samples) { |n| @classes[decision_values[n, true].max_index] })
end

#predict_proba(x) ⇒ Numo::DFloat

Predict probability for samples.

Parameters:

  • x (Numo::DFloat) —

    (shape: [n_samples, n_features]) The samples to predict the probailities.

Returns:

  • (Numo::DFloat) —

    (shape: [n_samples, n_classes]) Predicted probability of each class per sample.



120
121
122
123
124
125
126
127
128
129
130
131
 
# File 'lib/svmkit/linear_model/logistic_regression.rb', line 120

def predict_proba(x)
  check_sample_array(x)

  proba = 1.0 / (Numo::NMath.exp(-decision_function(x)) + 1.0)
  return (proba.transpose / proba.sum(axis: 1)).transpose if @classes.size > 2

  n_samples, = x.shape
  probs = Numo::DFloat.zeros(n_samples, 2)
  probs[true, 1] = proba
  probs[true, 0] = 1.0 - proba
  probs
end