Class: SVMKit::LinearModel::LogisticRegression

Inherits:
Object
  • Object
show all
Includes:
Base::BaseEstimator, Base::Classifier
Defined in:
lib/svmkit/linear_model/logistic_regression.rb

Overview

LogisticRegression is a class that implements Logistic Regression with stochastic gradient descent (SGD) optimization. Note that the class performs as a binary classifier.

Reference

    1. Shalev-Shwartz, Y. Singer, N. Srebro, and A. Cotter, “Pegasos: Primal Estimated sub-GrAdient SOlver for SVM,” Mathematical Programming, vol. 127 (1), pp. 3–30, 2011.

Examples:

estimator =
  SVMKit::LinearModel::LogisticRegression.new(reg_param: 1.0, max_iter: 100, batch_size: 20, random_seed: 1)
estimator.fit(training_samples, traininig_labels)
results = estimator.predict(testing_samples)

Instance Attribute Summary collapse

Attributes included from Base::BaseEstimator

#params

Instance Method Summary collapse

Constructor Details

#initialize(reg_param: 1.0, fit_bias: false, bias_scale: 1.0, max_iter: 100, batch_size: 50, random_seed: nil) ⇒ LogisticRegression

Create a new classifier with Logisitc Regression by the SGD optimization.

Parameters:

  • reg_param (Float) (defaults to: 1.0) —

    The regularization parameter.

  • fit_bias (Boolean) (defaults to: false) —

    The flag indicating whether to fit the bias term.

  • bias_scale (Float) (defaults to: 1.0) —

    The scale of the bias term. If fit_bias is true, the feature vector v becoms [v; bias_scale].

  • max_iter (Integer) (defaults to: 100) —

    The maximum number of iterations.

  • batch_size (Integer) (defaults to: 50) —

    The size of the mini batches.

  • random_seed (Integer) (defaults to: nil) —

    The seed value using to initialize the random generator.



44
45
46
47
48
49
50
51
52
53
54
55
56
 
# File 'lib/svmkit/linear_model/logistic_regression.rb', line 44

def initialize(reg_param: 1.0, fit_bias: false, bias_scale: 1.0, max_iter: 100, batch_size: 50, random_seed: nil)
  @params = {}
  @params[:reg_param] = reg_param
  @params[:fit_bias] = fit_bias
  @params[:bias_scale] = bias_scale
  @params[:max_iter] = max_iter
  @params[:batch_size] = batch_size
  @params[:random_seed] = random_seed
  @params[:random_seed] ||= srand
  @weight_vec = nil
  @bias_term = 0.0
  @rng = Random.new(@params[:random_seed])
end

Instance Attribute Details

#bias_term ⇒ Float (readonly)

Return the bias term (a.k.a. intercept) for Logistic Regression.

Returns:

  • (Float) 


29
30
31
 
# File 'lib/svmkit/linear_model/logistic_regression.rb', line 29

def bias_term
  @bias_term
end

#rng ⇒ Random (readonly)

Return the random generator for transformation.

Returns:

  • (Random) 


33
34
35
 
# File 'lib/svmkit/linear_model/logistic_regression.rb', line 33

def rng
  @rng
end

#weight_vec ⇒ Numo::DFloat (readonly)

Return the weight vector for Logistic Regression.

Returns:

  • (Numo::DFloat) —

    (shape: [n_features])



25
26
27
 
# File 'lib/svmkit/linear_model/logistic_regression.rb', line 25

def weight_vec
  @weight_vec
end

Instance Method Details

#decision_function(x) ⇒ Numo::DFloat

Calculate confidence scores for samples.

Parameters:

  • x (Numo::DFloat) —

    (shape: [n_samples, n_features]) The samples to compute the scores.

Returns:

  • (Numo::DFloat) —

    (shape: [n_samples]) Confidence score per sample.



114
115
116
117
 
# File 'lib/svmkit/linear_model/logistic_regression.rb', line 114

def decision_function(x)
  w = Numo::NMath.exp(((@weight_vec.dot(x.transpose) + @bias_term) * -1.0)) + 1.0
  w.map { |v| 1.0 / v }
end

#fit(x, y) ⇒ LogisticRegression

Fit the model with given training data.

Parameters:

  • x (Numo::DFloat) —

    (shape: [n_samples, n_features]) The training data to be used for fitting the model.

  • y (Numo::Int32) —

    (shape: [n_samples]) The categorical variables (e.g. labels) to be used for fitting the model.

Returns:



64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
 
# File 'lib/svmkit/linear_model/logistic_regression.rb', line 64

def fit(x, y)
  # Generate binary labels.
  negative_label = y.to_a.uniq.sort.shift
  bin_y = y.to_a.map { |l| l != negative_label ? 1 : 0 }
  # Expand feature vectors for bias term.
  samples = x
  if @params[:fit_bias]
    samples = Numo::NArray.hstack(
      [samples, Numo::DFloat.ones([x.shape[0], 1]) * @params[:bias_scale]]
    )
  end
  # Initialize some variables.
  n_samples, n_features = samples.shape
  rand_ids = [*0..n_samples - 1].shuffle(random: @rng)
  weight_vec = Numo::DFloat.zeros(n_features)
  # Start optimization.
  @params[:max_iter].times do |t|
    # random sampling
    subset_ids = rand_ids.shift(@params[:batch_size])
    rand_ids.concat(subset_ids)
    # update the weight vector.
    eta = 1.0 / (@params[:reg_param] * (t + 1))
    mean_vec = Numo::DFloat.zeros(n_features)
    subset_ids.each do |n|
      z = weight_vec.dot(samples[n, true])
      coef = bin_y[n] / (1.0 + Math.exp(bin_y[n] * z))
      mean_vec += samples[n, true] * coef
    end
    mean_vec *= eta / @params[:batch_size]
    weight_vec = weight_vec * (1.0 - eta * @params[:reg_param]) + mean_vec
    # scale the weight vector.
    norm = Math.sqrt(weight_vec.dot(weight_vec))
    scaler = (1.0 / @params[:reg_param]**0.5) / (norm + 1.0e-12)
    weight_vec *= [1.0, scaler].min
  end
  # Store the learned model.
  if @params[:fit_bias]
    @weight_vec = weight_vec[0...n_features - 1]
    @bias_term = weight_vec[n_features - 1]
  else
    @weight_vec = weight_vec[0...n_features]
    @bias_term = 0.0
  end
  self
end

#marshal_dump ⇒ Hash

Dump marshal data.

Returns:

  • (Hash) —

    The marshal data about LogisticRegression.



148
149
150
 
# File 'lib/svmkit/linear_model/logistic_regression.rb', line 148

def marshal_dump
  { params: @params, weight_vec: @weight_vec, bias_term: @bias_term, rng: @rng }
end

#marshal_load(obj) ⇒ nil

Load marshal data.

Returns:

  • (nil) 


154
155
156
157
158
159
160
 
# File 'lib/svmkit/linear_model/logistic_regression.rb', line 154

def marshal_load(obj)
  @params = obj[:params]
  @weight_vec = obj[:weight_vec]
  @bias_term = obj[:bias_term]
  @rng = obj[:rng]
  nil
end

#predict(x) ⇒ Numo::Int32

Predict class labels for samples.

Parameters:

  • x (Numo::DFloat) —

    (shape: [n_samples, n_features]) The samples to predict the labels.

Returns:

  • (Numo::Int32) —

    (shape: [n_samples]) Predicted class label per sample.



123
124
125
 
# File 'lib/svmkit/linear_model/logistic_regression.rb', line 123

def predict(x)
  Numo::Int32.cast(decision_function(x).map { |v| v >= 0.5 ? 1 : -1 })
end

#predict_proba(x) ⇒ Numo::DFloat

Predict probability for samples.

Parameters:

  • x (Numo::DFloat) —

    (shape: [n_samples, n_features]) The samples to predict the probailities.

Returns:

  • (Numo::DFloat) —

    (shape: [n_samples]) Predicted probability per sample.



131
132
133
 
# File 'lib/svmkit/linear_model/logistic_regression.rb', line 131

def predict_proba(x)
  decision_function(x)
end

#score(x, y) ⇒ Float

Claculate the mean accuracy of the given testing data.

Parameters:

  • x (Numo::DFloat) —

    (shape: [n_samples, n_features]) Testing data.

  • y (Numo::Int32) —

    (shape: [n_samples]) True labels for testing data.

Returns:

  • (Float) —

    Mean accuracy



140
141
142
143
144
 
# File 'lib/svmkit/linear_model/logistic_regression.rb', line 140

def score(x, y)
  p = predict(x)
  n_hits = (y.to_a.map.with_index { |l, n| l == p[n] ? 1 : 0 }).inject(:+)
  n_hits / y.size.to_f
end