Class: Ai4r::NeuralNetwork::Backpropagation

Inherits:
Object
  • Object
show all
Includes:
Data::Parameterizable
Defined in:
lib/ai4r/neural_network/backpropagation.rb

Overview

Introduction

This is an implementation of a multilayer perceptron network, using the backpropagation algorithm for learning.

Backpropagation is a supervised learning technique (described by Paul Werbos in 1974, and further developed by David E. Rumelhart, Geoffrey E. Hinton and Ronald J. Williams in 1986)

Features

  • Support for any network architecture (number of layers and neurons)

  • Configurable propagation function

  • Optional usage of bias

  • Configurable momentum

  • Configurable learning rate

  • Configurable initial weight function

  • 100% ruby code, no external dependency

Parameters

Use class method get_parameters_info to obtain details on the algorithm parameters. Use set_parameters to set values for this parameters.

  • :disable_bias => If true, the algorithm will not use bias nodes. False by default.

  • :initial_weight_function => f(n, i, j) must return the initial weight for the conection between the node i in layer n, and node j in layer n+1. By default a random number in [-1, 1) range.

  • :propagation_function => By default: lambda { |x| 1/(1+Math.exp(-1*(x))) }

  • :derivative_propagation_function => Derivative of the propagation function, based on propagation function output. By default: lambda { |y| y*(1-y) }, where y=propagation_function(x)

  • :activation => Built-in activation name (:sigmoid, :tanh or :relu). Selecting this overrides propagation_function and derivative_propagation_function. Default: :sigmoid

  • :learning_rate => By default 0.25

  • :momentum => By default 0.1. Set this parameter to 0 to disable momentum

How to use it

# Create the network with 4 inputs, 1 hidden layer with 3 neurons,
# and 2 outputs
net = Ai4r::NeuralNetwork::Backpropagation.new([4, 3, 2])

# Train the network
1000.times do |i|
  net.train(example[i], result[i])
end

# Use it: Evaluate data with the trained network
net.eval([12, 48, 12, 25])
  =>  [0.86, 0.01]

More about multilayer perceptron neural networks and backpropagation:

About the project

Author

Sergio Fierens

License

MPL 1.1

Url

github.com/SergioFierens/ai4r

Instance Attribute Summary collapse

Instance Method Summary collapse

Methods included from Data::Parameterizable

#get_parameters, included, #set_parameters

Constructor Details

#initialize(network_structure, activation = :sigmoid, weight_init = :uniform) ⇒ Object

Creates a new network specifying the its architecture. E.g.

net = Backpropagation.new([4, 3, 2])  # 4 inputs
                                      # 1 hidden layer with 3 neurons,
                                      # 2 outputs
net = Backpropagation.new([2, 3, 3, 4])   # 2 inputs
                                          # 2 hidden layer with 3 neurons each,
                                          # 4 outputs
net = Backpropagation.new([2, 1])   # 2 inputs
                                    # No hidden layer
                                    # 1 output

Parameters:

  • network_structure (Object)
  • activation (Object) (defaults to: :sigmoid)
  • weight_init (Object) (defaults to: :uniform) 


173
174
175
176
177
178
179
180
181
182
183
184
185
 
# File 'lib/ai4r/neural_network/backpropagation.rb', line 173

def initialize(network_structure, activation = :sigmoid, weight_init = :uniform)
  @structure = network_structure
  self.weight_init = weight_init
  @custom_propagation = false
  @set_by_loss = true
  self.activation = activation
  @activation_overridden = (activation != :sigmoid)
  @set_by_loss = false
  @disable_bias = false
  @learning_rate = 0.25
  @momentum = 0.1
  @loss_function = :mse
end

Instance Attribute Details

#activation_nodes ⇒ Object

Returns the value of attribute activation_nodes.



93
94
95
 
# File 'lib/ai4r/neural_network/backpropagation.rb', line 93

def activation_nodes
  @activation_nodes
end

#last_changes ⇒ Object

Returns the value of attribute last_changes.



93
94
95
 
# File 'lib/ai4r/neural_network/backpropagation.rb', line 93

def last_changes
  @last_changes
end

#structure ⇒ Object

Returns the value of attribute structure.



93
94
95
 
# File 'lib/ai4r/neural_network/backpropagation.rb', line 93

def structure
  @structure
end

#weights ⇒ Object

Returns the value of attribute weights.



93
94
95
 
# File 'lib/ai4r/neural_network/backpropagation.rb', line 93

def weights
  @weights
end

Instance Method Details

#activation ⇒ Object

Returns:

  • (Object) 


120
121
122
123
124
125
126
127
128
129
130
 
# File 'lib/ai4r/neural_network/backpropagation.rb', line 120

def activation
  if @activation.is_a?(Array)
    if @set_by_loss || (@loss_function == :cross_entropy && @activation_overridden)
      @activation.first
    else
      @activation
    end
  else
    @activation
  end
end

#activation=(symbols) ⇒ Object

When the activation parameter changes, update internal lambdas for each layer. Accepts a single symbol or an array of symbols (one for each layer except the input layer).

Parameters:

  • symbols (Object)

Returns:

  • (Object) 


100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
 
# File 'lib/ai4r/neural_network/backpropagation.rb', line 100

def activation=(symbols)
  symbols = [symbols] unless symbols.is_a?(Array)
  layer_count = @structure.length - 1
  if symbols.length == 1
    symbols = Array.new(layer_count, symbols.first)
  elsif symbols.length != layer_count
    raise ArgumentError, "Activation array size must match number of layers (#{layer_count})"
  end
  @activation = symbols
  @propagation_functions = @activation.map do |a|
    Ai4r::NeuralNetwork::ActivationFunctions::FUNCTIONS[a] ||
      Ai4r::NeuralNetwork::ActivationFunctions::FUNCTIONS[:sigmoid]
  end
  @derivative_functions = @activation.map do |a|
    Ai4r::NeuralNetwork::ActivationFunctions::DERIVATIVES[a] ||
      Ai4r::NeuralNetwork::ActivationFunctions::DERIVATIVES[:sigmoid]
  end
end

#eval(input_values) ⇒ Object

net.eval([25, 32.3, 12.8, 1.5])

# =>  [0.83, 0.03]

Parameters:

  • input_values (Object)

Returns:

  • (Object) 


191
192
193
194
195
196
 
# File 'lib/ai4r/neural_network/backpropagation.rb', line 191

def eval(input_values)
  check_input_dimension(input_values.length)
  init_network unless @weights
  feedforward(input_values)
  @activation_nodes.last.clone
end

#eval_result(input_values) ⇒ Object

Evaluates the input and returns most active node E.g.

net = Backpropagation.new([4, 3, 2])
net.eval_result([25, 32.3, 12.8, 1.5])
    # eval gives [0.83, 0.03]
    # =>  0

Parameters:

  • input_values (Object)

Returns:

  • (Object) 


206
207
208
209
 
# File 'lib/ai4r/neural_network/backpropagation.rb', line 206

def eval_result(input_values)
  result = eval(input_values)
  result.index(result.max)
end

#init_network ⇒ Object

Initialize (or reset) activation nodes and weights, with the provided net structure and parameters.

Returns:

  • (Object) 


347
348
349
350
351
352
 
# File 'lib/ai4r/neural_network/backpropagation.rb', line 347

def init_network
  init_activation_nodes
  init_weights
  init_last_changes
  self
end

#loss_function=(symbol) ⇒ Object

Parameters:

  • symbol (Object)

Returns:

  • (Object) 


148
149
150
151
152
153
154
155
 
# File 'lib/ai4r/neural_network/backpropagation.rb', line 148

def loss_function=(symbol)
  @loss_function = symbol
  return unless symbol == :cross_entropy && !@activation_overridden && !@custom_propagation

  @set_by_loss = true
  self.activation = :softmax
  @activation_overridden = false
end

#train(inputs, outputs) ⇒ Object

This method trains the network using the backpropagation algorithm.

input: Networks input

output: Expected output for the given input.

This method returns the training loss according to loss_function.

Parameters:

  • inputs (Object)
  • outputs (Object)

Returns:

  • (Object) 


221
222
223
224
225
 
# File 'lib/ai4r/neural_network/backpropagation.rb', line 221

def train(inputs, outputs)
  eval(inputs)
  backpropagate(outputs)
  calculate_loss(outputs, @activation_nodes.last)
end

#train_batch(batch_inputs, batch_outputs) ⇒ Object

Train a list of input/output pairs and return average loss.

Parameters:

  • batch_inputs (Object)
  • batch_outputs (Object)

Returns:

  • (Object) 


231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
 
# File 'lib/ai4r/neural_network/backpropagation.rb', line 231

def train_batch(batch_inputs, batch_outputs)
  if batch_inputs.length != batch_outputs.length
    raise ArgumentError,
          'Inputs and outputs size mismatch'
  end

  batch_size = batch_inputs.length
  init_network unless @weights

  accumulated_changes = Array.new(@weights.length) do |w|
    Array.new(@weights[w].length) do |i|
      Array.new(@weights[w][i].length, 0.0)
    end
  end

  sum_error = 0.0
  batch_inputs.each_index do |idx|
    inputs = batch_inputs[idx]
    outputs = batch_outputs[idx]
    eval(inputs)
    calculate_output_deltas(outputs)
    calculate_internal_deltas

    (@weights.length - 1).downto(0) do |n|
      @weights[n].each_index do |i|
        @weights[n][i].each_index do |j|
          change = @deltas[n][j] * @activation_nodes[n][i]
          accumulated_changes[n][i][j] += change
        end
      end
    end

    sum_error += calculate_loss(outputs, @activation_nodes.last)
  end

  (@weights.length - 1).downto(0) do |n|
    @weights[n].each_index do |i|
      @weights[n][i].each_index do |j|
        avg_change = accumulated_changes[n][i][j] / batch_size.to_f
        @weights[n][i][j] += (learning_rate * avg_change) + (momentum * @last_changes[n][i][j])
        @last_changes[n][i][j] = avg_change
      end
    end
  end

  sum_error / batch_size.to_f
end

#train_epochs(data_inputs, data_outputs, epochs:, batch_size: 1, early_stopping_patience: nil, min_delta: 0.0, shuffle: true, random_seed: nil, &block) ⇒ Object

Train for a number of epochs over the dataset. Optionally define a batch size. Data can be shuffled between epochs passing shuffle: true (default). Use random_seed to make shuffling deterministic. Returns an array with the average loss of each epoch.

Returns:

  • (Object) 


284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
 
# File 'lib/ai4r/neural_network/backpropagation.rb', line 284

def train_epochs(data_inputs, data_outputs, epochs:, batch_size: 1,
                 early_stopping_patience: nil, min_delta: 0.0,
                 shuffle: true, random_seed: nil, &block)
  if data_inputs.length != data_outputs.length
    raise ArgumentError,
          'Inputs and outputs size mismatch'
  end

  losses = []
  best_loss = Float::INFINITY
  patience = early_stopping_patience
  patience_counter = 0
  rng = random_seed.nil? ? Random.new : Random.new(random_seed)
  epochs.times do |epoch|
    epoch_error = 0.0
    epoch_inputs = data_inputs
    epoch_outputs = data_outputs
    if shuffle
      indices = (0...data_inputs.length).to_a.shuffle(random: rng)
      epoch_inputs = data_inputs.values_at(*indices)
      epoch_outputs = data_outputs.values_at(*indices)
    end
    index = 0
    while index < epoch_inputs.length
      batch_in = epoch_inputs[index, batch_size]
      batch_out = epoch_outputs[index, batch_size]
      batch_error = train_batch(batch_in, batch_out)
      epoch_error += batch_error * batch_in.length
      index += batch_size
    end
    epoch_loss = epoch_error / data_inputs.length.to_f
    losses << epoch_loss
    if block
      if block.arity >= 3
        correct = 0
        data_inputs.each_index do |i|
          output = eval(data_inputs[i])
          predicted = output.index(output.max)
          expected = data_outputs[i].index(data_outputs[i].max)
          correct += 1 if predicted == expected
        end
        accuracy = correct.to_f / data_inputs.length
        block.call(epoch, epoch_loss, accuracy)
      else
        block.call(epoch, epoch_loss)
      end
    end
    if patience
      if best_loss - epoch_loss > min_delta
        best_loss = epoch_loss
        patience_counter = 0
      else
        patience_counter += 1
        break if patience_counter >= patience
      end
    end
  end
  losses
end

#weight_init=(symbol) ⇒ Object

Parameters:

  • symbol (Object)

Returns:

  • (Object) 


134
135
136
137
138
139
140
141
142
143
144
 
# File 'lib/ai4r/neural_network/backpropagation.rb', line 134

def weight_init=(symbol)
  @weight_init = symbol
  @initial_weight_function = case symbol
                             when :xavier
                               Ai4r::NeuralNetwork::WeightInitializations.xavier(@structure)
                             when :he
                               Ai4r::NeuralNetwork::WeightInitializations.he(@structure)
                             else
                               Ai4r::NeuralNetwork::WeightInitializations.uniform
                             end
end