Class: TensorStream::Train::AdamOptimizer

Inherits:
Optimizer
  • Object
show all
Includes:
OpHelper
Defined in:
lib/tensor_stream/train/adam_optimizer.rb

Overview

High Level implementation of the ADAM algorithm

Instance Attribute Summary collapse

Attributes inherited from Optimizer

#name

Instance Method Summary collapse

Methods included from OpHelper

#_op, #cons, #format_source, #fp_type?, #i_cons, #i_op, #i_var, #int_type?, #reduced_shape, #shape_eval, #shape_full_specified, #shapes_fully_specified_and_equal

Methods inherited from Optimizer

#apply_gradients, #compute_gradients, #get_slot, #get_slot_names, #minimize

Methods included from SlotCreator

#create_slot, #create_slot_var, #create_slot_with_initializer, #create_zeros_slot

Methods included from Utils

#__v_scope_name, #apply_data_type_coercion, #assign, #check_allowed_types, #check_data_types, #check_if_dense, #colocate_with, #constant, #control_dependencies, #convert_to_tensor, #device, #disable_eager_execution, #dynamic_stitch, #enable_eager_execution, #executing_eagerly?, #float32, #get_collection, #get_default_graph, #get_variable, #get_variable_scope, #global_variables_initializer, #graph, #group, #image, #layers, #list_local_devices, #math, #name_scope, #placeholder, #program, #reset_default_graph, #session, #set_random_seed, #train, #trainable_variables, #variable, #variable_scope

Constructor Details

#initialize(learning_rate = 0.001, beta1 = 0.9, beta2 = 0.999, epsilon = 1e-8, use_locking: false, name: "Adam") ⇒ AdamOptimizer

Construct a new Adam optimizer.

Args: learning_rate: A Tensor or a floating point value. The learning rate. beta1: A float value or a constant float tensor.

The exponential decay rate for the 1st moment estimates.

beta2: A float value or a constant float tensor.

The exponential decay rate for the 2nd moment estimates.

epsilon: A small constant for numerical stability. This epsilon is

"epsilon hat" in the Kingma and Ba paper (in the formula just before
Section 2.1), not the epsilon in Algorithm 1 of the paper.

use_locking: If True use locks for update operations. name: Optional name for the operations created when applying gradients.

Defaults to "Adam".



24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
 
# File 'lib/tensor_stream/train/adam_optimizer.rb', line 24

def initialize(learning_rate = 0.001, beta1 = 0.9, beta2 = 0.999, epsilon = 1e-8,
  use_locking: false, name: "Adam")
  @learning_rate = learning_rate
  @beta1 = beta1
  @beta2 = beta2
  @epsilon = epsilon

  # Tensor versions of the constructor arguments, created in _prepare().
  @lr_t = nil
  @beta1_t = nil
  @beta2_t = nil
  @epsilon_t = nil

  # Created in SparseApply if needed.
  @updated_lr = nil
  super(name: name, use_locking: use_locking)
end

Instance Attribute Details

#learning_rate ⇒ Object

Returns the value of attribute learning_rate.



7
8
9
 
# File 'lib/tensor_stream/train/adam_optimizer.rb', line 7

def learning_rate
  @learning_rate
end