Module: Dataflow

Defined in:
lib/dataflow-rb.rb,
lib/dataflow-rb.rb,
lib/dataflow/node.rb,
lib/dataflow/logger.rb,
lib/dataflow/version.rb,
lib/dataflow/event_mixin.rb,
lib/dataflow/schema_mixin.rb,
lib/dataflow/nodes/map_node.rb,
lib/dataflow/nodes/data_node.rb,
lib/dataflow/nodes/join_node.rb,
lib/dataflow/nodes/merge_node.rb,
lib/dataflow/properties_mixin.rb,
lib/dataflow/adapters/settings.rb,
lib/dataflow/nodes/upsert_node.rb,
lib/dataflow/nodes/compute_node.rb,
lib/dataflow/nodes/snapshot_node.rb,
lib/dataflow/adapters/csv_adapter.rb,
lib/dataflow/adapters/sql_adapter.rb,
lib/dataflow/nodes/sql_query_node.rb,
lib/dataflow/adapters/psql_adapter.rb,
lib/dataflow/adapters/mysql_adapter.rb,
lib/dataflow/nodes/select_keys_node.rb,
lib/dataflow/nodes/filter/where_node.rb,
lib/dataflow/nodes/export/to_csv_node.rb,
lib/dataflow/nodes/filter/newest_node.rb,
lib/dataflow/nodes/runtime_query_node.rb,
lib/dataflow/adapters/mongo_db_adapter.rb,
lib/dataflow/nodes/read_only_data_node.rb,
lib/dataflow/errors/not_implemented_error.rb,
lib/dataflow/nodes/filter/drop_while_node.rb,
lib/dataflow/nodes/mixin/rename_dotted_fields.rb,
lib/dataflow/nodes/transformation/to_time_node.rb,
lib/dataflow/errors/invalid_configuration_error.rb,
lib/dataflow/nodes/mixin/add_internal_timestamp.rb

Overview

Override the #constantize in active_support/inflector/methods.rb to rescue from Dataflow::Nodes::… name errors. In such cases, we return a generic Dataflow::Nodes::DataNode instead. This is used within mongoid to instance the correct node types.

Defined Under Namespace

Modules: Adapters, ConstantizePatch, Errors, EventMixin, Node, Nodes, PropertiesMixin, SchemaMixin Classes: Logger

Constant Summary collapse

CsvPath = 
"#{Dir.pwd}/datanodes/csv"
VERSION = 
'0.13.0'

Class Method Summary collapse

Class Method Details

.clear_tmp_datasets ⇒ Object

helper that helps clearing un-used datasets NOTE: although there is a best attempt to not delete datasets that are currently being written to, this is not safe to use while executing in parallel.



78
79
80
 
# File 'lib/dataflow-rb.rb', line 78

def self.clear_tmp_datasets
  Dataflow::Nodes::DataNode.all.each(&:safely_clear_write_dataset)
end

.compute_node(id) ⇒ Object

helper that tries to find a computed node by id and then name



69
70
71
72
73
 
# File 'lib/dataflow-rb.rb', line 69

def self.compute_node(id)
  Dataflow::Nodes::ComputeNode.find(id)
rescue Mongoid::Errors::DocumentNotFound
  Dataflow::Nodes::ComputeNode.find_by(name: id)
end

.data_node(id) ⇒ Object

helper that tries to find a data node by id and then by name



62
63
64
65
66
 
# File 'lib/dataflow-rb.rb', line 62

def self.data_node(id)
  Dataflow::Nodes::DataNode.find(id)
rescue Mongoid::Errors::DocumentNotFound
  Dataflow::Nodes::DataNode.find_by(name: id)
end