Module: CircleCI::Parallel

Extended by:
SingleForwardable
Defined in:
lib/circleci/parallel.rb,
lib/circleci/parallel/hook.rb,
lib/circleci/parallel/node.rb,
lib/circleci/parallel/build.rb,
lib/circleci/parallel/version.rb,
lib/circleci/parallel/task/base.rb,
lib/circleci/parallel/task/slave.rb,
lib/circleci/parallel/environment.rb,
lib/circleci/parallel/task/master.rb,
lib/circleci/parallel/configuration.rb,
lib/circleci/parallel/task/mock_slave.rb,
lib/circleci/parallel/task/mock_master.rb,
lib/circleci/parallel/configuration/slave_node_configuration.rb,
lib/circleci/parallel/configuration/master_node_configuration.rb,
lib/circleci/parallel/configuration/configuration_collection_proxy.rb

Overview

Provides simple APIs for syncing CircleCI parallel nodes and transferring files between the nodes.

Examples:

merged_data = {}

CircleCI::Parallel.configure do |config|
  config.on_every_node.before_sync do
    data = do_something
    json = JSON.generate(data)
    File.write('data.json', json)
  end

  config.on_master_node.after_download do
    Dir.glob('*/data.json') do |path|
      json = File.read(path)
      data = JSON.parse(json)
      node_name = File.dirname(path)
      merged_data[node_name] = data
    end
  end
end

CircleCI::Parallel.sync

p merged_data

Defined Under Namespace

Modules: Task, Version Classes: Build, Configuration, Environment, Hook, Node

Constant Summary collapse

WORK_DIR =

This constant is part of a private API. You should avoid using this constant if possible, as it may be removed or be changed in the future. 

'/tmp/circleci-parallel'.freeze
BASE_DATA_DIR =

This constant is part of a private API. You should avoid using this constant if possible, as it may be removed or be changed in the future. 

File.join(WORK_DIR, 'data')
SYNC_MARKER_FILE =

This constant is part of a private API. You should avoid using this constant if possible, as it may be removed or be changed in the future. 

File.join(WORK_DIR, 'SYNCING')
DOWNLOAD_MARKER_FILE =

This constant is part of a private API. You should avoid using this constant if possible, as it may be removed or be changed in the future. 

File.join(WORK_DIR, 'DOWNLOADED')

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.configure {|config| ... }

This method returns an undefined value.

Provides a block for configuring CircleCI::Parallel.

Examples:

CircleCI::Parallel.configure do |config|
  config.silent = true
end

Yield Parameters:

See Also:



74
 
# File 'lib/circleci/parallel.rb', line 74

def_delegator :environment, :configure

.current_build ⇒ Build

Returns the current CircleCI build.

Returns:

  • (Build) —

    the current build

See Also:



85
 
# File 'lib/circleci/parallel.rb', line 85

def_delegator :environment, :current_build

.current_node ⇒ Node

Returns the current CircleCI node.

Returns:

  • (Node) —

    the current node

See Also:



96
 
# File 'lib/circleci/parallel.rb', line 96

def_delegator :environment, :current_node

.download_data_dir ⇒ String

Returns the download data directory where all node data will be downloaded. Note that only master node downloads data from other slave node. When the downloads are complete, the directory structure on the master node will be the following:

.
├── node0
│   └── node_specific_data_you_saved_on_node0.txt
├── node1
│   └── node_specific_data_you_saved_on_node1.txt
└── node2
    └── node_specific_data_you_saved_on_node2.txt

Examples:

Dir.chdir(CircleCI::Parallel.download_data_dir) do
  merged_data = Dir['*/data.json'].each_with_object({}) do |path, merged_data|
    data = JSON.parse(File.read(path))
    node_name = File.dirname(path)
    merged_data[node_name] = data
  end
end

Returns:

  • (String) —

    the download data directory

See Also:

  • CircleCI::Parallel::MasterNodeConfiguration#before_download
  • CircleCI::Parallel::MasterNodeConfiguration#after_download


165
166
167
168
169
 
# File 'lib/circleci/parallel.rb', line 165

def download_data_dir
  BASE_DATA_DIR.tap do |path|
    FileUtils.makedirs(path) unless Dir.exist?(path)
  end
end

.join ⇒ Object

Deprecated.

Use .sync instead.



178
179
180
 
# File 'lib/circleci/parallel.rb', line 178

def join
  sync
end

.local_data_dir ⇒ String

Returns the local data directory where node specific data should be saved in.

Examples:

path = File.join(CircleCI::Parallel.local_data_dir, 'data.json')
File.write(path, JSON.generate(some_data))

Returns:

  • (String) —

    the local data directory

See Also:

  • CircleCI::Parallel::MasterNodeConfiguration#before_sync
  • CircleCI::Parallel::MasterNodeConfiguration#after_sync
  • CircleCI::Parallel::SlaveNodeConfiguration#before_sync
  • CircleCI::Parallel::SlaveNodeConfiguration#after_sync


133
134
135
136
137
 
# File 'lib/circleci/parallel.rb', line 133

def local_data_dir
  current_node.data_dir.tap do |path|
    FileUtils.makedirs(path) unless Dir.exist?(path)
  end
end

.puts ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future. 



118
 
# File 'lib/circleci/parallel.rb', line 118

def_delegator :environment, :puts

.reset! ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future. 



172
173
174
175
 
# File 'lib/circleci/parallel.rb', line 172

def reset!
  environment.clean
  @environment = nil
end

.sync ⇒ Object

Sync all nodes in the same build and gather all node data into the master node. Invoking this method blocks until the sync and data downloads are complete.

Raises:

  • (RuntimeError) —

    when CIRCLECI environment variable is not set

See Also:

  • CircleCI::Parallel::MasterNodeConfiguration#before_sync
  • CircleCI::Parallel::MasterNodeConfiguration#before_download
  • CircleCI::Parallel::MasterNodeConfiguration#after_download
  • CircleCI::Parallel::MasterNodeConfiguration#after_sync
  • CircleCI::Parallel::SlaveNodeConfiguration#before_sync
  • CircleCI::Parallel::SlaveNodeConfiguration#after_sync


113
 
# File 'lib/circleci/parallel.rb', line 113

def_delegator :environment, :sync

Instance Method Details

#configuration ⇒ Object 




# File 'lib/circleci/parallel.rb', line 47