Class: HybridPlatformsConductor::HpcPlugins::Connector::Ssh

Inherits:
Connector
  • Object
show all
Defined in:
lib/hybrid_platforms_conductor/hpc_plugins/connector/ssh.rb

Overview

Connect to node using SSH

Defined Under Namespace

Modules: PlatformsDslSsh Classes: NotConnectableError

Constant Summary collapse

TMP_SSH_SUB_DIR =

String: Sub-path of the system’s temporary directory where temporary SSH config are generated

'hpc_ssh'
MAX_CMD_ARG_LENGTH =

Integer: Max size for an argument that can be executed without getting through an intermediary file

131_055

Constants included from LoggerHelpers

LoggerHelpers::LEVELS_MODIFIERS, LoggerHelpers::LEVELS_TO_STDERR

Instance Attribute Summary collapse

Instance Method Summary collapse

Methods inherited from Connector

#initialize, #prepare_for

Methods inherited from Plugin

extend_config_dsl_with, #initialize, valid?

Methods included from LoggerHelpers

#err, #init_loggers, #log_component=, #log_debug?, #log_level=, #out, #section, #set_loggers_format, #stderr_device, #stderr_device=, #stderr_displayed?, #stdout_device, #stdout_device=, #stdout_displayed?, #stdouts_to_s, #with_progress_bar

Constructor Details

This class inherits a constructor from HybridPlatformsConductor::Connector

Instance Attribute Details

#auth_password ⇒ Object

Do we expect some connections to require password authentication? [default: false] Boolean



122
123
124
 
# File 'lib/hybrid_platforms_conductor/hpc_plugins/connector/ssh.rb', line 122

def auth_password
  @auth_password
end

#passwords ⇒ Object

Passwords to be used, per node [default: {}] Hash<String, String>



118
119
120
 
# File 'lib/hybrid_platforms_conductor/hpc_plugins/connector/ssh.rb', line 118

def passwords
  @passwords
end

#ssh_gateway_user ⇒ Object

Name of the gateway user to be used. [default: ENV or ubradm]

String



98
99
100
 
# File 'lib/hybrid_platforms_conductor/hpc_plugins/connector/ssh.rb', line 98

def ssh_gateway_user
  @ssh_gateway_user
end

#ssh_gateways_conf ⇒ Object

Name of the gateways configuration, or nil if no gateway. [default: ENV or nil]

Symbol or nil



102
103
104
 
# File 'lib/hybrid_platforms_conductor/hpc_plugins/connector/ssh.rb', line 102

def ssh_gateways_conf
  @ssh_gateways_conf
end

#ssh_strict_host_key_checking ⇒ Object

Do we use strict host key checking in our SSH commands? [default: true] Boolean



110
111
112
 
# File 'lib/hybrid_platforms_conductor/hpc_plugins/connector/ssh.rb', line 110

def ssh_strict_host_key_checking
  @ssh_strict_host_key_checking
end

#ssh_use_control_master ⇒ Object

Do we use the control master? [default: true] Boolean



114
115
116
 
# File 'lib/hybrid_platforms_conductor/hpc_plugins/connector/ssh.rb', line 114

def ssh_use_control_master
  @ssh_use_control_master
end

#ssh_user ⇒ Object

User name used in SSH connections. [default: ENV or ENV]

String



106
107
108
 
# File 'lib/hybrid_platforms_conductor/hpc_plugins/connector/ssh.rb', line 106

def ssh_user
  @ssh_user
end

Instance Method Details

#connectable_nodes_from(nodes) ⇒ Object

Select nodes where this connector can connect.

API

  • This method is mandatory

API

  • @cmd_runner can be used

API

  • @nodes_handler can be used

Parameters

  • nodes (Array<String>): List of candidate nodes

Result

  • Array<String>: List of nodes we can connect to from the candidates



204
205
206
207
 
# File 'lib/hybrid_platforms_conductor/hpc_plugins/connector/ssh.rb', line 204

def connectable_nodes_from(nodes)
  @nodes_handler.prefetch_metadata_of nodes, :host_ip
  nodes.select { |node| @nodes_handler.get_host_ip_of(node) }
end

#init ⇒ Object

Initialize the connector. This can be used to initialize global variables that are used for this connector

API

  • This method is optional

API

  • @cmd_runner can be used

API

  • @nodes_handler can be used



132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
 
# File 'lib/hybrid_platforms_conductor/hpc_plugins/connector/ssh.rb', line 132

def init
  # Default values
  @ssh_user = ENV['hpc_ssh_user']
  @ssh_user = ENV['USER'] if @ssh_user.nil? || @ssh_user.empty?
  if @ssh_user.nil? || @ssh_user.empty?
    _exit_status, stdout = @cmd_runner.run_cmd 'whoami', log_to_stdout: log_debug?
    @ssh_user = stdout.strip
  end
  @ssh_use_control_master = true
  @ssh_strict_host_key_checking = true
  @passwords = {}
  @auth_password = false
  @ssh_gateways_conf = ENV['hpc_ssh_gateways_conf'].nil? ? nil : ENV['hpc_ssh_gateways_conf'].to_sym
  @ssh_gateway_user = ENV['hpc_ssh_gateway_user'].nil? ? 'ubradm' : ENV['hpc_ssh_gateway_user']
  # The map of existing ssh directories that have been created, per node that can access them
  # Array< String, Array<String> >
  @ssh_dirs = {}
  # Mutex protecting the map to make sure it's thread-safe
  @ssh_dirs_mutex = Mutex.new
  # Temporary directory used by all ActionsExecutors, even from different processes
  @tmp_dir = "#{Dir.tmpdir}/#{TMP_SSH_SUB_DIR}"
  FileUtils.mkdir_p @tmp_dir
end

#options_parse(options_parser) ⇒ Object

Complete an option parser with options meant to control this connector

API

  • This method is optional

API

  • @cmd_runner can be used

API

  • @nodes_handler can be used

Parameters

  • options_parser (OptionParser): The option parser to complete



163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
 
# File 'lib/hybrid_platforms_conductor/hpc_plugins/connector/ssh.rb', line 163

def options_parse(options_parser)
  options_parser.on('-g', '--ssh-gateway-user USER', "Name of the gateway user to be used by the gateways. Can also be set from environment variable hpc_ssh_gateway_user. Defaults to #{@ssh_gateway_user}.") do |user|
    @ssh_gateway_user = user
  end
  options_parser.on('-j', '--ssh-no-control-master', 'If used, don\'t create SSH control masters for connections.') do
    @ssh_use_control_master = false
  end
  options_parser.on('-q', '--ssh-no-host-key-checking', 'If used, don\'t check for SSH host keys.') do
    @ssh_strict_host_key_checking = false
  end
  options_parser.on('-u', '--ssh-user USER', 'Name of user to be used in SSH connections (defaults to hpc_ssh_user or USER environment variables)') do |user|
    @ssh_user = user
  end
  options_parser.on('-w', '--password', 'If used, then expect SSH connections to ask for a password.') do
    @auth_password = true
  end
  options_parser.on('-y', '--ssh-gateways-conf GATEWAYS_CONF', 'Name of the gateways configuration to be used. Can also be set from environment variable hpc_ssh_gateways_conf.') do |gateway|
    @ssh_gateways_conf = gateway.to_sym
  end
end

#remote_bash(bash_cmds) ⇒ Object

Run bash commands on a given node.

API

  • This method is mandatory

API

  • If defined, then with_connection_to has been called before this method.

API

  • @cmd_runner can be used

API

  • @nodes_handler can be used

API

  • @node can be used to access the node on which we execute the remote bash

API

  • @timeout can be used to know when the action should fail

API

  • @stdout_io can be used to send stdout output

API

  • @stderr_io can be used to send stderr output

Parameters

  • bash_cmds (String or SecretString): Bash commands to execute. Use #to_unprotected to access the real content (otherwise secrets are obfuscated).



240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
 
# File 'lib/hybrid_platforms_conductor/hpc_plugins/connector/ssh.rb', line 240

def remote_bash(bash_cmds)
  SecretString.protect(
    if @nodes_handler.get_ssh_session_exec_of(@node) == false
      # When ExecSession is disabled we need to use stdin directly
      "{ cat | #{ssh_exec} #{ssh_url} -T; } <<'HPC_EOF'\n#{bash_cmds.to_unprotected}\nHPC_EOF"
    else
      "#{ssh_exec} #{ssh_url} /bin/bash <<'HPC_EOF'\n#{bash_cmds.to_unprotected}\nHPC_EOF"
    end,
    silenced_str:
      if @nodes_handler.get_ssh_session_exec_of(@node) == false
        # When ExecSession is disabled we need to use stdin directly
        "{ cat | #{ssh_exec} #{ssh_url} -T; } <<'HPC_EOF'\n#{bash_cmds}\nHPC_EOF"
      else
        "#{ssh_exec} #{ssh_url} /bin/bash <<'HPC_EOF'\n#{bash_cmds}\nHPC_EOF"
      end
  ) do |ssh_cmd|
    # Due to a limitation of Process.spawn, each individual argument is limited to 128KB of size.
    # Therefore we need to make sure that if bash_cmds exceeds MAX_CMD_ARG_LENGTH bytes (considering EOF chars) then we use an intermediary shell script to store the commands.
    if bash_cmds.to_unprotected.size > MAX_CMD_ARG_LENGTH
      # Write the commands in a file
      temp_file = "#{Dir.tmpdir}/hpc_temp_cmds_#{Digest::MD5.hexdigest(bash_cmds.to_unprotected)}.sh"
      File.open(temp_file, 'w+') do |file|
        file.write ssh_cmd.to_unprotected
        file.chmod 0o700
      end
      begin
        run_cmd(temp_file)
      ensure
        File.unlink(temp_file)
      end
    else
      run_cmd ssh_cmd
    end
  end
end

#remote_copy(from, to, sudo: false, owner: nil, group: nil) ⇒ Object

Copy a file to the remote node in a directory

API

  • This method is mandatory

API

  • If defined, then with_connection_to has been called before this method.

API

  • @cmd_runner can be used

API

  • @nodes_handler can be used

API

  • @node can be used to access the node on which we execute the remote bash

API

  • @timeout can be used to know when the action should fail

API

  • @stdout_io can be used to send stdout output

API

  • @stderr_io can be used to send stderr output

Parameters

  • from (String): Local file to copy

  • to (String): Remote directory to copy to

  • sudo (Boolean): Do we use sudo on the remote to copy? [default: false]

  • owner (String or nil): Owner to be used when copying the files, or nil for current one [default: nil]

  • group (String or nil): Group to be used when copying the files, or nil for current one [default: nil]



312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
 
# File 'lib/hybrid_platforms_conductor/hpc_plugins/connector/ssh.rb', line 312

def remote_copy(from, to, sudo: false, owner: nil, group: nil)
  need_sudo = sudo && !@actions_executor.privileged_access?(@node)
  if @nodes_handler.get_ssh_session_exec_of(@node) == false
    # We don't have ExecSession, so don't use ssh, but scp instead.
    if need_sudo
      # We need to first copy the file in an accessible directory, and then sudo mv
      remote_bash('mkdir -p hpc_tmp_scp')
      run_cmd "scp -S #{ssh_exec} #{from} #{ssh_url}:./hpc_tmp_scp"
      remote_bash("#{@actions_executor.sudo_prefix(@node)}mv ./hpc_tmp_scp/#{File.basename(from)} #{to}")
    else
      run_cmd "scp -S #{ssh_exec} #{from} #{ssh_url}:#{to}"
    end
  else
    run_cmd "      cd \#{File.dirname(from)} && \\\n      tar \\\n        --create \\\n        --gzip \\\n        --file - \\\n        \#{owner.nil? ? '' : \"--owner \#{owner}\"} \\\n        \#{group.nil? ? '' : \"--group \#{group}\"} \\\n        \#{File.basename(from)} | \\\n      \#{ssh_exec} \\\n        \#{ssh_url} \\\n        \\\"\#{need_sudo ? @actions_executor.sudo_prefix(@node) : ''}tar \\\n          --extract \\\n          --gunzip \\\n          --file - \\\n          --directory \#{to} \\\n          --owner root \\\n        \\\"\n    EO_BASH\n  end\nend\n"

#remote_interactive ⇒ Object

Execute an interactive shell on the remote node

API

  • This method is mandatory

API

  • If defined, then with_connection_to has been called before this method.

API

  • @cmd_runner can be used

API

  • @nodes_handler can be used

API

  • @node can be used to access the node on which we execute the remote bash

API

  • @timeout can be used to know when the action should fail

API

  • @stdout_io can be used to send stdout output

API

  • @stderr_io can be used to send stderr output



285
286
287
288
289
290
291
292
293
294
 
# File 'lib/hybrid_platforms_conductor/hpc_plugins/connector/ssh.rb', line 285

def remote_interactive
  interactive_cmd = "#{ssh_exec} #{ssh_url}"
  out interactive_cmd
  # As we're not using run_cmd here, make sure we handle the dry_run switch ourselves
  if @cmd_runner.dry_run
    out 'Won\'t execute interactive shell in dry_run mode'
  else
    system interactive_cmd
  end
end

#ssh_config(ssh_exec: 'ssh', known_hosts_file: nil, nodes: @nodes_handler.known_nodes) ⇒ Object

Get an SSH configuration content giving access to nodes of the platforms with the current configuration

Parameters

  • ssh_exec (String): SSH command to be used [default: ‘ssh’]

  • known_hosts_file (String or nil): Path to the known hosts file, or nil for default [default: nil]

  • nodes (Array<String>): List of nodes to generate the config for [default: @nodes_handler.known_nodes]

Result

  • String: The SSH config



371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
 
# File 'lib/hybrid_platforms_conductor/hpc_plugins/connector/ssh.rb', line 371

def ssh_config(ssh_exec: 'ssh', known_hosts_file: nil, nodes: @nodes_handler.known_nodes)
  config_content = "    ############\n    # GATEWAYS #\n    ############\n\n    \#{@ssh_gateways_conf.nil? || [email protected]_gateways.include?(@ssh_gateways_conf) ? '' : @config.ssh_for_gateway(@ssh_gateways_conf, ssh_exec: ssh_exec, user: @ssh_user)}\n\n    #############\n    # ENDPOINTS #\n    #############\n\n  EO_SSH_CONFIG\n\n  # Add each node\n  # Query for the metadata of all nodes at once\n  @nodes_handler.prefetch_metadata_of nodes, %i[private_ips hostname host_ip description ssh_port]\n  nodes.sort.each do |node|\n    # Generate the conf for the node\n    connection, connection_user, gateway, gateway_user = connection_info_for(node, no_exception: true)\n    if connection.nil?\n      config_content << \"# \#{node} - Not connectable using SSH - \#{@nodes_handler.get_description_of(node) || ''}\\n\"\n    else\n      ssh_port = @nodes_handler.get_ssh_port_of(node)\n      config_content << \"# \#{node} - \#{connection} - \#{@nodes_handler.get_description_of(node) || ''}\\n\"\n      config_content << \"Host \#{ssh_aliases_for(node).join(' ')}\\n\"\n      config_content << \"  Hostname \#{connection}\\n\"\n      config_content << \"  Port \#{ssh_port}\\n\" unless ssh_port.nil?\n      config_content << \"  User \\\"\#{connection_user}\\\"\\n\" if connection_user != @ssh_user\n      config_content << \"  ProxyCommand \#{ssh_exec} -q -W %h:%p \#{gateway_user}@\#{gateway}\\n\" unless gateway.nil?\n      if @passwords.key?(node)\n        config_content << \"  PreferredAuthentications password\\n\"\n        config_content << \"  PubkeyAuthentication no\\n\"\n      end\n    end\n    config_content << \"\\n\"\n  end\n  # Add global definitions at the end of the SSH config, as they might be overriden by previous ones, and first match wins.\n  config_content << <<~EO_SSH_CONFIG\n    ###########\n    # GLOBALS #\n    ###########\n\n    Host *\n      User \#{@ssh_user}\n      # Default control socket path to be used when multiplexing SSH connections\n      ControlPath \#{control_master_file('%h', '%p', '%r')}\n      \#{open_ssh_major_version >= 7 ? 'PubkeyAcceptedKeyTypes +ssh-dss' : ''}\n      \#{known_hosts_file.nil? ? '' : \"UserKnownHostsFile \#{known_hosts_file}\"}\n      \#{@ssh_strict_host_key_checking ? '' : 'StrictHostKeyChecking no'}\n\n  EO_SSH_CONFIG\n  config_content\nend\n"

#ssh_exec ⇒ Object

Get the ssh executable to be used when connecting to the current node

Result

  • String: The ssh executable



351
352
353
 
# File 'lib/hybrid_platforms_conductor/hpc_plugins/connector/ssh.rb', line 351

def ssh_exec
  ssh_exec_for @node
end

#ssh_url ⇒ Object

Get the ssh URL to be used to connect to the current node

Result

  • String: The ssh URL connecting to the current node



359
360
361
 
# File 'lib/hybrid_platforms_conductor/hpc_plugins/connector/ssh.rb', line 359

def ssh_url
  "hpc.#{@node}"
end

#validate_params ⇒ Object

Validate that parsed parameters are valid

API

  • This method is optional

API

  • @cmd_runner can be used

API

  • @nodes_handler can be used



188
189
190
191
192
193
 
# File 'lib/hybrid_platforms_conductor/hpc_plugins/connector/ssh.rb', line 188

def validate_params
  raise 'No SSH user name specified. Please use --ssh-user option or hpc_ssh_user environment variable to set it.' if @ssh_user.nil? || @ssh_user.empty?

  known_gateways = @config.known_gateways
  raise "Unknown gateway configuration provided: #{@ssh_gateways_conf}. Possible values are: #{known_gateways.join(', ')}." if !@ssh_gateways_conf.nil? && !known_gateways.include?(@ssh_gateways_conf)
end

#with_connection_to(nodes, no_exception: false, &block) ⇒ Object

Prepare connections to a given set of nodes. Useful to prefetch metadata or open bulk connections.

API

  • This method is optional

API

  • @cmd_runner can be used

API

  • @nodes_handler can be used

Parameters

  • nodes (Array<String>): Nodes to prepare the connection to

  • no_exception (Boolean): Should we still continue if some nodes have connection errors? [default: false]

  • block (Proc): Code called with the connections prepared.

    • Parameters

      • connected_nodes (Array<String>): The list of connected nodes (should be equal to nodes unless no_exception == true and some nodes failed to connect)



221
222
223
 
# File 'lib/hybrid_platforms_conductor/hpc_plugins/connector/ssh.rb', line 221

def with_connection_to(nodes, no_exception: false, &block)
  with_ssh_master_to(nodes, no_exception: no_exception, &block)
end