Class: Net::SCP

Inherits:
Object
  • Object
show all
Includes:
Download, Upload, Net::SSH::Loggable
Defined in:
lib/net/scp.rb,
lib/net/scp/errors.rb,
lib/net/scp/upload.rb,
lib/net/scp/version.rb,
lib/net/scp/download.rb

Overview

Net::SCP implements the SCP (Secure CoPy) client protocol, allowing Ruby programs to securely and programmatically transfer individual files or entire directory trees to and from remote servers. It provides support for multiple simultaneous SCP copies working in parallel over the same connection, as well as for synchronous, serial copies.

Basic usage:

require 'net/scp'

Net::SCP.start("remote.host", "username", :password => "passwd") do |scp|
  # synchronous (blocking) upload; call blocks until upload completes
  scp.upload! "/local/path", "/remote/path"

  # asynchronous upload; call returns immediately and requires SSH
  # event loop to run
  channel = scp.upload("/local/path", "/remote/path")
  channel.wait
end

Net::SCP also provides an open-uri tie-in, so you can use the Kernel#open method to open and read a remote file:

# if you just want to parse SCP URL's:
require 'uri/scp'
url = URI.parse("scp://[email protected]/path/to/file")

# if you want to read from a URL voa SCP:
require 'uri/open-scp'
puts open("scp://[email protected]/path/to/file").read

Lastly, Net::SCP adds a method to the Net::SSH::Connection::Session class, allowing you to easily grab a Net::SCP reference from an existing Net::SSH session:

require 'net/ssh'
require 'net/scp'

Net::SSH.start("remote.host", "username", :password => "passwd") do |ssh|
  ssh.scp.download! "/remote/path", "/local/path"
end

Progress Reporting

By default, uploading and downloading proceed silently, without any outward indication of their progress. For long running uploads or downloads (and especially in interactive environments) it is desirable to report to the user the progress of the current operation.

To receive progress reports for the current operation, just pass a block to #upload or #download (or one of their variants):

scp.upload!("/path/to/local", "/path/to/remote") do |ch, name, sent, total|
  puts "#{name}: #{sent}/#{total}"
end

Whenever a new chunk of data is recieved for or sent to a file, the callback will be invoked, indicating the name of the file (local for downloads, remote for uploads), the number of bytes that have been sent or received so far for the file, and the size of the file.

–

Protocol Description

Although this information has zero relevance to consumers of the Net::SCP library, I’m documenting it here so that anyone else looking for documentation of the SCP protocol won’t be left high-and-dry like I was. The following is reversed engineered from the OpenSSH SCP implementation, and so may contain errors. You have been warned!

The first step is to invoke the “scp” command on the server. It accepts the following parameters, which must be set correctly to avoid errors:

  • “-t” – tells the remote scp process that data will be sent “to” it, e.g., that data will be uploaded and it should initialize itself accordingly.

  • “-f” – tells the remote scp process that data should come “from” it, e.g., that data will be downloaded and it should initialize itself accordingly.

  • “-v” – verbose mode; the remote scp process should chatter about what it is doing via stderr.

  • “-p” – preserve timestamps. ‘T’ directives (see below) should be/will be sent to indicate the modification and access times of each file.

  • “-r” – recursive transfers should be allowed. Without this, it is an error to upload or download a directory.

After those flags, the name of the remote file/directory should be passed as the sole non-switch argument to scp.

Then the fun begins. If you’re doing a download, enter the download_start_state. Otherwise, look for upload_start_state.

Net::SCP::Download#download_start_state

This is the start state for downloads. It simply sends a 0-byte to the server. The next state is Net::SCP::Download#read_directive_state.

Net::SCP::Upload#upload_start_state

Sets up the initial upload scaffolding and waits for a 0-byte from the server, and then switches to Net::SCP::Upload#upload_current_state.

Net::SCP::Download#read_directive_state

Reads a directive line from the input. The following directives are recognized:

  • T%d %d %d %d – a “times” packet. Indicates that the next file to be downloaded must have mtime/usec/atime/usec attributes preserved.

  • D%o %d %s – a directory change. The process is changing to a directory with the given permissions/size/name, and the recipient should create a directory with the same name and permissions. Subsequent files and directories will be children of this directory, until a matching ‘E’ directive.

  • C%o %d %s – a file is being sent next. The file will have the given permissions/size/name. Immediately following this line, size bytes will be sent, raw.

  • E – terminator directive. Indicates the end of a directory, and subsequent files and directories should be received by the parent of the current directory.

  • 0 – indicates a successful response from the other end.

  • 1 – warning directive. Indicates a warning from the other end. Text from this warning will be reported if the SCP results in an error.

  • 2 – error directive. Indicates an error from the other end. Text from this error will be reported if the SCP results in an error.

If a ‘C’ directive is received, we switch over to Net::SCP::Download#read_data_state. If an ‘E’ directive is received, and there is no parent directory, we switch over to Net::SCP#finish_state.

Regardless of what the next state is, we send a 0-byte to the server before moving to the next state.

Net::SCP::Download#read_data_state

Bytes are read to satisfy the size of the incoming file. When all pending data has been read, we wait for the server to send a 0-byte, and then we switch to the Net::SCP::Download#finish_read_state.

Net::SCP::Download#finish_read_state

We sent a 0-byte to the server to indicate that the file was successfully received. If there is no parent directory, then we’re downloading a single file and we switch to Net::SCP#finish_state. Otherwise we jump back to the Net::SCP::Download#read_directive state to see what we get to download next.

Net::SCP::Upload#upload_current_state

If the current item is a file, send a file. Sending a file starts with a ‘T’ directive (if :preserve is true), then a wait for the server to respond, and then a ‘C’ directive, and then a wait for the server to respond, and then a jump to Net::SCP::Upload#send_data_state.

If current item is a directory, send a ‘D’ directive, and wait for the server to respond with a 0-byte. Then jump to Net::SCP::Upload#next_item_state.

Net::SCP::Upload#send_data_state

Reads and sends the next chunk of data to the server. The state machine remains in this state until all data has been sent, at which point we send a 0-byte to the server, and wait for the server to respond with a 0-byte of its own. Then we jump back to Net::SCP::Upload#next_item_state.

Net::SCP::Upload#next_item_state

If there is nothing left to upload, and there is no parent directory, jump to Net::SCP#finish_state.

If there is nothing left to upload from the current directory, send an ‘E’ directive and wait for the server to respond with a 0-byte. Then go to Net::SCP::Upload#next_item_state.

Otherwise, set the current upload source and go to Net::SCP::Upload#upload_current_state.

Net::SCP#finish_state

Tells the server that no more data is forthcoming from this end of the pipe (via Net::SSH::Connection::Channel#eof!) and leaves the pipe to drain. It will be terminated when the remote process closes with an exit status of zero. ++

Defined Under Namespace

Modules: Download, Upload Classes: Error, Version

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(session) ⇒ SCP

Creates a new Net::SCP session on top of the given Net::SSH session object.



251
252
253
254
 
# File 'lib/net/scp.rb', line 251

def initialize(session)
  @session = session
  self.logger = session.logger
end

Instance Attribute Details

#session ⇒ Object (readonly)

The underlying Net::SSH session that acts as transport for the SCP packets.



247
248
249
 
# File 'lib/net/scp.rb', line 247

def session
  @session
end

Class Method Details

.download!(host, username, remote, local = nil, options = {}, &progress) ⇒ Object

Starts up a new SSH connection using the host and username parameters, instantiates a new SCP session on top of it, and then begins a download from remote to local. If the options hash includes an :ssh key, the value for that will be passed to the SSH connection as options (e.g., to set the password, etc.). All other options are passed to the #download! method. If a block is given, it will be used to report progress (see “Progress Reporting”, under Net::SCP).



238
239
240
241
242
243
 
# File 'lib/net/scp.rb', line 238

def self.download!(host, username, remote, local=nil, options={}, &progress)
  options = options.dup
  start(host, username, options.delete(:ssh) || {}) do |scp|
    return scp.download!(remote, local, options, &progress)
  end
end

.start(host, username, options = {}) ⇒ Object

Starts up a new SSH connection and instantiates a new SCP session on top of it. If a block is given, the SCP session is yielded, and the SSH session is closed automatically when the block terminates. If no block is given, the SCP session is returned.



201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
 
# File 'lib/net/scp.rb', line 201

def self.start(host, username, options={})
  session = Net::SSH.start(host, username, options)
  scp = new(session)

  if block_given?
    begin
      yield scp
      session.loop
    ensure
      session.close
    end
  else
    return scp
  end
end

.upload!(host, username, local, remote, options = {}, &progress) ⇒ Object

Starts up a new SSH connection using the host and username parameters, instantiates a new SCP session on top of it, and then begins an upload from local to remote. If the options hash includes an :ssh key, the value for that will be passed to the SSH connection as options (e.g., to set the password, etc.). All other options are passed to the #upload! method. If a block is given, it will be used to report progress (see “Progress Reporting”, under Net::SCP).



224
225
226
227
228
229
 
# File 'lib/net/scp.rb', line 224

def self.upload!(host, username, local, remote, options={}, &progress)
  options = options.dup
  start(host, username, options.delete(:ssh) || {}) do |scp|
    scp.upload!(local, remote, options, &progress)
  end
end

Instance Method Details

#download(remote, local, options = {}, &progress) ⇒ Object

Inititiate a synchronous (non-blocking) download from remote to local. The following options are recognized:

  • :recursive - the remote parameter refers to a remote directory, which should be downloaded to a new directory named local on the local machine.

  • :preserve - the atime and mtime of the file should be preserved.

  • :verbose - the process should result in verbose output on the server end (useful for debugging).

This method will return immediately, returning the Net::SSH::Connection::Channel object that will support the download. To wait for the download to finish, you can either call the #wait method on the channel, or otherwise run the Net::SSH event loop until the channel’s #active? method returns false.

channel = scp.download("/remote/path", "/local/path")
channel.wait



304
305
306
 
# File 'lib/net/scp.rb', line 304

def download(remote, local, options={}, &progress)
  start_command(:download, local, remote, options, &progress)
end

#download!(remote, local = nil, options = {}, &progress) ⇒ Object

Same as #download, but blocks until the download finishes. Identical to calling #download and then calling the #wait method on the channel object that is returned.

scp.download!("/remote/path", "/local/path")

If local is nil, and the download is not recursive (e.g., it is downloading only a single file), the file will be downloaded to an in-memory buffer and the resulting string returned.

data = download!("/remote/path")



319
320
321
322
323
 
# File 'lib/net/scp.rb', line 319

def download!(remote, local=nil, options={}, &progress)
  destination = local ? local : StringIO.new.tap { |io| io.set_encoding('BINARY') }
  download(remote, destination, options, &progress).wait
  local ? true : destination.string
end

#upload(local, remote, options = {}, &progress) ⇒ Object

Inititiate a synchronous (non-blocking) upload from local to remote. The following options are recognized:

  • :recursive - the local parameter refers to a local directory, which should be uploaded to a new directory named remote on the remote server.

  • :preserve - the atime and mtime of the file should be preserved.

  • :verbose - the process should result in verbose output on the server end (useful for debugging).

  • :chunk_size - the size of each “chunk” that should be sent. Defaults to 2048. Changing this value may improve throughput at the expense of decreasing interactivity.

This method will return immediately, returning the Net::SSH::Connection::Channel object that will support the upload. To wait for the upload to finish, you can either call the #wait method on the channel, or otherwise run the Net::SSH event loop until the channel’s #active? method returns false.

channel = scp.upload("/local/path", "/remote/path")
channel.wait



276
277
278
 
# File 'lib/net/scp.rb', line 276

def upload(local, remote, options={}, &progress)
  start_command(:upload, local, remote, options, &progress)
end

#upload!(local, remote, options = {}, &progress) ⇒ Object

Same as #upload, but blocks until the upload finishes. Identical to calling #upload and then calling the #wait method on the channel object that is returned. The return value is not defined.



283
284
285
 
# File 'lib/net/scp.rb', line 283

def upload!(local, remote, options={}, &progress)
  upload(local, remote, options, &progress).wait
end