Class: Chimps::Workflows::Downloader

Inherits:
Object
  • Object
show all
Includes:
Utils::UsesCurl
Defined in:
lib/chimps/workflows/downloader.rb

Overview

Downloads data from Infochimps by first making a request for a download token and, if granted one, proceeding to download the data.

Will download the latest package for a given dataset, optionally constrained to have given data and package formats.

Instance Attribute Summary collapse

Instance Method Summary collapse

Methods included from Utils::UsesCurl

#curl

Constructor Details

#initialize(options = {}) ⇒ Chimps::Workflows::Downloader

Create a new Downloader with the given parameters.

Parameters:

  • options (Hash) (defaults to: {})

Options Hash (options):

  • dataset (String, Integer) —

    the ID or handle of the dataset to download

  • fmt (String) —

    the data format to download

  • pkg_fmt (String) —

    the package format to download

  • local_path (String) —

    the local path to which the data will be downloaded



35
36
37
38
39
40
 
# File 'lib/chimps/workflows/downloader.rb', line 35

def initialize options={}
  @dataset    = options[:dataset]
  @fmt        = options[:fmt]
  @pkg_fmt    = options[:pkg_fmt]
  @local_path = options[:local_path]
end

Instance Attribute Details

#dataset ⇒ Object (readonly)

The ID or handle of the dataset to download.



19
20
21
 
# File 'lib/chimps/workflows/downloader.rb', line 19

def dataset
  @dataset
end

#fmt ⇒ Object (readonly)

The data format of the data to download.



22
23
24
 
# File 'lib/chimps/workflows/downloader.rb', line 22

def fmt
  @fmt
end

#pkg_fmt ⇒ Object (readonly)

The package format of the data to download.



25
26
27
 
# File 'lib/chimps/workflows/downloader.rb', line 25

def pkg_fmt
  @pkg_fmt
end

#token ⇒ Object (readonly)

The token received from Infochimps which contains a signed URL for the download.



16
17
18
 
# File 'lib/chimps/workflows/downloader.rb', line 16

def token
  @token
end

Instance Method Details

#ask_for_token! ⇒ Object

Ask for a download token for this dataset/package. If no or an invalid token is obtained, raise an error.



51
52
53
54
55
56
57
58
59
 
# File 'lib/chimps/workflows/downloader.rb', line 51

def ask_for_token!
  new_token = Request.new(download_tokens_path, :data => token_params, :sign_if_possible => true).post
  if new_token.error?
    new_token.print
    raise AuthenticationError.new("Unauthorized to download dataset #{dataset}")
  else
    @token = new_token
  end
end

#download! ⇒ Object

Issue the download request.

Uses curl for the data transfer.



88
89
90
91
92
 
# File 'lib/chimps/workflows/downloader.rb', line 88

def download!
  command = "#{curl} -o '#{local_path}' '#{download_url}'"
  puts command if Chimps.verbose?
  system(command)
end

#download_tokens_path ⇒ String

Path to submit download token requests to.

Returns:



64
65
66
 
# File 'lib/chimps/workflows/downloader.rb', line 64

def download_tokens_path
  "/download_tokens"
end

#download_url ⇒ String

The signed, remote URL from where the data can be downloaded.

Returns:



71
72
73
 
# File 'lib/chimps/workflows/downloader.rb', line 71

def download_url
  token['download_token']['package']['url']
end

#execute! ⇒ Object

Ask for a token and perform the download.



95
96
97
98
 
# File 'lib/chimps/workflows/downloader.rb', line 95

def execute!
  ask_for_token!
  download!
end

#local_path ⇒ String?

The local path where the downloaded data will be put.

Defaults to the current directory and the default basename of the downloaded package.

Returns:



81
82
83
 
# File 'lib/chimps/workflows/downloader.rb', line 81

def local_path
  @local_path || token["download_token"]["package"]["basename"]
end

#token_params ⇒ Hash

Params to send for the token.

Returns:



45
46
47
 
# File 'lib/chimps/workflows/downloader.rb', line 45

def token_params
  { :download_token => { :dataset_id => dataset, :fmt =>  fmt, :pkg_fmt => pkg_fmt} }
end