Class: Enigma::Download

Inherits:
Object
  • Object
show all
Defined in:
lib/enigma/download.rb

Overview

Handle polling the url returned by the Export api until we’re able to download it. Will load the zipped contents of the file into memory, and can then either write it to disk (‘write`), write the unzipped version to disk (`write_csv`) or parse the unzipped contents using the CSV library as an array of hashes

Constant Summary collapse

DELAY =

How long to wait between polling attempts, in seconds

1

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(res) ⇒ Download

Returns a new instance of Download.



14
15
16
 
# File 'lib/enigma/download.rb', line 14

def initialize(res)
  self.response = res
end

Instance Attribute Details

#download_contents ⇒ Object

Returns the value of attribute download_contents.



12
13
14
 
# File 'lib/enigma/download.rb', line 12

def download_contents
  @download_contents
end

#raw_download ⇒ Object

Returns the value of attribute raw_download.



12
13
14
 
# File 'lib/enigma/download.rb', line 12

def raw_download
  @raw_download
end

#response ⇒ Object

Returns the value of attribute response.



12
13
14
 
# File 'lib/enigma/download.rb', line 12

def response
  @response
end

Instance Method Details

#datapath ⇒ Object 



25
26
27
 
# File 'lib/enigma/download.rb', line 25

def datapath
  response.datapath
end

#do_download ⇒ Object 



74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
 
# File 'lib/enigma/download.rb', line 74

def do_download
  Enigma.logger.info "Trying to download #{download_url}"
  success = false
  until success
    req = Typhoeus::Request.new(download_url)
    req.on_complete do |response|
      if response.response_code == 404
        sleep DELAY
      else
        success = true
        return response.body
      end
    end
    req.run
  end
end

#download_url ⇒ Object

This url is where the file will eventually be available. Returns a 404 until then



21
22
23
 
# File 'lib/enigma/download.rb', line 21

def download_url
  response.export_url
end

#get ⇒ Object

Actually goes and fetches the download. Don’t return the raw_download because it’s a massive amount of data that will take over your terminal in IRB mode.

Returns:

  • true on success



35
36
37
38
 
# File 'lib/enigma/download.rb', line 35

def get
  @raw_download ||= do_download
  true
end

#parse(opts = {}) ⇒ Object 



69
70
71
72
 
# File 'lib/enigma/download.rb', line 69

def parse(opts = {})
  opts = { headers: true, header_converters: :symbol }
  CSV.parse(unzip, opts.merge(opts || {}))
end

#unzip ⇒ Object 



52
53
54
55
56
57
58
59
60
61
62
 
# File 'lib/enigma/download.rb', line 52

def unzip
  @download_contents ||=
    begin
      tmp = write_tmp
      contents = nil
      Zip::File.open(tmp.path) do |zipfile|
        contents = zipfile.first.get_input_stream.read
      end
      contents
    end
end

#write(io) ⇒ Object 



40
41
42
43
 
# File 'lib/enigma/download.rb', line 40

def write(io)
  get
  io.write raw_download
end

#write_csv(io) ⇒ Object 



64
65
66
67
 
# File 'lib/enigma/download.rb', line 64

def write_csv(io)
  unzip
  io.write download_contents
end

#write_tmp ⇒ Object 



45
46
47
48
49
50
 
# File 'lib/enigma/download.rb', line 45

def write_tmp
  tmp = Tempfile.new(datapath)
  write(tmp)
  tmp.rewind
  tmp
end