Class: CoreApi

Inherits:
Object
  • Object
show all
Includes:
Helper
Defined in:
lib/core_api.rb

Overview

module that is used for formatting numbers using metrics

Direct Known Subclasses

BadgeApi, RubygemsApi

Instance Attribute Summary collapse

Instance Method Summary collapse

Methods included from Helper

available_extension?, clean_image_label, dispatch_http_response, display_total, display_type, env_production?, fetch_content_type, find_version, force_utf8_encoding, format_error, get_latest_stable_version_details, get_string_from_cookie_data, http_valid_content_types?, http_valid_status_code?, last_version, metric_power, metric_prefixes, non_empty_http_response?, options_base_url, parse_gem_version, parse_json, parsed_url_property, print_to_output_buffer, root, rubygems_valid_response?, setup_options_for_url, shields_io_valid_response?, sorted_versions, stable_gem_versions, valid_http_code_returned?, valid_http_response?

Instance Attribute Details

#base_urlString

Returns THe base_url of the API.

Returns:

  • (String)

    THe base_url of the API


12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
# File 'lib/core_api.rb', line 12

class CoreApi
  include Helper

  # @return [Hash] The params that Sinatra received
  attr_reader :params

  # Returns the connection options used for connecting to API's
  # @param [String] url  The url that will be used for connection, this is needed so that we can set properly the SNI hostname
  #
  # @return [Hash] Returns the connection options used for connecting to API's
  def em_connection_options(url = nil)
    {
      connect_timeout: 5, # default connection setup timeout
      inactivity_timeout: 10, # default connection inactivity (post-setup) timeout
      ssl: {
        verify_peer: false,
        sni_hostname: parsed_url_property(url)
      },
      head: {
        'ACCEPT' => '*/*',
        'Connection' => 'keep-alive'
      }
    }
  end

  # Method that checks if the call to rubygems.org needs to be authorized
  # and adds the authorization header if is needed
  # @return [Hash] The additional headers needed for request, by default empty Hash
  def fetch_additional_headers
    {}
  end

  # Returns the request options used for connecting to API's
  # @param [Hash] data The optional data that will be used to inject additional headers to the request or providing a body to the request
  #
  # @return [Hash] Returns the request options used for connecting to API's
  def em_request_options(data = {})
    data = data.is_a?(Hash) ? data.with_indifferent_access : {}
    {
      redirects: 5,              # follow 3XX redirects up to depth 5
      keepalive: true,           # enable keep-alive (don't send Connection:close header)
      head: (data[:head] || {}).merge(
        'ACCEPT' => '*/*',
        'Connection' => 'keep-alive'
      ).remove_blank_values!,
      body: (params[:body] || {})
    }
  end

  # instantiates an eventmachine http request object that will be used to make the htpp request
  # @see EventMachine::HttpRequest#initialize
  #
  # @param [String] request_url The URL that will be used in the HTTP request
  # @param [Hash] options The options specific to this request which are needed for setting the SNI hostname and additional headers to the request
  #
  # @return [EventMachine::HttpRequest] Returns an http request object that will be used to set the success callbacks and error callbacks
  def em_request(request_url, options)
    em_request = EventMachine::HttpRequest.new(request_url, em_connection_options(request_url))
    em_request.send(options.fetch('http_method', 'get'), em_request_options(options))
  end # sets the full cookie string received from the response to the application's request cookies store

  # so that it can be used later when a new request cames for same URL
  #
  # This is in particular used for CloudFlare to be able to send the _cfuid cookie next time a new
  # request cames for same gem, This cookie is used to prevent CloudFlare to generate new cookies for same
  # visitor, and to bypass the cache miss functionality that is executed when it can't identify the visitor
  # and use instead the caching functionality as long as is not expired
  #
  # Otherwise it could happen that CloudFlare could block the new request due to its throttling capabilities
  # and due to his security checks. This prevents this to happen by making sure that CloudFlare could always identify
  # visitors , except for the first time requests.
  #
  # Currently only Shields.io uses CLoudFlare Nginx
  #
  # @see #setup_request_cookies_for_url
  # @see RubygemsDownloadShieldsApp#cookie_db
  #
  # @param [String] cookie_string The cookie String that will be persisted for the current URL that is used.
  #
  # @return [void]
  def persist_cookies_for_url(cookie_string)
    return if cookie_string.blank?
    cookie_db[@base_url] = cookie_string
  end

  # persist the cookies through requests
  # @see #persist_cookies_for_url
  # @see EM::HttpClient::SET_COOKIE
  #
  # @param [EventMachine::HttpRequest] http_client The http client that will be used to retrieve the headers as soon as they are returned and start persisting them
  #
  # @return [void]
  def persist_cookies(http_client)
    http_client.headers do |head|
      cookie_string = head[EM::HttpClient::SET_COOKIE]
      persist_cookies_for_url(cookie_string)
    end
  end

  # check if a cookie data already exist in the cookie store for the specified URL and if it is will return only the cookie value
  # without the other data, like expiration date, or other values that are specific to cookies
  #
  # @see RubygemsDownloadShieldsApp#cookie_db
  # @see RubygemsDownloadShieldsApp#cookie_hash
  # @see #get_string_from_cookie_data
  #
  # @param [String] base_url The URL that will be checked if exists in the cookie store of the application
  #
  # @return [String, nil] Returns the cookie value for the specified URL if exists in the cookie store and is not expired, or nil
  def get_cookie_string_for_base_url(base_url)
    cookie_h = cookie_db.key?(base_url) ? cookie_hash(base_url) : {}
    return if cookie_h.blank?
    get_string_from_cookie_data(cookie_h)
  end

  # Adds in the options received , inside the head key the cookie key with value of the cookie that
  # was persisted for the specified URL, if it exists
  #
  # @see #options_base_url
  # @see #get_cookie_string_for_base_url
  #
  # @param [Hash] options The options used for fetching the request name
  # @param [String] url The URL that is being used currently to fetch data from
  #
  # @return [void]
  def add_cookie_header(options, url)
    set_time_zone
    @base_url = options_base_url(options, url)
    cookie_string = get_cookie_string_for_base_url(@base_url)
    options['head']['cookie'] = cookie_string if cookie_string.present?
  end

  # Method that fetch the data from a URL and registers the error and success callback to the HTTP object
  # @see #setup_options_for_url
  # @see #fetch_real_data
  #
  # @param [url] url The URL that is used to fetch data from
  # @param [Hash] options The additional options needed for further processing
  # @param [Proc] block the callback that will be called when success
  # @return [void]
  def fetch_data(url, options = {}, &block)
    options = options.stringify_keys
    options['head'] = (options['head'] || {}).merge(fetch_additional_headers)
    setup_options_for_url(options, url)
    fetch_real_data(url, options, &block)
  end

  # Method that is used to add the cookie header to the request, instantiate the Request instance and
  # register the callbacks
  # @see #add_cookie_header
  # @see #em_request
  # @see #do_fetch_real_data
  #
  # @param [url] url The URL that is used to fetch data from
  # @param [Hash] options The additional options needed for further processing
  # @param [Proc] block the callback that will be called when success
  # @return [void]
  def fetch_real_data(url, options = {}, &block)
    add_cookie_header(options, url)
    http = em_request(url, options)
    do_fetch_real_data(http, options, &block)
  end

  # Method that is used to reqister the callbacks for success, erorr, and on receiving headers
  # @see #persist_cookies
  # @see #register_error_callback
  # @see #register_success_callback
  #
  # @param [EventMachine::HttpRequest] http_client The http client used for fetching data
  # @param [Hash] options The additional options needed for further processing
  # @param [Proc] block the callback that will be called when success
  # @return [void]
  def do_fetch_real_data(http_client, options, &block)
    persist_cookies(http_client)
    register_error_callback(http_client, options)
    register_success_callback(http_client, options, &block)
  end

  # Method that is used to register a success callback to a http object
  # @see #handle_http_callback
  #
  # @param [EventMachine::HttpRequest] http The HTTP object that will be used for registering the success callback
  # @param [Hash] options The additional options needed for further processing
  # @param [Proc] block If the response is not blank, the block will receive the response
  # @return [void]
  def register_success_callback(http, options, &block)
    http.callback do
      handle_http_callback(http, options, &block)
    end
  end

  # Method that is call an additional callback before calling the success callback for parsing response
  # and dispatch the http response to the appropiate callback
  # @see #callback_before_success
  # @see #dispatch_http_response
  #
  # @param [EventMachine::HttpRequest] http The HTTP object that will be used for registering the success callback
  # @param [Hash] options The additional options needed for further processing
  # @param [Proc] block If the response is not blank, the block will receive the response
  # @return [void]
  def handle_http_callback(http, options, &block)
    http_response = http.response
    res = callback_before_success(http_response)
    dispatch_http_response(res, options, &block)
  end

  # Callback that is used before returning the response the the instance, by default it does no additional processing
  #
  # if needed this can be overriden by child classes to provide the format processing
  # before calling the dispatch_http_response method
  #
  # @param [String] response The response that will be dispatched to the instance class that made the request
  # @return [String] Returns the response
  def callback_before_success(response)
    response
  end

  # This method is used to reqister a error callback to a HTTP request object
  # @see #callback_error
  # @param [EventMachine::HttpRequest] http The HTTP object that will be used for reqisteringt the error callback
  # @param [Hash] options The additional options needed for further processing
  # @return [void]
  def register_error_callback(http, options)
    http.errback { |error| callback_error(error, options) }
  end

  # Method that is used to react when an error happens in a HTTP request
  # and prints out an error message
  #
  # @param [Object] error The error that was raised by the HTTP request
  # @param [Hash] options The additional options needed for further processing
  # @return [void]
  def callback_error(error, options = {})
    debug = "#{error.inspect} with #{options.inspect}"
    logger.debug("Error during fetching data  : #{debug}")
  end
end

#hostnameString

Returns THe hostname from where the badges are fetched from.

Returns:

  • (String)

    THe hostname from where the badges are fetched from


12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
# File 'lib/core_api.rb', line 12

class CoreApi
  include Helper

  # @return [Hash] The params that Sinatra received
  attr_reader :params

  # Returns the connection options used for connecting to API's
  # @param [String] url  The url that will be used for connection, this is needed so that we can set properly the SNI hostname
  #
  # @return [Hash] Returns the connection options used for connecting to API's
  def em_connection_options(url = nil)
    {
      connect_timeout: 5, # default connection setup timeout
      inactivity_timeout: 10, # default connection inactivity (post-setup) timeout
      ssl: {
        verify_peer: false,
        sni_hostname: parsed_url_property(url)
      },
      head: {
        'ACCEPT' => '*/*',
        'Connection' => 'keep-alive'
      }
    }
  end

  # Method that checks if the call to rubygems.org needs to be authorized
  # and adds the authorization header if is needed
  # @return [Hash] The additional headers needed for request, by default empty Hash
  def fetch_additional_headers
    {}
  end

  # Returns the request options used for connecting to API's
  # @param [Hash] data The optional data that will be used to inject additional headers to the request or providing a body to the request
  #
  # @return [Hash] Returns the request options used for connecting to API's
  def em_request_options(data = {})
    data = data.is_a?(Hash) ? data.with_indifferent_access : {}
    {
      redirects: 5,              # follow 3XX redirects up to depth 5
      keepalive: true,           # enable keep-alive (don't send Connection:close header)
      head: (data[:head] || {}).merge(
        'ACCEPT' => '*/*',
        'Connection' => 'keep-alive'
      ).remove_blank_values!,
      body: (params[:body] || {})
    }
  end

  # instantiates an eventmachine http request object that will be used to make the htpp request
  # @see EventMachine::HttpRequest#initialize
  #
  # @param [String] request_url The URL that will be used in the HTTP request
  # @param [Hash] options The options specific to this request which are needed for setting the SNI hostname and additional headers to the request
  #
  # @return [EventMachine::HttpRequest] Returns an http request object that will be used to set the success callbacks and error callbacks
  def em_request(request_url, options)
    em_request = EventMachine::HttpRequest.new(request_url, em_connection_options(request_url))
    em_request.send(options.fetch('http_method', 'get'), em_request_options(options))
  end # sets the full cookie string received from the response to the application's request cookies store

  # so that it can be used later when a new request cames for same URL
  #
  # This is in particular used for CloudFlare to be able to send the _cfuid cookie next time a new
  # request cames for same gem, This cookie is used to prevent CloudFlare to generate new cookies for same
  # visitor, and to bypass the cache miss functionality that is executed when it can't identify the visitor
  # and use instead the caching functionality as long as is not expired
  #
  # Otherwise it could happen that CloudFlare could block the new request due to its throttling capabilities
  # and due to his security checks. This prevents this to happen by making sure that CloudFlare could always identify
  # visitors , except for the first time requests.
  #
  # Currently only Shields.io uses CLoudFlare Nginx
  #
  # @see #setup_request_cookies_for_url
  # @see RubygemsDownloadShieldsApp#cookie_db
  #
  # @param [String] cookie_string The cookie String that will be persisted for the current URL that is used.
  #
  # @return [void]
  def persist_cookies_for_url(cookie_string)
    return if cookie_string.blank?
    cookie_db[@base_url] = cookie_string
  end

  # persist the cookies through requests
  # @see #persist_cookies_for_url
  # @see EM::HttpClient::SET_COOKIE
  #
  # @param [EventMachine::HttpRequest] http_client The http client that will be used to retrieve the headers as soon as they are returned and start persisting them
  #
  # @return [void]
  def persist_cookies(http_client)
    http_client.headers do |head|
      cookie_string = head[EM::HttpClient::SET_COOKIE]
      persist_cookies_for_url(cookie_string)
    end
  end

  # check if a cookie data already exist in the cookie store for the specified URL and if it is will return only the cookie value
  # without the other data, like expiration date, or other values that are specific to cookies
  #
  # @see RubygemsDownloadShieldsApp#cookie_db
  # @see RubygemsDownloadShieldsApp#cookie_hash
  # @see #get_string_from_cookie_data
  #
  # @param [String] base_url The URL that will be checked if exists in the cookie store of the application
  #
  # @return [String, nil] Returns the cookie value for the specified URL if exists in the cookie store and is not expired, or nil
  def get_cookie_string_for_base_url(base_url)
    cookie_h = cookie_db.key?(base_url) ? cookie_hash(base_url) : {}
    return if cookie_h.blank?
    get_string_from_cookie_data(cookie_h)
  end

  # Adds in the options received , inside the head key the cookie key with value of the cookie that
  # was persisted for the specified URL, if it exists
  #
  # @see #options_base_url
  # @see #get_cookie_string_for_base_url
  #
  # @param [Hash] options The options used for fetching the request name
  # @param [String] url The URL that is being used currently to fetch data from
  #
  # @return [void]
  def add_cookie_header(options, url)
    set_time_zone
    @base_url = options_base_url(options, url)
    cookie_string = get_cookie_string_for_base_url(@base_url)
    options['head']['cookie'] = cookie_string if cookie_string.present?
  end

  # Method that fetch the data from a URL and registers the error and success callback to the HTTP object
  # @see #setup_options_for_url
  # @see #fetch_real_data
  #
  # @param [url] url The URL that is used to fetch data from
  # @param [Hash] options The additional options needed for further processing
  # @param [Proc] block the callback that will be called when success
  # @return [void]
  def fetch_data(url, options = {}, &block)
    options = options.stringify_keys
    options['head'] = (options['head'] || {}).merge(fetch_additional_headers)
    setup_options_for_url(options, url)
    fetch_real_data(url, options, &block)
  end

  # Method that is used to add the cookie header to the request, instantiate the Request instance and
  # register the callbacks
  # @see #add_cookie_header
  # @see #em_request
  # @see #do_fetch_real_data
  #
  # @param [url] url The URL that is used to fetch data from
  # @param [Hash] options The additional options needed for further processing
  # @param [Proc] block the callback that will be called when success
  # @return [void]
  def fetch_real_data(url, options = {}, &block)
    add_cookie_header(options, url)
    http = em_request(url, options)
    do_fetch_real_data(http, options, &block)
  end

  # Method that is used to reqister the callbacks for success, erorr, and on receiving headers
  # @see #persist_cookies
  # @see #register_error_callback
  # @see #register_success_callback
  #
  # @param [EventMachine::HttpRequest] http_client The http client used for fetching data
  # @param [Hash] options The additional options needed for further processing
  # @param [Proc] block the callback that will be called when success
  # @return [void]
  def do_fetch_real_data(http_client, options, &block)
    persist_cookies(http_client)
    register_error_callback(http_client, options)
    register_success_callback(http_client, options, &block)
  end

  # Method that is used to register a success callback to a http object
  # @see #handle_http_callback
  #
  # @param [EventMachine::HttpRequest] http The HTTP object that will be used for registering the success callback
  # @param [Hash] options The additional options needed for further processing
  # @param [Proc] block If the response is not blank, the block will receive the response
  # @return [void]
  def register_success_callback(http, options, &block)
    http.callback do
      handle_http_callback(http, options, &block)
    end
  end

  # Method that is call an additional callback before calling the success callback for parsing response
  # and dispatch the http response to the appropiate callback
  # @see #callback_before_success
  # @see #dispatch_http_response
  #
  # @param [EventMachine::HttpRequest] http The HTTP object that will be used for registering the success callback
  # @param [Hash] options The additional options needed for further processing
  # @param [Proc] block If the response is not blank, the block will receive the response
  # @return [void]
  def handle_http_callback(http, options, &block)
    http_response = http.response
    res = callback_before_success(http_response)
    dispatch_http_response(res, options, &block)
  end

  # Callback that is used before returning the response the the instance, by default it does no additional processing
  #
  # if needed this can be overriden by child classes to provide the format processing
  # before calling the dispatch_http_response method
  #
  # @param [String] response The response that will be dispatched to the instance class that made the request
  # @return [String] Returns the response
  def callback_before_success(response)
    response
  end

  # This method is used to reqister a error callback to a HTTP request object
  # @see #callback_error
  # @param [EventMachine::HttpRequest] http The HTTP object that will be used for reqisteringt the error callback
  # @param [Hash] options The additional options needed for further processing
  # @return [void]
  def register_error_callback(http, options)
    http.errback { |error| callback_error(error, options) }
  end

  # Method that is used to react when an error happens in a HTTP request
  # and prints out an error message
  #
  # @param [Object] error The error that was raised by the HTTP request
  # @param [Hash] options The additional options needed for further processing
  # @return [void]
  def callback_error(error, options = {})
    debug = "#{error.inspect} with #{options.inspect}"
    logger.debug("Error during fetching data  : #{debug}")
  end
end

#paramsHash

Returns The params that Sinatra received.

Returns:

  • (Hash)

    The params that Sinatra received


12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
# File 'lib/core_api.rb', line 12

class CoreApi
  include Helper

  # @return [Hash] The params that Sinatra received
  attr_reader :params

  # Returns the connection options used for connecting to API's
  # @param [String] url  The url that will be used for connection, this is needed so that we can set properly the SNI hostname
  #
  # @return [Hash] Returns the connection options used for connecting to API's
  def em_connection_options(url = nil)
    {
      connect_timeout: 5, # default connection setup timeout
      inactivity_timeout: 10, # default connection inactivity (post-setup) timeout
      ssl: {
        verify_peer: false,
        sni_hostname: parsed_url_property(url)
      },
      head: {
        'ACCEPT' => '*/*',
        'Connection' => 'keep-alive'
      }
    }
  end

  # Method that checks if the call to rubygems.org needs to be authorized
  # and adds the authorization header if is needed
  # @return [Hash] The additional headers needed for request, by default empty Hash
  def fetch_additional_headers
    {}
  end

  # Returns the request options used for connecting to API's
  # @param [Hash] data The optional data that will be used to inject additional headers to the request or providing a body to the request
  #
  # @return [Hash] Returns the request options used for connecting to API's
  def em_request_options(data = {})
    data = data.is_a?(Hash) ? data.with_indifferent_access : {}
    {
      redirects: 5,              # follow 3XX redirects up to depth 5
      keepalive: true,           # enable keep-alive (don't send Connection:close header)
      head: (data[:head] || {}).merge(
        'ACCEPT' => '*/*',
        'Connection' => 'keep-alive'
      ).remove_blank_values!,
      body: (params[:body] || {})
    }
  end

  # instantiates an eventmachine http request object that will be used to make the htpp request
  # @see EventMachine::HttpRequest#initialize
  #
  # @param [String] request_url The URL that will be used in the HTTP request
  # @param [Hash] options The options specific to this request which are needed for setting the SNI hostname and additional headers to the request
  #
  # @return [EventMachine::HttpRequest] Returns an http request object that will be used to set the success callbacks and error callbacks
  def em_request(request_url, options)
    em_request = EventMachine::HttpRequest.new(request_url, em_connection_options(request_url))
    em_request.send(options.fetch('http_method', 'get'), em_request_options(options))
  end # sets the full cookie string received from the response to the application's request cookies store

  # so that it can be used later when a new request cames for same URL
  #
  # This is in particular used for CloudFlare to be able to send the _cfuid cookie next time a new
  # request cames for same gem, This cookie is used to prevent CloudFlare to generate new cookies for same
  # visitor, and to bypass the cache miss functionality that is executed when it can't identify the visitor
  # and use instead the caching functionality as long as is not expired
  #
  # Otherwise it could happen that CloudFlare could block the new request due to its throttling capabilities
  # and due to his security checks. This prevents this to happen by making sure that CloudFlare could always identify
  # visitors , except for the first time requests.
  #
  # Currently only Shields.io uses CLoudFlare Nginx
  #
  # @see #setup_request_cookies_for_url
  # @see RubygemsDownloadShieldsApp#cookie_db
  #
  # @param [String] cookie_string The cookie String that will be persisted for the current URL that is used.
  #
  # @return [void]
  def persist_cookies_for_url(cookie_string)
    return if cookie_string.blank?
    cookie_db[@base_url] = cookie_string
  end

  # persist the cookies through requests
  # @see #persist_cookies_for_url
  # @see EM::HttpClient::SET_COOKIE
  #
  # @param [EventMachine::HttpRequest] http_client The http client that will be used to retrieve the headers as soon as they are returned and start persisting them
  #
  # @return [void]
  def persist_cookies(http_client)
    http_client.headers do |head|
      cookie_string = head[EM::HttpClient::SET_COOKIE]
      persist_cookies_for_url(cookie_string)
    end
  end

  # check if a cookie data already exist in the cookie store for the specified URL and if it is will return only the cookie value
  # without the other data, like expiration date, or other values that are specific to cookies
  #
  # @see RubygemsDownloadShieldsApp#cookie_db
  # @see RubygemsDownloadShieldsApp#cookie_hash
  # @see #get_string_from_cookie_data
  #
  # @param [String] base_url The URL that will be checked if exists in the cookie store of the application
  #
  # @return [String, nil] Returns the cookie value for the specified URL if exists in the cookie store and is not expired, or nil
  def get_cookie_string_for_base_url(base_url)
    cookie_h = cookie_db.key?(base_url) ? cookie_hash(base_url) : {}
    return if cookie_h.blank?
    get_string_from_cookie_data(cookie_h)
  end

  # Adds in the options received , inside the head key the cookie key with value of the cookie that
  # was persisted for the specified URL, if it exists
  #
  # @see #options_base_url
  # @see #get_cookie_string_for_base_url
  #
  # @param [Hash] options The options used for fetching the request name
  # @param [String] url The URL that is being used currently to fetch data from
  #
  # @return [void]
  def add_cookie_header(options, url)
    set_time_zone
    @base_url = options_base_url(options, url)
    cookie_string = get_cookie_string_for_base_url(@base_url)
    options['head']['cookie'] = cookie_string if cookie_string.present?
  end

  # Method that fetch the data from a URL and registers the error and success callback to the HTTP object
  # @see #setup_options_for_url
  # @see #fetch_real_data
  #
  # @param [url] url The URL that is used to fetch data from
  # @param [Hash] options The additional options needed for further processing
  # @param [Proc] block the callback that will be called when success
  # @return [void]
  def fetch_data(url, options = {}, &block)
    options = options.stringify_keys
    options['head'] = (options['head'] || {}).merge(fetch_additional_headers)
    setup_options_for_url(options, url)
    fetch_real_data(url, options, &block)
  end

  # Method that is used to add the cookie header to the request, instantiate the Request instance and
  # register the callbacks
  # @see #add_cookie_header
  # @see #em_request
  # @see #do_fetch_real_data
  #
  # @param [url] url The URL that is used to fetch data from
  # @param [Hash] options The additional options needed for further processing
  # @param [Proc] block the callback that will be called when success
  # @return [void]
  def fetch_real_data(url, options = {}, &block)
    add_cookie_header(options, url)
    http = em_request(url, options)
    do_fetch_real_data(http, options, &block)
  end

  # Method that is used to reqister the callbacks for success, erorr, and on receiving headers
  # @see #persist_cookies
  # @see #register_error_callback
  # @see #register_success_callback
  #
  # @param [EventMachine::HttpRequest] http_client The http client used for fetching data
  # @param [Hash] options The additional options needed for further processing
  # @param [Proc] block the callback that will be called when success
  # @return [void]
  def do_fetch_real_data(http_client, options, &block)
    persist_cookies(http_client)
    register_error_callback(http_client, options)
    register_success_callback(http_client, options, &block)
  end

  # Method that is used to register a success callback to a http object
  # @see #handle_http_callback
  #
  # @param [EventMachine::HttpRequest] http The HTTP object that will be used for registering the success callback
  # @param [Hash] options The additional options needed for further processing
  # @param [Proc] block If the response is not blank, the block will receive the response
  # @return [void]
  def register_success_callback(http, options, &block)
    http.callback do
      handle_http_callback(http, options, &block)
    end
  end

  # Method that is call an additional callback before calling the success callback for parsing response
  # and dispatch the http response to the appropiate callback
  # @see #callback_before_success
  # @see #dispatch_http_response
  #
  # @param [EventMachine::HttpRequest] http The HTTP object that will be used for registering the success callback
  # @param [Hash] options The additional options needed for further processing
  # @param [Proc] block If the response is not blank, the block will receive the response
  # @return [void]
  def handle_http_callback(http, options, &block)
    http_response = http.response
    res = callback_before_success(http_response)
    dispatch_http_response(res, options, &block)
  end

  # Callback that is used before returning the response the the instance, by default it does no additional processing
  #
  # if needed this can be overriden by child classes to provide the format processing
  # before calling the dispatch_http_response method
  #
  # @param [String] response The response that will be dispatched to the instance class that made the request
  # @return [String] Returns the response
  def callback_before_success(response)
    response
  end

  # This method is used to reqister a error callback to a HTTP request object
  # @see #callback_error
  # @param [EventMachine::HttpRequest] http The HTTP object that will be used for reqisteringt the error callback
  # @param [Hash] options The additional options needed for further processing
  # @return [void]
  def register_error_callback(http, options)
    http.errback { |error| callback_error(error, options) }
  end

  # Method that is used to react when an error happens in a HTTP request
  # and prints out an error message
  #
  # @param [Object] error The error that was raised by the HTTP request
  # @param [Hash] options The additional options needed for further processing
  # @return [void]
  def callback_error(error, options = {})
    debug = "#{error.inspect} with #{options.inspect}"
    logger.debug("Error during fetching data  : #{debug}")
  end
end

Instance Method Details

This method returns an undefined value.

Adds in the options received , inside the head key the cookie key with value of the cookie that was persisted for the specified URL, if it exists

Parameters:

  • options (Hash)

    The options used for fetching the request name

  • url (String)

    The URL that is being used currently to fetch data from

See Also:


137
138
139
140
141
142
# File 'lib/core_api.rb', line 137

def add_cookie_header(options, url)
  set_time_zone
  @base_url = options_base_url(options, url)
  cookie_string = get_cookie_string_for_base_url(@base_url)
  options['head']['cookie'] = cookie_string if cookie_string.present?
end

#callback_before_success(response) ⇒ String

Callback that is used before returning the response the the instance, by default it does no additional processing

if needed this can be overriden by child classes to provide the format processing before calling the dispatch_http_response method

Parameters:

  • response (String)

    The response that will be dispatched to the instance class that made the request

Returns:

  • (String)

    Returns the response


225
226
227
# File 'lib/core_api.rb', line 225

def callback_before_success(response)
  response
end

#callback_error(error, options = {}) ⇒ void

This method returns an undefined value.

Method that is used to react when an error happens in a HTTP request and prints out an error message

Parameters:

  • error (Object)

    The error that was raised by the HTTP request

  • options (Hash) (defaults to: {})

    The additional options needed for further processing


244
245
246
247
# File 'lib/core_api.rb', line 244

def callback_error(error, options = {})
  debug = "#{error.inspect} with #{options.inspect}"
  logger.debug("Error during fetching data  : #{debug}")
end

#do_fetch_real_data(http_client, options, &block) ⇒ void

This method returns an undefined value.

Method that is used to reqister the callbacks for success, erorr, and on receiving headers

Parameters:

  • http_client (EventMachine::HttpRequest)

    The http client used for fetching data

  • options (Hash)

    The additional options needed for further processing

  • block (Proc)

    the callback that will be called when success

See Also:


184
185
186
187
188
# File 'lib/core_api.rb', line 184

def do_fetch_real_data(http_client, options, &block)
  persist_cookies(http_client)
  register_error_callback(http_client, options)
  register_success_callback(http_client, options, &block)
end

#em_connection_options(url = nil) ⇒ Hash

Returns the connection options used for connecting to API's

Parameters:

  • url (String) (defaults to: nil)

    The url that will be used for connection, this is needed so that we can set properly the SNI hostname

Returns:

  • (Hash)

    Returns the connection options used for connecting to API's


22
23
24
25
26
27
28
29
30
31
32
33
34
35
# File 'lib/core_api.rb', line 22

def em_connection_options(url = nil)
  {
    connect_timeout: 5, # default connection setup timeout
    inactivity_timeout: 10, # default connection inactivity (post-setup) timeout
    ssl: {
      verify_peer: false,
      sni_hostname: parsed_url_property(url)
    },
    head: {
      'ACCEPT' => '*/*',
      'Connection' => 'keep-alive'
    }
  }
end

#em_request(request_url, options) ⇒ EventMachine::HttpRequest

instantiates an eventmachine http request object that will be used to make the htpp request

Parameters:

  • request_url (String)

    The URL that will be used in the HTTP request

  • options (Hash)

    The options specific to this request which are needed for setting the SNI hostname and additional headers to the request

Returns:

  • (EventMachine::HttpRequest)

    Returns an http request object that will be used to set the success callbacks and error callbacks

See Also:

  • EventMachine::HttpRequest#initialize

68
69
70
71
# File 'lib/core_api.rb', line 68

def em_request(request_url, options)
  em_request = EventMachine::HttpRequest.new(request_url, em_connection_options(request_url))
  em_request.send(options.fetch('http_method', 'get'), em_request_options(options))
end

#em_request_options(data = {}) ⇒ Hash

Returns the request options used for connecting to API's

Parameters:

  • data (Hash) (defaults to: {})

    The optional data that will be used to inject additional headers to the request or providing a body to the request

Returns:

  • (Hash)

    Returns the request options used for connecting to API's


48
49
50
51
52
53
54
55
56
57
58
59
# File 'lib/core_api.rb', line 48

def em_request_options(data = {})
  data = data.is_a?(Hash) ? data.with_indifferent_access : {}
  {
    redirects: 5,              # follow 3XX redirects up to depth 5
    keepalive: true,           # enable keep-alive (don't send Connection:close header)
    head: (data[:head] || {}).merge(
      'ACCEPT' => '*/*',
      'Connection' => 'keep-alive'
    ).remove_blank_values!,
    body: (params[:body] || {})
  }
end

#fetch_additional_headersHash

Method that checks if the call to rubygems.org needs to be authorized and adds the authorization header if is needed

Returns:

  • (Hash)

    The additional headers needed for request, by default empty Hash


40
41
42
# File 'lib/core_api.rb', line 40

def fetch_additional_headers
  {}
end

#fetch_data(url, options = {}, &block) ⇒ void

This method returns an undefined value.

Method that fetch the data from a URL and registers the error and success callback to the HTTP object

Parameters:

  • url (url)

    The URL that is used to fetch data from

  • options (Hash) (defaults to: {})

    The additional options needed for further processing

  • block (Proc)

    the callback that will be called when success

See Also:


152
153
154
155
156
157
# File 'lib/core_api.rb', line 152

def fetch_data(url, options = {}, &block)
  options = options.stringify_keys
  options['head'] = (options['head'] || {}).merge(fetch_additional_headers)
  setup_options_for_url(options, url)
  fetch_real_data(url, options, &block)
end

#fetch_real_data(url, options = {}, &block) ⇒ void

This method returns an undefined value.

Method that is used to add the cookie header to the request, instantiate the Request instance and register the callbacks

Parameters:

  • url (url)

    The URL that is used to fetch data from

  • options (Hash) (defaults to: {})

    The additional options needed for further processing

  • block (Proc)

    the callback that will be called when success

See Also:


169
170
171
172
173
# File 'lib/core_api.rb', line 169

def fetch_real_data(url, options = {}, &block)
  add_cookie_header(options, url)
  http = em_request(url, options)
  do_fetch_real_data(http, options, &block)
end

check if a cookie data already exist in the cookie store for the specified URL and if it is will return only the cookie value without the other data, like expiration date, or other values that are specific to cookies

Parameters:

  • base_url (String)

    The URL that will be checked if exists in the cookie store of the application

Returns:

  • (String, nil)

    Returns the cookie value for the specified URL if exists in the cookie store and is not expired, or nil

See Also:


121
122
123
124
125
# File 'lib/core_api.rb', line 121

def get_cookie_string_for_base_url(base_url)
  cookie_h = cookie_db.key?(base_url) ? cookie_hash(base_url) : {}
  return if cookie_h.blank?
  get_string_from_cookie_data(cookie_h)
end

#handle_http_callback(http, options, &block) ⇒ void

This method returns an undefined value.

Method that is call an additional callback before calling the success callback for parsing response and dispatch the http response to the appropiate callback

Parameters:

  • http (EventMachine::HttpRequest)

    The HTTP object that will be used for registering the success callback

  • options (Hash)

    The additional options needed for further processing

  • block (Proc)

    If the response is not blank, the block will receive the response

See Also:


212
213
214
215
216
# File 'lib/core_api.rb', line 212

def handle_http_callback(http, options, &block)
  http_response = http.response
  res = callback_before_success(http_response)
  dispatch_http_response(res, options, &block)
end

#persist_cookies(http_client) ⇒ void

This method returns an undefined value.

persist the cookies through requests

Parameters:

  • http_client (EventMachine::HttpRequest)

    The http client that will be used to retrieve the headers as soon as they are returned and start persisting them

See Also:


104
105
106
107
108
109
# File 'lib/core_api.rb', line 104

def persist_cookies(http_client)
  http_client.headers do |head|
    cookie_string = head[EM::HttpClient::SET_COOKIE]
    persist_cookies_for_url(cookie_string)
  end
end

#persist_cookies_for_url(cookie_string) ⇒ void

This method returns an undefined value.

so that it can be used later when a new request cames for same URL

This is in particular used for CloudFlare to be able to send the _cfuid cookie next time a new request cames for same gem, This cookie is used to prevent CloudFlare to generate new cookies for same visitor, and to bypass the cache miss functionality that is executed when it can't identify the visitor and use instead the caching functionality as long as is not expired

Otherwise it could happen that CloudFlare could block the new request due to its throttling capabilities and due to his security checks. This prevents this to happen by making sure that CloudFlare could always identify visitors , except for the first time requests.

Currently only Shields.io uses CLoudFlare Nginx

Parameters:

  • cookie_string (String)

    The cookie String that will be persisted for the current URL that is used.

See Also:

  • #setup_request_cookies_for_url
  • RubygemsDownloadShieldsApp#cookie_db

92
93
94
95
# File 'lib/core_api.rb', line 92

def persist_cookies_for_url(cookie_string)
  return if cookie_string.blank?
  cookie_db[@base_url] = cookie_string
end

#register_error_callback(http, options) ⇒ void

This method returns an undefined value.

This method is used to reqister a error callback to a HTTP request object

Parameters:

  • http (EventMachine::HttpRequest)

    The HTTP object that will be used for reqisteringt the error callback

  • options (Hash)

    The additional options needed for further processing

See Also:


234
235
236
# File 'lib/core_api.rb', line 234

def register_error_callback(http, options)
  http.errback { |error| callback_error(error, options) }
end

#register_success_callback(http, options, &block) ⇒ void

This method returns an undefined value.

Method that is used to register a success callback to a http object

Parameters:

  • http (EventMachine::HttpRequest)

    The HTTP object that will be used for registering the success callback

  • options (Hash)

    The additional options needed for further processing

  • block (Proc)

    If the response is not blank, the block will receive the response

See Also:


197
198
199
200
201
# File 'lib/core_api.rb', line 197

def register_success_callback(http, options, &block)
  http.callback do
    handle_http_callback(http, options, &block)
  end
end