Class: Gitlab::GithubImport::Client

Inherits:
Object
  • Object
show all
Includes:
Gitlab::GithubImport::Clients::SearchRepos, Utils::StrongMemoize
Defined in:
lib/gitlab/github_import/client.rb

Overview

HTTP client for interacting with the GitHub API.

This class is basically a fancy wrapped around Octokit while adding some functionality to deal with rate limiting and parallel imports. Usage is mostly the same as Octokit, for example:

client = GithubImport::Client.new('hunter2')

client.labels.each do |label|
  puts label.name
end

Defined Under Namespace

Classes: Page

Constant Summary collapse

SEARCH_MAX_REQUESTS_PER_MINUTE = 
30
DEFAULT_PER_PAGE = 
100
LOWER_PER_PAGE = 
50
CLIENT_CONNECTION_ERROR =

used/set in sawyer agent which octokit uses

::Faraday::ConnectionFailed
RATE_LIMIT_THRESHOLD =

The minimum number of requests we want to keep available.

We don’t use a value of 0 as multiple threads may be using the same token in parallel. This could result in all of them hitting the GitHub rate limit at once. The threshold is put in place to not hit the limit in most cases.

50
SEARCH_RATE_LIMIT_THRESHOLD = 
3

Instance Attribute Summary collapse

Instance Method Summary collapse

Methods included from Gitlab::GithubImport::Clients::SearchRepos

#count_repos_by_relation_type_graphql, #search_repos_by_name_graphql

Constructor Details

#initialize(token, host: nil, per_page: DEFAULT_PER_PAGE, parallel: true) ⇒ Client

token - The GitHub API token to use.

host - The GitHub hostname. If nil, github.com will be used.

per_page - The number of objects that should be displayed per page.

parallel - When set to true hitting the rate limit will result in a

dedicated error being raised. When set to `false` we will
instead just `sleep()` until the rate limit is reset. Setting
this value to `true` for parallel importing is crucial as
otherwise hitting the rate limit will result in a thread
being blocked in a `sleep()` call for up to an hour.



51
52
53
54
55
56
57
58
59
60
61
62
63
 
# File 'lib/gitlab/github_import/client.rb', line 51

def initialize(token, host: nil, per_page: DEFAULT_PER_PAGE, parallel: true)
  @host = host
  @octokit = ::Octokit::Client.new(
    access_token: token,
    per_page: per_page,
    api_endpoint: api_endpoint,
    web_endpoint: web_endpoint
  )

  @octokit.connection_options[:ssl] = { verify: verify_ssl }

  @parallel = parallel
end

Instance Attribute Details

#octokit ⇒ Object (readonly)

Returns the value of attribute octokit.



20
21
22
 
# File 'lib/gitlab/github_import/client.rb', line 20

def octokit
  @octokit
end

Instance Method Details

#api_endpoint ⇒ Object 



239
240
241
 
# File 'lib/gitlab/github_import/client.rb', line 239

def api_endpoint
  @host || custom_api_endpoint || default_api_endpoint
end

#branch_protection(repo_name, branch_name) ⇒ Object 



127
128
129
 
# File 'lib/gitlab/github_import/client.rb', line 127

def branch_protection(repo_name, branch_name)
  with_rate_limit { octokit.branch_protection(repo_name, branch_name).to_h }
end

#branches(*args) ⇒ Object 



119
120
121
 
# File 'lib/gitlab/github_import/client.rb', line 119

def branches(*args)
  each_object(:branches, *args)
end

#collaborators(*args) ⇒ Object 



123
124
125
 
# File 'lib/gitlab/github_import/client.rb', line 123

def collaborators(*args)
  each_object(:collaborators, *args)
end

#custom_api_endpoint ⇒ Object 



247
248
249
 
# File 'lib/gitlab/github_import/client.rb', line 247

def custom_api_endpoint
  github_omniauth_provider.dig('args', 'client_options', 'site')
end

#default_api_endpoint ⇒ Object 



251
252
253
 
# File 'lib/gitlab/github_import/client.rb', line 251

def default_api_endpoint
  OmniAuth::Strategies::GitHub.default_options[:client_options][:site] || ::Octokit::Default.api_endpoint
end

#each_object(method, *args, &block) ⇒ Object

Iterates over all of the objects for the given method (e.g. ‘:labels`).

method - The method to send to Octokit for querying data. args - Any arguments to pass to the Octokit method.



165
166
167
168
169
170
171
172
173
 
# File 'lib/gitlab/github_import/client.rb', line 165

def each_object(method, *args, &block)
  return to_enum(__method__, method, *args) unless block

  each_page(method, *args) do |page|
    page.objects.each do |object|
      yield object.to_h
    end
  end
end

#each_page(method, *args) {|Page.new(collection, page)| ... } ⇒ Object

Fetches data from the GitHub API and yields a Page object for every page of data, without loading all of them into memory.

method - The Octokit method to use for getting the data. args - Arguments to pass to the Octokit method.

rubocop: disable GitlabSecurity/PublicSend

Yields:

  • (Page.new(collection, page)) 


138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
 
# File 'lib/gitlab/github_import/client.rb', line 138

def each_page(method, *args, &block)
  return to_enum(__method__, method, *args) unless block

  page =
    if args.last.is_a?(Hash) && args.last[:page]
      args.last[:page]
    else
      1
    end

  collection = with_rate_limit { octokit.public_send(method, *args) }
  next_url = octokit.last_response.rels[:next]

  yield Page.new(collection, page)

  while next_url
    response = with_rate_limit { next_url.get }
    next_url = response.rels[:next]

    yield Page.new(response.data, page += 1)
  end
end

#github_omniauth_provider ⇒ Object 



259
260
261
 
# File 'lib/gitlab/github_import/client.rb', line 259

def github_omniauth_provider
  @github_omniauth_provider ||= Gitlab::Auth::OAuth::Provider.config_for('github').to_h
end

#labels(*args) ⇒ Object 



107
108
109
 
# File 'lib/gitlab/github_import/client.rb', line 107

def labels(*args)
  each_object(:labels, *args)
end

#milestones(*args) ⇒ Object 



111
112
113
 
# File 'lib/gitlab/github_import/client.rb', line 111

def milestones(*args)
  each_object(:milestones, *args)
end

#parallel? ⇒ Boolean

Returns:

  • (Boolean) 


65
66
67
 
# File 'lib/gitlab/github_import/client.rb', line 65

def parallel?
  @parallel
end

#pull_request(repo_name, iid) ⇒ Object 



103
104
105
 
# File 'lib/gitlab/github_import/client.rb', line 103

def pull_request(repo_name, iid)
  with_rate_limit { octokit.pull_request(repo_name, iid).to_h }
end

#pull_request_review_requests(repo_name, iid) ⇒ Object 



88
89
90
 
# File 'lib/gitlab/github_import/client.rb', line 88

def pull_request_review_requests(repo_name, iid)
  with_rate_limit { octokit.pull_request_review_requests(repo_name, iid).to_h }
end

#pull_request_reviews(repo_name, iid) ⇒ Object 



84
85
86
 
# File 'lib/gitlab/github_import/client.rb', line 84

def pull_request_reviews(repo_name, iid)
  each_object(:pull_request_reviews, repo_name, iid)
end

#raise_or_wait_for_rate_limit ⇒ Object 



216
217
218
219
220
221
222
223
224
 
# File 'lib/gitlab/github_import/client.rb', line 216

def raise_or_wait_for_rate_limit
  rate_limit_counter.increment

  if parallel?
    raise RateLimitError
  else
    sleep(rate_limit_resets_in)
  end
end

#rate_limit_counter ⇒ Object 



263
264
265
266
267
268
 
# File 'lib/gitlab/github_import/client.rb', line 263

def rate_limit_counter
  @rate_limit_counter ||= Gitlab::Metrics.counter(
    :github_importer_rate_limit_hits,
    'The number of times we hit the GitHub rate limit when importing projects'
  )
end

#rate_limit_resets_in ⇒ Object 



226
227
228
229
230
231
 
# File 'lib/gitlab/github_import/client.rb', line 226

def rate_limit_resets_in
  # We add a few seconds to the rate limit so we don't _immediately_
  # resume when the rate limit resets as this may result in us performing
  # a request before GitHub has a chance to reset the limit.
  octokit.rate_limit.resets_in + 5
end

#rate_limiting_enabled? ⇒ Boolean

Returns:

  • (Boolean) 


233
234
235
236
237
 
# File 'lib/gitlab/github_import/client.rb', line 233

def rate_limiting_enabled?
  strong_memoize(:rate_limiting_enabled) do
    api_endpoint.include?('.github.com')
  end
end

#releases(*args) ⇒ Object 



115
116
117
 
# File 'lib/gitlab/github_import/client.rb', line 115

def releases(*args)
  each_object(:releases, *args)
end

#remaining_requests ⇒ Object 



208
209
210
 
# File 'lib/gitlab/github_import/client.rb', line 208

def remaining_requests
  octokit.rate_limit.remaining
end

#repos(options = {}) ⇒ Object 



92
93
94
 
# File 'lib/gitlab/github_import/client.rb', line 92

def repos(options = {})
  octokit.repos(nil, options).map(&:to_h)
end

#repository(name) ⇒ Object

Returns the details of a GitHub repository.

name - The path (in the form ‘owner/repository`) of the repository.



99
100
101
 
# File 'lib/gitlab/github_import/client.rb', line 99

def repository(name)
  with_rate_limit { octokit.repo(name).to_h }
end

#request_count_counter ⇒ Object 



270
271
272
273
274
275
 
# File 'lib/gitlab/github_import/client.rb', line 270

def request_count_counter
  @request_counter ||= Gitlab::Metrics.counter(
    :github_importer_request_count,
    'The number of GitHub API calls performed when importing projects'
  )
end

#requests_limit ⇒ Object 



212
213
214
 
# File 'lib/gitlab/github_import/client.rb', line 212

def requests_limit
  octokit.rate_limit.limit
end

#requests_remaining? ⇒ Boolean

Returns ‘true` if we’re still allowed to perform API calls. Search API has rate limit of 30, use lowered threshold when search is used.

Returns:

  • (Boolean) 


200
201
202
203
204
205
206
 
# File 'lib/gitlab/github_import/client.rb', line 200

def requests_remaining?
  if requests_limit == SEARCH_MAX_REQUESTS_PER_MINUTE
    return remaining_requests > SEARCH_RATE_LIMIT_THRESHOLD
  end

  remaining_requests > RATE_LIMIT_THRESHOLD
end

#user(username, options = {}) ⇒ Object

Returns the details of a GitHub user. 304 (Not Modified) status means the user is cached - API won’t return user data.

Parameters:

  • username (String) —

    the username of the user.

  • options (Hash) (defaults to: {}) —

    the optional parameters.



74
75
76
77
78
79
80
81
82
 
# File 'lib/gitlab/github_import/client.rb', line 74

def user(username, options = {})
  with_rate_limit do
    user = octokit.user(username, options)

    next if octokit.last_response&.status == 304

    user.to_h
  end
end

#verify_ssl ⇒ Object 



255
256
257
 
# File 'lib/gitlab/github_import/client.rb', line 255

def verify_ssl
  github_omniauth_provider.fetch('verify_ssl', true)
end

#web_endpoint ⇒ Object 



243
244
245
 
# File 'lib/gitlab/github_import/client.rb', line 243

def web_endpoint
  @host || custom_api_endpoint || ::Octokit::Default.web_endpoint
end

#with_rate_limit ⇒ Object

Yields the supplied block, responding to any rate limit errors.

The exact strategy used for handling rate limiting errors depends on whether we are running in parallel mode or not. For more information see ‘#rate_or_wait_for_rate_limit`.



180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
 
# File 'lib/gitlab/github_import/client.rb', line 180

def with_rate_limit
  return with_retry { yield } unless rate_limiting_enabled?

  request_count_counter.increment

  raise_or_wait_for_rate_limit unless requests_remaining?

  begin
    with_retry { yield }
  rescue ::Octokit::TooManyRequests
    raise_or_wait_for_rate_limit

    # This retry will only happen when running in sequential mode as we'll
    # raise an error in parallel mode.
    retry
  end
end