Class: Gitlab::GithubImport::Client
Overview
HTTP client for interacting with the GitHub API.
This class is basically a fancy wrapped around Octokit while adding some functionality to deal with rate limiting and parallel imports. Usage is mostly the same as Octokit, for example:
client = GithubImport::Client.new('hunter2')
client.labels.each do |label|
puts label.name
end
Defined Under Namespace
Classes: Page
Constant Summary
collapse
- SEARCH_MAX_REQUESTS_PER_MINUTE =
30
- DEFAULT_PER_PAGE =
100
- LOWER_PER_PAGE =
50
- CLIENT_CONNECTION_ERROR =
used/set in sawyer agent which octokit uses
::Faraday::ConnectionFailed
- RATE_LIMIT_THRESHOLD =
The minimum number of requests we want to keep available.
We don’t use a value of 0 as multiple threads may be using the same token in parallel. This could result in all of them hitting the GitHub rate limit at once. The threshold is put in place to not hit the limit in most cases.
50
- SEARCH_RATE_LIMIT_THRESHOLD =
3
Instance Attribute Summary collapse
Instance Method Summary
collapse
-
#api_endpoint ⇒ Object
-
#branch_protection(repo_name, branch_name) ⇒ Object
-
#branches(*args) ⇒ Object
-
#collaborators(*args) ⇒ Object
-
#custom_api_endpoint ⇒ Object
-
#default_api_endpoint ⇒ Object
-
#each_object(method, *args, &block) ⇒ Object
Iterates over all of the objects for the given method (e.g. ‘:labels`).
-
#each_page(method, *args) {|Page.new(collection, page)| ... } ⇒ Object
Fetches data from the GitHub API and yields a Page object for every page of data, without loading all of them into memory.
-
#github_omniauth_provider ⇒ Object
-
#initialize(token, host: nil, per_page: DEFAULT_PER_PAGE, parallel: true) ⇒ Client
constructor
token - The GitHub API token to use.
-
#labels(*args) ⇒ Object
-
#milestones(*args) ⇒ Object
-
#parallel? ⇒ Boolean
-
#pull_request(repo_name, iid) ⇒ Object
-
#pull_request_review_requests(repo_name, iid) ⇒ Object
-
#pull_request_reviews(repo_name, iid) ⇒ Object
-
#raise_or_wait_for_rate_limit ⇒ Object
-
#rate_limit_counter ⇒ Object
-
#rate_limit_resets_in ⇒ Object
-
#rate_limiting_enabled? ⇒ Boolean
-
#releases(*args) ⇒ Object
-
#remaining_requests ⇒ Object
-
#repos(options = {}) ⇒ Object
-
#repository(name) ⇒ Object
Returns the details of a GitHub repository.
-
#request_count_counter ⇒ Object
-
#requests_limit ⇒ Object
-
#requests_remaining? ⇒ Boolean
Returns ‘true` if we’re still allowed to perform API calls.
-
#user(username, options = {}) ⇒ Object
Returns the details of a GitHub user.
-
#verify_ssl ⇒ Object
-
#web_endpoint ⇒ Object
-
#with_rate_limit ⇒ Object
Yields the supplied block, responding to any rate limit errors.
#count_repos_by_relation_type_graphql, #search_repos_by_name_graphql
Constructor Details
#initialize(token, host: nil, per_page: DEFAULT_PER_PAGE, parallel: true) ⇒ Client
token - The GitHub API token to use.
host - The GitHub hostname. If nil, github.com will be used.
per_page - The number of objects that should be displayed per page.
parallel - When set to true hitting the rate limit will result in a
dedicated error being raised. When set to `false` we will
instead just `sleep()` until the rate limit is reset. Setting
this value to `true` for parallel importing is crucial as
otherwise hitting the rate limit will result in a thread
being blocked in a `sleep()` call for up to an hour.
51
52
53
54
55
56
57
58
59
60
61
62
63
|
# File 'lib/gitlab/github_import/client.rb', line 51
def initialize(token, host: nil, per_page: DEFAULT_PER_PAGE, parallel: true)
@host = host
@octokit = ::Octokit::Client.new(
access_token: token,
per_page: per_page,
api_endpoint: api_endpoint,
web_endpoint: web_endpoint
)
@octokit.connection_options[:ssl] = { verify: verify_ssl }
@parallel = parallel
end
|
Instance Attribute Details
#octokit ⇒ Object
Returns the value of attribute octokit.
20
21
22
|
# File 'lib/gitlab/github_import/client.rb', line 20
def octokit
@octokit
end
|
Instance Method Details
#api_endpoint ⇒ Object
239
240
241
|
# File 'lib/gitlab/github_import/client.rb', line 239
def api_endpoint
@host || custom_api_endpoint || default_api_endpoint
end
|
#branch_protection(repo_name, branch_name) ⇒ Object
127
128
129
|
# File 'lib/gitlab/github_import/client.rb', line 127
def branch_protection(repo_name, branch_name)
with_rate_limit { octokit.branch_protection(repo_name, branch_name).to_h }
end
|
#branches(*args) ⇒ Object
119
120
121
|
# File 'lib/gitlab/github_import/client.rb', line 119
def branches(*args)
each_object(:branches, *args)
end
|
#collaborators(*args) ⇒ Object
123
124
125
|
# File 'lib/gitlab/github_import/client.rb', line 123
def collaborators(*args)
each_object(:collaborators, *args)
end
|
#custom_api_endpoint ⇒ Object
247
248
249
|
# File 'lib/gitlab/github_import/client.rb', line 247
def custom_api_endpoint
github_omniauth_provider.dig('args', 'client_options', 'site')
end
|
#default_api_endpoint ⇒ Object
251
252
253
|
# File 'lib/gitlab/github_import/client.rb', line 251
def default_api_endpoint
OmniAuth::Strategies::GitHub.default_options[:client_options][:site] || ::Octokit::Default.api_endpoint
end
|
#each_object(method, *args, &block) ⇒ Object
Iterates over all of the objects for the given method (e.g. ‘:labels`).
method - The method to send to Octokit for querying data. args - Any arguments to pass to the Octokit method.
165
166
167
168
169
170
171
172
173
|
# File 'lib/gitlab/github_import/client.rb', line 165
def each_object(method, *args, &block)
return to_enum(__method__, method, *args) unless block
each_page(method, *args) do |page|
page.objects.each do |object|
yield object.to_h
end
end
end
|
#each_page(method, *args) {|Page.new(collection, page)| ... } ⇒ Object
Fetches data from the GitHub API and yields a Page object for every page of data, without loading all of them into memory.
method - The Octokit method to use for getting the data. args - Arguments to pass to the Octokit method.
rubocop: disable GitlabSecurity/PublicSend
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
|
# File 'lib/gitlab/github_import/client.rb', line 138
def each_page(method, *args, &block)
return to_enum(__method__, method, *args) unless block
page =
if args.last.is_a?(Hash) && args.last[:page]
args.last[:page]
else
1
end
collection = with_rate_limit { octokit.public_send(method, *args) }
next_url = octokit.last_response.rels[:next]
yield Page.new(collection, page)
while next_url
response = with_rate_limit { next_url.get }
next_url = response.rels[:next]
yield Page.new(response.data, page += 1)
end
end
|
#github_omniauth_provider ⇒ Object
259
260
261
|
# File 'lib/gitlab/github_import/client.rb', line 259
def github_omniauth_provider
@github_omniauth_provider ||= Gitlab::Auth::OAuth::Provider.config_for('github').to_h
end
|
#labels(*args) ⇒ Object
107
108
109
|
# File 'lib/gitlab/github_import/client.rb', line 107
def labels(*args)
each_object(:labels, *args)
end
|
#milestones(*args) ⇒ Object
111
112
113
|
# File 'lib/gitlab/github_import/client.rb', line 111
def milestones(*args)
each_object(:milestones, *args)
end
|
#parallel? ⇒ Boolean
65
66
67
|
# File 'lib/gitlab/github_import/client.rb', line 65
def parallel?
@parallel
end
|
#pull_request(repo_name, iid) ⇒ Object
103
104
105
|
# File 'lib/gitlab/github_import/client.rb', line 103
def pull_request(repo_name, iid)
with_rate_limit { octokit.pull_request(repo_name, iid).to_h }
end
|
#pull_request_review_requests(repo_name, iid) ⇒ Object
88
89
90
|
# File 'lib/gitlab/github_import/client.rb', line 88
def pull_request_review_requests(repo_name, iid)
with_rate_limit { octokit.pull_request_review_requests(repo_name, iid).to_h }
end
|
#pull_request_reviews(repo_name, iid) ⇒ Object
84
85
86
|
# File 'lib/gitlab/github_import/client.rb', line 84
def pull_request_reviews(repo_name, iid)
each_object(:pull_request_reviews, repo_name, iid)
end
|
#raise_or_wait_for_rate_limit ⇒ Object
216
217
218
219
220
221
222
223
224
|
# File 'lib/gitlab/github_import/client.rb', line 216
def raise_or_wait_for_rate_limit
rate_limit_counter.increment
if parallel?
raise RateLimitError
else
sleep(rate_limit_resets_in)
end
end
|
#rate_limit_counter ⇒ Object
263
264
265
266
267
268
|
# File 'lib/gitlab/github_import/client.rb', line 263
def rate_limit_counter
@rate_limit_counter ||= Gitlab::Metrics.counter(
:github_importer_rate_limit_hits,
'The number of times we hit the GitHub rate limit when importing projects'
)
end
|
#rate_limit_resets_in ⇒ Object
226
227
228
229
230
231
|
# File 'lib/gitlab/github_import/client.rb', line 226
def rate_limit_resets_in
octokit.rate_limit.resets_in + 5
end
|
#rate_limiting_enabled? ⇒ Boolean
233
234
235
236
237
|
# File 'lib/gitlab/github_import/client.rb', line 233
def rate_limiting_enabled?
strong_memoize(:rate_limiting_enabled) do
api_endpoint.include?('.github.com')
end
end
|
#releases(*args) ⇒ Object
115
116
117
|
# File 'lib/gitlab/github_import/client.rb', line 115
def releases(*args)
each_object(:releases, *args)
end
|
#remaining_requests ⇒ Object
208
209
210
|
# File 'lib/gitlab/github_import/client.rb', line 208
def remaining_requests
octokit.rate_limit.remaining
end
|
#repos(options = {}) ⇒ Object
92
93
94
|
# File 'lib/gitlab/github_import/client.rb', line 92
def repos(options = {})
octokit.repos(nil, options).map(&:to_h)
end
|
#repository(name) ⇒ Object
Returns the details of a GitHub repository.
name - The path (in the form ‘owner/repository`) of the repository.
99
100
101
|
# File 'lib/gitlab/github_import/client.rb', line 99
def repository(name)
with_rate_limit { octokit.repo(name).to_h }
end
|
#request_count_counter ⇒ Object
270
271
272
273
274
275
|
# File 'lib/gitlab/github_import/client.rb', line 270
def request_count_counter
@request_counter ||= Gitlab::Metrics.counter(
:github_importer_request_count,
'The number of GitHub API calls performed when importing projects'
)
end
|
#requests_limit ⇒ Object
212
213
214
|
# File 'lib/gitlab/github_import/client.rb', line 212
def requests_limit
octokit.rate_limit.limit
end
|
#requests_remaining? ⇒ Boolean
Returns ‘true` if we’re still allowed to perform API calls. Search API has rate limit of 30, use lowered threshold when search is used.
#user(username, options = {}) ⇒ Object
Returns the details of a GitHub user. 304 (Not Modified) status means the user is cached - API won’t return user data.
74
75
76
77
78
79
80
81
82
|
# File 'lib/gitlab/github_import/client.rb', line 74
def user(username, options = {})
with_rate_limit do
user = octokit.user(username, options)
next if octokit.last_response&.status == 304
user.to_h
end
end
|
#verify_ssl ⇒ Object
255
256
257
|
# File 'lib/gitlab/github_import/client.rb', line 255
def verify_ssl
github_omniauth_provider.fetch('verify_ssl', true)
end
|
#web_endpoint ⇒ Object
243
244
245
|
# File 'lib/gitlab/github_import/client.rb', line 243
def web_endpoint
@host || custom_api_endpoint || ::Octokit::Default.web_endpoint
end
|
#with_rate_limit ⇒ Object
Yields the supplied block, responding to any rate limit errors.
The exact strategy used for handling rate limiting errors depends on whether we are running in parallel mode or not. For more information see ‘#rate_or_wait_for_rate_limit`.
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
|
# File 'lib/gitlab/github_import/client.rb', line 180
def with_rate_limit
return with_retry { yield } unless rate_limiting_enabled?
request_count_counter.increment
raise_or_wait_for_rate_limit unless requests_remaining?
begin
with_retry { yield }
rescue ::Octokit::TooManyRequests
raise_or_wait_for_rate_limit
retry
end
end
|