Class: Gitlab::GithubImport::Client
- Inherits:
-
Object
- Object
- Gitlab::GithubImport::Client
- Includes:
- Utils::StrongMemoize
- Defined in:
- lib/gitlab/github_import/client.rb
Overview
HTTP client for interacting with the GitHub API.
This class is basically a fancy wrapped around Octokit while adding some functionality to deal with rate limiting and parallel imports. Usage is mostly the same as Octokit, for example:
client = GithubImport::Client.new('hunter2')
client.labels.each do |label|
puts label.name
end
Defined Under Namespace
Classes: Page
Constant Summary collapse
- RATE_LIMIT_THRESHOLD =
The minimum number of requests we want to keep available.
We don't use a value of 0 as multiple threads may be using the same token in parallel. This could result in all of them hitting the GitHub rate limit at once. The threshold is put in place to not hit the limit in most cases.
50
Instance Attribute Summary collapse
-
#octokit ⇒ Object
readonly
Returns the value of attribute octokit.
Instance Method Summary collapse
- #api_endpoint ⇒ Object
- #custom_api_endpoint ⇒ Object
- #default_api_endpoint ⇒ Object
-
#each_object(method, *args, &block) ⇒ Object
Iterates over all of the objects for the given method (e.g. `:labels`).
-
#each_page(method, *args) {|Page.new(collection, page)| ... } ⇒ Object
Fetches data from the GitHub API and yields a Page object for every page of data, without loading all of them into memory.
- #github_omniauth_provider ⇒ Object
-
#initialize(token, per_page: 100, parallel: true) ⇒ Client
constructor
token - The GitHub API token to use.
- #labels(*args) ⇒ Object
- #milestones(*args) ⇒ Object
- #parallel? ⇒ Boolean
- #raise_or_wait_for_rate_limit ⇒ Object
- #rate_limit_counter ⇒ Object
- #rate_limit_resets_in ⇒ Object
- #rate_limiting_enabled? ⇒ Boolean
- #releases(*args) ⇒ Object
- #remaining_requests ⇒ Object
-
#repository(name) ⇒ Object
Returns the details of a GitHub repository.
- #request_count_counter ⇒ Object
-
#requests_remaining? ⇒ Boolean
Returns `true` if we're still allowed to perform API calls.
-
#user(username) ⇒ Object
Returns the details of a GitHub user.
- #verify_ssl ⇒ Object
-
#with_rate_limit ⇒ Object
Yields the supplied block, responding to any rate limit errors.
Methods included from Utils::StrongMemoize
#clear_memoization, #strong_memoize, #strong_memoized?
Constructor Details
#initialize(token, per_page: 100, parallel: true) ⇒ Client
token - The GitHub API token to use.
per_page - The number of objects that should be displayed per page.
parallel - When set to true hitting the rate limit will result in a
dedicated error being raised. When set to `false` we will
instead just `sleep()` until the rate limit is reset. Setting
this value to `true` for parallel importing is crucial as
otherwise hitting the rate limit will result in a thread
being blocked in a `sleep()` call for up to an hour.
42 43 44 45 46 47 48 49 50 51 52 |
# File 'lib/gitlab/github_import/client.rb', line 42 def initialize(token, per_page: 100, parallel: true) @octokit = ::Octokit::Client.new( access_token: token, per_page: per_page, api_endpoint: api_endpoint ) @octokit.[:ssl] = { verify: verify_ssl } @parallel = parallel end |
Instance Attribute Details
#octokit ⇒ Object (readonly)
Returns the value of attribute octokit
19 20 21 |
# File 'lib/gitlab/github_import/client.rb', line 19 def octokit @octokit end |
Instance Method Details
#api_endpoint ⇒ Object
183 184 185 |
# File 'lib/gitlab/github_import/client.rb', line 183 def api_endpoint custom_api_endpoint || default_api_endpoint end |
#custom_api_endpoint ⇒ Object
187 188 189 |
# File 'lib/gitlab/github_import/client.rb', line 187 def custom_api_endpoint github_omniauth_provider.dig('args', 'client_options', 'site') end |
#default_api_endpoint ⇒ Object
191 192 193 |
# File 'lib/gitlab/github_import/client.rb', line 191 def default_api_endpoint OmniAuth::Strategies::GitHub.[:client_options][:site] || ::Octokit::Default.api_endpoint end |
#each_object(method, *args, &block) ⇒ Object
Iterates over all of the objects for the given method (e.g. `:labels`).
method - The method to send to Octokit for querying data. args - Any arguments to pass to the Octokit method.
118 119 120 121 122 123 124 125 126 |
# File 'lib/gitlab/github_import/client.rb', line 118 def each_object(method, *args, &block) return to_enum(__method__, method, *args) unless block_given? each_page(method, *args) do |page| page.objects.each do |object| yield object end end end |
#each_page(method, *args) {|Page.new(collection, page)| ... } ⇒ Object
Fetches data from the GitHub API and yields a Page object for every page of data, without loading all of them into memory.
method - The Octokit method to use for getting the data. args - Arguments to pass to the Octokit method.
rubocop: disable GitlabSecurity/PublicSend
91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 |
# File 'lib/gitlab/github_import/client.rb', line 91 def each_page(method, *args, &block) return to_enum(__method__, method, *args) unless block_given? page = if args.last.is_a?(Hash) && args.last[:page] args.last[:page] else 1 end collection = with_rate_limit { octokit.public_send(method, *args) } next_url = octokit.last_response.rels[:next] yield Page.new(collection, page) while next_url response = with_rate_limit { next_url.get } next_url = response.rels[:next] yield Page.new(response.data, page += 1) end end |
#github_omniauth_provider ⇒ Object
199 200 201 |
# File 'lib/gitlab/github_import/client.rb', line 199 def github_omniauth_provider @github_omniauth_provider ||= Gitlab::Auth::OAuth::Provider.config_for('github').to_h end |
#labels(*args) ⇒ Object
72 73 74 |
# File 'lib/gitlab/github_import/client.rb', line 72 def labels(*args) each_object(:labels, *args) end |
#milestones(*args) ⇒ Object
76 77 78 |
# File 'lib/gitlab/github_import/client.rb', line 76 def milestones(*args) each_object(:milestones, *args) end |
#parallel? ⇒ Boolean
54 55 56 |
# File 'lib/gitlab/github_import/client.rb', line 54 def parallel? @parallel end |
#raise_or_wait_for_rate_limit ⇒ Object
160 161 162 163 164 165 166 167 168 |
# File 'lib/gitlab/github_import/client.rb', line 160 def raise_or_wait_for_rate_limit rate_limit_counter.increment if parallel? raise RateLimitError else sleep(rate_limit_resets_in) end end |
#rate_limit_counter ⇒ Object
203 204 205 206 207 208 |
# File 'lib/gitlab/github_import/client.rb', line 203 def rate_limit_counter @rate_limit_counter ||= Gitlab::Metrics.counter( :github_importer_rate_limit_hits, 'The number of times we hit the GitHub rate limit when importing projects' ) end |
#rate_limit_resets_in ⇒ Object
170 171 172 173 174 175 |
# File 'lib/gitlab/github_import/client.rb', line 170 def rate_limit_resets_in # We add a few seconds to the rate limit so we don't _immediately_ # resume when the rate limit resets as this may result in us performing # a request before GitHub has a chance to reset the limit. octokit.rate_limit.resets_in + 5 end |
#rate_limiting_enabled? ⇒ Boolean
177 178 179 180 181 |
# File 'lib/gitlab/github_import/client.rb', line 177 def rate_limiting_enabled? strong_memoize(:rate_limiting_enabled) do api_endpoint.include?('.github.com') end end |
#releases(*args) ⇒ Object
80 81 82 |
# File 'lib/gitlab/github_import/client.rb', line 80 def releases(*args) each_object(:releases, *args) end |
#remaining_requests ⇒ Object
156 157 158 |
# File 'lib/gitlab/github_import/client.rb', line 156 def remaining_requests octokit.rate_limit.remaining end |
#repository(name) ⇒ Object
Returns the details of a GitHub repository.
name - The path (in the form `owner/repository`) of the repository.
68 69 70 |
# File 'lib/gitlab/github_import/client.rb', line 68 def repository(name) with_rate_limit { octokit.repo(name) } end |
#request_count_counter ⇒ Object
210 211 212 213 214 215 |
# File 'lib/gitlab/github_import/client.rb', line 210 def request_count_counter @request_counter ||= Gitlab::Metrics.counter( :github_importer_request_count, 'The number of GitHub API calls performed when importing projects' ) end |
#requests_remaining? ⇒ Boolean
Returns `true` if we're still allowed to perform API calls.
152 153 154 |
# File 'lib/gitlab/github_import/client.rb', line 152 def requests_remaining? remaining_requests > RATE_LIMIT_THRESHOLD end |
#user(username) ⇒ Object
Returns the details of a GitHub user.
username - The username of the user.
61 62 63 |
# File 'lib/gitlab/github_import/client.rb', line 61 def user(username) with_rate_limit { octokit.user(username) } end |
#verify_ssl ⇒ Object
195 196 197 |
# File 'lib/gitlab/github_import/client.rb', line 195 def verify_ssl github_omniauth_provider.fetch('verify_ssl', true) end |
#with_rate_limit ⇒ Object
Yields the supplied block, responding to any rate limit errors.
The exact strategy used for handling rate limiting errors depends on whether we are running in parallel mode or not. For more information see `#rate_or_wait_for_rate_limit`.
133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 |
# File 'lib/gitlab/github_import/client.rb', line 133 def with_rate_limit return yield unless rate_limiting_enabled? request_count_counter.increment raise_or_wait_for_rate_limit unless requests_remaining? begin yield rescue ::Octokit::TooManyRequests raise_or_wait_for_rate_limit # This retry will only happen when running in sequential mode as we'll # raise an error in parallel mode. retry end end |