Class: Gitlab::GithubImport::Client

Inherits:
Object
  • Object
show all
Includes:
Utils::StrongMemoize
Defined in:
lib/gitlab/github_import/client.rb

Overview

HTTP client for interacting with the GitHub API.

This class is basically a fancy wrapped around Octokit while adding some functionality to deal with rate limiting and parallel imports. Usage is mostly the same as Octokit, for example:

client = GithubImport::Client.new('hunter2')

client.labels.each do |label|
  puts label.name
end

Defined Under Namespace

Classes: Page

Constant Summary collapse

RATE_LIMIT_THRESHOLD =

The minimum number of requests we want to keep available.

We don't use a value of 0 as multiple threads may be using the same token in parallel. This could result in all of them hitting the GitHub rate limit at once. The threshold is put in place to not hit the limit in most cases.

50

Instance Attribute Summary collapse

Instance Method Summary collapse

Methods included from Utils::StrongMemoize

#clear_memoization, #strong_memoize, #strong_memoized?

Constructor Details

#initialize(token, per_page: 100, parallel: true) ⇒ Client

token - The GitHub API token to use.

per_page - The number of objects that should be displayed per page.

parallel - When set to true hitting the rate limit will result in a

dedicated error being raised. When set to `false` we will
instead just `sleep()` until the rate limit is reset. Setting
this value to `true` for parallel importing is crucial as
otherwise hitting the rate limit will result in a thread
being blocked in a `sleep()` call for up to an hour.


42
43
44
45
46
47
48
49
50
51
52
 
# File 'lib/gitlab/github_import/client.rb', line 42

def initialize(token, per_page: 100, parallel: true)
  @octokit = ::Octokit::Client.new(
    access_token: token,
    per_page: per_page,
    api_endpoint: api_endpoint
  )

  @octokit.connection_options[:ssl] = { verify: verify_ssl }

  @parallel = parallel
end

Instance Attribute Details

#octokit ⇒ Object (readonly)

Returns the value of attribute octokit


19
20
21
 
# File 'lib/gitlab/github_import/client.rb', line 19

def octokit
  @octokit
end

Instance Method Details

#api_endpoint ⇒ Object 


183
184
185
 
# File 'lib/gitlab/github_import/client.rb', line 183

def api_endpoint
  custom_api_endpoint || default_api_endpoint
end

#custom_api_endpoint ⇒ Object 


187
188
189
 
# File 'lib/gitlab/github_import/client.rb', line 187

def custom_api_endpoint
  github_omniauth_provider.dig('args', 'client_options', 'site')
end

#default_api_endpoint ⇒ Object 


191
192
193
 
# File 'lib/gitlab/github_import/client.rb', line 191

def default_api_endpoint
  OmniAuth::Strategies::GitHub.default_options[:client_options][:site] || ::Octokit::Default.api_endpoint
end

#each_object(method, *args, &block) ⇒ Object

Iterates over all of the objects for the given method (e.g. `:labels`).

method - The method to send to Octokit for querying data. args - Any arguments to pass to the Octokit method.


118
119
120
121
122
123
124
125
126
 
# File 'lib/gitlab/github_import/client.rb', line 118

def each_object(method, *args, &block)
  return to_enum(__method__, method, *args) unless block_given?

  each_page(method, *args) do |page|
    page.objects.each do |object|
      yield object
    end
  end
end

#each_page(method, *args) {|Page.new(collection, page)| ... } ⇒ Object

Fetches data from the GitHub API and yields a Page object for every page of data, without loading all of them into memory.

method - The Octokit method to use for getting the data. args - Arguments to pass to the Octokit method.

rubocop: disable GitlabSecurity/PublicSend

Yields:

  • (Page.new(collection, page)) 

91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
 
# File 'lib/gitlab/github_import/client.rb', line 91

def each_page(method, *args, &block)
  return to_enum(__method__, method, *args) unless block_given?

  page =
    if args.last.is_a?(Hash) && args.last[:page]
      args.last[:page]
    else
      1
    end

  collection = with_rate_limit { octokit.public_send(method, *args) }
  next_url = octokit.last_response.rels[:next]

  yield Page.new(collection, page)

  while next_url
    response = with_rate_limit { next_url.get }
    next_url = response.rels[:next]

    yield Page.new(response.data, page += 1)
  end
end

#github_omniauth_provider ⇒ Object 


199
200
201
 
# File 'lib/gitlab/github_import/client.rb', line 199

def github_omniauth_provider
  @github_omniauth_provider ||= Gitlab::Auth::OAuth::Provider.config_for('github').to_h
end

#labels(*args) ⇒ Object 


72
73
74
 
# File 'lib/gitlab/github_import/client.rb', line 72

def labels(*args)
  each_object(:labels, *args)
end

#milestones(*args) ⇒ Object 


76
77
78
 
# File 'lib/gitlab/github_import/client.rb', line 76

def milestones(*args)
  each_object(:milestones, *args)
end

#parallel? ⇒ Boolean

Returns:

  • (Boolean) 

54
55
56
 
# File 'lib/gitlab/github_import/client.rb', line 54

def parallel?
  @parallel
end

#raise_or_wait_for_rate_limit ⇒ Object 


160
161
162
163
164
165
166
167
168
 
# File 'lib/gitlab/github_import/client.rb', line 160

def raise_or_wait_for_rate_limit
  rate_limit_counter.increment

  if parallel?
    raise RateLimitError
  else
    sleep(rate_limit_resets_in)
  end
end

#rate_limit_counter ⇒ Object 


203
204
205
206
207
208
 
# File 'lib/gitlab/github_import/client.rb', line 203

def rate_limit_counter
  @rate_limit_counter ||= Gitlab::Metrics.counter(
    :github_importer_rate_limit_hits,
    'The number of times we hit the GitHub rate limit when importing projects'
  )
end

#rate_limit_resets_in ⇒ Object 


170
171
172
173
174
175
 
# File 'lib/gitlab/github_import/client.rb', line 170

def rate_limit_resets_in
  # We add a few seconds to the rate limit so we don't _immediately_
  # resume when the rate limit resets as this may result in us performing
  # a request before GitHub has a chance to reset the limit.
  octokit.rate_limit.resets_in + 5
end

#rate_limiting_enabled? ⇒ Boolean

Returns:

  • (Boolean) 

177
178
179
180
181
 
# File 'lib/gitlab/github_import/client.rb', line 177

def rate_limiting_enabled?
  strong_memoize(:rate_limiting_enabled) do
    api_endpoint.include?('.github.com')
  end
end

#releases(*args) ⇒ Object 


80
81
82
 
# File 'lib/gitlab/github_import/client.rb', line 80

def releases(*args)
  each_object(:releases, *args)
end

#remaining_requests ⇒ Object 


156
157
158
 
# File 'lib/gitlab/github_import/client.rb', line 156

def remaining_requests
  octokit.rate_limit.remaining
end

#repository(name) ⇒ Object

Returns the details of a GitHub repository.

name - The path (in the form `owner/repository`) of the repository.


68
69
70
 
# File 'lib/gitlab/github_import/client.rb', line 68

def repository(name)
  with_rate_limit { octokit.repo(name) }
end

#request_count_counter ⇒ Object 


210
211
212
213
214
215
 
# File 'lib/gitlab/github_import/client.rb', line 210

def request_count_counter
  @request_counter ||= Gitlab::Metrics.counter(
    :github_importer_request_count,
    'The number of GitHub API calls performed when importing projects'
  )
end

#requests_remaining? ⇒ Boolean

Returns `true` if we're still allowed to perform API calls.

Returns:

  • (Boolean) 

152
153
154
 
# File 'lib/gitlab/github_import/client.rb', line 152

def requests_remaining?
  remaining_requests > RATE_LIMIT_THRESHOLD
end

#user(username) ⇒ Object

Returns the details of a GitHub user.

username - The username of the user.


61
62
63
 
# File 'lib/gitlab/github_import/client.rb', line 61

def user(username)
  with_rate_limit { octokit.user(username) }
end

#verify_ssl ⇒ Object 


195
196
197
 
# File 'lib/gitlab/github_import/client.rb', line 195

def verify_ssl
  github_omniauth_provider.fetch('verify_ssl', true)
end

#with_rate_limit ⇒ Object

Yields the supplied block, responding to any rate limit errors.

The exact strategy used for handling rate limiting errors depends on whether we are running in parallel mode or not. For more information see `#rate_or_wait_for_rate_limit`.


133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
 
# File 'lib/gitlab/github_import/client.rb', line 133

def with_rate_limit
  return yield unless rate_limiting_enabled?

  request_count_counter.increment

  raise_or_wait_for_rate_limit unless requests_remaining?

  begin
    yield
  rescue ::Octokit::TooManyRequests
    raise_or_wait_for_rate_limit

    # This retry will only happen when running in sequential mode as we'll
    # raise an error in parallel mode.
    retry
  end
end