Class: ConnectorsSdk::GitLab::Extractor
- Inherits:
-
Base::Extractor
- Object
- Base::Extractor
- ConnectorsSdk::GitLab::Extractor
- Defined in:
- lib/connectors_sdk/gitlab/extractor.rb
Constant Summary collapse
- PAGE_SIZE =
max is 100
100
Constants inherited from Base::Extractor
Base::Extractor::DEFAULT_CURSOR_KEY, Base::Extractor::MAX_CONNECTION_ATTEMPTS, Base::Extractor::TRANSIENT_SERVER_ERROR_CLASSES
Instance Attribute Summary
Attributes inherited from Base::Extractor
#client_proc, #completed, #config, #content_source_id, #features, #monitor, #original_cursors, #service_type
Instance Method Summary collapse
- #yield_deleted_ids(ids) ⇒ Object
- #yield_document_changes(modified_since: nil) ⇒ Object
- #yield_permissions(source_user_id) {|result| ... } ⇒ Object
Methods inherited from Base::Extractor
#authorization_data, #authorization_data!, #client, #client!, #convert_transient_server_errors, #cursors_modified_since_start?, #deleted_ids, #document_changes, #download_args_and_proc, #evictable?, #identifying_error_message, #initialize, #permissions, #retrieve_latest_cursors, #transient_error?, #with_auth_tokens_and_retry, #yield_single_document_change
Constructor Details
This class inherits a constructor from ConnectorsSdk::Base::Extractor
Instance Method Details
#yield_deleted_ids(ids) ⇒ Object
59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 |
# File 'lib/connectors_sdk/gitlab/extractor.rb', line 59 def yield_deleted_ids(ids) if ids.present? ids.each do |id| response = client.get("projects/#{id}") if response.status == 404 # not found - assume deleted yield id else unless response.success? raise "Could not get a project by ID: #{id}, response code: #{response.status}, response: #{response.body}" end end end end end |
#yield_document_changes(modified_since: nil) ⇒ Object
20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 |
# File 'lib/connectors_sdk/gitlab/extractor.rb', line 20 def yield_document_changes(modified_since: nil) query_params = { :pagination => :keyset, :per_page => PAGE_SIZE, :order_by => :id, :sort => :desc } # looks like it's an incremental sync if modified_since.present? date_since = modified_since.is_a?(Time) ? modified_since : Time.new(modified_since) query_params[:last_activity_after] = date_since.iso8601 end next_page_link = nil loop do if next_page_link.present? if (matcher = /(https?:[^>]*)/.match(next_page_link)) clean_query = URI.parse(matcher.captures[0]).query query_params = Rack::Utils.parse_query(clean_query) else raise "Next page link has unexpected format: #{next_page_link}" end end response = client.get('projects', query_params) JSON.parse(response.body).map do |doc| doc = doc.with_indifferent_access if config. doc = doc.merge((doc[:id], doc[:visibility])) end yield :create_or_update, ConnectorsSdk::GitLab::Adapter.to_es_document(:project, doc), nil end next_page_link = response.headers['Link'] || nil break unless next_page_link.present? end end |
#yield_permissions(source_user_id) {|result| ... } ⇒ Object
75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 |
# File 'lib/connectors_sdk/gitlab/extractor.rb', line 75 def (source_user_id) result = [] if source_user_id.present? result.push("user:#{source_user_id}") user_response = client.get("users/#{source_user_id}") if user_response.success? username = JSON.parse(user_response.body).with_indifferent_access[:username] query = { :external => true, :username => username } external_response = client.get('users', query) if external_response.success? external_users = Hashie::Array.new(JSON.parse(external_response.body)) if external_users.empty? # the user is not external result.push('type:internal') end else raise "Could not check external user status by ID: #{source_user_id}" end else raise "User isn't found by ID: #{source_user_id}" end end yield result end |