Class: Autobuild::Importer

Inherits:
Object
  • Object
show all
Defined in:
lib/autobuild/importer.rb

Direct Known Subclasses

ArchiveImporter, CVSImporter, DarcsImporter, Git, Hg, SVN

Defined Under Namespace

Classes: Hook, Status

Class Attribute Summary collapse

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(options) ⇒ Importer

Creates a new Importer object. The options known to Importer are:

:patches

a list of patch to apply after import

More options are specific to each importer type.



146
147
148
149
150
151
152
153
 
# File 'lib/autobuild/importer.rb', line 146

def initialize(options)
    @options = options.dup
    @options[:retry_count] = Integer(@options[:retry_count] || 0)
    @repository_id = options[:repository_id] || "#{self.class.name}:#{object_id}"
    @interactive = options[:interactive]
    @source_id = options[:source_id] || @repository_id
    @post_hooks = Array.new
end

Class Attribute Details

.fallback_handlers ⇒ Object (readonly)

The set of handlers registered by Importer.fallback



24
25
26
 
# File 'lib/autobuild/importer.rb', line 24

def fallback_handlers
  @fallback_handlers
end

Instance Attribute Details

#interactive=(value) ⇒ Object (writeonly)

Changes whether this importer is interactive or not



184
185
186
 
# File 'lib/autobuild/importer.rb', line 184

def interactive=(value)
  @interactive = value
end

#options ⇒ Hash (readonly)

Returns the original option hash as given to #initialize.

Returns:

  • (Hash) —

    the original option hash as given to #initialize



140
141
142
 
# File 'lib/autobuild/importer.rb', line 140

def options
  @options
end

#post_hooks ⇒ Object (readonly)

A list of hooks that are called after a successful checkout or update

They are added either at the instance level with #add_post_hook or globally for all importers of a given type with add_post_hook



274
275
276
 
# File 'lib/autobuild/importer.rb', line 274

def post_hooks
  @post_hooks
end

#repository_id ⇒ String (readonly)

Returns a string that identifies the remote repository uniquely

This can be used to check whether two importers are pointing to the same repository, regardless of e.g. the access protocol used. For instance, two git importers that point to the same repository but different branches would have the same repository_id but different source_id

Returns:

  • (String)

See Also:



164
165
166
 
# File 'lib/autobuild/importer.rb', line 164

def repository_id
  @repository_id
end

#source_id ⇒ String (readonly)

Returns a string that identifies the remote source uniquely

This can be used to check whether two importers are pointing to the same code base inside the same repository. For instance, two git importers that point to the same repository but different branches would have the same repository_id but different source_id

Returns:

  • (String)

See Also:



175
176
177
 
# File 'lib/autobuild/importer.rb', line 175

def source_id
  @source_id
end

Class Method Details

.add_post_hook(always: false) {|importer, package| ... } ⇒ Object

Define a post-import hook for all instances of this class

Yield Parameters:

  • importer (Importer) —

    the importer that finished

  • package (Package) —

    the package we’re acting on

See Also:



283
284
285
286
287
 
# File 'lib/autobuild/importer.rb', line 283

def self.add_post_hook(always: false, &hook)
    @post_hooks ||= Array.new
    @post_hooks << Hook.new(always, hook)
    nil
end

.cache_dirs(type) ⇒ nil, Array<String>

The cache directories for the given importer type.

This is used by some importers to save disk space and/or avoid downloading the same things over and over again

The default global cache directory is initialized from the AUTOBUILD_CACHE_DIR environment variable. Per-importer cache directories can be overriden by setting AUTOBUILD_TYPE_CACHE_DIR (e.g. AUTOBUILD_GIT_CACHE_DIR)

The following importers use caches:

  • the archive importer saves downloaded files in the cache. They are saved under an archives/ subdirectory of the default cache if set, or to the value of AUTOBUILD_ARCHIVES_CACHE_DIR

  • the git importer uses the cache directories as alternates for the git checkouts

Parameters:

  • type (String) —

    the importer type. If set, it Given a root cache directory X, and importer specific cache is setup as a subdirectory of X with e.g. X/git or X/archives. The subdirectory name is defined by this argument

Returns:

  • (nil, Array<String>)

See Also:



94
95
96
97
98
99
100
 
# File 'lib/autobuild/importer.rb', line 94

def self.cache_dirs(type)
    if @cache_dirs[type] || (env = ENV["AUTOBUILD_#{type.upcase}_CACHE_DIR"])
        @cache_dirs[type] ||= env.split(":")
    elsif (dirs = default_cache_dirs)
        dirs.map { |d| File.join(d, type) }
    end
end

.default_cache_dirs ⇒ Array<String>?

Returns the default cache directory if there is one

Returns:

  • (Array<String>, nil)

See Also:



106
107
108
109
110
111
112
 
# File 'lib/autobuild/importer.rb', line 106

def self.default_cache_dirs
    if @default_cache_dirs
        @default_cache_dirs
    elsif (from_env = ENV['AUTOBUILD_CACHE_DIR'])
        @default_cache_dirs = [from_env]
    end
end

.default_cache_dirs=(dirs) ⇒ Object

Sets the default cache directory

Parameters:

  • the (Array<String>, String) —

    directories

See Also:



127
128
129
 
# File 'lib/autobuild/importer.rb', line 127

def self.default_cache_dirs=(dirs)
    @default_cache_dirs = Array(dirs)
end

.each_post_hook(error: false) ⇒ Object

Enumerate the post-import hooks defined for all instances of this class



290
291
292
293
294
295
296
 
# File 'lib/autobuild/importer.rb', line 290

def self.each_post_hook(error: false)
    return enum_for(__method__) unless block_given?

    (@post_hooks ||= Array.new).each do |hook|
        yield(hook.callback) if hook.always || !error
    end
end

.fallback(&block) ⇒ Object

call-seq:

Autobuild::Importer.fallback { |package, importer| ... }

If called, registers the given block as a fallback mechanism for failing imports.

Fallbacks are tried in reverse order with the failing importer object as argument. The first valid importer object that has been returned will be used instead.

It is the responsibility of the fallback handler to make sure that it does not do infinite recursions and stuff like that.



18
19
20
 
# File 'lib/autobuild/importer.rb', line 18

def self.fallback(&block)
    @fallback_handlers.unshift(block)
end

.set_cache_dirs(type, *dirs) ⇒ Object

Sets the cache directory for a given importer type

Parameters:

  • type (String) —

    the importer type

  • dir (String) —

    the cache directory

See Also:



119
120
121
 
# File 'lib/autobuild/importer.rb', line 119

def self.set_cache_dirs(type, *dirs)
    @cache_dirs[type] = dirs
end

.unset_cache_dirs ⇒ Object

Unset all cache directories



132
133
134
135
 
# File 'lib/autobuild/importer.rb', line 132

def self.unset_cache_dirs
    @cache_dirs = Hash.new
    @default_cache_dirs = nil
end

Instance Method Details

#add_post_hook(always: false) {|importer, package| ... } ⇒ Object

Add a block that should be called when the import has successfully finished

Yield Parameters:

  • importer (Importer) —

    the importer that finished

  • package (Package) —

    the package we’re acting on

See Also:



313
314
315
 
# File 'lib/autobuild/importer.rb', line 313

def add_post_hook(always: false, &hook)
    post_hooks << Hook.new(always, hook)
end

#apply(package, path, patch_level = 0) ⇒ Object 



528
529
530
 
# File 'lib/autobuild/importer.rb', line 528

def apply(package, path, patch_level = 0)
    call_patch(package, false, path, patch_level)
end

#call_patch(package, reverse, file, patch_level) ⇒ Object 



522
523
524
525
526
 
# File 'lib/autobuild/importer.rb', line 522

def call_patch(package, reverse, file, patch_level)
    package.run(:patch, Autobuild.tool('patch'),
                "-p#{patch_level}", (reverse ? '-R' : nil), '--forward',
                input: file, working_directory: package.importdir)
end

#currently_applied_patches(package) ⇒ Object 



550
551
552
553
554
555
556
557
558
559
560
561
562
563
 
# File 'lib/autobuild/importer.rb', line 550

def currently_applied_patches(package)
    patches_file = patchlist(package)
    return parse_patch_list(package, patches_file) if File.exist?(patches_file)

    patches_file = File.join(package.importdir, "patches-autobuild-stamp")
    if File.exist?(patches_file)
        cur_patches = parse_patch_list(package, patches_file)
        save_patch_state(package, cur_patches)
        FileUtils.rm_f patches_file
        return currently_applied_patches(package)
    end

    []
end

#each_post_hook(error: false, &block) ⇒ Object

Enumerate the post-import hooks for this importer



318
319
320
321
322
323
324
325
326
 
# File 'lib/autobuild/importer.rb', line 318

def each_post_hook(error: false, &block)
    return enum_for(__method__, error: false) unless block_given?

    self.class.each_post_hook(error: error, &block)

    post_hooks.each do |hook|
        yield(hook.callback) if hook.always || !error
    end
end

#execute_post_hooks(package, error: false) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Call the post-import hooks added with #add_post_hook



301
302
303
304
305
 
# File 'lib/autobuild/importer.rb', line 301

def execute_post_hooks(package, error: false)
    each_post_hook(error: error) do |block|
        block.call(self, package)
    end
end

#fallback(error, package, *args, &block) ⇒ Object

Tries to find a fallback importer because of the given error.



498
499
500
501
502
503
504
505
506
507
508
509
510
 
# File 'lib/autobuild/importer.rb', line 498

def fallback(error, package, *args, &block)
    Importer.fallback_handlers.each do |handler|
        fallback_importer = handler.call(package, self)
        if fallback_importer.kind_of?(Importer)
            begin
                return fallback_importer.send(*args, &block)
            rescue Exception
                raise error
            end
        end
    end
    raise error
end

#fingerprint(package) ⇒ Object

Returns a unique hash representing the state of the imported package as a whole unit, including its dependencies and patches



197
198
199
200
201
202
203
204
205
206
207
208
 
# File 'lib/autobuild/importer.rb', line 197

def fingerprint(package)
    vcs_fingerprint_string = vcs_fingerprint(package)
    return unless vcs_fingerprint_string

    patches_fingerprint_string = patches_fingerprint(package)
    if patches_fingerprint_string
        Digest::SHA1.hexdigest(vcs_fingerprint_string +
                               patches_fingerprint_string)
    elsif patches.empty?
        vcs_fingerprint_string
    end
end

#import(package, *old_boolean, ignore_errors: false, checkout_only: false, allow_interactive: true, **options) ⇒ Object

Imports the given package

The importer will checkout or update code in package.importdir. No update will be done if update? returns false.

Parameters:

  • options (Hash) —

    a customizable set of options

Options Hash (**options):

  • :checkout_only (Boolean) — default: false —

    if true, the importer will not update an already checked-out package.

  • :only_local (Boolean) — default: false —

    if true, will only perform actions that do not require network access. Importers that do not support this mode will simply do nothing

  • :reset (Boolean) — default: false —

    if true, the importer’s configuration is interpreted as a hard state in which it should put the working copy. Otherwise, it tries to update the local repository with the remote information. For instance, a git importer for which a commit ID is given will, in this mode, reset the repository to the requested ID (if that does not involve losing commits). Otherwise, it will only ensure that the requested commit ID is present in the current HEAD.



460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
 
# File 'lib/autobuild/importer.rb', line 460

def import( # rubocop:disable Metrics/ParameterLists
    package, *old_boolean,
    ignore_errors: false, checkout_only: false, allow_interactive: true, **options
)
    # Backward compatibility
    unless old_boolean.empty?
        old_boolean = old_boolean.first
        Autoproj.warn "calling #import with a boolean as second argument "\
            "is deprecated, switch to the named argument interface instead"
        Autoproj.warn "   e.g. call import(package, only_local: #{old_boolean})"
        Autoproj.warn "   #{caller(1..1).first}"
        options[:only_local] = old_boolean
    end

    importdir = package.importdir
    if File.directory?(importdir)
        package.isolate_errors(mark_as_failed: false,
                               ignore_errors: ignore_errors) do
            if !checkout_only && package.update?
                perform_update(package, checkout_only: false, **options)
            elsif Autobuild.verbose
                package.message "%s: not updating"
            end
        end

    elsif File.exist?(importdir)
        raise ConfigException.new(package, 'import'),
              "#{importdir} exists but is not a directory"
    else
        package.isolate_errors(mark_as_failed: true,
                               ignore_errors: ignore_errors) do
            perform_checkout(package, allow_interactive: allow_interactive)
            true
        end
    end
end

#interactive? ⇒ Boolean

Whether this importer will need interaction with the user, for instance to give credentials

Returns:

  • (Boolean) 


179
180
181
 
# File 'lib/autobuild/importer.rb', line 179

def interactive?
    @interactive
end

#parse_patch_list(package, patches_file) ⇒ Object 



536
537
538
539
540
541
542
543
544
545
546
547
548
 
# File 'lib/autobuild/importer.rb', line 536

def parse_patch_list(package, patches_file)
    File.readlines(patches_file).map do |line|
        line = line.rstrip
        if line =~ /^(.*)\s+(\d+)$/
            path = File.expand_path($1, package.importdir)
            level = Integer($2)
        else
            path = File.expand_path(line, package.importdir)
            level = 0
        end
        [path, level, File.read(path)]
    end
end

#patch(package, patches = self.patches) ⇒ Object 



565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
 
# File 'lib/autobuild/importer.rb', line 565

def patch(package, patches = self.patches)
    # Get the list of already applied patches
    cur_patches = currently_applied_patches(package)

    cur_patches_state = cur_patches.map { |_, level, content| [level, content] }
    patches_state     = patches.map { |_, level, content| [level, content] }
    return false if cur_patches_state == patches_state

    # Do not be smart, remove all already applied patches
    # and then apply the new ones
    begin
        apply_count = (patches - cur_patches).size
        unapply_count = (cur_patches - patches).size
        if apply_count > 0 && unapply_count > 0
            package.message "patching %s: applying #{apply_count} and "\
                "unapplying #{unapply_count} patch(es)"
        elsif apply_count > 0
            package.message "patching %s: applying #{apply_count} patch(es)"
        elsif unapply_count > 0
            package.message "patching %s: unapplying #{unapply_count} patch(es)"
        end

        while (p = cur_patches.last)
            p, level, = *p
            unapply(package, p, level)
            cur_patches.pop
        end

        patches.to_a.each do |new_patch, new_patch_level, content|
            apply(package, new_patch, new_patch_level)
            cur_patches << [new_patch, new_patch_level, content]
        end
    ensure
        save_patch_state(package, cur_patches)
    end

    true
end

#patchdir(package) ⇒ Object 



512
513
514
 
# File 'lib/autobuild/importer.rb', line 512

def patchdir(package)
    File.join(package.importdir, ".autobuild-patches")
end

#patches ⇒ Object 



235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
 
# File 'lib/autobuild/importer.rb', line 235

def patches
    patches =
        if @options[:patches].respond_to?(:to_ary)
            @options[:patches]
        elsif !@options[:patches]
            []
        else
            [[@options[:patches], 0]]
        end

    single_patch = (patches.size == 2 &&
        patches[0].respond_to?(:to_str) &&
        patches[1].respond_to?(:to_int))

    patches = [patches] if single_patch
    patches.map do |obj|
        if obj.respond_to?(:to_str)
            path  = obj
            level = 0
        elsif obj.respond_to?(:to_ary)
            path, level = obj
        else
            raise Arguments, "wrong patch specification #{obj.inspect}"
        end
        [path, level, File.read(path)]
    end
end

#patches_fingerprint(package) ⇒ Object

fingerprint for patches associated to this package



219
220
221
222
223
224
225
 
# File 'lib/autobuild/importer.rb', line 219

def patches_fingerprint(package)
    cur_patches = currently_applied_patches(package)
    cur_patches.map(&:shift) # leave only level and source information
    if !patches.empty? && cur_patches
      Digest::SHA1.hexdigest(cur_patches.sort.flatten.join(""))
    end
end

#patchlist(package) ⇒ Object

We assume that package.importdir already exists (checkout is supposed to have been called)



518
519
520
 
# File 'lib/autobuild/importer.rb', line 518

def patchlist(package)
    File.join(patchdir(package), "list")
end

#perform_checkout(package, **options) ⇒ Object 



402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
 
# File 'lib/autobuild/importer.rb', line 402

def perform_checkout(package, **options)
    last_error = nil
    package.progress_start "checking out %s", :done_message => 'checked out %s' do
        retry_count = 0
        begin
            checkout(package, **options)
            execute_post_hooks(package)
        rescue Interrupt
            if last_error then raise last_error
            else
                raise
            end
        rescue ::Exception => e
            last_error = e
            retry_count = update_retry_count(e, retry_count)
            raise unless retry_count

            package.message "checkout of %s failed, "\
                "deleting the source directory #{package.importdir} "\
                "and retrying (#{retry_count}/#{self.retry_count})"
            FileUtils.rm_rf package.importdir
            retry
        end
    end

    patch(package)
    package.updated = true
rescue Interrupt
    raise
rescue ::Exception # rubocop:disable Lint/ShadowedException
    package.message "checkout of %s failed, "\
        "deleting the source directory #{package.importdir}"
    FileUtils.rm_rf package.importdir
    raise
rescue Autobuild::Exception => e
    FileUtils.rm_rf package.importdir
    fallback(e, package, :import, package)
end

#perform_update(package, only_local = false) ⇒ Object 



328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
 
# File 'lib/autobuild/importer.rb', line 328

def perform_update(package, only_local = false)
    cur_patches    = currently_applied_patches(package)
    needed_patches = patches
    patch_changed = cur_patches.map(&:last) != needed_patches.map(&:last)
    patch(package, []) if patch_changed

    last_error = nil
    retry_count = 0
    package.progress_start "updating %s"
    begin
        begin
            did_update = update(package, only_local)
            execute_post_hooks(package, error: false)
        rescue ::Exception
            execute_post_hooks(package, error: true)
            raise
        end

        message = if did_update == false
                      Autobuild.color('already up-to-date', :green)
                  else
                      Autobuild.color('updated', :yellow)
                  end

        did_update
    rescue Interrupt
        message = Autobuild.color('interrupted', :red)
        if last_error
            raise last_error
        else
            raise
        end
    rescue ::Exception => e
        message = Autobuild.color('update failed', :red)
        last_error = e
        # If the package is patched, it might be that the update
        # failed because we needed to unpatch first. Try it out
        #
        # This assumes that importing data with conflict will
        # make the import fail, but not make the patch
        # un-appliable. Importers that do not follow this rule
        # will have to unpatch by themselves.
        cur_patches = currently_applied_patches(package)
        unless cur_patches.empty?
            package.progress_done
            package.message "update failed and some patches are applied, "\
                "removing all patches and retrying"
            begin
                patch(package, [])
                return perform_update(package, only_local)
            rescue Interrupt
                raise
            rescue ::Exception
                raise e
            end
        end

        retry_count = update_retry_count(e, retry_count)
        raise unless retry_count

        package.message "update failed in #{package.importdir}, "\
            "retrying (#{retry_count}/#{self.retry_count})"
        retry
    ensure
        package.progress_done "#{message} %s"
    end

    patch(package)
    package.updated = true
    did_update
rescue Autobuild::Exception => e
    fallback(e, package, :import, package)
end

#retry_count ⇒ Object

The number of times update / checkout should be retried before giving up. The default is 0 (do not retry)

Set either with #retry_count= or by setting the :retry_count option when constructing this importer



191
192
193
 
# File 'lib/autobuild/importer.rb', line 191

def retry_count
    @options[:retry_count] || 0
end

#retry_count=(count) ⇒ Object

Sets the number of times update / checkout should be retried before giving up. 0 (the default) disables retrying.

See also #retry_count



231
232
233
 
# File 'lib/autobuild/importer.rb', line 231

def retry_count=(count)
    @options[:retry_count] = Integer(count)
end

#save_patch_state(package, cur_patches) ⇒ Object 



604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
 
# File 'lib/autobuild/importer.rb', line 604

def save_patch_state(package, cur_patches)
    patch_dir = patchdir(package)
    FileUtils.mkdir_p patch_dir
    cur_patches = cur_patches.each_with_index.
        map do |(_path, level, content), idx|
            path = File.join(patch_dir, idx.to_s)
            File.open(path, 'w') do |patch_io|
                patch_io.write content
            end
            [path, level]
        end
    File.open(patchlist(package), 'w') do |f|
        patch_state = cur_patches.map do |path, level|
            path = Pathname.new(path).
                relative_path_from(Pathname.new(package.importdir)).to_s
            "#{path} #{level}"
        end
        f.write(patch_state.join("\n"))
    end
end

#supports_relocation? ⇒ Boolean

Returns:

  • (Boolean) 


625
626
627
 
# File 'lib/autobuild/importer.rb', line 625

def supports_relocation?
    false
end

#unapply(package, path, patch_level = 0) ⇒ Object 



532
533
534
 
# File 'lib/autobuild/importer.rb', line 532

def unapply(package, path, patch_level = 0)
    call_patch(package, true, path, patch_level)
end

#update_retry_count(original_error, retry_count) ⇒ Object 



263
264
265
266
267
268
 
# File 'lib/autobuild/importer.rb', line 263

def update_retry_count(original_error, retry_count)
    return if !original_error.respond_to?(:retry?) || !original_error.retry?

    retry_count += 1
    retry_count if retry_count <= self.retry_count
end

#vcs_fingerprint(package) ⇒ Object

basic fingerprint of the package and its dependencies



211
212
213
214
215
216
 
# File 'lib/autobuild/importer.rb', line 211

def vcs_fingerprint(package)
    # each importer type should implement its own
    Autoproj.warn "Fingerprint in #{package.name} has not been implemented "\
                  "for this type of packages, results should be discarded"
    nil
end