Class: DevStructure::Blueprint

Inherits:
Hash
  • Object
show all
Includes:
DevStructure
Defined in:
lib/devstructure/blueprint.rb

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(name, hash) ⇒ Blueprint

Blueprints are usually created from a name and a hash, being gathered from the URL and response of a DevStructure API request.



18
19
20
21
22
23
24
 
# File 'lib/devstructure/blueprint.rb', line 18

def initialize(name, hash)
  super nil
  clear
  hash.each { |k, v| self[k.to_s] = v }
  @name = name
  self["name"] = name
end

Dynamic Method Handling

This class handles dynamic methods through the method_missing method

#method_missing(symbol, *args) ⇒ Object

This is the preferred method for reading the blueprint metadata but hash-like access will work just fine, you’ll just have to use strings.



42
43
44
 
# File 'lib/devstructure/blueprint.rb', line 42

def method_missing(symbol, *args)
  self[symbol.to_s]
end

Instance Attribute Details

#name ⇒ Object

Returns the value of attribute name.



26
27
28
 
# File 'lib/devstructure/blueprint.rb', line 26

def name
  @name
end

Instance Method Details

#-(other) ⇒ Object

Subtracting one blueprint from another is an important strategy used to keep superfluous packages managed by ‘apt` out of blueprints in most cases. Improvements in the 1.1.0 release reduced the need for this optimization but it doesn’t hurt.

It takes three passes through the package tree. The first two remove superfluous packages and the final one accounts for some special dependencies by adding them back.



73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
 
# File 'lib/devstructure/blueprint.rb', line 73

def -(other)
  b = Marshal.load(Marshal.dump(self))

  # The first pass removes all duplicate packages that are not
  # themselves managers.  The introduction of multiple-version
  # complicates this slightly.  For each package, each version
  # that appears in the other blueprint is removed from this
  # blueprint.  After that is finished, this blueprint is
  # normalized.  If no versions remain, the package is removed.
  # If only one version remains, it is converted to a string.
  other.walk(
    :package => lambda { |manager, command, package, version|
      return if b.packages[package]
      return unless b.packages[manager]
      versions = [b.packages[manager]["_packages"][package]].flatten
      versions.reject! { |v| v.nil? }
      [version].flatten.each { |v| versions.delete v }
      case versions.length
      when 0
        b.packages[manager]["_packages"].delete package
      when 1
        b.packages[manager]["_packages"][package] = versions.to_s
      else
        b.packages[manager]["_packages"][package] = versions
      end
    }
  )

  # The second pass removes managers that manage no packages, which
  # is a potential side effect of the first pass.
  b.walk(
    :package => lambda { |manager, command, package, version|
      return unless b.packages[package]
      p = b.packages[manager]["_packages"]
      b.packages.delete manager if 0 == p.length
    }
  )

  # The third pass adds back special dependencies like `ruby*-dev`.
  # It isn't apparent by the rules above that a manager like RubyGems
  # needs more than just itself to function.  In some sense, this
  # might be considered a missing dependency in the Debian archive
  # but in reality, it's only _likely_ that you need `ruby*-dev` to
  # use `rubygems*`.
  #
  # The only dependency that fits this bill currently is in fact
  # RubyGems, which gets `ruby*-dev` added for the matching version
  # of the Ruby language.
  b.walk(
    :after => lambda { |manager, command|
      case manager
      when /^rubygems(\d+\.\d+(?:\.\d+)?)$/
        b.packages["apt"]["_packages"]["ruby#{$1}-dev"] =
          packages["apt"]["_packages"]["ruby#{$1}-dev"]
      end
    }
  )

  b
end

#arch ⇒ Object

The architecture of the server used to create a blueprint will be “amd64” or “i386” if it’s set at all. When possible, the architecture is left unspecified to allow blueprints to be applied to any server.



49
50
51
 
# File 'lib/devstructure/blueprint.rb', line 49

def arch
  self["blueprint"]["_arch"]
end

#chef(io = StringIO.new) ⇒ Object

This is the Chef code generator. It is the first of its kind in creating a tarball rather than a single plaintext file. Because of this added output complexity, an ‘IO`-like object is accepted and returned after being used by the underlying `Chef::Cookbook`.



440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
 
# File 'lib/devstructure/blueprint.rb', line 440

def chef(io=StringIO.new)
  cookbook = Chef::Cookbook.new(@name, disclaimer)

  # Files are handled by the `cookbook_file` resource type and are
  # preceded by the `directory` which contains them.  Just like the
  # shell and Puppet generators, this code handles binary files and
  # symbolic links (handled by the `link` resource type), too.
  files.sort.each do |pathname, content|
    cookbook.directory File.dirname(pathname),
      :group => "root",
      :mode => "755",
      :owner => "root",
      :recursive => true
    owner = group = mode = nil
    if Hash === content
      owner = content["_owner"]
      group = content["_group"]
      if content["_target"]
        cookbook.link pathname,
          :group => group || "root",
          :owner => owner || "root",
          :to => content["_target"]
        next
      else
        mode = content["_mode"]
        if content["_base64"]
          content = content["_base64"]
        else
          content = content["_content"]
        end
      end
    end
    cookbook.cookbook_file pathname, content,
      :backup => false,
      :group => group || "root",
      :mode => mode || "644",
      :owner => owner || "root",
      :source => pathname[1..-1]
  end if files

  # Packages come after files so managers can be configured before they
  # run.  Packages are traversed in the same order as with the shell
  # code generator but passed off to the cookbook.
  walk(
    :before => lambda { |manager, command|
      cookbook.execute "apt-get -q update" if "apt" == manager
    },
    :package => lambda { |manager, command, package, version|
      return if manager == package
      command =~ /gem(\d+\.\d+(?:\.\d+)?) install/
      ruby_version = $1
      case manager

      # `apt` packages use the builtin `package` resource type with the
      # standard added resources to handle upgrading an old RubyGems
      # install.
      when "apt"
        cookbook.apt_package package, :version => version
        if package =~ /^rubygems(\d+\.\d+(?:\.\d+)?)$/ && rubygems_update?
          v = $1
          cookbook.gem_package "rubygems-update",
            :gem_binary => "/usr/bin/gem#{v}"
          cookbook.execute "/bin/sh -c '/usr/bin/ruby#{v
            } $(PATH=\"$PATH:/var/lib/gems/#{v
            }/bin\" which update_rubygems)'"
        end

      # Gems themselves, no matter what version of Ruby they're for,
      # can be managed by the builtin `package` resource type because
      # Opscode thoughtfully allows specifying the `gem` command
      # to run for each resource individually.
      when /ruby(?:gems)?(\d+\.\d+(?:\.\d+)?)(?:-dev)?/
        v = $1
        cookbook.gem_package package,
          :gem_binary => "/usr/bin/gem#{v}",
          :version => version

      # Because dependencies are handled by order, not declaration,
      # Python packages that need to come after `python-setuptools`
      # just do because `apt` is traversed first.  Everything else
      # gets an `execute` resource here, too..
      else
        cookbook.execute sprintf(command, package, version)

      end
    }
  )

  # Source tarballs are lifted directly from the shell code generator
  # and wrapped in `execute` resources.
  sources.sort.each do |dirname, filename|
    cookbook.execute \
      "wget http://s3.amazonaws.com/blueprint-sources/#{filename}"
    cookbook.execute "tar xf #{filename} -C #{dirname}"
    cookbook.execute "rm #{filename}"
  end if sources

  # With the list of resources specified, generate a tarball.  The
  # tarball will contain a minimal `metadata.rb`, a recipe, and any
  # files that should be included.  The tarball is written to the
  # `IO` object passed to this method.
  cookbook.to_gz(io)

end

#disclaimer ⇒ Object

Everyone loves a good disclaimer. This one’s chock full of useful links so folks can find their way back to DevStructure when the need strikes. Marketing!



137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
 
# File 'lib/devstructure/blueprint.rb', line 137

def disclaimer
  disclaimer = <<EOF
\#
\# #{owner}'s #{name}
\# https://devstructure.com/blueprints/#{owner}/#{name}
EOF
  if token
    disclaimer << <<EOF
\#
\# This blueprint is private.  You may share it by adding trusted users
\# or by sharing this secret URL:
\#   https://devstructure.com/blueprints/#{owner}/#{name}/#{token}
EOF
  end
  disclaimer << <<EOF
\#
\# This file was automatically generated by ruby-devstructure(7).
\#
EOF
  disclaimer
end

#files ⇒ Object

The files, packages, and sources hashes are completely described in [‘blueprint`(5)](devstructure.github.com/contractor/blueprint.5.html).



55
56
57
 
# File 'lib/devstructure/blueprint.rb', line 55

def files
  self["blueprint"]["_files"]
end

#lsb_release_codename ⇒ Object

Return this release’s codename.



599
600
601
602
603
 
# File 'lib/devstructure/blueprint.rb', line 599

def lsb_release_codename
  @lsb_release_codename if defined?(@lsb_release_codename)
  /\t(\w+)$/ =~ `/usr/bin/lsb_release -c`
  @lsb_release_codename = $1
end

#packages ⇒ Object 



58
59
60
 
# File 'lib/devstructure/blueprint.rb', line 58

def packages
  self["blueprint"]["_packages"]
end

#puppet(io = StringIO.new) ⇒ Object

This is the Puppet code generator.



240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
 
# File 'lib/devstructure/blueprint.rb', line 240

def puppet(io=StringIO.new)
  manifest = Puppet::Manifest.new(@name, nil, disclaimer)

  # Right out of the gate, we set a default `PATH` because Puppet does not.
  manifest << Puppet::Exec.defaults(:path => ENV["PATH"].split(":"))

  # We need a pre-built map of each manager to its own manager so we can
  # easily declare dependencies within the Puppet manifest tree.
  managers = {"apt" => nil}
  walk(:package => lambda { |manager, command, package, version|
    managers[package] = manager if packages[package] && manager != package
  })

  # System files must be placed before packages are installed so we set
  # Puppet's dependencies to put all packages ahead of all files.  This
  # is a change from the past when files were placed later.  The new
  # way allows `/etc/apt/sources.list.d/*` and `/root/.gemrc` (for
  # example) to be placed prior to running package manager commands.
  #
  # File content is handled much like the shell version except base 64
  # decoding takes place here in Ruby rather than in the `base64`(1)
  # tool.
  if files && 0 < files.length
    classes = managers.keys.
      reject { |manager| 0 == packages[manager]["_packages"].length }.
      map { |manager| Puppet::Class[manager] }
    if 0 < classes.length
      manifest << Puppet::File.defaults(:before => classes)
    end
    files.each do |pathname, content|

      # Resources for all parent directories are created ahead of
      # any file.  Puppet's autorequire mechanism will ensure that a
      # file's parent directories are realized before the file itself.
      dirnames = File.dirname(pathname).split("/")
      dirnames.shift
      (0..(dirnames.length - 1)).each do |i|
        manifest << Puppet::File.new("/#{dirnames[0..i].join("/")}", nil,
          :ensure => :directory)
      end

      # Convert the blueprint file attributes to those that Puppet
      # understands.  Everything's optional.
      options = {}
      if Hash === content
        options[:owner] = content["_owner"] if content["_owner"]
        options[:group] = content["_group"] if content["_group"]
        if content["_target"]
          options[:ensure] = content["_target"]
          content = nil
        else
          options[:mode] = content["_mode"] if content["_mode"]
          options[:ensure] = :file
          options[:content] = :"template(\"#{manifest.name}#{pathname}\")"
          content = if content["_base64"]
            Base64.decode64(content["_base64"])
          else
            content["_content"]
          end
        end

      # The easiest way to specify a file is to include just the
      # content.  The owner and group will be `root` and the mode will
      # be set according to the `umask`.
      else
        options = {
          :content => :"template(\"#{manifest.name}#{pathname}\")",
          :ensure => :file,
        }
      end

      manifest << Puppet::File.new(pathname, content, options)
    end
  end

  # Each manager's packages are grouped to create a series of subclasses
  # that allow dependency management to be handled coursely using Puppet
  # classes.  Because of Puppet's rules about resource name uniqueness,
  # we have to check that managers aren't managing themselves and bail
  # early.
  #
  # As always, RubyGems is getting ready to get bossy so we're prepared
  # by having the version of Ruby in question available.
  walk(
    :before => lambda { |manager, command|
      p = packages[manager]["_packages"]
      return if 0 == p.length || 1 == p.length && p[manager]
      if "apt" == manager
        manifest << Puppet::Exec.new("apt-get -q update",
          :before => Puppet::Class["apt"])
      end
    },
    :package => lambda { |manager, command, package, version|
      command =~ /gem(\d+\.\d+(?:\.\d+)?) install/
      ruby_version = $1
      case manager

      # `apt` packages are natively supported by Puppet so they're
      # very easy.
      when "apt"
        unless manager == package
          manifest[manager] << Puppet::Package.new(package,
            :ensure => version
          )
        end

        # However, if this package happens to be RubyGems, we introduce
        # a couple of new resources that update it to the latest version.
        if package =~ /^rubygems(\d+\.\d+(?:\.\d+)?)$/ && rubygems_update?
          manifest[manager] << Puppet::Package.new("rubygems-update",
            :require => Puppet::Package["rubygems1.8"],
            :provider => :gem,
            :ensure => :latest
          )
          manifest[manager] << Puppet::Exec.new(
            "/bin/sh -c '/usr/bin/ruby#{v} $(PATH=$PATH:/var/lib/gems/#{
              v}/bin which update_rubygems)'",
            :require => Puppet::Package["rubygems-update"]
          )
        end

      # RubyGems 1.8 is supported well enough by Puppet to include a
      # native provider, which means it is almost as simple as `apt`.
      when "rubygems1.8"
        options = {
          :require => [
            Puppet::Class[managers[manager]],
            Puppet::Package["rubygems1.8"],
          ],
          :provider => :gem,
          :ensure => version
        }
        manifest[manager] << Puppet::Package.new(package, options)

      # Other RubyGems versions use `exec` resources for installation
      # but follow a predictable enough directory layout that we can
      # avoid running the `gem` command after the first run with an
      # inexpensive check to see if the directory exists.
      when /ruby/
        manifest[manager] << Puppet::Exec.new(
          sprintf(command, package, version),
          :creates =>
            "#{rubygems_path}/#{ruby_version}/gems/#{package}-#{version}",
          :require => [
            Puppet::Class[managers[manager]],
            Puppet::Package[manager],
          ]
        )

      # Python packages follow a far less predictable directory naming
      # scheme so we aren't able to optimize them like RubyGems.  They
      # use `exec` resources and leave it to the Python tools (probably
      # `easy_install`).
      when /python/
        manifest[manager] << Puppet::Exec.new(sprintf(command, package),
          :require => [
            Puppet::Class[managers[manager]],
            Puppet::Package[manager, "python-setuptools"],
          ]
        )

      # It would be at this point where we would tackle Java, Erlang,
      # and every other possible package manager.  Slow and steady.

      # As a last resort, we execute the command exactly as the shell
      # code generator would.
      else
        manifest[manager] << Puppet::Exec.new(sprintf(command, package,
          version), :require => [
            Puppet::Class[managers[manager]],
            Puppet::Package[manager],
          ]
        )
      end
    }
  )

  # Source tarballs are downloaded, extracted, and removed in one fell
  # swoop.  Puppet 2.6 introduced the requirement to wrap compound commands
  # such as this in a shell invocation.  This is a pretty direct Puppet
  # equivalent to the shell version above.
  if sources && 0 < sources.length
    sources.each do |dirname, filename|
      manifest << Puppet::Exec.new(filename,
        :command => "/bin/sh -c 'wget http://s3.amazonaws.com/blueprint-sources/#{filename}; tar xf #{filename}; rm #{filename}'",
        :cwd => dirname
      )
    end
  end

  # Generate a module tarball containing the manifest and any templates
  # needed to compile a catalog that includes this module.
  manifest.to_gz(io)

end

#rubygems_path ⇒ Object

Platforms that need the rubygems-update gem install gems in a different location than typical Debian-based systems.



612
613
614
615
616
617
618
 
# File 'lib/devstructure/blueprint.rb', line 612

def rubygems_path
  if rubygems_update?
    "/usr/lib/ruby/gems"
  else
    "/var/lib/gems"
  end
end

#rubygems_update? ⇒ Boolean

Lucid and older need the rubygems-update gem.

Returns:

  • (Boolean) 


606
607
608
 
# File 'lib/devstructure/blueprint.rb', line 606

def rubygems_update?
  lsb_release_codename[0] < "m"[0]
end

#save(ds = nil) ⇒ Object

Attempt to save the blueprint, returning directly from the API. This is purely for convenience.



30
31
32
33
 
# File 'lib/devstructure/blueprint.rb', line 30

def save(ds=nil)
  ds ||= API.new
  ds.blueprints(owner).post(name, self)
end

#sh(io = StringIO.new) ⇒ Object

This is the POSIX shell code generator.



160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
 
# File 'lib/devstructure/blueprint.rb', line 160

def sh(io=StringIO.new)
  io.puts disclaimer

  # Plaintext files are written to their final destination using `cat`(1)
  # and heredoc syntax.  Binary files are written using `base64`(1) and
  # symbolic links are placed using `ln`(1).  Shocking stuff.
  files.sort.each do |pathname, content|
    io.puts "mkdir -p #{File.dirname(pathname)}"
    owner = group = mode = nil
    command = "cat"
    if Hash === content
      owner = content["_owner"]
      group = content["_group"]
      if content["_target"]
        io.puts "ln -s #{content["_target"]} #{pathname}"
        next
      else
        mode = content["_mode"]
        if content["_base64"]
          content = content["_base64"]
          command = "base64 --decode"
        else
          content = content["_content"]
        end
      end
    end
    content.gsub!(/\\/, "\\\\\\\\")
    content.gsub!(/\$/, "\\$")
    eof = "EOF"
    eof << "EOF" while content =~ /^#{eof}$/
    io.puts "#{command} >#{pathname} <<#{eof}"
    io.puts content
    io.puts eof
    io.puts "chown #{owner} #{pathname}" if owner
    io.puts "chgrp #{group} #{pathname}" if group
    io.puts "chmod #{mode} #{pathname}" if mode
  end if files

  # Packages used to come first.  Now they follow files so things like
  # `/etc/apt/sources.list.d/*` or `/root/.gemrc` can be in place before
  # package managers are called.  The algorithm used to walk the package
  # tree is pluggable and in this case we only need to take action as
  # each package name is encountered.  Each manager is annotated with a
  # shell command suitable for installing packages in `printf`(3)-style.
  #
  # RubyGems is, once again, a special case.  Ubuntu ships with a very
  # old version so we take the liberty of upgrading it anytime it is
  # encountered.
  walk(
    :before => lambda { |manager, command|
      io.puts "apt-get -q update" if "apt" == manager
    },
    :package => lambda { |manager, command, package, version|
      return if manager == package
      io.printf "#{command}\n", package, version
      if "apt" == manager \
        && package =~ /^rubygems(\d+\.\d+(?:\.\d+)?)$/ \
        && rubygems_update?
        io.puts \
          "/usr/bin/gem#{$1} install --no-rdoc --no-ri rubygems-update"
        io.puts "/usr/bin/ruby#{$1} $(PATH=\"$PATH:/var/lib/gems/#{
          $1}/bin\" which update_rubygems)"
      end
    }
  )

  # Source tarballs are downloaded, extracted, and removed.  Generally,
  # the directory in question is `/usr/local` but the future could hold
  # anything.
  sources.sort.each do |dirname, filename|
    io.puts "wget http://s3.amazonaws.com/blueprint-sources/#{filename}"
    io.puts "tar xf #{filename} -C #{dirname}"
    io.puts "rm #{filename}"
  end if sources

  io.close
  io
end

#sources ⇒ Object 



61
62
63
 
# File 'lib/devstructure/blueprint.rb', line 61

def sources
  self["blueprint"]["_sources"]
end

#walk(options = {}) ⇒ Object

Walk a package tree and execute callbacks along the way. Callbacks are listed in the options hash. These are supported:

  • ‘:before => lambda { |manager, command| }`: Executed before a manager’s dependencies are enumerated.

  • ‘:package => lambda { |manager, command, package, version| }`: Executed when a package is enumerated.

  • ‘:after => lambda { |manager, command| }`: Executed after a manager’s dependencies are enumerated.



554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
 
# File 'lib/devstructure/blueprint.rb', line 554

def walk(options={})

  # Start with packages installed with `apt`.
  options[:manager] ||= "apt"

  # Each manager gets its chance to take action before we loop over all
  # its managed packages.
  if options[:before].respond_to?(:call)
    options[:before].call options[:manager],
      packages[options[:manager]]["_command"]
  end

  # Each package gets its chance to take action.  While we're looping
  # over all the packages, we must note which ones are themselves
  # managers so we can recurse later.  We don't start the recursion
  # now because of the possibility that packages managed by this
  # just-encountered manager may also depend indirectly on packages
  # yet to be installed at this level.
  managers = []
  packages[options[:manager]]["_packages"].sort.each do |package, version|
    if options[:package].respond_to?(:call)
      [version].flatten.each do |v|
        options[:package].call options[:manager],
          packages[options[:manager]]["_command"], package, v
      end
    end
    if options[:manager] != package && packages[package]
      managers << package
    end
  end

  # Once more, each manager gets its chance to take action.
  if options[:after].respond_to?(:call)
    options[:after].call options[:manager],
      packages[options[:manager]]["_command"]
  end

  # Now we can recurse into each manager that was just installed.
  managers.each do |manager|
    walk options.merge(:manager => manager)
  end

end