StackedConfig

If you need to manage config files accross the system, some of them belonging to the administrator some to the user running the application, some coming from the command line and more, This Gem is made for you !

The purpose of this gem is to provide a simple way to handle the inheritance of config files for a ruby script. By default, it will handle already few config layers:

The system layer, which is a layer common to all applications using this gem.
The gem layer, which is the layer that will enable a gem to embed its own config. You may consider this level as the layer where you will set the default values for your properties.
The global layer, which is the layer to declare options for all users that use the ruby script using this gem.
The user layer, which is the layer, where a user can set options for the ruby script using this gem.
The extra layer, which provides the possibility to specify another config file from the config itself.
The enviroment variables layer, which provides the possibility to include in the config variables coming from the shell variables. See below for more info.
The command-line layer, which provides the ability to specify options from the command line.
The override layer, which will contain all modifications done to the config at run time.

The different layers are evaluated by default in that order using the super_stack gem (Please read for more detail). The StackedConfig::Orchestrator will expose a merged view of all its layers without modifying them.

All the config files are following the YAML syntax.

If you're looking for a complete solution for your command line scripts, including some logging features, then you are probably looking for the easy_app_helper Gem, which is itself internally relying on stacked_config.

Installation

Add this line to your application's Gemfile:

gem 'stacked_config'

And then execute:

$ bundle

Or install it yourself as:

$ gem install stacked_config

Usage

Basics

stacked_config is a very versatile configuration system for your scripts. The default behaviour may satisfy most of the needs out of the box.

require 'stacked_config'

config = StackedConfig::Orchestrator.new

config[:runtime_property] = 'A runtime property'

# Displays all the internals of the config:
puts config.detailed_layers_info

# Normally you may probably only need to access some properties
if config[:help]
    puts config.command_line_help
end

Try this little script and then create some config files to test how it is handled (see next section to know where to create the config files).

You are supposed to access the merged config through the [] method of the orchestrator (config[] in the previous example). Still you can directly modify any of the layers automatically created, you are not supposed to, and if you modify or add any new property (a so called "runtime property"), it will actually set the property in the override layer.

That's why you can always come back to the initial state by calling reset on the orchestrator. This actually just clears the override layer.

Every layer is accessible through the following orchestrator properties:

system_layer
gem_layer
global_layer
user_layer
provided_config_file_layer
env_layer
command_line_layer
write_layer

Where are my config files ?

stacked_config will look for config files in different places depending on the layer we are talking about. Have a look at the source to understand where exactly your config files can be, but basically it is following the Unix way of doing things...

Sources for the system layer
Sources for the gem layer
Sources for the global layer
Sources for the user layer

As you can see in the sources, paths are expressed using kind of 'templates', which meaning should be obvious

##SYSTEM_CONFIG_ROOT## is where the system config is stored. On Unix systems, it should be /etc.
##USER_CONFIG_ROOT## is where the user config is stored. On Unix systems, it should be your $HOME directory.
##GEM_CONFIG_ROOT## is the path to the "current" Gem root. The current gem being the one containing the currently executing script.
##PROGRAM_NAME## is by default the name of the script you are running (with no extension). You can if you want change this name at runtime. Changing it (using the config_file_base_name orchestrator property ) will trigger a re-search and reload of all the config files. config_file_base_name defines the base name to search for config files accross the system. by default the name of the gem will be used.
##EXTENSION## is one of the following extensions : conf CONF cfg CFG yml YML yaml YAML.

The search of the config files for a layer is done according to the order defined in sources just above and then extensions are tried according to the extensions just above in that exact order.

The first file matching for a particular level is used ! And there can be only one per level.

Thus according to the rules above, and assuming my script is named my_script.rb if the two following files exists at user config level, only the first is taken in account:

~/.my_script.yml
~/.config/my_script.conf

Environment variables

stacked_config can read information from your ENV variables thanks to the dedicated env_layer. This layer is not created by default, it is up to you to add it to your application, due to the fact that you will use it mostly with an optional filter and that you may want to add it with different priorities regarding your use case:

require 'stacked_config'

config = StackedConfig::Orchestrator.new
env_layer =  StackedConfig::Layers::EnvLayer.new(optional_filter)
config.add_layer(envLayer)
env_layer.name = 'Environment variables level'
env_layer.priority = 60

Or the exactly same using the convenience method of the orchestrator:

require 'stacked_config'

config = StackedConfig::Orchestrator.new
config.include_env_layer(optional_filter, optional_priority)

Which does not prevent you changing afterwards its priority using the config's env_layer accessor.

The constructor parameter optional_filter parameter aims at filtering the ENV variables:

if nil, all the ENV[] content will be added to the env_layer
it can be a single string if you want only one variable
it can be an array of accepted names of variables. ex: StackedConfig::Layers::EnvLayer.new 'VAR_NAME_1' StackedConfig::Layers::EnvLayer.new ['VAR_NAME_1', 'VAR_NAME_2']
it can be a regexp that variables names have to match. ex: StackedConfig::Layers::EnvLayer.new /somePattern/

Script command line options

stacked_config uses internally the fantastic Slop gem to manage options coming from the command line within the command line layer. This layer will be simply part of the complete config that the orchestrator exposes.

stacked_config is using the v3 branch of Slop and has not yet been adapted to the newest v4 branch.

Command line help

You can easily display a help using the orchestrator command_line_help method.

To even have a better command line help displayed you can provide optional information:

The application name through the app_name orchestrator property.
The application version through the app_version orchestrator property.
The application description through the app_description orchestrator property.

You could as well do this with the describes_application method (see complete example at the end of this page).

require 'stacked_config'

config = StackedConfig::Orchestrator.new

config.app_name = 'My super Application'
config.app_version = '1.0.0'
config.app_description = <<EOD
You can have a multiline description of your application.
This may help a lot having a consistent command-line help.
EOD
puts config.command_line_help

This would issue:

Usage: myscript [options]
My super Application Version: 1.0.0

You can have a multiline description of your application.
This may help a lot having a consistent command-line help.

-- Generic options -------------------------------------------------------------
        --auto                 Auto mode. Bypasses questions to user.
        --simulate             Do not perform the actual underlying actions.
    -v, --verbose              Enable verbose mode.
    -h, --help                 Displays this help.
-- Configuration options -------------------------------------------------------
        --config-file          Specify a config file.
        --config-override      If specified override all other config.

Which are the default options available.

Default command line options

By default the following command line options are available:

auto Auto mode. Bypasses questions to user. Just provided for convenience. Not used anywhere in the code.
simulate Do not perform the actual underlying actions. Just provided for convenience. Not used anywhere in the code.
verbose Enable verbose mode. Just provided for convenience. Not used anywhere in the code.
help Displays this help. Just provided for convenience. Not used anywhere in the code.
config-file Specify a config file for the extra layer.
config-override If specified, means that the config file specified will override all other config (actually changes the merge policy see below) of the extra layer.

Flags that are said "Just provided for convenience. Not used anywhere in the code." are just there because they are standard options and thus you can easily test in your code.

puts "Something very important" if config[:verbose]

stacked_config provides a convenient method to display the command line help every user expects from a decent script:

require 'stacked_config'

config = StackedConfig::Orchestrator.new

if config[:help]
    # command_line_help will provide a formatted help to display on the command line
    puts config.command_line_help
    exit 0
end

# ... do something else

Adding new command line options

To define your options the command-line layer exposes a slop_definition method that enables to directly configure slop.

For most usages you can use the higher level method add_command_line_section from the orchestrator.

require 'stacked_config'

config = StackedConfig::Orchestrator.new
config.add_command_line_section do |slop|
  slop.on :u, :useless, 'Stupid option', :argument => false
  slop.on :an_int, 'Stupid option with integer argument', :argument => true, :as => Integer
end

The add_command_line_section method supports a parameter to define the name of the section. Check Slop documentation for further information.

Of course adding new command line options will adapt the display of the command_line_help method of the orchestrator.

Advanced usage

stacked_config is internally relying on the super_stack Gem for the management of the layers, their priorities and the way they are merged in order to provide the "merged config". Check its documentation for further info.

Re-ordering layers

The way layers are processed is done according to their priority. By default the existing layers have the following priorities:

The system layer has a priority of 10
The gem layer has a priority of 20
The global layer has a priority of 30
The user layer has a priority of 40
The extra layer has a priority of 50
The command-line layer has a priority of 100
The override layer has a priority of 1000

But imagine you want to say that no-one could override properties defined at the system and global layer even from the command-line, then you just have to change the priorities of those 2 layers.

require 'stacked_config'

config = StackedConfig::Orchestrator.new
config.system_layer.priority = 1500
config.global_layer.priority = 1600

By doing such the system and global layers will be evaluated after the command line layer and therefore properties set in those files cannot be overridden even at command line level thanks to super_stack mechanisms.

Adding extra layers

Imagine you want to add a specific layer in your config, coming from let's say a web-service or a database, you may create your own layers for this purpose. Have a look at super_stack Gem for further info about how to create layers.

But basically just create your new layer, gives it a priority and add it to the orchestrator.

Changing the way things are merged

The super_stack gem defines various merge policies. By default stacked_config will use the SuperStack::MergePolicies::FullMergePolicy that merges hashes and arrays at all levels. But you can choose to completely change the merge behaviour by changing the merge policy. See super_stack gem documentation for other available merge policies.

Merge policies can be changed either at orchestrator level (globally) or at layer level (for the layer only) by setting the merge_policy property.

This is actually exactly what happens when the config-override flag is passed on the command line. It triggers the change of the merge policy of the extra layer from SuperStack::MergePolicies::FullMergePolicy to SuperStack::MergePolicies::OverridePolicy.

A complete example of a program using this Gem

Save the following file somewhere as example.rb.

You can use this as a template for your own scripts:

#!/usr/bin/env ruby

require 'stacked_config'

class MyApp

  VERSION = '0.0.1'
  NAME = 'My brand new Application'
  DESCRIPTION = 'Best app ever'

  attr_reader :config

  def initialize
    @config = StackedConfig::Orchestrator.new
    config.describes_application app_name: NAME, app_version: VERSION, app_description: DESCRIPTION
    add_script_options
  end


  def add_script_options
    config.add_command_line_section('Options for the script') do |slop|
      slop.on :u, :useless, 'Stupid option', :argument => false
      slop.on :an_int, 'Stupid option with integer argument', :argument => true, :as => Integer
    end
  end

  def run
    if config[:help]
      puts config.command_line_help
      exit 0
    end
    do_some_processing
  end

  def do_some_processing
    # Here you would really start your process
    config[:something] = 'Added something...'

    if config[:verbose]
      puts ' ## Here is a display of the config sources and contents'
      puts config.detailed_layers_info
      puts ' ## This the resulting merged config'
      puts config[].to_yaml
    end

    puts ' ## Bye...'
  end

end

MyApp.new.run

If you run:

$ ./example.rb

You would get:

 ## Bye...

If you run:

$ ./example.rb --help

You would get:

Usage: example [options]
My brand new Application Version: 0.0.1

Best app ever
-- Generic options -------------------------------------------------------------
        --auto                 Auto mode. Bypasses questions to user.
        --simulate             Do not perform the actual underlying actions.
    -v, --verbose              Enable verbose mode.
    -h, --help                 Displays this help.
-- Configuration options -------------------------------------------------------
        --config-file          Specify a config file.
        --config-override      If specified override all other config.
-- Options for the script ------------------------------------------------------
    -u, --useless              Stupid option
        --an_int               Stupid option with integer argument

If you run:

$ ./example.rb --verbose

You would get:

 ## Here is a display of the config sources and contents
--------------------------------------------------------------------------------
System-wide configuration level
There is no file attached to this level.
There is no data in this layer
--------------------------------------------------------------------------------
Gem configuration level
There is no file attached to this level.
There is no data in this layer
--------------------------------------------------------------------------------
Global configuration level
There is no file attached to this level.
There is no data in this layer
--------------------------------------------------------------------------------
User configuration level
There is no file attached to this level.
There is no data in this layer
--------------------------------------------------------------------------------
Specific config file configuration level
There is no file attached to this level.
There is no data in this layer
--------------------------------------------------------------------------------
Command line configuration level
There is no file attached to this level.
This layer contains the following data:
--- !ruby/hash:StackedConfig::Layers::CommandLineLayer
:verbose: true

--------------------------------------------------------------------------------
Overridden configuration level
There is no file attached to this level.
This layer contains the following data:
--- !ruby/hash:SuperStack::Layer
:something: Added something...

--------------------------------------------------------------------------------
 ## This the resulting merged config
---
:verbose: true
:something: Added something...
 ## Bye...

Contributing

Fork it, clone your fork.
Create your feature branch (git checkout -b my-new-feature) and develop your super extra feature.
Commit your changes (git commit -am 'Add some feature').
Push to the branch (git push origin my-new-feature).
Create a Pull Request.