Module: Lipsiadmin::Loops

Defined in:
lib/loops.rb,
lib/loops/base.rb,
lib/loops/worker.rb,
lib/loops/daemonize.rb,
lib/loops/worker_pool.rb,
lib/loops/process_manager.rb

Overview

Simple background loops framework for rails

Loops is a small and lightweight framework for Ruby on Rails created to support simple background loops in your application which are usually used to do some background data processing on your servers (queue workers, batch tasks processors, etc).

Authors

Alexey Kovyrin and Dmytro Shteflyuk

Comments

This plugin has been created in Scribd.com for internal use.

What tasks could you use it for?

Originally loops plugin was created to make our own loops code more organized. We used to have tens of different modules with methods that were called with script/runner and then used with nohup and other not so convenient backgrounding techniques. When you have such a number of loops/workers to run in background it becomes a nightmare to manage them on a regular basis (restarts, code upgrades, status/health checking, etc).

After a short time of writing our loops in more organized ways we were able to generalize most of the loops code so now our loops look like a classes with a single mandatory public method called run. Everything else (spawning many workers, managing them, logging, backgrounding, pid-files management, etc) is handled by the plugin it

But there are dozens of libraries like this! Why do we need one more?

The major idea behind this small project was to create a deadly simple and yet robust framework to be able to run some tasks in background and do not think about spawning many workers, restarting them when they die, etc. So, if you need to be able to run either one or many copies of your worker or you do not want to think about re-spawning dead workers and do not want to spend megabytes of RAM on separate copies of Ruby interpreter (when you run each copy of your loop as a separate process controlled by monit/god/etc), then I’d recommend you to try this framework – you’d like it.

How to use?

Generate binary and configuration files by running

script/generate loops

This will create the following list of files:

script/loops        # binary file that will be used to manage your loops
config/loops.yml    # example configuration file
app/loops/simple.rb # REALLY simple loop example

Here is a simple loop scaffold for you to start from (put this file to app/loops/hello_world_loop.rb):

class HelloWorldLoop < Lipsiadmin::Loops::Base
  def run
    debug("Hello, debug log!")
    sleep(config['sleep_period']) # Do something "useful" and make it configurable
    debug("Hello, debug log (yes, once again)!")
  end
end

When you have your loop ready to use, add the following lines to your (maybe empty yet) config/loops.yml file:

hello_world:
  sleep_period: 10

This is it! To start your loop, just run one of the following commands:

# Generates: list all configured loops:
$ script/loops -L

# Generates: run all enabled (actually non-disabled) loops in foreground:
$ script/loops -a

# Generates: run all enabled loops in background:
$ script/loops -d -a

# Generates: run specific loop in background:
$ ./script/loops -d -l hello_world

# Generates: all possible options:
$ ./script/loops -h

How to run more than one worker?

If you want to have more than one copy of your worker running, that is as simple as adding one option to your loop configuration:

hello_world:
  sleep_period: 10
  workers_number: 1

This workers_number option would say loops manager to spawn more than one copy of your loop and run them in parallel. The only thing you’d need to do is to think about concurrent work of your loops. For example, if you have some kind of database table with elements you need to process, you can create a simple database-based locks system or use any memcache-based locks.

There is this workers_engine option in config file. What it could be used for?

There are two so called “workers engines” in this plugin: fork and thread. They’re used to control the way process manager would spawn new loops workers: with fork engine we’ll load all loops classes and then fork ruby interpreter as many times as many workers we need. With thread engine we’d do Thread.new instead of forks. Thread engine could be useful if you are sure your loop won’t lock ruby interpreter (it does not do native calls, etc) or if you use some interpreter that does not support forks (like jruby).

Default engine is fork.

What Ruby implementations does it work for?

We’ve tested and used the plugin on MRI 1.8.6 and on JRuby 1.1.5. At this point we do not support demonization in JRuby and never tested the code on Ruby 1.9. Obviously because of JVM limitations you won’t be able to use fork workers engine in JRuby, but threaded workers do pretty well.

Defined Under Namespace

Modules: Daemonize Classes: Base, ProcessManager, Worker, WorkerPool

Class Method Summary collapse

Class Method Details

.config ⇒ Object

Set/Return the main config



122
123
124
 
# File 'lib/loops.rb', line 122

def config
  @@config
end

.global_config ⇒ Object

Set/Return the global config



132
133
134
 
# File 'lib/loops.rb', line 132

def global_config
  @@global_config
end

.load_config(file) ⇒ Object

Load the yml config file, default config/loops.yml



137
138
139
140
141
142
143
 
# File 'lib/loops.rb', line 137

def load_config(file)
  @@config = YAML.load_file(file)
  @@global_config = @@config['global']
  @@loops_config = @@config['loops']

  @@logger = create_logger('global', global_config)
end

.loops_config ⇒ Object

Set/Return the loops config



127
128
129
 
# File 'lib/loops.rb', line 127

def loops_config
  @@loops_config
end

.start_loops!(loops_to_start = :all) ⇒ Object

Start loops, default :all



146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
 
# File 'lib/loops.rb', line 146

def start_loops!(loops_to_start = :all)
  @@running_loops = []
  @@pm = Loops::ProcessManager.new(global_config, @@logger)

  # Start all loops
  loops_config.each do |name, config|
    next if config['disabled']
    next unless loops_to_start == :all || loops_to_start.member?(name)
    klass = load_loop_class(name)
    next unless klass

    start_loop(name, klass, config)
    @@running_loops << name
  end

  # Do not continue if there is nothing to run
  if @@running_loops.empty?
    puts "WARNING: No loops to run! Exiting..."
    return
  end

  # Start monitoring loop
  setup_signals
  @@pm.monitor_workers

  info "Loops are stopped now!"
end