Querrel

Playing around with the idea of parallel queries to multiple databases with ActiveRecord, more info on my blog post.

Installation

Add this line to your application's Gemfile:

gem 'querrel'

And then execute:

$ bundle

Or install it yourself as:

$ gem install querrel

Usage

Basic

You can use with Querrel directly via it's top level namespace, or by creating an instance:

all_brands = Querrel.query(Brand.all, on: ['db1', 'db2', 'db3'])
# is the same as
all_brands = Querrel.new(['db1', 'db2', 'db3']).query(Brand.all)

Either of the above will give you an array of all the Brand objects from all the given databases. The records will be marked as readonly.

Advanced

query will yield a block with the passed in ActiveRecord::Relation, this allows you to do additional operations on the results before they are merged, for example you could use pluck:

all_brand_names = q.query(Brand.all) do |s|
  s.pluck(:name)
end

There is also a run method which instead of running a preprescribed scope and merging, will yield the ConnectedModelFactory class which allows you to wrap any ActiveRecord class so that you can query it in the thread, for instance:

require 'thread'
all_brands = Brand.all
b_s = Mutex.new
all_products = Product.all
p_s = Mutex.new

Querrel.run(on: dbs) do |q|
  b_s.synchronize { all_brands += q[Brand].all.to_a }
  p_s.synchronize { all_products += q[Product].all.to_a }
end

At the moment, any queries using run and the ConnectedModelFactory won't return objects of the class passed in, but an anonymous class instead. However, query can marshall the results back into the original class.

Databases

There are three ways in which you can instruct Querrel which databases to use:

1. Pass in an array of environments, e.g. Querrel.new([:customer1, :customer2])
2. Pass in an array of database names, e.g. Querrel.new(['dbs/customer1.sqlite3', 'dbs/customer2.sqlite3'], db_names: true)

3. Pass in a hash of named connection configurations, e.g.:

   Querrel.new({
     one: {
       adapter: "sqlite3",
       database: "test/dbs/test_db_1.sqlite3"
     },
     two: {
       adapter: "sqlite3",
       database: "test/dbs/test_db_2.sqlite3"
     }
   })
   

Configuration

By default Querrel will use a maximum of 20 threads, but you can adjust this using the :threads option:

q = Querrel.new(dbs, threads: 50)

Contributing

1. Fork it ( https://github.com/meritec/querrel/fork )
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request