MineShaft: Seed data from your stakeholders

Overview

MineShaft uses mechanize to walk through a Remine project site and parse content which can’t be easily accessed via the REST API (yet). The gem also provides a simple way to deserialize an HTML table into an Array of Hash objects which are key-value pairs of the table heading(s) and the corresponding value for each row in the table.

Motivation

We were looking for some domain-specific information to work with in an application and wanted to feed off of the knowledge of some of our stakeholders. We didn’t want to build out an admin interface which would be replaced with an automated system down the road and having our users edit text files and send them to us was not going to work. We created a Seed_Data wiki page and threw in a few tables with ID’s and wrote MineShaft to parse those tables and convert them into Hashes we could use to add/update content in the application database. Using this method allow(s) us to spread data entry & review requests for to a wide variety of people in an efficient manner.

Usage

Assuming you have a Redmine installation set up at http://your.rm-install.com and a project named twitter4biz set up.

Given a wiki page titled “Seed_Data” with the following table definition somewhere in that page (note: the table ID must be present):

table(#companies).
|Ticker|Name     |
|AAPL  |Apple    |
|MSFT  |Microsoft|
|GOOG  |Google   |
|YHOO  |Yahoo    |

Then in IRB (or whatever):

require 'mine_shaft'
include MineShaft
shaft = Shaft.new('rm-username', 'rm-password', 'http://your.rm-install.com')
companies = shaft.grab("companies", '/projects/twitter4biz/wiki/Seed_Data')
=> [{:name => 'Apple', :ticker => 'AAPL'}, {:name => 'Microsoft', :ticker => 'MSFT'},...]

So, in the db/seeds.rb file of your Rails app, you could put something like the following (assuming you have also included the previous code example):

companies.each do |attributes|
  company = Company.find_by_ticker(attributes[:ticker])
  if company.nil?
    Company.create!(attributes)
    puts "Added '#{attributes[:ticker]}'"
  else
    ticker = attributes.delete(:ticker)
    company.update(attributes)
    puts "Updated '#{ticker}'"
  end
end

…and then run:

rake db:seed

Installation

gem install mine_shaft

Contributing

  1. Fork it…
  2. bundle install
  3. …make awesomeness
  4. Commit (ideally) to a feature-branch
  5. Send a pull request

Found a bug?

File an issue on the project’s issues page

Dependencies

  • mechanize

License

Refer to LICENSE file (hint: MIT)

Future Plans

The gem is meeting our needs at the moment, so we don’t have any plans to add significant functionality at the moment. However, it has come in quite handy so far, so we may end up expanding it further if a new need arises.