Rigrate

Overview

This gem is used to migrate data between different data source in an easy way. SQLite3、MySQL、Oracle supported in current.

Installation

gem install rigrate

if you are using oracle, you need install instant oracle clent at least.

Basic Usage

An easy example, migrate entries hrdb users table to accounts table in oadb.

Firstly， Write the migrate script

# file : hr_users_to_oa_accounts.rigrate

# define the data sources, use URI pattern like  DB://USR:PWD@HOST/DBNAME
ds :hr, "oracle://scott:tiger@hrdb" 
ds :oa, "mysql://root:[email protected]:oadb"

# migrate snippet code
from hr.users to oa.accounts

Then, you can run as following:

$ rigrate execute -f hr_users_to_oa_accounts.rigrate

More Examples

Using native SQL. Just migrate DepNO is D00001 users to target data source.

from 
  hr.sql("select * from users where deptno = 'D00001'")
to
  oa.accounts

Specify the sync condition columns. When the migration has condition columns, it will update target data instead of delete the records directly.
```
from 
  hr.sql("select * from users")
to
  oa.accounts
on :user_code => :job_code
```

Just migrate a part of entry source columns. In this situation migration with one condition is advised.

from 
  hr.users(:user_code, :name, :age, :passwd)
to
  oa.account(:job_code, :name, :age, :password)
on :user_code => :job_code

Union two data source.

from 
  hr.users union oa.accounts 
to 
  erp.users

Minus two data source

from
  hr.users minus oa.accounts
to
  erp.users

Join two data source. JOIN must with condition.

from
  hr.users join oa.account on :user_code => :job_code
to
  erp.users

Data source select with after hooks

from
  hr.users(:user_code, :user_name, :password, :age) do |row|
    row[2] = "useless_password"  # make password useless
    row[3] = 16                  # make everyone 16 forever :-)
  end
to
  oa.accounts(:job_code, :name, :pwd, :age)
on :user_code => :job_code

Advance Tips

Sync Mode
Rigrate support support three mode, inspired by SyncToy a file sync tool by Microsoft.

:echo is the default mode, will delete all records of target ds which not exists in source ds
:contrbiute is same as :echo, but keep the records even it deleted in source ds
:sync mode will make two side of migration the same

use :contribute mode in migration:
```
from hr.users to oa.users mode :contribute
```
Using Transaction Rigrate will do not use transaction as default. Switch it on, you can do it througt command line --strict.
```
  $ rigrate -f script.rigrate --strict
```

Using Rigrate in other ruby script

require 'rigrate'

str ="ds :hr, \"oracle://scott:tiger@hrdb\" \nds :oa, \"mysql://root:[email protected]:oadb\"\n\nfrom hr.users to oa.accounts\n"

parser = Rigrate::Parser.new
parser.lex(str).parsing

QuickScript. a fast way to operate data, Rigrate will load a predefined data sources file in home/.rigrate/ds. you can define some frequently used data source, and run in migration throught command line. like below:
```
$ rigrate execute -c "from hr.users to oa.accounts"
```

for more usage to check Rigrate help

TODO Lists

Integrate with schedule framework
Add pg/sql sever/access/csv support
Row data type and length checking
Rigrate file checking
More logger details

Others

None :-)