es-reindex - simple ruby gem for copying ElasticSearch index
Simple ruby gem to copy and reindex ElasticSearch index, copying index settings and mapping(s).
Progress and time estimation is displayed during the scrolling process.
Requirements
- Ruby 1.9.3 or newer
- Gems:
Usage (command line)
Refer to script's help:
$ ./es-reindex.rb -h
Script to copy particular ES index including its (re)creation w/options set
and mapping copied.
Usage:
./es-reindex.rb [-r] [-f <frame>] [source_url/]<index> [destination_url/]<index>
- -r - remove the index in the new location first
- -f - specify frame size to be obtained with one fetch during scrolling
- -u - update existing documents (default: only create non-existing)
- optional source/destination urls default to http://127.0.0.1:9200
Usage (in project)
You can also use it as a PORO:
To Copy
# Options:
# remove: same as -r
# frame: same as -f
# update: same as -u cli option
= {
remove: true,
update: true
}
ESReindex.copy! 'http://my_server/index', 'http://my_server/index_copy',
To Reindex
If you want to reindex the destination from the source without copying the mappings/settings from the source, you can do it as such:
ESReindex.reindex! 'http://my_server/index', 'http://my_server/index_copy',
mappings: -> { set_of_mappings },
settings: -> { set_of_settings}
If using the .reindex!
method, you MUST pass valid mappings/settings in via the options.
Callbacks
There also a set of callbacks you can use:
ESReindex.copy! 'http://my_server/index', 'http://my_server/index_copy',
before_create: -> { do_something }, # Runs before the (re)creation of the destination index
after_create: -> { do_something_else }, # Runs after the (re)creation of the destinatino index
before_each: ->doc { use_the doc }, # Runs before each document is copied
after_each: ->doc { doc }, # Runs after each document is copied
after_copy: -> { finish_thing } # Runs after everything is copied over
Changelog
- 0.2.0: Lots of bugfixes, use elasticsearch client gem, add .reindex! method and callbacks
- 0.1.0: First gem release
- 0.0.9: Gemification, Oj -> MultiJSON
- 0.0.8: Optimization in string concat (@nara)
- 0.0.7: Document header arguments
_timestamp
and_ttl
are copied as well - 0.0.6: Document headers in bulks are now assembled and properly JSON dumped
- 0.0.5: Merge fix for trailing slash in urls (@ichinco), formatting cleanup
- 0.0.4: Force create only, update is optional (@pgaertig)
- 0.0.3: Yajl -> Oj
- 0.0.2: repated document count comparison
- 0.0.1: first revision
Credits
Original script:
- @geronime
- @pgaertig
Developed by Mojo Lingo.
License
es-reindex the gem is copyright (c)2014 Mojo Lingo, and released under the terms of the MIT license. See the LICENSE file for the gory details.
es-reindex is copyright (c)2012 Jiri Nemecek, and released under the terms of the MIT license. See the LICENSE file for the gory details.