Ruby json_stream_trigger Gem

Instead of parsing a huge JSON files and loading it into memory, this library will stream the bytes through json-stream and only creates a small buffer for objects whose JSONPath matches a pattern you specify. When the object is completed, the specified block will be called.

Install with gem "json_stream_trigger" in your Gemfile.


f ='really_big_file.json')
stream =

# Match each array item. If you wanted to whole array use $.data
stream.on('$.data[*]') do |json_string|
  import JSON.parse(json_string, :quirks_mode => true)

# Will match for $.any.sub[*].item.meta
stream.on('$..meta') do |json_string|
  save_meta JSON.parse(json_string, :quirks_mode => true)

# read in 1MB chunks
while chunk =
  stream << chunk

The captured JSON strinb buffer will be passed to the block. Note, Ruby's JSON library expects JSON documents to be passed to it - not primatives - this is why :quirks_mode => true has been added.

Path Details

The JSONPaths are similar to XPath notation. $ is the root, single wild card keys can be done with $.*.version, or you can do muli-level wildcard with $ More info on JSONPath

A few more examples:

  meta: {version: 0.1},
  docs: [
    {id: 1},
    {id: 2},
    {id: 3},
    {id: 4},
      id: 5,
      user: {
        name: "Tyler"
on('$.docs[*].id') # triggers for id property of every item in docs array
on('$.docs') # returns full array of items
on('$.docs[*]') # triggers for each item in the array
on('$.docs[1].id') # returns value of ID 1
on('$.docs[*].*.name') # returns 'Tyler'
on('$') # matches any value who's key is 'name'


rake test