Gem Version Tests (Ruby)

An API for streaming files from remote locations one line at a time.


Some applications run in production environments without a writable file system; usually this is a security measure. Futhermore, with the proliferation of container-based production environments, containers may not have access to tremendous amounts of memory. Thus, it can be impossible to read large files unless you read the file into memory in small doses. A common pattern is to use a line-delimited file like JSON Lines or a CSV and to read the file one line at a time in order to iterate over a dataset. This gem aims to provide an Enumerable interface for iterating over remote, line-delimited datasets.


Add this line to your application's Gemfile:

gem 'stream_lines'

And then execute:

$ bundle

Or install it yourself as:

$ gem install stream_lines



From a URL

url = 'https://my.remote.file/file.txt'
stream =

stream.each do |line|
  # Do something with the line of data (the line will be a String)

# A StreamLines::Reading::Stream object is Enumerable, so you can also use
# any Enumerable methods.

stream.each_slice(100) do |lines|
  # Do something with the 100 lines of data

This library strives to provide streamed data via an Enumerable interface. In order to be memory-efficient, however, each time the stream is iterated over, a new GET request is made to fetch the data from its remote URL. For example,

url = 'https://my.remote.file/file.txt'
stream =
do_something_with_first_row(stream.first) # GET request made

stream.each do |line| # same GET request made
  # Do something with the line of data (the line will be a String)

makes two GET requests. The call to first makes a GET request to fetch the first row of data. The subsequent call to each makes the same GET request. To avoid unnecessary requests, I recommend a slightly different approach, which may not be intuitive but does make only one network request:

url = 'https://my.remote.file/file.txt'
stream =

stream.each_with_index do |line, i|
  do_something_with_first_row(line) if
  # Do something with the line of data (the line will be a String)

This gem provides first-class support for streaming CSVs from a remote URL.

url = 'https://my.remote.file/file.csv'
stream =

stream.each do |row|
  # each row will be an Array

# Supports most Ruby CSV options (see ignored options below)
stream =, headers: true)

stream.each do |row|
  # each row will be a CSV::Row object that you can access like row['column_name']

Most options that you can pass to Ruby's CSV library are supported; however, the following options are explicitly ignored:

  • return_headers
  • header_converters
  • skip_lines

I suspect that these options are not used terribly frequently, and each would require additional logic in the StreamLines::Reading::CSV#each method. Rather than attempting to implement sensible solutions for these options, I am choosing to explicitly ignore them until there is enough outcry to support them.

JSON Lines/Streaming JSON

This gem provides first-class support for streaming JSON lines from a remote URL.

url = 'https://my.remote.file/file.jsonl'
stream =

stream.each do |row|
  # each row will be an Hash

# Supports all Ruby JSON::parse options
stream =, symbolize_names: true)

stream.each do |row|
  # each row will be a Hash


After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install.


After merging in the new functionality to the main branch:

git checkout main
git pull --prune
bundle exec rake version:bump:<major, minor, or patch>
bundle exec rubocop -a
git commit -a --amend
bundle exec rake release


Bug reports and pull requests are welcome on GitHub at This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the code of conduct.


The gem is available as open source under the terms of the MIT License.