waterslide
Unix-style pipes for Ruby programs
TL;DR
class FacebookFriendsController < ApplicationController
def index
render json:
FacebookFriends.new(current_user) >>
DeserializeFacebookUsers >>
MergeWithAttributesFromDatabase >>
RemoveRecordsNotInDatabase >>
Serialize
end
end
Wat
A common problem in programming is the need to perform a series of transformations on data. Ruby enumerables have lots of nice functional-style methods (map
, reduce
) to make this easy, but in some cases this approach breaks down.
Let's say you're building a social app where users log in with Facebook. You want to have a page where users can see which of their Facebook friends are using your site.
Doing this requires calling the Facebook API to get the current user's friends, merging the data from Facebook with the user data in your own database, filtering out the Facebook users who don't have accounts on your site, and serializing the remaining users to HTML or JSON so you can render them on your page.
The usual solution might look something like this:
Facebook.friends_of(current_user)
.map { |friend_datum| FacebookUserDeserializer.deserialize(friend_datum) }
.map { |user| user.merge_attributes User.find_by_facebook_id(user.facebook_id) }
.reject { |user| user.id.nil? }
.map { |user| UserSerializer.serialize(user) }
Later, you find that the page takes a long time to load for users with many facebook friends, and you isolate the problem to the O(n) User.find_by_facebook_id
calls, most of which don't actually find a user. Fortunately, that's not hard to fix.
facebook_friends = Facebook.friends_of(current_user).map do |friend_datum|
FacebookUserDeserializer.deserialize(friend_datum)
end
facebook_ids = facebook_friends.map(&:facebook_id)
facebook_friends_from_database = User.where(facebook_id: facebook_ids)
facebook_friends.map! do |friend|
db_record = facebook_friends_from_database.find do |db_record|
db_record.facebook_id == friend.facebook_id
end
friend.merge_attributes db_record
end
facebook_friends.map do |user|
UserSerializer.serialize(user)
end
In making the code more efficient, its elegance has been destroyed. Now one of the map
blocks is dependent on an invariant - the users pulled from the database - and that dependency makes the code harder to read and harder to refactor.
With Waterslide, the various data transformations can easily be broken into their own classes.
class FacebookFriends
include Waterslide::Pipe
def initialize(current_user)
@current_user = current_user
end
def each(&block)
@friends ||= Facebook.friends_of(@current_user)
@friends.each(&block)
end
end
class DeserializeFacebookUsers
include Waterslide::Pipe
def pipe_one(user_json)
yield User.new#( ... )
end
end
class MergeWithAttributesFromDatabase
include Waterslide::Pipe
def pipe_one(user)
record = database_records.find do |record|
record.facebook_id == user.facebook_id
end
yield user.merge_attributes record
end
def database_records
@records ||= User.where(facebook_id: incoming.map(&:facebook_id)).to_a
end
end
class RemoveRecordsNotInDatabase
include Waterslide::Pipe
def pipe_one(record)
yield record if record.id
end
end
class Serialize
include Waterslide::Pipe
def as_json
map(&:as_json)
end
end
# ...
FacebookFriends.new(current_user) >>
DeserializeFacebookUsers >>
MergeWithAttributesFromDatabase >>
RemoveRecordsNotInDatabase >>
SerializeUsers
There's obviously a lot more lines of code in the new version, but the sequence of transformations reads naturally, and, perhaps more importantly, every step can now be unit-tested individually.
Installation
Add this line to your application's Gemfile:
gem 'waterslide'
And then execute:
$ bundle
Or install it yourself as:
$ gem install waterslide
Usage
TODO: Write usage instructions here
Serving Suggestions
If you like syntactic sugar on your cerealizables, you may want to monkey-patch Array with the Waterslide right-shift operator override. That will let you do stuff like this:
[1, 2, 3] >> MultiplyByTwo # => [2, 4, 6]
Here's how to do the monkey-patch:
class Array
include Waterslide::RightShiftOverride
end
You should probably only do this if everyone on your team is on board with Waterslide and knows how to use it; otherwise, they'll have a hell of time deciphering your code.
Contributing
- Fork it ( https://github.com/benchristel/waterslide/fork )
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Add some feature'
) - Push to the branch (
git push origin my-new-feature
) - Create a new Pull Request