ActiveRecord extension to kick the speed of allocating ActiveRecord object
How it works
It provides functionality to load ActiveRecord records with patched attribute related methods. This make AR objects as read-only but it makes up to 5 times less object allocations.
Each time when you retrieve objects via
.light_records it will create annonymous class to work with given set of attributes.
Extension Class ↓ Your AR Model ↓ ActiveRecord::Base
gem 'light_record', github: 'paxa/light_record'
records = User.limit(1_000_000).light_records records # => array of records. Very fast and very memory efficient
Idea is to skip all magic related to attributes and object initialization. This creates new class inherited from your model. That allows us to create only one extra object when we initialize new record.
Simply it become something like this:
class User_light_record < User def initialize(attributes) @attributes = attributes # hash of data "as is" from database library end def email @attributes[:email] end end
.light_records_each, it will utilize
stream: true feature from mysql2 client. So it will initialize objects one by one for every interation:
User.limit(1_000_000).light_records_each do |user| user.do_something end
This allow you to interate big amount of data without using
find_in_batches because with
light_records_each it will use very low memory. Or allow you to use
find_in_batches with bigger batch size
Still on a way, but I try to use in some project and it gives 3-5 times improvement, and 2-3 times less memory usage
Sometimes this can break functionality because it will override attribute methods and disable some of features in activerecord.
There is mechanism to override attribute methods created by LightRecord:
class User < ActiveRecord::Base # this module will be included in extending class when we use light_records and light_records_each module def sometihng end def success attributes[:success] == 1 end def success_time return attributes[:success_time] unless attributes[:success_time] @success_time ||= attributes[:success_time].in_time_zone(Time.zone) end def success_time=(val) @success_time = nil super(val) end end end
Note: when you use LightRecord instances it will break type casting
This gem supports MySQL and PostgreSQL