BenchmarkDriver Build Status

Fully-featured accurate benchmark driver for Ruby


Accurate Measurement

  • Low overhead benchmark by running generated script instead of calling Proc
  • Profiling memory and high-precision real time
  • Running multiple times to minimize measurement errors
  • Flexible and real-time output format in comparison, markdown table, graph, etc.
  • Measuring various metrics by specifying runners
  • Integrated benchmark support using external libraries
  • Runner and output format are all pluggable

Flexible Interface

  • Ruby interface similar to stdlib benchmark.rb, benchmark-ips
  • YAML input to easily manage structured benchmark set
  • Comparing multiple Ruby binaries, even with miniruby


$ gem install benchmark_driver


Ruby Interface

This interface generates code to profile with low overhead and executes it.

require 'benchmark_driver'

Benchmark.driver do |x|
  x.prelude <<~RUBY
    require 'active_support/all'
    array = []
  RUBY 'blank?', %{ array.blank? } 'empty?', %{ array.empty? }

or simply:

require 'benchmark_driver'

Benchmark.driver do |x|
  x.prelude <<~RUBY
    require 'active_support/all'
    array = []
  RUBY %{ array.blank? } %{ array.empty? }

Structured YAML Input

With benchmark-driver command, you can describe benchmark with YAML input.

$ benchmark-driver -h
Usage: benchmark-driver [options] [YAML|RUBY]
    -r, --runner [TYPE]              Specify runner type: ips, time, memory, once (default: ips)
    -o, --output [TYPE]              Specify output type: compare, simple, markdown, record (default: compare)
    -e, --executables [EXECS]        Ruby executables (e1::path1 arg1; e2::path2 arg2;...)
        --rbenv [VERSIONS]           Ruby executables in rbenv (x.x.x arg1;y.y.y arg2;...)
        --rvm [VERSIONS]             Ruby executables in rvm (x.x.x arg1;y.y.y arg2;...)
        --repeat-count [NUM]         Try benchmark NUM times and use the fastest result or the worst memory usage
        --repeat-result [TYPE]       Yield "best", "average" or "worst" result with --repeat-count (default: best)
        --bundler                    Install and use gems specified in Gemfile
        --filter [REGEXP]            Filter out benchmarks with given regexp
        --run-duration [SECONDS]     Warmup estimates loop_count to run for this duration (default: 3)
    -v, --verbose                    Verbose mode. Multiple -v options increase visilibity (max: 2)

Running single script

With following example_single.yml,

prelude: |
  require 'erb'
  erb =[Hello <%= 'World' %>])
benchmark: erb.result

you can benchmark the script with multiple ruby executables.

$ benchmark-driver example_single.yml --rbenv '2.4.1;2.5.0'
Warming up --------------------------------------
          erb.result    71.683k i/s
Calculating -------------------------------------
                          2.4.1       2.5.0
          erb.result    72.387k     75.046k i/s -    215.049k times in 2.970833s 2.865581s

               2.5.0:     75045.5 i/s
               2.4.1:     72386.8 i/s - 1.04x  slower

Running multiple scripts

One YAML file can contain multiple benchmark scripts. With following example_multi.yml,

prelude: |
  a = 'a' * 100
  b = 'b' * 100
  join: '[a, b].join'
  str-interp: '"#{a}#{b}"'

you can benchmark the scripts with multiple ruby executables.

$ benchmark-driver example_multi.yml --rbenv '2.4.1;2.5.0'
Warming up --------------------------------------
                join     2.509M i/s
          str-interp     1.772M i/s
Calculating -------------------------------------
                          2.4.1       2.5.0
                join     2.661M      2.863M i/s -      7.527M times in 2.828771s 2.629191s
          str-interp     1.890M      3.258M i/s -      5.315M times in 2.812240s 1.630997s

               2.5.0:   2862755.1 i/s
               2.4.1:   2660777.4 i/s - 1.08x  slower

               2.5.0:   3258489.7 i/s
               2.4.1:   1889805.6 i/s - 1.72x  slower

Output Options

By default, there are following output options.

Output Description
compare benchmark-ips's compare!-like output (default)
simple ruby's original benchmark/driver.rb-like simple output
markdown output in markdown table
record serialize results in benchmark_driver.record.yml, to change outputs later as you like

With benchmark-driver CLI, you can specify it with -o [output] or --output [output].

With Ruby interface, you can specify it like:

Benchmark.driver do |x|
  x.prelude %{ array = [] } 'Array#empty?', %{ array.empty? }
  x.output 'markdown'

With following blank_loop.yml,

loop_count: 20000000
prelude: |
  class Array
    alias_method :blank?, :empty?
  array = []
  empty: array.empty?
  blank: array.blank?

you can output results in various ways like:


$ benchmark-driver blank_loop.yml --output compare --rbenv '2.4.2;2.5.0'
Calculating -------------------------------------
                          2.4.2       2.5.0
               empty   195.957M    129.970M i/s -     20.000M times in 0.102063s 0.153882s
               blank    66.554M     55.630M i/s -     20.000M times in 0.300507s 0.359519s

               2.4.2: 195957398.5 i/s
               2.5.0: 129969716.9 i/s - 1.51x  slower

               2.4.2:  66554190.0 i/s
               2.5.0:  55629883.2 i/s - 1.20x  slower


$ benchmark-driver blank_loop.yml --output simple --rbenv '2.4.2;2.5.0'
benchmark results (i/s):
          2.4.2     2.5.0
empty  184.084M  117.319M
blank   65.843M   62.093M


$ benchmark-driver blank_loop.yml --output markdown --rbenv '2.4.2;2.5.0'

benchmark results (i/s)

2.4.2 2.5.0
empty 187.296M 117.662M
blank 58.895M 58.852M


Measure first, output with various formats later.

$ benchmark-driver blank_loop.yml --output record --rbenv '2.4.2;2.5.0'

$ benchmark-driver benchmark_driver.record.yml --output compare
Calculating -------------------------------------
               empty   153.380M    114.228M i/s -     20.000M times in 0.130395s 0.175088s
               blank    67.834M     64.328M i/s -     20.000M times in 0.294836s 0.310906s

               empty: 153380113.9 i/s
               blank:  67834321.4 i/s - 2.26x  slower

$ benchmark-driver benchmark_driver.record.yml --output simple
benchmark results (i/s):
empty  153.380M  114.228M
blank   67.834M   64.328M


There is benchmark_driver-output-gruff plugin that renders a graph using gruff.gem.

Benchmark results for are also stored by output plugin. You can integrate any system as you like.

Runner Options

Runner decides metrics to be collected. There are following default runner options.

ips, time, memory, once

Runner Description
ips Iteration per second (default)
time Elapsed seconds
memory Max resident set. This is supported only on Linux and macOS for now.
once Forces loop_count to 1 for testing
ruby_stdout Special runner to integrate existing benchmarks


$ benchmark-driver blank_loop.yml --runner ips --rbenv '2.4.3;2.5.0'
Calculating -------------------------------------
                          2.4.3       2.5.0
               empty   228.802M    180.125M i/s -     20.000M times in 0.087412s 0.111034s
               blank    90.012M     88.853M i/s -     20.000M times in 0.222193s 0.225090s

               2.4.3: 228801720.5 i/s
               2.5.0: 180124821.8 i/s - 1.27x  slower

               2.4.3:  90012021.7 i/s
               2.5.0:  88853269.4 i/s - 1.01x  slower


$ benchmark-driver blank_loop.yml --runner time --rbenv '2.4.3;2.5.0'
Calculating -------------------------------------
                          2.4.3       2.5.0
               empty      0.087       0.110 s -     20.000M times
               blank      0.217       0.219 s -     20.000M times

               2.5.0:         0.1 s
               2.4.3:         0.1 s - 1.26x  slower

               2.5.0:         0.2 s
               2.4.3:         0.2 s - 1.01x  slower


$ benchmark-driver blank_loop.yml --runner memory --rbenv '2.4.3;2.5.0'
Calculating -------------------------------------
                          2.4.3       2.5.0
               empty     9.192M      9.364M bytes -     20.000M times
               blank     9.080M      9.372M bytes -     20.000M times

               2.4.3:   9192000.0 bytes
               2.5.0:   9364000.0 bytes - 1.02x  larger

               2.4.3:   9080000.0 bytes
               2.5.0:   9372000.0 bytes - 1.03x  larger


Only for testing purpose.

$ benchmark-driver blank_loop.yml --runner once --rbenv '2.4.3;2.5.0'
Calculating -------------------------------------
                          2.4.3       2.5.0
               empty     1.818M      2.681M i/s -       1.000 times in 0.000001s 0.000000s
               blank     1.531M      2.421M i/s -       1.000 times in 0.000001s 0.000000s

               2.5.0:   2680965.1 i/s
               2.4.3:   1818181.8 i/s - 1.47x  slower

               2.5.0:   2421307.5 i/s
               2.4.3:   1531393.6 i/s - 1.58x  slower


See following example:

If your benchmark can run with ruby foo bar, specify foo bar to command:. Then write stdout_to_metrics: to convert stdout to metrics. This runner can be used only with YAML interface for now.

$ benchmark-driver benchmark.yml --verbose 1 --rbenv '2.6.0-dev;2.6.0-dev --jit'
2.6.0-dev: ruby 2.6.0dev (2018-03-21 trunk 62870) [x86_64-linux]
2.6.0-dev --jit: ruby 2.6.0dev (2018-03-21 trunk 62870) +JIT [x86_64-linux]
Calculating -------------------------------------
                      2.6.0-dev  2.6.0-dev --jit
           optcarrot     51.866           67.445 fps

     2.6.0-dev --jit:        67.4 fps
           2.6.0-dev:        51.9 fps - 1.30x  slower


Bug reports and pull requests are welcome on GitHub at


The gem is available as open source under the terms of the MIT License.