Searchkick

:rocket: Intelligent search made easy

Searchkick learns what your users are looking for. As more people search, it gets smarter and the results get better. It’s friendly for developers - and magical for your users.

Searchkick handles:

stemming - tomatoes matches tomato
special characters - jalapeno matches jalapeño
extra whitespace - dishwasher matches dish washer
misspellings - zuchini matches zucchini
custom synonyms - qtip matches cotton swab

Plus:

query like SQL - no need to learn a new query language
reindex without downtime

:tangerine: Battle-tested at Instacart

Get Started

Install Elasticsearch. For Homebrew, use:

brew install elasticsearch

Add this line to your application’s Gemfile:

gem "searchkick"

Add searchkick to models you want to search.

class Product < ActiveRecord::Base
  searchkick
end

Add data to the search index.

Product.reindex

And to query, use:

products = Product.search "2% Milk"
products.each do |product|
  puts product.name
end

Queries

Query like SQL

Product.search "2% Milk", where: {in_stock: true}, limit: 10, offset: 50

Search specific fields

fields: [:name, :brand]

Where

where: {
  expires_at: {gt: Time.now}, # lt, gte, lte also available
  orders_count: 1..10,        # equivalent to {gte: 1, lte: 10}
  aisle_id: [25, 30],         # in
  store_id: {not: 2},         # not
  aisle_id: {not: [25, 30]},  # not in
  or: [
    [{in_stock: true}, {backordered: true}]
  ]
}

Order

order: {_score: :desc} # most relevant first - default

Limit / offset

limit: 20, offset: 40

Boost by a field

boost: "orders_count" # give popular documents a little boost

Pagination

Plays nicely with kaminari and will_paginate.

# controller
@products = Product.search "milk", page: params[:page], per_page: 20

# view
<%= paginate @products %>

Partial Matches

By default, results must match all words in the query.

Product.search "fresh honey" # fresh AND honey

To change this, use:

Product.search "fresh honey", partial: true # fresh OR honey

Synonyms

class Product < ActiveRecord::Base
  searchkick synonyms: [["scallion", "green onion"], ["qtip", "cotton swab"]]
end

You must call Product.reindex after changing synonyms.

Indexing

Control what data is indexed with the search_data method.

class Product < ActiveRecord::Base
  def search_data
    as_json only: [:name, :active], include: {brand: {only: [:city]}}
    # or equivalently
    {
      name: name,
      active: active,
      brand: {
        city: brand.city
      }
    }
  end
end

Searchkick uses find_in_batches to import documents. To eager load associations, use the search_import scope.

class Product < ActiveRecord::Base
  scope :search_import, includes(:searches)
end

Keep Getting Better

Searchkick uses conversion data to learn what users are looking for. If a user searches for “ice cream” and adds Ben & Jerry’s Chunky Monkey to the cart (our conversion metric at Instacart), that item gets a little more weight for similar searches.

The first step is to define your conversion metric and start tracking conversions. The database works well for low volume, but feel free to use Redis or another datastore.

class Search < ActiveRecord::Base
  belongs_to :product
  # fields: id, query, searched_at, converted_at, product_id
end

Add conversions to the index.

class Product < ActiveRecord::Base
  has_many :searches

  def search_data
    {
      name: name,
      conversions: searches.group("query").count
    }
  end
end

Reindex and set up a cron job to add new conversions daily.

rake searchkick:reindex CLASS=Product

search = Product.search "2% Milk", facets: [:store_id, :aisle_id]
p search.facets

Advanced

Product.search "2% Milk", facets: {store_id: {where: {in_stock: true}}}

Deployment

Heroku

Choose an add-on: SearchBox, Bonsai, or Found.

# SearchBox
heroku addons:add searchbox:starter

# Bonsai
heroku addons:add bonsai

# Found
heroku addons:add foundelasticsearch

And create an initializer config/initializers/elasticsearch.rb with:

# SearchBox
ENV["ELASTICSEARCH_URL"] = ENV["SEARCHBOX_URL"]

# Bonsai
ENV["ELASTICSEARCH_URL"] = ENV["BONSAI_URL"]

# Found
ENV["ELASTICSEARCH_URL"] = ENV["FOUNDELASTICSEARCH_URL"]

Then deploy and reindex:

heroku run rake searchkick:reindex CLASS=Product

Reference

Reindex one record

product = Product.find 10
product.reindex

Use a different index name

class Product < ActiveRecord::Base
  searchkick index_name: "products_v2"
end

Eagar load associations

Product.search "milk", include: [:brand, :stores]

Do not load models

Product.search "milk", load: false

Migrating from Tire

Change search methods to tire.search and add index name in existing search calls

  Product.search "fruit"

should be replaced with

  Product.tire.search "fruit", index: "products"

Replace tire mapping w/ searchkick method

  class Product < ActiveRecord::Base
      searchkick
  end

Deploy and reindex

  rake searchkick:reindex CLASS=Product # or Product.reindex in the console

Once it finishes, replace search calls w/ searchkick calls

Elasticsearch Gotchas

Inconsistent Scores

Due to the distributed nature of Elasticsearch, you can get incorrect results when the number of documents in the index is low. You can read more about it here. To fix this, do:

class Product < ActiveRecord::Base
  searchkick settings: {number_of_shards: 1}
end

Thanks

Thanks to Karel Minarik for Tire and Jaroslav Kalistsuk for zero downtime reindexing.

TODO

Custom results for each user
Make Searchkick work with any language
Built-in synonyms from WordNet

Contributing

Fork it
Create your feature branch (git checkout -b my-new-feature)
Commit your changes (git commit -am 'Add some feature')
Push to the branch (git push origin my-new-feature)
Create new Pull Request