Searchkick
:rocket: Intelligent search made easy
Searchkick learns what your users are looking for. As more people search, it gets smarter and the results get better. It’s friendly for developers - and magical for your users.
Searchkick handles:
- stemming -
tomatoes
matchestomato
- special characters -
jalapeno
matchesjalapeño
- extra whitespace -
dishwasher
matchesdish washer
- misspellings -
zuchini
matcheszucchini
- custom synonyms -
pop
matchessoda
Plus:
- query like SQL - no need to learn a new query language
- reindex without downtime
- easily personalize results for each user
- autocomplete
- “Did you mean” suggestions
- supports many languages
- works with ActiveRecord, Mongoid, and NoBrainer
Check out Searchjoy for analytics and Autosuggest for query suggestions
:tangerine: Battle-tested at Instacart
Contents
- Getting Started
- Querying
- Indexing
- Intelligent Search
- Instant Search / Autocomplete
- Aggregations
- Testing
- Deployment
- Performance
- Elasticsearch DSL
- Reference
- Contributing
Getting Started
Install Elasticsearch. For Homebrew, use:
sh
brew install elasticsearch
brew services start elasticsearch
Add this line to your application’s Gemfile:
ruby
gem 'searchkick'
The latest version works with Elasticsearch 6 and 7. For Elasticsearch 5, use version 3.1.3 and this readme.
Add searchkick to models you want to search.
ruby
class Product < ApplicationRecord
searchkick
end
Add data to the search index.
ruby
Product.reindex
And to query, use:
ruby
products = Product.search("apples")
products.each do |product|
puts product.name
end
Searchkick supports the complete Elasticsearch Search API. As your search becomes more advanced, we recommend you use the Elasticsearch DSL for maximum flexibility.
Querying
Query like SQL
ruby
Product.search "apples", where: {in_stock: true}, limit: 10, offset: 50
Search specific fields
ruby
fields: [:name, :brand]
Where
ruby
where: {
expires_at: {gt: Time.now}, # lt, gte, lte also available
orders_count: 1..10, # equivalent to {gte: 1, lte: 10}
aisle_id: [25, 30], # in
store_id: {not: 2}, # not
aisle_id: {not: [25, 30]}, # not in
user_ids: {all: [1, 3]}, # all elements in array
category: {like: "%frozen%"}, # like
category: /frozen .+/, # regexp
category: {prefix: "frozen"}, # prefix
store_id: {exists: true}, # exists
_or: [{in_stock: true}, {backordered: true}],
_and: [{in_stock: true}, {backordered: true}],
_not: {store_id: 1} # negate a condition
}
Order
ruby
order: {_score: :desc} # most relevant first - default
All of these sort options are supported
Limit / offset
ruby
limit: 20, offset: 40
Select
ruby
select: [:name]
These source filtering options are supported
Results
Searches return a Searchkick::Results
object. This responds like an array to most methods.
ruby
results = Product.search("milk")
results.size
results.any?
results.each { |result| ... }
By default, ids are fetched from Elasticsearch and records are fetched from your database. To fetch everything from Elasticsearch, use:
ruby
Product.search("apples", load: false)
Get total results
ruby
results.total_count
Get the time the search took (in milliseconds)
ruby
results.took
Get the full response from Elasticsearch
ruby
results.response
Note: By default, Elasticsearch limits paging to the first 10,000 results for performance. With Elasticsearch 7, this applies to the total count as well.
Boosting
Boost important fields
ruby
fields: ["title^10", "description"]
Boost by the value of a field (field must be numeric)
ruby
boost_by: [:orders_count] # give popular documents a little boost
boost_by: {orders_count: {factor: 10}} # default factor is 1
Boost matching documents
ruby
boost_where: {user_id: 1}
boost_where: {user_id: {value: 1, factor: 100}} # default factor is 1000
boost_where: {user_id: [{value: 1, factor: 100}, {value: 2, factor: 200}]}
Boost by recency
ruby
boost_by_recency: {created_at: {scale: "7d", decay: 0.5}}
You can also boost by:
Get Everything
Use a *
for the query.
ruby
Product.search "*"
Pagination
Plays nicely with kaminari and will_paginate.
ruby
# controller
@products = Product.search "milk", page: params[:page], per_page: 20
View with kaminari
erb
<%= paginate @products %>
View with will_paginate
erb
<%= will_paginate @products %>
Partial Matches
By default, results must match all words in the query.
ruby
Product.search "fresh honey" # fresh AND honey
To change this, use:
ruby
Product.search "fresh honey", operator: "or" # fresh OR honey
By default, results must match the entire word - back
will not match backpack
. You can change this behavior with:
ruby
class Product < ApplicationRecord
searchkick word_start: [:name]
end
And to search (after you reindex):
ruby
Product.search "back", fields: [:name], match: :word_start
Available options are:
Option | Matches | Example |
---|---|---|
:word |
entire word | apple matches apple |
:word_start |
start of word | app matches apple |
:word_middle |
any part of word | ppl matches apple |
:word_end |
end of word | ple matches apple |
:text_start |
start of text | gre matches green apple , app does not match |
:text_middle |
any part of text | een app matches green apple |
:text_end |
end of text | ple matches green apple , een does not match |
The default is :word
. The most matches will happen with :word_middle
.
Exact Matches
To match a field exactly (case-sensitive), use:
ruby
User.search query, fields: [{email: :exact}, :name]
Phrase Matches
To only match the exact order, use:
ruby
User.search "fresh honey", match: :phrase
Stemming and Language
Searchkick stems words by default for better matching. apple
and apples
both stem to appl
, so searches for either term will have the same matches.
Searchkick defaults to English for stemming. To change this, use:
ruby
class Product < ApplicationRecord
searchkick language: "german"
end
See the list of languages. A few languages require plugins:
chinese
- analysis-ik pluginchinese2
- analysis-smartcn pluginjapanese
- analysis-kuromoji pluginkorean
- analysis-openkoreantext pluginkorean2
- analysis-nori pluginpolish
- analysis-stempel pluginukrainian
- analysis-ukrainian pluginvietnamese
- analysis-vietnamese plugin
You can also use a Hunspell dictionary for stemming.
ruby
class Product < ApplicationRecord
searchkick stemmer: {type: "hunspell", locale: "en_US"}
end
Disable stemming with:
ruby
class Image < ApplicationRecord
searchkick stem: false
end
Exclude certain words from stemming with:
ruby
class Image < ApplicationRecord
searchkick stem_exclusion: ["apples"]
end
Or change how words are stemmed:
ruby
class Image < ApplicationRecord
searchkick stemmer_override: ["apples => other"]
end
Synonyms
ruby
class Product < ApplicationRecord
searchkick search_synonyms: [["pop", "soda"], ["burger", "hamburger"]]
end
Call Product.reindex
after changing synonyms. Synonyms are applied at search time before stemming, and can be a single word or multiple words.
For directional synonyms, use:
ruby
search_synonyms: ["lightbulb => halogenlamp"]
Dynamic Synonyms
The above approach works well when your synonym list is static, but in practice, this is often not the case. When you analyze search conversions, you often want to add new synonyms without a full reindex.
Elasticsearch 7.3+
For Elasticsearch 7.3+, we recommend placing synonyms in a file on the Elasticsearch server (in the config
directory). This allows you to reload synonyms without reindexing.
txt
pop, soda
burger, hamburger
Then use:
ruby
search_synonyms: "synonyms.txt"
Add elasticsearch-xpack to your Gemfile:
ruby
gem 'elasticsearch-xpack', '>= 7.8.0'
And use:
ruby
Product.search_index.reload_synonyms
Elasticsearch < 7.3
You can use a library like ActsAsTaggableOn and do:
```ruby class Product < ApplicationRecord acts_as_taggable scope :search_import, -> { includes(:tags) }
def search_data { name_tagged: “#name #“)” } end end ```
Search with:
ruby
Product.search query, fields: [:name_tagged]
Misspellings
By default, Searchkick handles misspelled queries by returning results with an edit distance of one.
You can change this with:
ruby
Product.search "zucini", misspellings: {edit_distance: 2} # zucchini
To prevent poor precision and improve performance for correctly spelled queries (which should be a majority for most applications), Searchkick can first perform a search without misspellings, and if there are too few results, perform another with them.
ruby
Product.search "zuchini", misspellings: {below: 5}
If there are fewer than 5 results, a 2nd search is performed with misspellings enabled. The result of this query is returned.
Turn off misspellings with:
ruby
Product.search "zuchini", misspellings: false # no zucchini
Specify which fields can include misspellings with:
ruby
Product.search "zucini", fields: [:name, :color], misspellings: {fields: [:name]}
When doing this, you must also specify fields to search
Bad Matches
If a user searches butter
, they may also get results for peanut butter
. To prevent this, use:
ruby
Product.search "butter", exclude: ["peanut butter"]
You can map queries and terms to exclude with:
```ruby exclude_queries = { “butter” => [“peanut butter”], “cream” => [“ice cream”, “whipped cream”] }
Product.search query, exclude: exclude_queries[query] ```
You can demote results by boosting by a factor less than one:
ruby
Product.search("butter", boost_where: {category: {value: "pantry", factor: 0.5}})
Emoji
Search :ice_cream::cake: and get ice cream cake
!
Add this line to your application’s Gemfile:
ruby
gem 'gemoji-parser'
And use:
ruby
Product.search "🍨🍰", emoji: true
Indexing
Control what data is indexed with the search_data
method. Call Product.reindex
after changing this method.
```ruby class Product < ApplicationRecord belongs_to :department
def search_data { name: name, department_name: department.name, on_sale: sale_price.present? } end end ```
Searchkick uses find_in_batches
to import documents. To eager load associations, use the search_import
scope.
ruby
class Product < ApplicationRecord
scope :search_import, -> { includes(:department) }
end
By default, all records are indexed. To control which records are indexed, use the should_index?
method together with the search_import
scope.
```ruby class Product < ApplicationRecord scope :search_import, -> { where(active: true) }
def should_index? active # only index active records end end ```
If a reindex is interrupted, you can resume it with:
ruby
Product.reindex(resume: true)
For large data sets, try parallel reindexing.
To Reindex, or Not to Reindex
Reindex
- when you install or upgrade searchkick
- change the
search_data
method - change the
searchkick
method
No need to reindex
- app starts
Strategies
There are four strategies for keeping the index synced with your database.
- Inline (default)
Anytime a record is inserted, updated, or deleted
- Asynchronous
Use background jobs for better performance
ruby
class Product < ApplicationRecord
searchkick callbacks: :async
end
Jobs are added to a queue named searchkick
.
- Queuing
Push ids of records that need updated to a queue and reindex in the background in batches. This is more performant than the asynchronous method, which updates records individually. See how to set up.
- Manual
Turn off automatic syncing
ruby
class Product < ApplicationRecord
searchkick callbacks: false
end
You can also do bulk updates.
ruby
Searchkick.callbacks(:bulk) do
User.find_each(&:update_fields)
end
Or temporarily skip updates.
ruby
Searchkick.callbacks(false) do
User.find_each(&:update_fields)
end
Associations
Data is not automatically synced when an association is updated. If this is desired, add a callback to reindex:
```ruby class Image < ApplicationRecord belongs_to :product
after_commit :reindex_product
def reindex_product product.reindex end end ```
Intelligent Search
The best starting point to improve your search by far is to track searches and conversions. Searchjoy makes it easy.
ruby
Product.search "apple", track: {user_id: current_user.id}
See the docs for how to install and use.
Focus on:
- top searches with low conversions
- top searches with no results
Searchkick can then use the conversion data to learn what users are looking for. If a user searches for “ice cream” and adds Ben & Jerry’s Chunky Monkey to the cart (our conversion metric at Instacart), that item gets a little more weight for similar searches.
Add conversion data with:
```ruby class Product < ApplicationRecord has_many :searches, class_name: “Searchjoy::Search”, as: :convertable
searchkick conversions: [:conversions] # name of field
def search_data { name: name, conversions: searches.group(:query).uniq.count(:user_id) # cream” => 234, “chocolate” => 67, “cream” => 2 } end end ```
Reindex and set up a cron job to add new conversions daily.
ruby
rake searchkick:reindex CLASS=Product
This can make a huge difference on the quality of your search.
For a more performant way to reindex conversion data, check out performant conversions.
Personalized Results
Order results differently for each user. For example, show a user’s previously purchased products before other results.
ruby
class Product < ApplicationRecord
def search_data
{
name: name,
orderer_ids: orders.pluck(:user_id) # boost this product for these users
}
end
end
Reindex and search with:
ruby
Product.search "milk", boost_where: {orderer_ids: current_user.id}
Instant Search / Autocomplete
Autocomplete predicts what a user will type, making the search experience faster and easier.
Note: To autocomplete on search terms rather than results, check out Autosuggest.
Note 2: If you only have a few thousand records, don’t use Searchkick for autocomplete. It’s much faster to load all records into JavaScript and autocomplete there (eliminates network requests).
First, specify which fields use this feature. This is necessary since autocomplete can increase the index size significantly, but don’t worry - this gives you blazing faster queries.
ruby
class Movie < ApplicationRecord
searchkick word_start: [:title, :director]
end
Reindex and search with:
ruby
Movie.search "jurassic pa", fields: [:title], match: :word_start
Typically, you want to use a JavaScript library like typeahead.js or jQuery UI.
Here’s how to make it work with Rails
First, add a route and controller action.
ruby
class MoviesController < ApplicationController
def autocomplete
render json: Movie.search(params[:query], {
fields: ["title^5", "director"],
match: :word_start,
limit: 10,
load: false,
misspellings: {below: 5}
}).map(&:title)
end
end
Note: Use load: false
and misspellings: {below: n}
(or misspellings: false
) for best performance.
Then add the search box and JavaScript code to a view.
```html
```
Suggestions
ruby
class Product < ApplicationRecord
searchkick suggest: [:name] # fields to generate suggestions
end
Reindex and search with:
ruby
products = Product.search "peantu butta", suggest: true
products.suggestions # ["peanut butter"]
Aggregations
Aggregations provide aggregated search data.
ruby
products = Product.search "chuck taylor", aggs: [:product_type, :gender, :brand]
products.aggs
By default, where
conditions apply to aggregations.
ruby
Product.search "wingtips", where: {color: "brandy"}, aggs: [:size]
# aggregations for brandy wingtips are returned
Change this with:
ruby
Product.search "wingtips", where: {color: "brandy"}, aggs: [:size], smart_aggs: false
# aggregations for all wingtips are returned
Set where
conditions for each aggregation separately with:
ruby
Product.search "wingtips", aggs: {size: {where: {color: "brandy"}}}
Limit
ruby
Product.search "apples", aggs: {store_id: {limit: 10}}
Order
ruby
Product.search "wingtips", aggs: {color: {order: {"_key" => "asc"}}} # alphabetically
All of these options are supported
Ranges
ruby
price_ranges = [{to: 20}, {from: 20, to: 50}, {from: 50}]
Product.search "*", aggs: {price: {ranges: price_ranges}}
Minimum document count
ruby
Product.search "apples", aggs: {store_id: {min_doc_count: 2}}
Script support
ruby
Product.search "*", aggs: {color: {script: {source: "'Color: ' + _value"}}}
Date histogram
ruby
Product.search "pear", aggs: {products_per_year: {date_histogram: {field: :created_at, interval: :year}}}
For other aggregation types, including sub-aggregations, use body_options
:
ruby
Product.search "orange", body_options: {aggs: {price: {histogram: {field: :price, interval: 10}}}}
Highlight
Specify which fields to index with highlighting.
ruby
class Product < ApplicationRecord
searchkick highlight: [:name]
end
Highlight the search query in the results.
ruby
bands = Band.search "cinema", highlight: true
View the highlighted fields with:
ruby
bands.with_highlights.each do |band, highlights|
highlights[:name] # "Two Door <em>Cinema</em> Club"
end
To change the tag, use:
ruby
Band.search "cinema", highlight: {tag: "<strong>"}
To highlight and search different fields, use:
ruby
Band.search "cinema", fields: [:name], highlight: {fields: [:description]}
By default, the entire field is highlighted. To get small snippets instead, use:
ruby
bands = Band.search "cinema", highlight: {fragment_size: 20}
bands.with_highlights(multiple: true).each do |band, highlights|
highlights[:name].join(" and ")
end
Additional options can be specified for each field:
ruby
Band.search "cinema", fields: [:name], highlight: {fields: {name: {fragment_size: 200}}}
You can find available highlight options in the Elasticsearch reference.
Similar Items
Find similar items.
ruby
product = Product.first
product.similar(fields: [:name], where: {size: "12 oz"})
Geospatial Searches
```ruby class Restaurant < ApplicationRecord searchkick locations: [:location]
def search_data attributes.merge(location: latitude, lon: longitude) end end ```
Reindex and search with:
ruby
Restaurant.search "pizza", where: {location: {near: {lat: 37, lon: -114}, within: "100mi"}} # or 160km
Bounded by a box
ruby
Restaurant.search "sushi", where: {location: {top_left: {lat: 38, lon: -123}, bottom_right: {lat: 37, lon: -122}}}
Note: top_right
and bottom_left
also work
Bounded by a polygon
ruby
Restaurant.search "dessert", where: {location: {geo_polygon: {points: [{lat: 38, lon: -123}, {lat: 39, lon: -123}, {lat: 37, lon: 122}]}}}
Boost By Distance
Boost results by distance - closer results are boosted more
ruby
Restaurant.search "noodles", boost_by_distance: {location: {origin: {lat: 37, lon: -122}}}
Also supports additional options
ruby
Restaurant.search "wings", boost_by_distance: {location: {origin: {lat: 37, lon: -122}, function: "linear", scale: "30mi", decay: 0.5}}
Geo Shapes
You can also index and search geo shapes.
```ruby class Restaurant < ApplicationRecord searchkick geo_shape: [:bounds]
def search_data attributes.merge( bounds: { type: “envelope”, coordinates: [4, lon: 1, 2, lon: 3] } ) end end ```
See the Elasticsearch documentation for details.
Find shapes intersecting with the query shape
ruby
Restaurant.search "soup", where: {bounds: {geo_shape: {type: "polygon", coordinates: [[{lat: 38, lon: -123}, ...]]}}}
Falling entirely within the query shape
ruby
Restaurant.search "salad", where: {bounds: {geo_shape: {type: "circle", relation: "within", coordinates: [{lat: 38, lon: -123}], radius: "1km"}}}
Not touching the query shape
ruby
Restaurant.search "burger", where: {bounds: {geo_shape: {type: "envelope", relation: "disjoint", coordinates: [{lat: 38, lon: -123}, {lat: 37, lon: -122}]}}}
Inheritance
Searchkick supports single table inheritance.
ruby
class Dog < Animal
end
In your parent model, set:
ruby
class Animal < ApplicationRecord
searchkick inheritance: true
end
The parent and child model can both reindex.
ruby
Animal.reindex
Dog.reindex # equivalent, all animals reindexed
And to search, use:
ruby
Animal.search "*" # all animals
Dog.search "*" # just dogs
Animal.search "*", type: [Dog, Cat] # just cats and dogs
Notes:
-
The
suggest
option retrieves suggestions from the parent at the moment.ruby Dog.search "airbudd", suggest: true # suggestions for all animals
-
This relies on a
type
field that is automatically added to the indexed document. Be wary of defining your owntype
field insearch_data
, as it will take precedence.
Debugging Queries
To help with debugging queries, you can use:
ruby
Product.search("soap", debug: true)
This prints useful info to stdout
.
See how Elasticsearch scores your queries with:
ruby
Product.search("soap", explain: true).response
See how Elasticsearch tokenizes your queries with:
```ruby Product.search_index.tokens(“Dish Washer Soap”, analyzer: “searchkick_index”) # [“dish”, “dishwash”, “washer”, “washersoap”, “soap”]
Product.search_index.tokens(“dishwasher soap”, analyzer: “searchkick_search”) # [“dishwashersoap”] - no match
Product.search_index.tokens(“dishwasher soap”, analyzer: “searchkick_search2”) # [“dishwash”, “soap”] - match!! ```
Partial matches
```ruby Product.search_index.tokens(“San Diego”, analyzer: “searchkick_word_start_index”) # [”s”, “sa”, “san”, “d”, “di”, “die”, “dieg”, “diego”]
Product.search_index.tokens(“dieg”, analyzer: “searchkick_word_search”) # [“dieg”] - match!! ```
See the complete list of analyzers.
Testing
As you iterate on your search, it’s a good idea to add tests.
For performance, only enable Searchkick callbacks for the tests that need it.
Parallel Tests
Rails 6 enables parallel tests by default. Add to your test/test_helper.rb
:
```ruby class ActiveSupport::TestCase parallelize_setup do |worker| Searchkick.index_suffix = worker
# reindex models
Product.reindex
# and disable callbacks
Searchkick.disable_callbacks end end ```
And use:
```ruby class ProductTest < ActiveSupport::TestCase def setup Searchkick.enable_callbacks end
def teardown Searchkick.disable_callbacks end
def test_search Product.create!(name: “Apple”) Product.search_index.refresh assert_equal [“Apple”], Product.search(“apple”).map(&:name) end end ```
Minitest
Add to your test/test_helper.rb
:
```ruby # reindex models Product.reindex
and disable callbacks
Searchkick.disable_callbacks ```
And use:
```ruby class ProductTest < Minitest::Test def setup Searchkick.enable_callbacks end
def teardown Searchkick.disable_callbacks end
def test_search Product.create!(name: “Apple”) Product.search_index.refresh assert_equal [“Apple”], Product.search(“apple”).map(&:name) end end ```
RSpec
Add to your spec/spec_helper.rb
:
```ruby RSpec.configure do |config| config.before(:suite) do # reindex models Product.reindex
# and disable callbacks
Searchkick.disable_callbacks end
config.around(:each, search: true) do |example| Searchkick.callbacks(nil) do example.run end end end ```
And use:
ruby
describe Product, search: true do
it "searches" do
Product.create!(name: "Apple")
Product.search_index.refresh
assert_equal ["Apple"], Product.search("apple").map(&:name)
end
end
Factory Bot
Use a trait and an after create
hook for each indexed model:
```ruby FactoryBot.define do factory :product do # …
# Note: This should be the last trait in the list so `reindex` is called
# after all the other callbacks complete.
trait :reindex do
after(:create) do |product, _evaluator|
product.reindex(refresh: true)
end
end end end
use it
FactoryBot.create(:product, :some_trait, :reindex, some_attribute: “foo”) ```
GitHub Actions
Check out setup-elasticsearch for an easy way to install Elasticsearch.
yml
- uses: ankane/setup-elasticsearch@v1
Deployment
Searchkick uses ENV["ELASTICSEARCH_URL"]
for the Elasticsearch server. This defaults to http://localhost:9200
.
Elastic Cloud
Create an initializer config/initializers/elasticsearch.rb
with:
ruby
ENV["ELASTICSEARCH_URL"] = "https://user:password@host:port"
Then deploy and reindex:
sh
rake searchkick:reindex:all
Heroku
Choose an add-on: Bonsai, SearchBox, or Elastic Cloud.
For Bonsai:
sh
heroku addons:create bonsai
heroku config:set ELASTICSEARCH_URL=`heroku config:get BONSAI_URL`
For SearchBox:
sh
heroku addons:create searchbox:starter
heroku config:set ELASTICSEARCH_URL=`heroku config:get SEARCHBOX_URL`
For Elastic Cloud (previously Found):
sh
heroku addons:create foundelasticsearch
heroku addons:open foundelasticsearch
Visit the Shield page and reset your password. You’ll need to add the username and password to your url. Get the existing url with:
sh
heroku config:get FOUNDELASTICSEARCH_URL
And add elastic:password@
right after https://
and add port 9243
at the end:
sh
heroku config:set ELASTICSEARCH_URL=https://elastic:[email protected]:9243
Then deploy and reindex:
sh
heroku run rake searchkick:reindex:all
Amazon Elasticsearch Service
Create an initializer config/initializers/elasticsearch.rb
with:
ruby
ENV["ELASTICSEARCH_URL"] = "https://es-domain-1234.us-east-1.es.amazonaws.com:443"
To use signed requests, include in your Gemfile:
ruby
gem 'faraday_middleware-aws-sigv4'
and add to your initializer:
ruby
Searchkick.aws_credentials = {
access_key_id: ENV["AWS_ACCESS_KEY_ID"],
secret_access_key: ENV["AWS_SECRET_ACCESS_KEY"],
region: "us-east-1"
}
Then deploy and reindex:
sh
rake searchkick:reindex:all
Self-Hosted and Other
Create an initializer config/initializers/elasticsearch.rb
with:
ruby
ENV["ELASTICSEARCH_URL"] = "https://user:password@host:port"
Then deploy and reindex:
sh
rake searchkick:reindex:all
Data Protection
We recommend encrypting data at rest and in transit (even inside your own network). This is especially important if you send personal data of your users to Elasticsearch.
Bonsai, Elastic Cloud, and Amazon Elasticsearch all support encryption at rest and HTTPS.
Automatic Failover
Create an initializer config/initializers/elasticsearch.rb
with multiple hosts:
ruby
ENV["ELASTICSEARCH_URL"] = "https://user:password@host1,https://user:password@host2"
See elasticsearch-transport for a complete list of options.
Lograge
Add the following to config/environments/production.rb
:
ruby
config.lograge.custom_options = lambda do |event|
options = {}
options[:search] = event.payload[:searchkick_runtime] if event.payload[:searchkick_runtime].to_f > 0
options
end
See Production Rails for other good practices.
Performance
JSON Generation
Significantly increase performance with faster JSON generation. Add Oj to your Gemfile.
ruby
gem 'oj'
This speeds up all JSON generation and parsing in your application (automatically!)
Persistent HTTP Connections
Significantly increase performance with persistent HTTP connections. Add Typhoeus to your Gemfile and it’ll automatically be used.
ruby
gem 'typhoeus'
To reduce log noise, create an initializer with:
ruby
Ethon.logger = Logger.new(nil)
If you run into issues on Windows, check out this post.
Searchable Fields
By default, all string fields are searchable (can be used in fields
option). Speed up indexing and reduce index size by only making some fields searchable.
ruby
class Product < ApplicationRecord
searchkick searchable: [:name]
end
Filterable Fields
By default, all string fields are filterable (can be used in where
option). Speed up indexing and reduce index size by only making some fields filterable.
ruby
class Product < ApplicationRecord
searchkick filterable: [:brand]
end
Note: Non-string fields are always filterable and should not be passed to this option.
Parallel Reindexing
For large data sets, you can use background jobs to parallelize reindexing.
ruby
Product.reindex(async: true)
# {index_name: "products_production_20170111210018065"}
Once the jobs complete, promote the new index with:
ruby
Product.search_index.promote(index_name)
You can optionally track the status with Redis:
ruby
Searchkick.redis = Redis.new
And use:
ruby
Searchkick.reindex_status(index_name)
You can also have Searchkick wait for reindexing to complete
ruby
Product.reindex(async: {wait: true})
You can use ActiveJob::TrafficControl to control concurrency. Install the gem:
ruby
gem 'activejob-traffic_control', '>= 0.1.3'
And create an initializer with:
```ruby ActiveJob::TrafficControl.client = Searchkick.redis
class Searchkick::BulkReindexJob concurrency 3 end ```
This will allow only 3 jobs to run at once.
Refresh Interval
You can specify a longer refresh interval while reindexing to increase performance.
ruby
Product.reindex(async: true, refresh_interval: "30s")
Note: This only makes a noticable difference with parallel reindexing.
When promoting, have it restored to the value in your mapping (defaults to 1s
).
ruby
Product.search_index.promote(index_name, update_refresh_interval: true)
Queuing
Push ids of records needing reindexing to a queue and reindex in bulk for better performance. First, set up Redis in an initializer. We recommend using connection_pool.
ruby
Searchkick.redis = ConnectionPool.new { Redis.new }
And ask your models to queue updates.
ruby
class Product < ApplicationRecord
searchkick callbacks: :queue
end
Then, set up a background job to run.
ruby
Searchkick::ProcessQueueJob.perform_later(class_name: "Product")
You can check the queue length with:
ruby
Product.search_index.reindex_queue.length
For more tips, check out Keeping Elasticsearch in Sync.
Routing
Searchkick supports Elasticsearch’s routing feature, which can significantly speed up searches.
```ruby class Business < ApplicationRecord searchkick routing: true
def search_routing city_id end end ```
Reindex and search with:
ruby
Business.search "ice cream", routing: params[:city_id]
Partial Reindexing
Reindex a subset of attributes to reduce time spent generating search data and cut down on network traffic.
```ruby class Product < ApplicationRecord def search_data { name: name }.merge(search_prices) end
def search_prices { price: price, sale_price: sale_price } end end ```
And use:
ruby
Product.reindex(:search_prices)
Performant Conversions
Split out conversions into a separate method so you can use partial reindexing, and cache conversions to prevent N+1 queries. Be sure to use a centralized cache store like Memcached or Redis.
```ruby class Product < ApplicationRecord def search_data { name: name }.merge(search_conversions) end
def search_conversions { conversions: Rails.cache.read(“search_conversions:#selfself.classself.class.name:#id”) || {} } end end ```
Create a job to update the cache and reindex records with new conversions.
```ruby class ReindexConversionsJob < ApplicationJob def perform(class_name) # get records that have a recent conversion recently_converted_ids = Searchjoy::Search.where(“convertable_type = ? AND converted_at > ?”, class_name, 1.day.ago) .order(:convertable_id).uniq.pluck(:convertable_id)
# split into groups
recently_converted_ids.in_groups_of(1000, false) do |ids|
# fetch conversions
conversions =
Searchjoy::Search.where(convertable_id: ids, convertable_type: class_name)
.group(:convertable_id, :query).uniq.count(:user_id)
# group conversions by record
conversions_by_record = {}
conversions.each do |(id, query), count|
(conversions_by_record[id] ||= {})[query] = count
end
# write to cache
conversions_by_record.each do |id, conversions|
Rails.cache.write("search_conversions:#{class_name}:#{id}", conversions)
end
# partial reindex
class_name.constantize.where(id: ids).reindex(:search_conversions)
end end end ```
Run the job with:
ruby
ReindexConversionsJob.perform_later("Product")
Advanced
Searchkick makes it easy to use the Elasticsearch DSL on its own.
Advanced Mapping
Create a custom mapping:
ruby
class Product < ApplicationRecord
searchkick mappings: {
properties: {
name: {type: "keyword"}
}
}
end
Note: If you use a custom mapping, you’ll need to use custom searching as well.
To keep the mappings and settings generated by Searchkick, use:
ruby
class Product < ApplicationRecord
searchkick merge_mappings: true, mappings: {...}
end
Advanced Search
And use the body
option to search:
ruby
products = Product.search body: {query: {match: {name: "milk"}}}
View the response with:
ruby
products.response
To modify the query generated by Searchkick, use:
ruby
products = Product.search "milk", body_options: {min_score: 1}
or
ruby
products =
Product.search "apples" do |body|
body[:min_score] = 1
end
Elasticsearch Gem
Searchkick is built on top of the elasticsearch gem. To access the client directly, use:
ruby
Searchkick.client
Multi Search
To batch search requests for performance, use:
ruby
products = Product.search("snacks", execute: false)
coupons = Coupon.search("snacks", execute: false)
Searchkick.multi_search([products, coupons])
Then use products
and coupons
as typical results.
Note: Errors are not raised as with single requests. Use the error
method on each query to check for errors.
Multiple Models
Search across multiple models with:
ruby
Searchkick.search "milk", models: [Product, Category]
Boost specific models with:
ruby
indices_boost: {Category => 2, Product => 1}
Multi-Tenancy
Check out this great post on the Apartment gem. Follow a similar pattern if you use another gem.
Scroll API
Searchkick also supports the scroll API. Scrolling is not intended for real time user requests, but rather for processing large amounts of data.
ruby
Product.search("*", scroll: "1m").scroll do |batch|
# process batch ...
end
You can also scroll batches manually.
```ruby products = Product.search “*”, scroll: “1m” while products.any? # process batch …
products = products.scroll end
products.clear_scroll ```
Deep Paging
By default, Elasticsearch limits paging to the first 10,000 results. Here’s why. We don’t recommend changing this, but if you really need all results, you can use:
ruby
class Product < ApplicationRecord
searchkick deep_paging: true
end
If you just need an accurate total count with Elasticsearch 7, you can instead use:
ruby
Product.search("pears", body_options: {track_total_hits: true})
Nested Data
To query nested data, use dot notation.
ruby
User.search "san", fields: ["address.city"], where: {"address.zip_code" => 12345}
Reference
Reindex one record
ruby
product = Product.find(1)
product.reindex
Reindex multiple records
ruby
Product.where(store_id: 1).reindex
Reindex associations
ruby
store.products.reindex
Remove old indices
ruby
Product.search_index.clean_indices
Use custom settings
ruby
class Product < ApplicationRecord
searchkick settings: {number_of_shards: 3}
end
Use a different index name
ruby
class Product < ApplicationRecord
searchkick index_name: "products_v2"
end
Use a dynamic index name
ruby
class Product < ApplicationRecord
searchkick index_name: -> { "#{name.tableize}-#{I18n.locale}" }
end
Prefix the index name
ruby
class Product < ApplicationRecord
searchkick index_prefix: "datakick"
end
For all models
ruby
Searchkick.index_prefix = "datakick"
Use a different term for boosting by conversions
ruby
Product.search("banana", conversions_term: "organic banana")
Multiple conversion fields
```ruby class Product < ApplicationRecord has_many :searches, class_name: “Searchjoy::Search”
# searchkick also supports multiple “conversions” fields searchkick conversions: [“unique_user_conversions”, “total_conversions”]
def search_data { name: name, unique_user_conversions: searches.group(:query).uniq.count(:user_id), # cream” => 234, “chocolate” => 67, “cream” => 2 total_conversions: searches.group(:query).count # cream” => 412, “chocolate” => 117, “cream” => 6 } end end ```
and during query time:
ruby
Product.search("banana") # boost by both fields (default)
Product.search("banana", conversions: "total_conversions") # only boost by total_conversions
Product.search("banana", conversions: false) # no conversion boosting
Change timeout
ruby
Searchkick.timeout = 15 # defaults to 10
Set a lower timeout for searches
ruby
Searchkick.search_timeout = 3
Change the search method name
ruby
Searchkick.search_method_name = :lookup
Change search queue name
ruby
Searchkick.queue_name = :search_reindex
Eager load associations
ruby
Product.search "milk", includes: [:brand, :stores]
Eager load different associations by model
ruby
Searchkick.search("*", models: [Product, Store], model_includes: {Product => [:store], Store => [:product]})
Run additional scopes on results
ruby
Product.search "milk", scope_results: ->(r) { r.with_attached_images }
Specify default fields to search
ruby
class Product < ApplicationRecord
searchkick default_fields: [:name]
end
Turn off special characters
ruby
class Product < ApplicationRecord
# A will not match Ä
searchkick special_characters: false
end
Turn on stemming for conversions
ruby
class Product < ApplicationRecord
searchkick stem_conversions: true
end
Make search case-sensitive
ruby
class Product < ApplicationRecord
searchkick case_sensitive: true
end
Note: If misspellings are enabled (default), results with a single character case difference will match. Turn off misspellings if this is not desired.
Change import batch size
ruby
class Product < ApplicationRecord
searchkick batch_size: 200 # defaults to 1000
end
Create index without importing
ruby
Product.reindex(import: false)
Use a different id
ruby
class Product < ApplicationRecord
def search_document_id
custom_id
end
end
Lazy searching
ruby
products = Product.search("carrots", execute: false)
products.each { ... } # search not executed until here
Add request parameters, like search_type
and query_cache
ruby
Product.search("carrots", request_params: {search_type: "dfs_query_then_fetch"})
Set options across all models
ruby
Searchkick.model_options = {
batch_size: 200
}
Reindex conditionally
```ruby class Product < ApplicationRecord searchkick callbacks: false
# add the callbacks manually after_commit :reindex, if: -> (model) { model.previous_changes.key?(“name”) } # use your own condition end ```
Reindex all models - Rails only
sh
rake searchkick:reindex:all
Turn on misspellings after a certain number of characters
ruby
Product.search "api", misspellings: {prefix_length: 2} # api, apt, no ahi
Note: With this option, if the query length is the same as prefix_length
, misspellings are turned off
ruby
Product.search "ah", misspellings: {prefix_length: 2} # ah, no aha
Elasticsearch 6 to 7 Upgrade
- Install Searchkick 4
- Upgrade your Elasticsearch cluster
Elasticsearch Gotchas
Consistency
Elasticsearch is eventually consistent, meaning it can take up to a second for a change to reflect in search. You can use the refresh
method to have it show up immediately.
ruby
product.save!
Product.search_index.refresh
Inconsistent Scores
Due to the distributed nature of Elasticsearch, you can get incorrect results when the number of documents in the index is low. You can read more about it here. To fix this, do:
ruby
class Product < ApplicationRecord
searchkick settings: {number_of_shards: 1}
end
For convenience, this is set by default in the test environment.
History
View the changelog.
Thanks
Thanks to Karel Minarik for Elasticsearch Ruby and Tire, Jaroslav Kalistsuk for zero downtime reindexing, and Alex Leschenko for Elasticsearch autocomplete.
Contributing
Everyone is encouraged to help improve this project. Here are a few ways you can help:
- Report bugs
- Fix bugs and submit pull requests
- Write, clarify, or fix documentation
- Suggest or add new features
To get started with development:
sh
git clone https://github.com/ankane/searchkick.git
cd searchkick
bundle install
bundle exec rake test
Feel free to open an issue to get feedback on your idea before spending too much time on it.