Method: Chewy::Search::Scrolling#scroll_batches

Defined in:
lib/chewy/search/scrolling.rb

#scroll_batches(batch_size: 1000, scroll: '1m') {|batch| ... } ⇒ Object #scroll_batches(batch_size: 1000, scroll: '1m') ⇒ Enumerator

Iterates through the documents of the scope in batches. Limit if overrided by the batch_size. There are 2 possible use-cases: with a block or without.

Overloads:

  • #scroll_batches(batch_size: 1000, scroll: '1m') {|batch| ... } ⇒ Object

    Examples:

    PlaceIndex.scroll_batches { |batch| batch.each { |hit| p hit['_id'] } }

    Yield Parameters:

    • batch (Array<Hash>) —

      block is executed for each batch of hits

  • #scroll_batches(batch_size: 1000, scroll: '1m') ⇒ Enumerator

    Returns a standard ruby Enumerator.

    Examples:

    PlaceIndex.scroll_batches.flat_map { |batch| batch.map { |hit| hit['_id'] } }

    Returns:

    • (Enumerator) —

      a standard ruby Enumerator

Parameters:

  • batch_size (Integer) (defaults to: Request::DEFAULT_BATCH_SIZE) —

    batch size obviously, replaces size query parameter

  • scroll (String) (defaults to: Request::DEFAULT_SCROLL) —

    cursor expiration time



27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
 
# File 'lib/chewy/search/scrolling.rb', line 27

def scroll_batches(batch_size: Request::DEFAULT_BATCH_SIZE, scroll: Request::DEFAULT_SCROLL)
  return enum_for(:scroll_batches, batch_size: batch_size, scroll: scroll) unless block_given?

  result = perform(size: batch_size, scroll: scroll)
  total = [raw_limit_value, result.fetch('hits', {}).fetch('total', {}).fetch('value', 0)].compact.min
  last_batch_size = total % batch_size
  fetched = 0
  scroll_id = nil

  loop do
    hits = result.fetch('hits', {}).fetch('hits', [])
    fetched += hits.size
    hits = hits.first(last_batch_size) if last_batch_size != 0 && fetched >= total
    yield(hits) if hits.present?
    scroll_id = result['_scroll_id']

    break if result['terminated_early'] || fetched >= total

    result = perform_scroll(scroll: scroll, scroll_id: scroll_id)
  end
ensure
  Chewy.client.clear_scroll(body: {scroll_id: scroll_id}) if scroll_id
end