Class: Turbine::Pipeline::Unique

Inherits:
Filter show all
Defined in:
lib/turbine/pipeline/unique.rb

Overview

A Pipeline segment which only emits values which it hasn’t emitted previously.

In order to determine if a value is a duplicate, Unique needs to keep a reference to each input it sees. For large result sets, you may prefer to sacrifice performance for reduced space complexity by passing a block; this used to reduce each input to a simpler value for storage and comparison:

pipeline.uniq { |value| value.hash }

See also: Array#uniq.

Instance Attribute Summary

Attributes inherited from Segment

#source

Instance Method Summary collapse

Methods inherited from Filter

#next

Methods included from Trace::Transparent

#trace

Methods inherited from Segment

#append, #each, #inspect, #next, #to_s, #trace, #tracing=

Constructor Details

#initialize(&block) ⇒ Unique

Public: Creates a new Unique segment.

block - An optional block which is used to “reduce” each value for

comparison with previously seen value.

Returns a Unique.



23
24
25
26
27
28
29
30
31
32
33
34
 
# File 'lib/turbine/pipeline/unique.rb', line 23

def initialize(&block)
  @seen = Set.new

  super do |value|
    key  = block ? block.call(value) : value
    seen = @seen.include?(key)

    @seen.add(key)

    not seen
  end
end

Instance Method Details

#rewind ⇒ Object

Public: Rewinds the segment so that iteration can happen from the first input again.

Returns nothing.



40
41
42
43
 
# File 'lib/turbine/pipeline/unique.rb', line 40

def rewind
  @seen.clear
  super
end