Class: UV::AbstractTokenizer

Inherits:
Object
  • Object
show all
Defined in:
lib/uv-rays/abstract_tokenizer.rb

Overview

AbstractTokenizer is similar to BufferedTokernizer however should only be used when there is no delimiter to work with. It uses a callback based system for application level tokenization without the heavy lifting.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(options) ⇒ AbstractTokenizer

Returns a new instance of AbstractTokenizer.

Parameters:

  • options (Hash)

Raises:

  • (ArgumentError) 


13
14
15
16
17
18
19
20
21
22
23
 
# File 'lib/uv-rays/abstract_tokenizer.rb', line 13

def initialize(options)
    @callback  = options[:callback]
    @indicator  = options[:indicator]
    @size_limit = options[:size_limit]
    @verbose  = options[:verbose] if @size_limit

    raise ArgumentError, 'no indicator provided' unless @indicator
    raise ArgumentError, 'no callback provided' unless @callback

    @input = ''
end

Instance Attribute Details

#callback ⇒ Object

Returns the value of attribute callback.



10
11
12
 
# File 'lib/uv-rays/abstract_tokenizer.rb', line 10

def callback
  @callback
end

#indicator ⇒ Object

Returns the value of attribute indicator.



10
11
12
 
# File 'lib/uv-rays/abstract_tokenizer.rb', line 10

def indicator
  @indicator
end

#size_limit ⇒ Object

Returns the value of attribute size_limit.



10
11
12
 
# File 'lib/uv-rays/abstract_tokenizer.rb', line 10

def size_limit
  @size_limit
end

#verbose ⇒ Object

Returns the value of attribute verbose.



10
11
12
 
# File 'lib/uv-rays/abstract_tokenizer.rb', line 10

def verbose
  @verbose
end

Instance Method Details

#empty? ⇒ Boolean

Returns:

  • (Boolean) 


96
97
98
 
# File 'lib/uv-rays/abstract_tokenizer.rb', line 96

def empty?
    @input.empty?
end

#extract(data) ⇒ Object

Extract takes an arbitrary string of input data and returns an array of tokenized entities using a message start indicator

Examples:


tokenizer.extract(data).
    map { |entity| Decode(entity) }.each { ... }

Parameters:

  • data (String) 


34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
 
# File 'lib/uv-rays/abstract_tokenizer.rb', line 34

def extract(data)
    @input << data

    messages = @input.split(@indicator, -1)
    if messages.length > 1
        messages.shift      # the first item will always be junk
        last = messages.pop # the last item may require buffering

        entities = []
        messages.each do |msg|
            entities << msg if @callback.call(msg)
        end

        # Check if buffering is required
        result = @callback.call(last)
        if result
            # Check for multi-byte indicator edge case
            if result.is_a? Fixnum
                entities << last[0...result]
                @input = last[result..-1]
            else
                @input = ''
                entities << last
            end
        else
            # This will work with a regex
            index = messages.last.nil? ? 0 : @input[0...-last.length].rindex(messages.last) + messages.last.length
            indicator_val = @input[index...-last.length]
            @input = indicator_val + last
        end
    else
        @input = messages.pop
        entities = messages
    end

    # Check to see if the buffer has exceeded capacity, if we're imposing a limit
    if @size_limit && @input.size > @size_limit
        if @indicator.respond_to?(:length) # check for regex
            # save enough of the buffer that if one character of the indicator were
            # missing we would match on next extract (very much an edge case) and
            # best we can do with a full buffer.
            @input = @input[-(@indicator.length - 1)..-1]
        else
            @input = ''
        end
        raise 'input buffer exceeded limit' if @verbose
    end

    return entities
end

#flush ⇒ String

Flush the contents of the input buffer, i.e. return the input buffer even though a token has not yet been encountered.

Returns:

  • (String) 


89
90
91
92
93
 
# File 'lib/uv-rays/abstract_tokenizer.rb', line 89

def flush
    buffer = @input
    @input = ''
    buffer
end