Method: BufferedTokenizer#initialize

Defined in:
lib/em/buftok.rb

#initialize(delimiter = $/) ⇒ BufferedTokenizer

New BufferedTokenizers will operate on lines delimited by a delimiter, which is by default the global input delimiter $/ (“\n”).

The input buffer is stored as an array. This is by far the most efficient approach given language constraints (in C a linked list would be a more appropriate data structure). Segments of input data are stored in a list which is only joined when a token is reached, substantially reducing the number of objects required for the operation.



15
16
17
18
19
20
 
# File 'lib/em/buftok.rb', line 15

def initialize(delimiter = $/)
  @delimiter = delimiter
  @input = []
  @tail = ''
  @trim = @delimiter.length - 1
end