Class: CodeRay::TokenStream

Inherits:
Tokens
  • Object
show all
Defined in:
lib/coderay/tokens.rb

Overview

TokenStream

The TokenStream class is a fake Array without elements.

It redirects the method << to a block given at creation.

This allows scanners and Encoders to use streaming (no tokens are saved, the input is highlighted the same time it is scanned) with the same code.

See CodeRay.encode_stream and CodeRay.scan_stream

Constant Summary

Constants inherited from Tokens

CodeRay::Tokens::ClassOfKind

Instance Attribute Summary collapse

Attributes inherited from Tokens

#scanner

Instance Method Summary collapse

Methods inherited from Tokens

#each, #each_text_token, #encode, #fix, #fix!, load, #method_missing, #optimize!, #split_into_lines, #split_into_lines!, #text, #to_s

Constructor Details

#initialize(&block) ⇒ TokenStream

Creates a new TokenStream that calls block whenever its << method is called.

Example:

require 'coderay'

token_stream = CodeRay::TokenStream.new do |text, kind|
  puts 'kind: %s, text size: %d.' % [kind, text.size]
end

token_stream << ['/\d+/', :regexp]
#-> kind: rexpexp, text size: 5.

Raises:

  • (ArgumentError)


313
314
315
316
317
# File 'lib/coderay/tokens.rb', line 313

def initialize &block
  raise ArgumentError, 'Block expected for streaming.' unless block
  @callback = block
  @size = 0
end

Dynamic Method Handling

This class handles dynamic methods through the method_missing method in the class CodeRay::Tokens

Instance Attribute Details

#sizeObject (readonly)

The Array is empty, but size counts the tokens given by <<.



297
298
299
# File 'lib/coderay/tokens.rb', line 297

def size
  @size
end

Instance Method Details

#<<(token) ⇒ Object

Calls block with token and increments size.

Returns self.



322
323
324
325
326
# File 'lib/coderay/tokens.rb', line 322

def << token
  @callback.call(*token)
  @size += 1
  self
end

#dumpObject

A TokenStream cannot be dumped. Use Tokens.

Raises:

  • (NotImplementedError)


335
336
337
# File 'lib/coderay/tokens.rb', line 335

def dump
  raise NotImplementedError, 'A TokenStream cannot be dumped.'
end

#optimizeObject

A TokenStream cannot be optimized. Use Tokens.

Raises:

  • (NotImplementedError)


340
341
342
# File 'lib/coderay/tokens.rb', line 340

def optimize
  raise NotImplementedError, 'A TokenStream cannot be optimized.'
end

#stream?Boolean

Whether the object is a TokenStream.

Returns true.

Returns:

  • (Boolean)


292
293
294
# File 'lib/coderay/tokens.rb', line 292

def stream?
  true
end

#text_sizeObject

This method is not implemented due to speed reasons. Use Tokens.

Raises:

  • (NotImplementedError)


329
330
331
332
# File 'lib/coderay/tokens.rb', line 329

def text_size
  raise NotImplementedError,
    'This method is not implemented due to speed reasons.'
end