Class: Puppet::Pops::Parser::Lexer::TokenList

Inherits:
Object
  • Object
show all
Extended by:
Forwardable
Defined in:
lib/puppet/pops/parser/lexer.rb

Overview

Maintains a list of tokens.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize ⇒ TokenList

Creates an empty token list



108
109
110
111
112
113
 
# File 'lib/puppet/pops/parser/lexer.rb', line 108

def initialize
  @tokens = {}
  @regex_tokens = []
  @string_tokens = []
  @tokens_by_string = {}
end

Instance Attribute Details

#regex_tokens ⇒ Object (readonly) 



81
82
83
 
# File 'lib/puppet/pops/parser/lexer.rb', line 81

def regex_tokens
  @regex_tokens
end

#string_tokens ⇒ Object (readonly) 



81
82
83
 
# File 'lib/puppet/pops/parser/lexer.rb', line 81

def string_tokens
  @string_tokens
end

Instance Method Details

#add_token(name, regex, options = {}, &block) ⇒ Object

Adds a new token to the set of recognized tokens

Parameters:

  • name (String) —

    the token name

  • regex (Regexp, String) —

    source text token matcher, a litral string or regular expression

  • options (Hash) (defaults to: {}) —

    see Token::set_options

  • block (Proc) —

    optional block set as the created tokens convert method

Raises:

  • (ArgumentError) —

    if the token with the given name is already defined



90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
 
# File 'lib/puppet/pops/parser/lexer.rb', line 90

def add_token(name, regex, options = {}, &block)
  raise(ArgumentError, "Token #{name} already exists") if @tokens.include?(name)
  token = Token.new(regex, name, options)
  @tokens[token.name] = token
  if token.string
    @string_tokens << token
    @tokens_by_string[token.string] = token
  else
    @regex_tokens << token
  end

  token.meta_def(:convert, &block) if block_given?

  token
end

#add_tokens(hash) ⇒ void

This method returns an undefined value.

Adds tokens from a hash where key is a matcher (literal string or regexp) and the value is the token’s name

Parameters:



126
127
128
129
130
 
# File 'lib/puppet/pops/parser/lexer.rb', line 126

def add_tokens(hash)
  hash.each do |regex, name|
    add_token(name, regex)
  end
end

#each ⇒ Object

Yield each token name and value in turn.



141
142
143
 
# File 'lib/puppet/pops/parser/lexer.rb', line 141

def each
  @tokens.each {|name, value| yield name, value }
end

#lookup(string) ⇒ Object

Look up a token by its literal (match) value, rather than name.

Parameters:



117
118
119
 
# File 'lib/puppet/pops/parser/lexer.rb', line 117

def lookup(string)
  @tokens_by_string[string]
end

#sort_tokens ⇒ void

This method returns an undefined value.

Sort literal (string-) tokens by length, so we know once we match, we’re done. This helps avoid the O(n^2) nature of token matching. The tokens are sorted in place.



136
137
138
 
# File 'lib/puppet/pops/parser/lexer.rb', line 136

def sort_tokens
  @string_tokens.sort! { |a, b| b.string.length <=> a.string.length }
end