Class: Oga::XPath::Lexer Private

Inherits:

Object

• Object
• Oga::XPath::Lexer

Defined in:

lib/oga/xpath/lexer.rb

Overview

This class is part of a private API. You should avoid using this class if possible, as it may be removed or be changed in the future.

Lexer for turning XPath expressions into a set of tokens. Tokens are returned as arrays with every array having two values:

1. The token type as a symbol
2. The token value or nil if there is no value

Basic usage of this lexer is as following:

lexer  = Oga::XPath::Lexer.new('//foo/bar')
tokens = lexer.lex

Alternatively you can stream tokens instead of returning them as a whole:

lexer = Oga::XPath::Lexer.new('//foo/bar')

lexer.advance do |type, value|

end

Unlike the XML lexer the XPath lexer does not support IO instances, it can only lex strings.

Thread Safety

This class keeps track of an internal state. As a result it's not safe to share a single instance between multiple threads. However, you're free to use separate instances per thread as there is no global (= class level) shared state.

Constant Summary

AXIS_MAPPING =

This constant is part of a private API. You should avoid using this constant if possible, as it may be removed or be changed in the future.

Maps certain XPath axes written in their short form to their long form equivalents.

Returns:

• (Hash)

{
  '@'  => 'attribute',
  '//' => 'descendant-or-self',
  '..' => 'parent',
  '.'  => 'self'
}

AXIS_EMIT_NODE =

This constant is part of a private API. You should avoid using this constant if possible, as it may be removed or be changed in the future.

Axes that require a separate node() call to be emitted.

Returns:

• (Array)

%w{descendant-or-self parent self}

AXIS_EMIT_EXTRA_SLASH =

This constant is part of a private API. You should avoid using this constant if possible, as it may be removed or be changed in the future.

Axes that require an extra T_SLASH token to be emitted.

Returns:

• (Array)

%w{descendant-or-self}

Class Attribute Summary

• ._xpath_lexer_eof_trans ⇒ Object private
• ._xpath_lexer_from_state_actions ⇒ Object private
• ._xpath_lexer_index_offsets ⇒ Object private
• ._xpath_lexer_indicies ⇒ Object private
• ._xpath_lexer_key_spans ⇒ Object private
• ._xpath_lexer_to_state_actions ⇒ Object private
• ._xpath_lexer_trans_actions ⇒ Object private
• ._xpath_lexer_trans_keys ⇒ Object private
• ._xpath_lexer_trans_targs ⇒ Object private
• .xpath_lexer_en_main ⇒ Object private
• .xpath_lexer_error ⇒ Object private
• .xpath_lexer_first_final ⇒ Object private
• .xpath_lexer_start ⇒ Object private

Instance Method Summary

• #advance(&block) ⇒ Object private

  Advances through the input and generates the corresponding tokens.

• #initialize(data) ⇒ Lexer constructor private

  A new instance of Lexer.

• #lex ⇒ Array private

  Gathers all the tokens for the input and returns them as an Array.

Constructor Details

#initialize(data) ⇒ Lexer

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns a new instance of Lexer.

Parameters:

• data (String) —

  The data to lex.

# File 'lib/oga/xpath/lexer.rb', line 1375

def initialize(data)
  @data = data
end

Class Attribute Details

._xpath_lexer_eof_trans ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

# File 'lib/oga/xpath/lexer.rb', line 1298

def _xpath_lexer_eof_trans
  @_xpath_lexer_eof_trans
end

._xpath_lexer_from_state_actions ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

# File 'lib/oga/xpath/lexer.rb', line 1273

def _xpath_lexer_from_state_actions
  @_xpath_lexer_from_state_actions
end

._xpath_lexer_index_offsets ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

# File 'lib/oga/xpath/lexer.rb', line 128

def _xpath_lexer_index_offsets
  @_xpath_lexer_index_offsets
end

._xpath_lexer_indicies ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

# File 'lib/oga/xpath/lexer.rb', line 153

def _xpath_lexer_indicies
  @_xpath_lexer_indicies
end

._xpath_lexer_key_spans ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

# File 'lib/oga/xpath/lexer.rb', line 103

def _xpath_lexer_key_spans
  @_xpath_lexer_key_spans
end

._xpath_lexer_to_state_actions ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

# File 'lib/oga/xpath/lexer.rb', line 1248

def _xpath_lexer_to_state_actions
  @_xpath_lexer_to_state_actions
end

._xpath_lexer_trans_actions ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

# File 'lib/oga/xpath/lexer.rb', line 1218

def _xpath_lexer_trans_actions
  @_xpath_lexer_trans_actions
end

._xpath_lexer_trans_keys ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

# File 'lib/oga/xpath/lexer.rb', line 43

def _xpath_lexer_trans_keys
  @_xpath_lexer_trans_keys
end

._xpath_lexer_trans_targs ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

# File 'lib/oga/xpath/lexer.rb', line 1188

def _xpath_lexer_trans_targs
  @_xpath_lexer_trans_targs
end

.xpath_lexer_en_main ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

# File 'lib/oga/xpath/lexer.rb', line 1336

def xpath_lexer_en_main
  @xpath_lexer_en_main
end

.xpath_lexer_error ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

# File 'lib/oga/xpath/lexer.rb', line 1331

def xpath_lexer_error
  @xpath_lexer_error
end

.xpath_lexer_first_final ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

# File 'lib/oga/xpath/lexer.rb', line 1327

def xpath_lexer_first_final
  @xpath_lexer_first_final
end

.xpath_lexer_start ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

# File 'lib/oga/xpath/lexer.rb', line 1323

def xpath_lexer_start
  @xpath_lexer_start
end

Instance Method Details

#advance(&block) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Advances through the input and generates the corresponding tokens. Each token is yielded to the supplied block.

Each token is an Array in the following format:

[TYPE, VALUE]

The type is a symbol, the value is either nil or a String.

This method stores the supplied block in @block and resets it after the lexer loop has finished.

See Also:

• Oga::XPath::Lexer.[[#add_token]

# File 'lib/oga/xpath/lexer.rb', line 1410

def advance(&block)
  @block = block

  data  = @data # saves ivar lookups while lexing.
  ts    = nil
  te    = nil
  stack = []
  top   = 0
  cs    = self.class.xpath_lexer_start
  act   = 0
  eof   = @data.bytesize
  p     = 0
  pe    = eof

  _xpath_lexer_eof_trans          = self.class.send(:_xpath_lexer_eof_trans)
  _xpath_lexer_from_state_actions = self.class.send(:_xpath_lexer_from_state_actions)
  _xpath_lexer_index_offsets      = self.class.send(:_xpath_lexer_index_offsets)
  _xpath_lexer_indicies           = self.class.send(:_xpath_lexer_indicies)
  _xpath_lexer_key_spans          = self.class.send(:_xpath_lexer_key_spans)
  _xpath_lexer_to_state_actions   = self.class.send(:_xpath_lexer_to_state_actions)
  _xpath_lexer_trans_actions      = self.class.send(:_xpath_lexer_trans_actions)
  _xpath_lexer_trans_keys         = self.class.send(:_xpath_lexer_trans_keys)
  _xpath_lexer_trans_targs        = self.class.send(:_xpath_lexer_trans_targs)

  
# line 1436 "lib/oga/xpath/lexer.rb"
begin
	testEof = false
	_slen, _trans, _keys, _inds, _acts, _nacts = nil
	_goto_level = 0
	_resume = 10
	_eof_trans = 15
	_again = 20
	_test_eof = 30
	_out = 40
	while true
	if _goto_level <= 0
	if p == pe
		_goto_level = _test_eof
		next
	end
	if cs == 0
		_goto_level = _out
		next
	end
	end
	if _goto_level <= _resume
	case _xpath_lexer_from_state_actions[cs] 
	when 11 then
# line 1 "NONE"
		begin
ts = p
		end
# line 1464 "lib/oga/xpath/lexer.rb"
	end
	_keys = cs << 1
	_inds = _xpath_lexer_index_offsets[cs]
	_slen = _xpath_lexer_key_spans[cs]
	_trans = if (   _slen > 0 && 
			_xpath_lexer_trans_keys[_keys] <= ( (data.getbyte(p) || 0)) && 
			( (data.getbyte(p) || 0)) <= _xpath_lexer_trans_keys[_keys + 1] 
) then
			_xpath_lexer_indicies[ _inds + ( (data.getbyte(p) || 0)) - _xpath_lexer_trans_keys[_keys] ] 
		 else 
			_xpath_lexer_indicies[ _inds + _slen ]
		 end
	end
	if _goto_level <= _eof_trans
	cs = _xpath_lexer_trans_targs[_trans]
	if _xpath_lexer_trans_actions[_trans] != 0
	case _xpath_lexer_trans_actions[_trans]
	when 19 then
# line 186 "lib/oga/xpath/lexer.rl"
		begin
 add_token(:T_SLASH) 		end
	when 34 then
# line 298 "lib/oga/xpath/lexer.rl"
		begin
 add_token(:T_ADD) 		end
	when 13 then
# line 1 "NONE"
		begin
te = p+1
		end
	when 12 then
# line 370 "lib/oga/xpath/lexer.rl"
		begin
te = p+1
		end
	when 9 then
# line 351 "lib/oga/xpath/lexer.rl"
		begin
te = p+1
 begin 
    emit(:T_TYPE_TEST, ts, te - 2)
   end
		end
	when 3 then
# line 364 "lib/oga/xpath/lexer.rl"
		begin
te = p+1
 begin 
    emit(:T_VAR, ts + 1, te)
   end
		end
	when 2 then
# line 240 "lib/oga/xpath/lexer.rl"
		begin
te = p+1
 begin 
    emit(:T_STRING, ts + 1, te - 1)
   end
		end
	when 8 then
# line 262 "lib/oga/xpath/lexer.rl"
		begin
te = p+1
 begin 
    emit(:T_AXIS, ts, te - 2)
   end
		end
	when 21 then
# line 274 "lib/oga/xpath/lexer.rl"
		begin
te = p+1
 begin 
    value = AXIS_MAPPING[slice_input(ts, te)]

    add_token(:T_AXIS, value)

    # Short axes that use node() as their default, implicit test. This is
    # added on lexer level to make it easier to handle these cases on
    # parser/evaluator level.
    if AXIS_EMIT_NODE.include?(value)
      add_token(:T_TYPE_TEST, 'node')

      if AXIS_EMIT_EXTRA_SLASH.include?(value) and te != eof
        add_token(:T_SLASH)
      end
    end
   end
		end
	when 16 then
# line 201 "lib/oga/xpath/lexer.rl"
		begin
te = p+1
 begin 
    emit(:T_IDENT, ts, te)
   end
		end
	when 24 then
# line 370 "lib/oga/xpath/lexer.rl"
		begin
te = p
p = p - 1;		end
	when 32 then
# line 364 "lib/oga/xpath/lexer.rl"
		begin
te = p
p = p - 1; begin 
    emit(:T_VAR, ts + 1, te)
   end
		end
	when 36 then
# line 215 "lib/oga/xpath/lexer.rl"
		begin
te = p
p = p - 1; begin 
    value = slice_input(ts, te).to_i

    add_token(:T_INT, value)
   end
		end
	when 37 then
# line 221 "lib/oga/xpath/lexer.rl"
		begin
te = p
p = p - 1; begin 
    value = slice_input(ts, te).to_f

    add_token(:T_FLOAT, value)
   end
		end
	when 38 then
# line 274 "lib/oga/xpath/lexer.rl"
		begin
te = p
p = p - 1; begin 
    value = AXIS_MAPPING[slice_input(ts, te)]

    add_token(:T_AXIS, value)

    # Short axes that use node() as their default, implicit test. This is
    # added on lexer level to make it easier to handle these cases on
    # parser/evaluator level.
    if AXIS_EMIT_NODE.include?(value)
      add_token(:T_TYPE_TEST, 'node')

      if AXIS_EMIT_EXTRA_SLASH.include?(value) and te != eof
        add_token(:T_SLASH)
      end
    end
   end
		end
	when 44 then
# line 201 "lib/oga/xpath/lexer.rl"
		begin
te = p
p = p - 1; begin 
    emit(:T_IDENT, ts, te)
   end
		end
	when 1 then
# line 370 "lib/oga/xpath/lexer.rl"
		begin
 begin p = ((te))-1; end
		end
	when 7 then
# line 201 "lib/oga/xpath/lexer.rl"
		begin
 begin p = ((te))-1; end
 begin 
    emit(:T_IDENT, ts, te)
   end
		end
	when 4 then
# line 1 "NONE"
		begin
	case act
	when 0 then
	begin	begin
		cs = 0
		_goto_level = _again
		next
	end
end
	when 6 then
	begin begin p = ((te))-1; end

    value = slice_input(ts, te).to_i

    add_token(:T_INT, value)
  end
	when 7 then
	begin begin p = ((te))-1; end

    value = slice_input(ts, te).to_f

    add_token(:T_FLOAT, value)
  end
	else
	begin begin p = ((te))-1; end
end
end 
			end
	when 14 then
# line 187 "lib/oga/xpath/lexer.rl"
		begin
 add_token(:T_LPAREN) 		end
# line 370 "lib/oga/xpath/lexer.rl"
		begin
te = p+1
		end
	when 15 then
# line 188 "lib/oga/xpath/lexer.rl"
		begin
 add_token(:T_RPAREN) 		end
# line 370 "lib/oga/xpath/lexer.rl"
		begin
te = p+1
		end
	when 18 then
# line 189 "lib/oga/xpath/lexer.rl"
		begin
 add_token(:T_COMMA) 		end
# line 370 "lib/oga/xpath/lexer.rl"
		begin
te = p+1
		end
	when 20 then
# line 190 "lib/oga/xpath/lexer.rl"
		begin
 add_token(:T_COLON) 		end
# line 370 "lib/oga/xpath/lexer.rl"
		begin
te = p+1
		end
	when 22 then
# line 191 "lib/oga/xpath/lexer.rl"
		begin
 add_token(:T_LBRACK) 		end
# line 370 "lib/oga/xpath/lexer.rl"
		begin
te = p+1
		end
	when 23 then
# line 192 "lib/oga/xpath/lexer.rl"
		begin
 add_token(:T_RBRACK) 		end
# line 370 "lib/oga/xpath/lexer.rl"
		begin
te = p+1
		end
	when 45 then
# line 297 "lib/oga/xpath/lexer.rl"
		begin
 add_token(:T_PIPE) 		end
# line 369 "lib/oga/xpath/lexer.rl"
		begin
te = p
p = p - 1;		end
	when 33 then
# line 298 "lib/oga/xpath/lexer.rl"
		begin
 add_token(:T_ADD) 		end
# line 369 "lib/oga/xpath/lexer.rl"
		begin
te = p
p = p - 1;		end
	when 41 then
# line 299 "lib/oga/xpath/lexer.rl"
		begin
 add_token(:T_EQ) 		end
# line 369 "lib/oga/xpath/lexer.rl"
		begin
te = p
p = p - 1;		end
	when 31 then
# line 300 "lib/oga/xpath/lexer.rl"
		begin
 add_token(:T_NEQ) 		end
# line 369 "lib/oga/xpath/lexer.rl"
		begin
te = p
p = p - 1;		end
	when 39 then
# line 301 "lib/oga/xpath/lexer.rl"
		begin
 add_token(:T_LT) 		end
# line 369 "lib/oga/xpath/lexer.rl"
		begin
te = p
p = p - 1;		end
	when 42 then
# line 302 "lib/oga/xpath/lexer.rl"
		begin
 add_token(:T_GT) 		end
# line 369 "lib/oga/xpath/lexer.rl"
		begin
te = p
p = p - 1;		end
	when 40 then
# line 303 "lib/oga/xpath/lexer.rl"
		begin
 add_token(:T_LTE) 		end
# line 369 "lib/oga/xpath/lexer.rl"
		begin
te = p
p = p - 1;		end
	when 43 then
# line 304 "lib/oga/xpath/lexer.rl"
		begin
 add_token(:T_GTE) 		end
# line 369 "lib/oga/xpath/lexer.rl"
		begin
te = p
p = p - 1;		end
	when 27 then
# line 314 "lib/oga/xpath/lexer.rl"
		begin
 add_token(:T_AND) 		end
# line 369 "lib/oga/xpath/lexer.rl"
		begin
te = p
p = p - 1;		end
	when 30 then
# line 315 "lib/oga/xpath/lexer.rl"
		begin
 add_token(:T_OR) 		end
# line 369 "lib/oga/xpath/lexer.rl"
		begin
te = p
p = p - 1;		end
	when 28 then
# line 316 "lib/oga/xpath/lexer.rl"
		begin
 add_token(:T_DIV) 		end
# line 369 "lib/oga/xpath/lexer.rl"
		begin
te = p
p = p - 1;		end
	when 29 then
# line 317 "lib/oga/xpath/lexer.rl"
		begin
 add_token(:T_MOD) 		end
# line 369 "lib/oga/xpath/lexer.rl"
		begin
te = p
p = p - 1;		end
	when 25 then
# line 318 "lib/oga/xpath/lexer.rl"
		begin
 add_token(:T_MUL) 		end
# line 369 "lib/oga/xpath/lexer.rl"
		begin
te = p
p = p - 1;		end
	when 26 then
# line 319 "lib/oga/xpath/lexer.rl"
		begin
 add_token(:T_SUB) 		end
# line 369 "lib/oga/xpath/lexer.rl"
		begin
te = p
p = p - 1;		end
	when 17 then
# line 1 "NONE"
		begin
te = p+1
		end
# line 369 "lib/oga/xpath/lexer.rl"
		begin
act = 1;		end
	when 5 then
# line 1 "NONE"
		begin
te = p+1
		end
# line 215 "lib/oga/xpath/lexer.rl"
		begin
act = 6;		end
	when 6 then
# line 1 "NONE"
		begin
te = p+1
		end
# line 221 "lib/oga/xpath/lexer.rl"
		begin
act = 7;		end
	when 35 then
# line 1 "NONE"
		begin
te = p+1
		end
# line 298 "lib/oga/xpath/lexer.rl"
		begin
 add_token(:T_ADD) 		end
# line 215 "lib/oga/xpath/lexer.rl"
		begin
act = 6;		end
# line 1861 "lib/oga/xpath/lexer.rb"
	end
	end
	end
	if _goto_level <= _again
	case _xpath_lexer_to_state_actions[cs] 
	when 10 then
# line 1 "NONE"
		begin
ts = nil;		end
# line 1 "NONE"
		begin
act = 0
		end
# line 1875 "lib/oga/xpath/lexer.rb"
	end

	if cs == 0
		_goto_level = _out
		next
	end
	p += 1
	if p != pe
		_goto_level = _resume
		next
	end
	end
	if _goto_level <= _test_eof
	if p == eof
	if _xpath_lexer_eof_trans[cs] > 0
		_trans = _xpath_lexer_eof_trans[cs] - 1;
		_goto_level = _eof_trans
		next;
	end
	end

	end
	if _goto_level <= _out
		break
	end
end
	end

# line 132 "lib/oga/xpath/lexer.rl"

  # % fix highlight
ensure
  @block = nil
end

#lex ⇒ Array

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Gathers all the tokens for the input and returns them as an Array.

Returns:

• (Array)

See Also:

• Oga::XPath::Lexer.[[#advance]

# File 'lib/oga/xpath/lexer.rb', line 1385

def lex
  tokens = []

  advance do |type, value|
    tokens << [type, value]
  end

  return tokens
end