Class: SrlRuby::Tokenizer

Inherits:
Object
  • Object
show all
Defined in:
lib/srl_ruby/tokenizer.rb

Overview

The tokenizer should recognize: Keywords: as, capture, letter Integer literals including single digit String literals (quote delimited) Single character literal Delimiters: parentheses '(' and ')' Separators: comma (optional)

Defined Under Namespace

Classes: ScanError

Constant Summary collapse

@@lexeme2name =

attr_reader(:column)

{
  '(' => 'LPAREN',
  ')' => 'RPAREN',
  ',' => 'COMMA'
}.freeze
@@keywords =

Here are all the SRL keywords (in uppercase)

%w[
  ALL
  ALREADY
  AND
  ANY
  ANYTHING
  AS
  AT
  BACKSLASH
  BEGIN
  BETWEEN
  BY
  CAPTURE
  CASE
  CHARACTER
  DIGIT
  END
  EXACTLY
  FOLLOWED
  FROM
  HAD
  IF
  INSENSITIVE
  LAZY
  LEAST
  LETTER
  LINE
  LITERALLY
  MORE
  MULTI
  MUST
  NEVER
  NEW
  NO
  NOT
  NUMBER
  OF
  ONCE
  ONE
  OPTIONAL
  OR
  STARTS
  TAB
  TIMES
  TO
  TWICE
  UNTIL
  UPPERCASE
  WHITESPACE
  WITH
].map { |x| [x, x] } .to_h

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(source) ⇒ Tokenizer

Returns a new instance of Tokenizer.



82
83
84
85
86
 
# File 'lib/srl_ruby/tokenizer.rb', line 82

def initialize(source)
  @scanner = StringScanner.new(source)
  @lineno = 1
  @line_start = 0
end

Instance Attribute Details

#line_start ⇒ Object (readonly)

Returns the value of attribute line_start.



18
19
20
 
# File 'lib/srl_ruby/tokenizer.rb', line 18

def line_start
  @line_start
end

#lineno ⇒ Object (readonly)

Returns the value of attribute lineno.



17
18
19
 
# File 'lib/srl_ruby/tokenizer.rb', line 17

def lineno
  @lineno
end

#scanner ⇒ Object (readonly)

Returns the value of attribute scanner.



16
17
18
 
# File 'lib/srl_ruby/tokenizer.rb', line 16

def scanner
  @scanner
end

Instance Method Details

#tokens ⇒ Object 



88
89
90
91
92
93
94
95
96
 
# File 'lib/srl_ruby/tokenizer.rb', line 88

def tokens()
  tok_sequence = []
  until @scanner.eos?
    token = _next_token
    tok_sequence << token unless token.nil?
  end

  return tok_sequence
end