Class: Loxxy::FrontEnd::Scanner

Inherits:
Object
  • Object
show all
Defined in:
lib/loxxy/front_end/scanner.rb

Overview

A scanner (tokenizer) for the Lox language. Reference material: https://craftinginterpreters.com/the-lox-language.html Section 4.2.1 Token types Appendix A1.2 Lexical Grammar Responsibility: break input into a sequence of token objects. The tokenizer should recognize: Identifiers, Number literals including single digit String literals (quote delimited) Delimiters: e.g. parentheses '(', ')' Separators: e.g. comma

Constant Summary collapse

@@lexeme2name =

One or two special character tokens. These are enumerated in section 4.2.1 Token type

{
  '(' => 'LEFT_PAREN',
  ')' => 'RIGHT_PAREN',
  '{' => 'LEFT_BRACE',
  '}' => 'RIGHT_BRACE',
  ',' => 'COMMA',
  '.' => 'DOT',
  '-' =>  'MINUS',
  '+' => 'PLUS',
  ';' => 'SEMICOLON',
  '/' => 'SLASH',
  '*' => 'STAR',
  '!' => 'BANG',
  '!=' => 'BANG_EQUAL',
  '=' => 'EQUAL',
  '==' => 'EQUAL_EQUAL',
  '>' => 'GREATER',
  '>=' => 'GREATER_EQUAL',
  '<' => 'LESS',
  '<=' => 'LESS_EQUAL'
}.freeze
@@keywords =

Here are all the implemented Lox keywords (in uppercase) These are enumerated in section 4.2.1 Token type

%w[
  AND CLASS ELSE FALSE FUN FOR IF NIL OR
  PRINT RETURN SUPER THIS TRUE VAR WHILE
].map { |x| [x, x] }.to_h

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(source = nil) ⇒ Scanner

Constructor. Initialize a tokenizer for Lox input.



66
67
68
69
 
# File 'lib/loxxy/front_end/scanner.rb', line 66

def initialize(source = nil)
  @scanner = StringScanner.new('')
  start_with(source) if source
end

Instance Attribute Details

#line_start ⇒ Integer (readonly) 



31
32
33
 
# File 'lib/loxxy/front_end/scanner.rb', line 31

def line_start
  @line_start
end

#lineno ⇒ Integer (readonly) 



28
29
30
 
# File 'lib/loxxy/front_end/scanner.rb', line 28

def lineno
  @lineno
end

#scanner ⇒ StringScanner (readonly) 



25
26
27
 
# File 'lib/loxxy/front_end/scanner.rb', line 25

def scanner
  @scanner
end

Instance Method Details

#start_with(source) ⇒ Object

Reset the tokenizer and make the given text, the current input.



73
74
75
76
77
 
# File 'lib/loxxy/front_end/scanner.rb', line 73

def start_with(source)
  @scanner.string = source
  @lineno = 1
  @line_start = 0
end

#tokens ⇒ Array<Rley::Lexical::Token>

Scan the source and return an array of tokens.



81
82
83
84
85
86
87
88
89
90
 
# File 'lib/loxxy/front_end/scanner.rb', line 81

def tokens
  tok_sequence = []
  until @scanner.eos?
    token = _next_token
    tok_sequence << token unless token.nil?
  end
  tok_sequence << build_token('EOF', '')

  return tok_sequence
end