Class: Loxxy::FrontEnd::Scanner

Inherits:
Object
  • Object
show all
Defined in:
lib/loxxy/front_end/scanner.rb

Overview

A scanner (tokenizer) for the Lox language. Reference material: https://craftinginterpreters.com/the-lox-language.html Section 4.2.1 Token types Appendix A1.2 Lexical Grammar Responsibility: break input into a sequence of token objects. The tokenizer should recognize: Identifiers, Number literals including single digit String literals (quote delimited) Delimiters: e.g. parentheses '(', ')' Separators: e.g. comma

Defined Under Namespace

Classes: ScanError

Constant Summary collapse

@@lexeme2name =

One or two special character tokens. These are enumerated in section 4.2.1 Token type

{
  '(' => 'LEFT_PAREN',
  ')' => 'RIGHT_PAREN',
  '{' => 'LEFT_BRACE',
  '}' => 'RIGHT_BRACE',
  ',' => 'COMMA',
  '.' => 'DOT',
  '-' =>  'MINUS',
  '+' => 'PLUS',
  ';' => 'SEMICOLON',
  '/' => 'SLASH',
  '*' => 'STAR',
  '!' => 'BANG',
  '!=' => 'BANG_EQUAL',
  '=' => 'EQUAL',
  '==' => 'EQUAL_EQUAL',
  '>' => 'GREATER',
  '>=' => 'GREATER_EQUAL',
  '<' => 'LESS',
  '<=' => 'LESS_EQUAL'
}.freeze
@@keywords =

Here are all the implemented Lox keywords (in uppercase) These are enumerated in section 4.2.1 Token type

%w[
  AND CLASS ELSE FALSE FUN FOR IF NIL OR
  PRINT RETURN SUPER THIS TRUE VAR WHILE
].map { |x| [x, x] }.to_h

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(source = nil) ⇒ Scanner

Constructor. Initialize a tokenizer for Lox input.

Parameters:

  • source (String) (defaults to: nil) —

    Lox text to tokenize.



67
68
69
70
 
# File 'lib/loxxy/front_end/scanner.rb', line 67

def initialize(source = nil)
  @scanner = StringScanner.new('')
  start_with(source) if source
end

Instance Attribute Details

#line_start ⇒ Integer (readonly)

Returns Position of last start of line in the input.

Returns:

  • (Integer) —

    Position of last start of line in the input



30
31
32
 
# File 'lib/loxxy/front_end/scanner.rb', line 30

def line_start
  @line_start
end

#lineno ⇒ Integer (readonly)

Returns The current line number.

Returns:

  • (Integer) —

    The current line number



27
28
29
 
# File 'lib/loxxy/front_end/scanner.rb', line 27

def lineno
  @lineno
end

#scanner ⇒ StringScanner (readonly)

Returns Low-level input scanner.

Returns:

  • (StringScanner) —

    Low-level input scanner



24
25
26
 
# File 'lib/loxxy/front_end/scanner.rb', line 24

def scanner
  @scanner
end

Instance Method Details

#start_with(source) ⇒ Object

Reset the tokenizer and make the given text, the current input.

Parameters:

  • source (String) —

    Lox text to tokenize.



74
75
76
77
78
 
# File 'lib/loxxy/front_end/scanner.rb', line 74

def start_with(source)
  @scanner.string = source
  @lineno = 1
  @line_start = 0
end

#tokens ⇒ Array<Rley::Lexical::Token>

Scan the source and return an array of tokens.

Returns:

  • (Array<Rley::Lexical::Token>) —

    | Returns a sequence of tokens



82
83
84
85
86
87
88
89
90
91
 
# File 'lib/loxxy/front_end/scanner.rb', line 82

def tokens
  tok_sequence = []
  until @scanner.eos?
    token = _next_token
    tok_sequence << token unless token.nil?
  end
  tok_sequence << build_token('EOF', '')

  return tok_sequence
end