Class: TRuby::Scanner

Inherits:
Object
  • Object
show all
Defined in:
lib/t_ruby/scanner.rb

Overview

Scanner - T-Ruby 소스 코드를 토큰 스트림으로 변환TypeScript 컴파일러와 유사한 구조로, 파서와 분리되어 증분 파싱을 지원

Defined Under Namespace

Classes: ScanError, Token

Constant Summary collapse

KEYWORDS =

키워드 맵

{
  "def" => :def,
  "end" => :end,
  "class" => :class,
  "module" => :module,
  "if" => :if,
  "unless" => :unless,
  "else" => :else,
  "elsif" => :elsif,
  "return" => :return,
  "type" => :type,
  "interface" => :interface,
  "public" => :public,
  "private" => :private,
  "protected" => :protected,
  "true" => true,
  "false" => false,
  "nil" => :nil,
  "while" => :while,
  "until" => :until,
  "for" => :for,
  "do" => :do,
  "begin" => :begin,
  "rescue" => :rescue,
  "ensure" => :ensure,
  "case" => :case,
  "when" => :when,
  "then" => :then,
  "and" => :and,
  "or" => :or,
  "not" => :not,
  "in" => :in,
  "self" => :self,
  "super" => :super,
  "yield" => :yield,
  "break" => :break,
  "next" => :next,
  "redo" => :redo,
  "retry" => :retry,
  "raise" => :raise,
  "alias" => :alias,
  "defined?" => :defined,
  "__FILE__" => :__file__,
  "__LINE__" => :__line__,
  "__ENCODING__" => :__encoding__,
}.freeze

Instance Method Summary collapse

Constructor Details

#initialize(source) ⇒ Scanner

Returns a new instance of Scanner.



70
71
72
73
74
75
76
77
78
 
# File 'lib/t_ruby/scanner.rb', line 70

def initialize(source)
  @source = source
  @position = 0
  @line = 1
  @column = 1
  @tokens = []
  @token_index = 0
  @scanned = false
end

Instance Method Details

#next_token ⇒ Object

단일 토큰 반환 (스트리밍용)



100
101
102
103
104
105
106
 
# File 'lib/t_ruby/scanner.rb', line 100

def next_token
  scan_all unless @scanned

  token = @tokens[@token_index]
  @token_index += 1 unless token&.type == :eof
  token || @tokens.last
end

#peek(n = 1) ⇒ Object

lookahead



109
110
111
112
113
114
115
116
117
 
# File 'lib/t_ruby/scanner.rb', line 109

def peek(n = 1)
  scan_all unless @scanned

  if n == 1
    @tokens[@token_index] || @tokens.last
  else
    @tokens[@token_index, n] || [@tokens.last]
  end
end

#reset ⇒ Object

토큰 인덱스 리셋



120
121
122
 
# File 'lib/t_ruby/scanner.rb', line 120

def reset
  @token_index = 0
end

#scan_all ⇒ Object

전체 토큰화 (캐싱용)



81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
 
# File 'lib/t_ruby/scanner.rb', line 81

def scan_all
  return @tokens if @scanned

  @tokens = []
  @position = 0
  @line = 1
  @column = 1

  while @position < @source.length
    token = scan_token
    @tokens << token if token
  end

  @tokens << Token.new(:eof, "", @position, @position, @line, @column)
  @scanned = true
  @tokens
end