Class: Pgsqlarbiter::Lexer

Inherits:
Object
  • Object
show all
Includes:
TokenType
Defined in:
lib/pgsqlarbiter/lexer.rb

Overview

SQL lexer that converts a query string into an array of Token objects.

Handles all PostgreSQL token types including keywords, identifiers (plain and double-quoted), strings (single-quoted, dollar-quoted, and prefixed), numbers, parameters, operators, and punctuation.

Constant Summary

Constants included from TokenType

TokenType::COMMA, TokenType::DOT, TokenType::EOF, TokenType::IDENT, TokenType::KEYWORD, TokenType::LBRACKET, TokenType::LPAREN, TokenType::NUMBER, TokenType::OP, TokenType::PARAM, TokenType::QUOTED_IDENT, TokenType::RBRACKET, TokenType::RPAREN, TokenType::SEMICOLON, TokenType::STAR, TokenType::STRING, TokenType::TYPECAST

Instance Method Summary collapse

Instance Method Details

#tokenize(sql) ⇒ Array<Token>

Tokenize a SQL query string.

Parameters:

  • sql (String) —

    the SQL string to tokenize

Returns:

  • (Array<Token>) —

    list of tokens ending with an EOF token

Raises:

  • (LexError) —

    on invalid syntax such as unexpected characters or unterminated strings/comments



20
21
22
23
24
25
26
27
28
29
30
 
# File 'lib/pgsqlarbiter/lexer.rb', line 20

def tokenize(sql)
  @scanner = StringScanner.new(sql)
  @tokens = []

  until @scanner.eos?
    scan_token
  end

  @tokens << Token.new(type: EOF, value: nil, position: @scanner.pos)
  @tokens
end