Class: Liquid2::Scanner

Inherits:
Object
  • Object
show all
Defined in:
lib/liquid2/scanner.rb

Overview

Liquid template source text lexical scanner.

This is a single pass tokenizer. We support tag and output delimiters inside string literals, so we must scan expressions as we go.

We give comment and raw tags special consideration here.

Constant Summary collapse

TOKEN_MAP =

Keywords and symbols that get their own token kind.

{
  "true" => :token_true,
  "false" => :token_false,
  "nil" => :token_nil,
  "null" => :token_nil,
  "and" => :token_and,
  "or" => :token_or,
  "not" => :token_not,
  "in" => :token_in,
  "contains" => :token_contains,
  "if" => :token_if,
  "else" => :token_else,
  "with" => :token_with,
  "required" => :token_required,
  "as" => :token_as,
  "for" => :token_for,
  "blank" => :token_blank,
  "empty" => :token_empty,
  "?" => :token_question,
  "[" => :token_lbracket,
  "]" => :token_rbracket,
  "|" => :token_pipe,
  "||" => :token_double_pipe,
  "." => :token_dot,
  ".." => :token_double_dot,
  "..." => :token_spread,
  "," => :token_comma,
  ":" => :token_colon,
  "(" => :token_lparen,
  ")" => :token_rparen,
  "=" => :token_assign,
  "<" => :token_lt,
  "<=" => :token_le,
  "<>" => :token_lg,
  ">" => :token_gt,
  ">=" => :token_ge,
  "==" => :token_eq,
  "!=" => :token_ne,
  "=>" => :token_arrow,
  "+" => :token_plus,
  "-" => :token_minus,
  "%" => :token_mod,
  "*" => :token_times,
  "/" => :token_divide,
  "//" => :token_floor_div,
  "**" => :token_pow
}.freeze

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(env, source, scanner) ⇒ Scanner

Returns a new instance of Scanner.

Parameters:

  • env (Environment)
  • source (String)
  • scanner (StringScanner) 


73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
 
# File 'lib/liquid2/scanner.rb', line 73

def initialize(env, source, scanner)
  @source = source
  @scanner = scanner
  @scanner.string = @source

  # A pointer to the start of the current token.
  @start = 0

  # Tokens are arrays of (kind, value, start index).
  # Sometimes we set value to `nil` when the symbol is unambiguous.
  @tokens = [] # : Array[[Symbol, String?, Integer]]

  @s_out_start = env.markup_out_start
  @s_out_end = env.markup_out_end
  @s_tag_start = env.markup_tag_start
  @s_tag_end = env.markup_tag_end
  @s_comment_prefix = env.markup_comment_prefix
  @s_comment_suffix = env.markup_comment_suffix

  @re_tag_name = env.re_tag_name
  @re_word = env.re_word
  @re_int = env.re_int
  @re_float = env.re_float
  @re_double_quote_string_special = env.re_double_quote_string_special
  @re_single_quote_string_special = env.re_single_quote_string_special
  @re_markup_start = env.re_markup_start
  @re_markup_end = env.re_markup_end
  @re_markup_end_chars = env.re_markup_end_chars
  @re_up_to_markup_start = env.re_up_to_markup_start
  @re_punctuation = env.re_punctuation
  @re_up_to_inline_comment_end = env.re_up_to_inline_comment_end
  @re_up_to_raw_end = env.re_up_to_raw_end
  @re_block_comment_chunk = env.re_block_comment_chunk
  @re_up_to_doc_end = env.re_up_to_doc_end
  @re_line_statement_comment = env.re_line_statement_comment
end

Instance Attribute Details

#tokens ⇒ Object (readonly)

Returns the value of attribute tokens.



13
14
15
 
# File 'lib/liquid2/scanner.rb', line 13

def tokens
  @tokens
end

Class Method Details

.tokenize(env, source, scanner) ⇒ Object 



64
65
66
67
68
 
# File 'lib/liquid2/scanner.rb', line 64

def self.tokenize(env, source, scanner)
  lexer = new(env, source, scanner)
  lexer.run
  lexer.tokens
end

Instance Method Details

#run ⇒ Object 



110
111
112
113
 
# File 'lib/liquid2/scanner.rb', line 110

def run
  state = :lex_markup
  state = send(state) until state.nil?
end