Class: Sass::Script::Lexer

Inherits:
Object
  • Object
show all
Includes:
Sass::SCSS::RX
Defined in:
lib/sass/script/lexer.rb

Overview

The lexical analyzer for SassScript. It takes a raw string and converts it to individual tokens that are easier to parse.

Direct Known Subclasses

CssLexer

Defined Under Namespace

Classes: Token

Constant Summary collapse

OPERATORS =

A hash from operator strings to the corresponding token types.

{
  '+' => :plus,
  '-' => :minus,
  '*' => :times,
  '/' => :div,
  '%' => :mod,
  '=' => :single_eq,
  ':' => :colon,
  '(' => :lparen,
  ')' => :rparen,
  '[' => :lsquare,
  ']' => :rsquare,
  ',' => :comma,
  'and' => :and,
  'or' => :or,
  'not' => :not,
  '==' => :eq,
  '!=' => :neq,
  '>=' => :gte,
  '<=' => :lte,
  '>' => :gt,
  '<' => :lt,
  '#{' => :begin_interpolation,
  '}' => :end_interpolation,
  ';' => :semicolon,
  '{' => :lcurly,
  '...' => :splat,
}
OPERATORS_REVERSE = 
Sass::Util.map_hash(OPERATORS) {|k, v| [v, k]}
TOKEN_NAMES = 
Sass::Util.map_hash(OPERATORS_REVERSE) {|k, v| [k, v.inspect]}.merge(
:const => "variable (e.g. $foo)",
:ident => "identifier (e.g. middle)")
OP_NAMES =

A list of operator strings ordered with longer names first so that > and < don't clobber >= and <=.

OPERATORS.keys.sort_by {|o| -o.size}
IDENT_OP_NAMES =

A sub-list of OP_NAMES that only includes operators with identifier names.

OP_NAMES.select {|k, _v| k =~ /^\w+/}
PARSEABLE_NUMBER = 
/(?:(\d*\.\d+)|(\d+))(?:[eE]([+-]?\d+))?(#{UNIT})?/
REGULAR_EXPRESSIONS =

A hash of regular expressions that are used for tokenizing.

{
  :whitespace => /\s+/,
  :comment => COMMENT,
  :single_line_comment => SINGLE_LINE_COMMENT,
  :variable => /(\$)(#{IDENT})/,
  :ident => /(#{IDENT})(\()?/,
  :number => PARSEABLE_NUMBER,
  :unary_minus_number => /-#{PARSEABLE_NUMBER}/,
  :color => HEXCOLOR,
  :id => /##{IDENT}/,
  :selector => /&/,
  :ident_op => /(#{Regexp.union(*IDENT_OP_NAMES.map do |s|
    Regexp.new(Regexp.escape(s) + "(?!#{NMCHAR}|\Z)")
  end)})/,
  :op => /(#{Regexp.union(*OP_NAMES)})/,
}
STRING_REGULAR_EXPRESSIONS =

A hash of regular expressions that are used for tokenizing strings.

The key is a [Symbol, Boolean] pair. The symbol represents which style of quotation to use, while the boolean represents whether or not the string is following an interpolated segment.

{
  :double => {
    false => string_re('"', '"'),
    true => string_re('', '"')
  },
  :single => {
    false => string_re("'", "'"),
    true => string_re('', "'")
  },
  :uri => {
    false => /url\(#{W}(#{URLCHAR}*?)(#{W}\)|#\{)/,
    true => /(#{URLCHAR}*?)(#{W}\)|#\{)/
  },
  # Defined in https://developer.mozilla.org/en/CSS/@-moz-document as a
  # non-standard version of http://www.w3.org/TR/css3-conditional/
  :url_prefix => {
    false => /url-prefix\(#{W}(#{URLCHAR}*?)(#{W}\)|#\{)/,
    true => /(#{URLCHAR}*?)(#{W}\)|#\{)/
  },
  :domain => {
    false => /domain\(#{W}(#{URLCHAR}*?)(#{W}\)|#\{)/,
    true => /(#{URLCHAR}*?)(#{W}\)|#\{)/
  }
}

Constants included from Sass::SCSS::RX

Sass::SCSS::RX::ANY, Sass::SCSS::RX::CDC, Sass::SCSS::RX::CDO, Sass::SCSS::RX::COMMENT, Sass::SCSS::RX::DASHMATCH, Sass::SCSS::RX::DOMAIN, Sass::SCSS::RX::ESCAPE, Sass::SCSS::RX::FUNCTION, Sass::SCSS::RX::GREATER, Sass::SCSS::RX::H, Sass::SCSS::RX::HASH, Sass::SCSS::RX::HEXCOLOR, Sass::SCSS::RX::IDENT, Sass::SCSS::RX::IDENT_HYPHEN_INTERP, Sass::SCSS::RX::IDENT_START, Sass::SCSS::RX::IMPORTANT, Sass::SCSS::RX::INCLUDES, Sass::SCSS::RX::INTERP_START, Sass::SCSS::RX::NAME, Sass::SCSS::RX::NL, Sass::SCSS::RX::NMCHAR, Sass::SCSS::RX::NMSTART, Sass::SCSS::RX::NONASCII, Sass::SCSS::RX::NOT, Sass::SCSS::RX::NUMBER, Sass::SCSS::RX::OPTIONAL, Sass::SCSS::RX::PERCENTAGE, Sass::SCSS::RX::PLUS, Sass::SCSS::RX::PREFIXMATCH, Sass::SCSS::RX::RANGE, Sass::SCSS::RX::S, Sass::SCSS::RX::SINGLE_LINE_COMMENT, Sass::SCSS::RX::STATIC_COMPONENT, Sass::SCSS::RX::STATIC_SELECTOR, Sass::SCSS::RX::STATIC_VALUE, Sass::SCSS::RX::STRING, Sass::SCSS::RX::STRING1, Sass::SCSS::RX::STRING1_NOINTERP, Sass::SCSS::RX::STRING2, Sass::SCSS::RX::STRING2_NOINTERP, Sass::SCSS::RX::STRING_NOINTERP, Sass::SCSS::RX::SUBSTRINGMATCH, Sass::SCSS::RX::SUFFIXMATCH, Sass::SCSS::RX::TILDE, Sass::SCSS::RX::UNICODE, Sass::SCSS::RX::UNICODERANGE, Sass::SCSS::RX::UNIT, Sass::SCSS::RX::UNITLESS_NUMBER, Sass::SCSS::RX::URI, Sass::SCSS::RX::URL, Sass::SCSS::RX::URLCHAR, Sass::SCSS::RX::URL_PREFIX, Sass::SCSS::RX::VARIABLE, Sass::SCSS::RX::W

Instance Method Summary collapse

Methods included from Sass::SCSS::RX

escape_ident

Constructor Details

#initialize(str, line, offset, options) ⇒ Lexer

Returns a new instance of Lexer.

Parameters:

  • str (String, StringScanner) —

    The source text to lex

  • line (Integer) —

    The 1-based line on which the SassScript appears. Used for error reporting and sourcemap building

  • offset (Integer) —

    The 1-based character (not byte) offset in the line in the source. Used for error reporting and sourcemap building

  • options ({Symbol => Object}) —

    An options hash; see the Sass options documentation



153
154
155
156
157
158
159
160
161
162
 
# File 'lib/sass/script/lexer.rb', line 153

def initialize(str, line, offset, options)
  @scanner = str.is_a?(StringScanner) ? str : Sass::Util::MultibyteStringScanner.new(str)
  @line = line
  @offset = offset
  @options = options
  @interpolation_stack = []
  @prev = nil
  @tok = nil
  @next_tok = nil
end

Instance Method Details

#after_interpolation? ⇒ Boolean

Returns Whether or not the last token lexed was :end_interpolation.

Returns:

  • (Boolean) —

    Whether or not the last token lexed was :end_interpolation.



226
227
228
 
# File 'lib/sass/script/lexer.rb', line 226

def after_interpolation?
  @prev && @prev.type == :end_interpolation
end

#char(pos = @scanner.pos) ⇒ String

Returns the given character.

Returns:

  • (String) 


189
190
191
 
# File 'lib/sass/script/lexer.rb', line 189

def char(pos = @scanner.pos)
  @scanner.string[pos, 1]
end

#done? ⇒ Boolean

Returns Whether or not there's more source text to lex.

Returns:

  • (Boolean) —

    Whether or not there's more source text to lex.



219
220
221
222
223
 
# File 'lib/sass/script/lexer.rb', line 219

def done?
  return if @next_tok
  whitespace unless after_interpolation? && !@interpolation_stack.empty?
  @scanner.eos? && @tok.nil?
end

#expected!(name)

Raise an error to the effect that name was expected in the input stream and wasn't found.

This calls #unpeek! to rewind the scanner to immediately after the last returned token.

Parameters:

  • name (String) —

    The name of the entity that was expected but not found

Raises:



238
239
240
241
 
# File 'lib/sass/script/lexer.rb', line 238

def expected!(name)
  unpeek!
  Sass::SCSS::Parser.expected(@scanner, name, @line)
end

#line ⇒ Integer

The line number of the lexer's current position.

Returns:

  • (Integer) 


29
30
31
32
 
# File 'lib/sass/script/lexer.rb', line 29

def line
  return @line unless @tok
  @tok.source_range.start_pos.line
end

#next ⇒ Token

Moves the lexer forward one token.

Returns:

  • (Token) —

    The token that was moved past



167
168
169
170
171
172
 
# File 'lib/sass/script/lexer.rb', line 167

def next
  @tok ||= read_token
  @tok, tok = nil, @tok
  @prev = tok
  tok
end

#next_char ⇒ String

Consumes and returns single raw character from the input stream.

Returns:

  • (String) 


196
197
198
199
 
# File 'lib/sass/script/lexer.rb', line 196

def next_char
  unpeek!
  scan(/./)
end

#offset ⇒ Integer

The number of bytes into the current line of the lexer's current position (1-based).

Returns:

  • (Integer) 


38
39
40
41
 
# File 'lib/sass/script/lexer.rb', line 38

def offset
  return @offset unless @tok
  @tok.source_range.start_pos.offset
end

#peek ⇒ Token

Returns the next token without moving the lexer forward.

Returns:

  • (Token) —

    The next token



204
205
206
 
# File 'lib/sass/script/lexer.rb', line 204

def peek
  @tok ||= read_token
end

#str { ... } ⇒ String

Records all non-comment text the lexer consumes within the block and returns it as a string.

Yields:

  • A block in which text is recorded

Returns:

  • (String) 


248
249
250
251
252
253
 
# File 'lib/sass/script/lexer.rb', line 248

def str
  old_pos = @tok ? @tok.pos : @scanner.pos
  yield
  new_pos = @tok ? @tok.pos : @scanner.pos
  @scanner.string[old_pos...new_pos]
end

#try

Runs a block, and rewinds the state of the lexer to the beginning of the block if it returns nil or false.



257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
 
# File 'lib/sass/script/lexer.rb', line 257

def try
  old_pos = @scanner.pos
  old_line = @line
  old_offset = @offset
  old_interpolation_stack = @interpolation_stack.dup
  old_prev = @prev
  old_tok = @tok
  old_next_tok = @next_tok

  result = yield
  return result if result

  @scanner.pos = old_pos
  @line = old_line
  @offset = old_offset
  @interpolation_stack = old_interpolation_stack
  @prev = old_prev
  @tok = old_tok
  @next_tok = old_next_tok
  nil
end

#unpeek!

Rewinds the underlying StringScanner to before the token returned by #peek.



210
211
212
213
214
215
216
 
# File 'lib/sass/script/lexer.rb', line 210

def unpeek!
  raise "[BUG] Can't unpeek before a queued token!" if @next_tok
  return unless @tok
  @scanner.pos = @tok.pos
  @line = @tok.source_range.start_pos.line
  @offset = @tok.source_range.start_pos.offset
end

#whitespace?(tok = @tok) ⇒ Boolean

Returns whether or not there's whitespace before the next token.

Returns:

  • (Boolean) 


177
178
179
180
181
182
183
184
 
# File 'lib/sass/script/lexer.rb', line 177

def whitespace?(tok = @tok)
  if tok
    @scanner.string[0...tok.pos] =~ /\s\Z/
  else
    @scanner.string[@scanner.pos, 1] =~ /^\s/ ||
      @scanner.string[@scanner.pos - 1, 1] =~ /\s\Z/
  end
end