Class: Rouge::Lexer Abstract

Inherits:
Object
  • Object
show all
Includes:
Token::Tokens
Defined in:
lib/rouge/lexer.rb

Overview

This class is abstract.

A lexer transforms text into a stream of ‘[token, chunk]` pairs.

Constant Summary

Constants included from Token::Tokens

Token::Tokens::Num, Token::Tokens::Str

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from Token::Tokens

token

Constructor Details

#initialize(opts = {}) ⇒ Lexer

Create a new lexer with the given options. Individual lexers may specify extra options. The only current globally accepted option is ‘:debug`.

Parameters:

  • opts (Hash) (defaults to: {}) —

    a customizable set of options

Options Hash (opts):

  • :debug (Object) —

    Prints debug information to stdout. The particular info depends on the lexer in question. In regex lexers, this will log the state stack at the beginning of each step, along with each regex tried and each stream consumed. Try it, it’s pretty useful.



283
284
285
286
287
288
 
# File 'lib/rouge/lexer.rb', line 283

def initialize(opts={})
  @options = {}
  opts.each { |k, v| @options[k.to_s] = v }

  @debug = Lexer.debug_enabled? && bool_option(:debug)
end

Instance Attribute Details

#options ⇒ Object (readonly)

-*- instance methods -*- #



273
274
275
 
# File 'lib/rouge/lexer.rb', line 273

def options
  @options
end

Class Method Details

.aliases(*args) ⇒ Object

Used to specify alternate names this lexer class may be found by.

Examples:

class Erb < Lexer
  tag 'erb'
  aliases 'eruby', 'rhtml'
end

Lexer.find('eruby') # => Erb


230
231
232
233
234
 
# File 'lib/rouge/lexer.rb', line 230

def aliases(*args)
  args.map!(&:to_s)
  args.each { |arg| Lexer.register(arg, self) }
  (@aliases ||= []).concat(args)
end

.all ⇒ Object

Returns a list of all lexers.

Returns:

  • a list of all lexers.



119
120
121
 
# File 'lib/rouge/lexer.rb', line 119

def all
  registry.values.uniq
end

.assert_utf8!(str) ⇒ Object

Raises:

  • (EncodingError) 


257
258
259
260
261
262
263
 
# File 'lib/rouge/lexer.rb', line 257

def assert_utf8!(str)
  return if %w(US-ASCII UTF-8 ASCII-8BIT).include? str.encoding.name
  raise EncodingError.new(
    "Bad encoding: #{str.encoding.names.join(',')}. " +
    "Please convert your string to UTF-8."
  )
end

.debug_enabled? ⇒ Boolean

Returns:

  • (Boolean) 


193
194
195
 
# File 'lib/rouge/lexer.rb', line 193

def debug_enabled?
  !!@debug_enabled
end

.demo(arg = :absent) ⇒ Object

Specify or get a small demo string for this lexer



112
113
114
115
116
 
# File 'lib/rouge/lexer.rb', line 112

def demo(arg=:absent)
  return @demo = arg unless arg == :absent

  @demo = File.read(demo_file, mode: 'rt:bom|utf-8')
end

.demo_file(arg = :absent) ⇒ Object

Specify or get the path name containing a small demo for this lexer (can be overriden by demo).



105
106
107
108
109
 
# File 'lib/rouge/lexer.rb', line 105

def demo_file(arg=:absent)
  return @demo_file = Pathname.new(arg) unless arg == :absent

  @demo_file = Pathname.new(__FILE__).dirname.join('demos', tag)
end

.desc(arg = :absent) ⇒ Object

Specify or get this lexer’s description.



87
88
89
90
91
92
93
 
# File 'lib/rouge/lexer.rb', line 87

def desc(arg=:absent)
  if arg == :absent
    @desc
  else
    @desc = arg
  end
end

.detect?(text) ⇒ Boolean

This method is abstract.

Return true if there is an in-text indication (such as a shebang or DOCTYPE declaration) that this lexer should be used.

Parameters:

Returns:

  • (Boolean) 


445
446
447
 
# File 'lib/rouge/lexer.rb', line 445

def self.detect?(text)
  false
end

.disable_debug! ⇒ Object 



189
190
191
 
# File 'lib/rouge/lexer.rb', line 189

def disable_debug!
  @debug_enabled = false
end

.enable_debug! ⇒ Object 



185
186
187
 
# File 'lib/rouge/lexer.rb', line 185

def enable_debug!
  @debug_enabled = true
end

.filenames(*fnames) ⇒ Object

Specify a list of filename globs associated with this lexer.

Examples:

class Ruby < Lexer
  filenames '*.rb', '*.ruby', 'Gemfile', 'Rakefile'
end


242
243
244
 
# File 'lib/rouge/lexer.rb', line 242

def filenames(*fnames)
  (@filenames ||= []).concat(fnames)
end

.find(name) ⇒ Class<Rouge::Lexer>?

Given a name in string, return the correct lexer class.

Parameters:

  • name (String)

Returns:



28
29
30
 
# File 'lib/rouge/lexer.rb', line 28

def find(name)
  registry[name.to_s]
end

.find_fancy(str, code = nil, additional_options = {}) ⇒ Object

Find a lexer, with fancy shiny features.

  • The string you pass can include CGI-style options

    Lexer.find_fancy('erb?parent=tex')

  • You can pass the special name ‘guess’ so we guess for you, and you can pass a second argument of the code to guess by

    Lexer.find_fancy('guess', "#!/bin/bash\necho Hello, world")

This is used in the Redcarpet plugin as well as Rouge’s own markdown lexer for highlighting internal code blocks.



46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
 
# File 'lib/rouge/lexer.rb', line 46

def find_fancy(str, code=nil, additional_options={})

  if str && !str.include?('?') && str != 'guess'
    lexer_class = find(str)
    return lexer_class && lexer_class.new(additional_options)
  end

  name, opts = str ? str.split('?', 2) : [nil, '']

  # parse the options hash from a cgi-style string
  opts = CGI.parse(opts || '').map do |k, vals|
    val = case vals.size
    when 0 then true
    when 1 then vals[0]
    else vals
    end

    [ k.to_s, val ]
  end

  opts = additional_options.merge(Hash[opts])

  lexer_class = case name
  when 'guess', nil
    self.guess(:source => code, :mimetype => opts['mimetype'])
  when String
    self.find(name)
  end

  lexer_class && lexer_class.new(opts)
end

.guess(info = {}, &fallback) ⇒ Class<Rouge::Lexer>

Guess which lexer to use based on a hash of info.

Parameters:

  • fallback (Proc) —

    called if multiple lexers are detected. If omitted, Guesser::Ambiguous is raised.

  • info (Hash) (defaults to: {}) —

    a customizable set of options

Options Hash (info):

  • :mimetype (Object) —

    A mimetype to guess by

  • :filename (Object) —

    A filename to guess by

  • :source (Object) —

    The source itself, which, if guessing by mimetype or filename fails, will be searched for shebangs, <!DOCTYPE …> tags, and other hints.

Returns:

See Also:



160
161
162
163
164
165
166
167
168
169
170
171
 
# File 'lib/rouge/lexer.rb', line 160

def guess(info={}, &fallback)
  lexers = guesses(info)

  return Lexers::PlainText if lexers.empty?
  return lexers[0] if lexers.size == 1

  if fallback
    fallback.call(lexers)
  else
    raise Guesser::Ambiguous.new(lexers)
  end
end

.guess_by_filename(fname) ⇒ Object 



177
178
179
 
# File 'lib/rouge/lexer.rb', line 177

def guess_by_filename(fname)
  guess :filename => fname
end

.guess_by_mimetype(mt) ⇒ Object 



173
174
175
 
# File 'lib/rouge/lexer.rb', line 173

def guess_by_mimetype(mt)
  guess :mimetype => mt
end

.guess_by_source(source) ⇒ Object 



181
182
183
 
# File 'lib/rouge/lexer.rb', line 181

def guess_by_source(source)
  guess :source => source
end

.guesses(info = {}) ⇒ Object

Guess which lexer to use based on a hash of info.

This accepts the same arguments as Lexer.guess, but will never throw an error. It will return a (possibly empty) list of potential lexers to use.



128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
 
# File 'lib/rouge/lexer.rb', line 128

def guesses(info={})
  mimetype, filename, source = info.values_at(:mimetype, :filename, :source)
  custom_globs = info[:custom_globs]

  guessers = (info[:guessers] || []).dup

  guessers << Guessers::Mimetype.new(mimetype) if mimetype
  guessers << Guessers::GlobMapping.by_pairs(custom_globs, filename) if custom_globs && filename
  guessers << Guessers::Filename.new(filename) if filename
  guessers << Guessers::Modeline.new(source) if source
  guessers << Guessers::Source.new(source) if source
  guessers << Guessers::Disambiguation.new(filename, source) if source && filename

  Guesser.guess(guessers, Lexer.all)
end

.lex(stream, opts = {}, &b) ⇒ Object

Lexes ‘stream` with the given options. The lex is delegated to a new instance.

See Also:



21
22
23
 
# File 'lib/rouge/lexer.rb', line 21

def lex(stream, opts={}, &b)
  new(opts).lex(stream, &b)
end

.mimetypes(*mts) ⇒ Object

Specify a list of mimetypes associated with this lexer.

Examples:

class Html < Lexer
  mimetypes 'text/html', 'application/xhtml+xml'
end


252
253
254
 
# File 'lib/rouge/lexer.rb', line 252

def mimetypes(*mts)
  (@mimetypes ||= []).concat(mts)
end

.option(name, desc) ⇒ Object 



99
100
101
 
# File 'lib/rouge/lexer.rb', line 99

def option(name, desc)
  option_docs[name.to_s] = desc
end

.option_docs ⇒ Object 



95
96
97
 
# File 'lib/rouge/lexer.rb', line 95

def option_docs
  @option_docs ||= InheritableHash.new(superclass.option_docs)
end

.tag(t = nil) ⇒ Object

Used to specify or get the canonical name of this lexer class.

Examples:

class MyLexer < Lexer
  tag 'foo'
end

MyLexer.tag # => 'foo'

Lexer.find('foo') # => MyLexer


214
215
216
217
218
219
 
# File 'lib/rouge/lexer.rb', line 214

def tag(t=nil)
  return @tag if t.nil?

  @tag = t.to_s
  Lexer.register(@tag, self)
end

.title(t = nil) ⇒ Object

Specify or get this lexer’s title. Meant to be human-readable.



79
80
81
82
83
84
 
# File 'lib/rouge/lexer.rb', line 79

def title(t=nil)
  if t.nil?
    t = tag.capitalize
  end
  @title ||= t
end

Instance Method Details

#as_bool(val) ⇒ Object 



290
291
292
293
294
295
296
297
298
299
 
# File 'lib/rouge/lexer.rb', line 290

def as_bool(val)
  case val
  when nil, false, 0, '0', 'off'
    false
  when Array
    val.empty? ? true : as_bool(val.last)
  else
    true
  end
end

#as_lexer(val) ⇒ Object 



318
319
320
321
322
323
324
325
326
327
328
329
 
# File 'lib/rouge/lexer.rb', line 318

def as_lexer(val)
  return as_lexer(val.last) if val.is_a?(Array)
  return val.new(@options) if val.is_a?(Class) && val < Lexer

  case val
  when Lexer
    val
  when String
    lexer_class = Lexer.find(val)
    lexer_class && lexer_class.new(@options)
  end
end

#as_list(val) ⇒ Object 



307
308
309
310
311
312
313
314
315
316
 
# File 'lib/rouge/lexer.rb', line 307

def as_list(val)
  case val
  when Array
    val.flat_map { |v| as_list(v) }
  when String
    val.split(',')
  else
    []
  end
end

#as_string(val) ⇒ Object 



301
302
303
304
305
 
# File 'lib/rouge/lexer.rb', line 301

def as_string(val)
  return as_string(val.last) if val.is_a?(Array)

  val ? val.to_s : nil
end

#as_token(val) ⇒ Object 



331
332
333
334
335
336
337
338
339
 
# File 'lib/rouge/lexer.rb', line 331

def as_token(val)
  return as_token(val.last) if val.is_a?(Array)
  case val
  when Token
    val
  else
    Token[val]
  end
end

#bool_option(name, &default) ⇒ Object 



341
342
343
344
345
346
347
 
# File 'lib/rouge/lexer.rb', line 341

def bool_option(name, &default)
  if @options.key?(name.to_s)
    as_bool(@options[name.to_s])
  else
    default ? default.call : false
  end
end

#hash_option(name, defaults, &val_cast) ⇒ Object 



365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
 
# File 'lib/rouge/lexer.rb', line 365

def hash_option(name, defaults, &val_cast)
  name = name.to_s
  out = defaults.dup

  base = @options.delete(name.to_s)
  base = {} unless base.is_a?(Hash)
  base.each { |k, v| out[k.to_s] = val_cast ? val_cast.call(v) : v }

  @options.keys.each do |key|
    next unless key =~ /(\w+)\[(\w+)\]/ and $1 == name
    value = @options.delete(key)

    out[$2] = val_cast ? val_cast.call(value) : value
  end

  out
end

#lex(string, opts = {}, &b) ⇒ Object

Given a string, yield [token, chunk] pairs. If no block is given, an enumerator is returned.

Parameters:

  • opts (Hash) (defaults to: {}) —

    a customizable set of options

Options Hash (opts):

  • :continue (Object) —

    Continue the lex from the previous state (i.e. don’t call #reset!)



395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
 
# File 'lib/rouge/lexer.rb', line 395

def lex(string, opts={}, &b)
  return enum_for(:lex, string, opts) unless block_given?

  Lexer.assert_utf8!(string)

  reset! unless opts[:continue]

  # consolidate consecutive tokens of the same type
  last_token = nil
  last_val = nil
  stream_tokens(string) do |tok, val|
    next if val.empty?

    if tok == last_token
      last_val << val
      next
    end

    b.call(last_token, last_val) if last_token
    last_token = tok
    last_val = val
  end

  b.call(last_token, last_val) if last_token
end

#lexer_option(name, &default) ⇒ Object 



353
354
355
 
# File 'lib/rouge/lexer.rb', line 353

def lexer_option(name, &default)
  as_lexer(@options.delete(name.to_s, &default))
end

#list_option(name, &default) ⇒ Object 



357
358
359
 
# File 'lib/rouge/lexer.rb', line 357

def list_option(name, &default)
  as_list(@options.delete(name.to_s, &default))
end

#reset! ⇒ Object

This method is abstract.

Called after each lex is finished. The default implementation is a noop.



387
388
 
# File 'lib/rouge/lexer.rb', line 387

def reset!
end

#stream_tokens(stream, &b) ⇒ Object

This method is abstract.

Yield ‘[token, chunk]` pairs, given a prepared input stream. This must be implemented.

Parameters:

  • stream (StringScanner) —

    the stream



433
434
435
 
# File 'lib/rouge/lexer.rb', line 433

def stream_tokens(stream, &b)
  raise 'abstract'
end

#string_option(name, &default) ⇒ Object 



349
350
351
 
# File 'lib/rouge/lexer.rb', line 349

def string_option(name, &default)
  as_string(@options.delete(name.to_s, &default))
end

#tag ⇒ Object

delegated to tag



422
423
424
 
# File 'lib/rouge/lexer.rb', line 422

def tag
  self.class.tag
end

#token_option(name, &default) ⇒ Object 



361
362
363
 
# File 'lib/rouge/lexer.rb', line 361

def token_option(name, &default)
  as_token(@options.delete(name.to_s, &default))
end