Class: Rouge::Lexer Abstract

Inherits:
Object
  • Object
show all
Defined in:
lib/rouge/lexer.rb

Overview

This class is abstract.

A lexer transforms text into a stream of `[token, chunk]` pairs.

Direct Known Subclasses

Rouge::Lexers::Text, RegexLexer

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(opts = {}) ⇒ Lexer

Create a new lexer with the given options. Individual lexers may specify extra options. The only current globally accepted option is `:debug`.

Parameters:

  • opts (Hash) (defaults to: {})

    a customizable set of options

Options Hash (opts):

  • :debug (Object)

    Prints debug information to stdout. The particular info depends on the lexer in question. In regex lexers, this will log the state stack at the beginning of each step, along with each regex tried and each stream consumed. Try it, it's pretty useful.


236
237
238
# File 'lib/rouge/lexer.rb', line 236

def initialize(opts={})
  options(opts)
end

Class Method Details

.aliases(*args) ⇒ Object

Used to specify alternate names this lexer class may be found by.

Examples:

class Erb < Lexer
  tag 'erb'
  aliases 'eruby', 'rhtml'
end

Lexer.find('eruby') # => Erb

184
185
186
187
188
# File 'lib/rouge/lexer.rb', line 184

def aliases(*args)
  args.map!(&:to_s)
  args.each { |arg| Lexer.register(arg, self) }
  (@aliases ||= []).concat(args)
end

.allObject

Returns a list of all lexers.

Returns:

  • a list of all lexers.


88
89
90
# File 'lib/rouge/lexer.rb', line 88

def all
  registry.values.uniq
end

.analyze_text(text) ⇒ Object

This method is abstract.

Return a number between 0 and 1 indicating the likelihood that the text given should be lexed with this lexer. The default implementation returns 0.

Parameters:


340
341
342
# File 'lib/rouge/lexer.rb', line 340

def self.analyze_text(text)
  0
end

.assert_utf8!(str) ⇒ Object

Raises:

  • (EncodingError)

211
212
213
214
215
216
217
# File 'lib/rouge/lexer.rb', line 211

def assert_utf8!(str)
  return if %w(US-ASCII UTF-8).include? str.encoding.name
  raise EncodingError.new(
    "Bad encoding: #{str.encoding.names.join(',')}. " +
    "Please convert your string to UTF-8."
  )
end

.default_options(o = {}) ⇒ Object


18
19
20
21
22
# File 'lib/rouge/lexer.rb', line 18

def default_options(o={})
  @default_options ||= {}
  @default_options.merge!(o)
  @default_options
end

.demo(arg = :absent) ⇒ Object

Specify or get a small demo string for this lexer


81
82
83
84
85
# File 'lib/rouge/lexer.rb', line 81

def demo(arg=:absent)
  return @demo = arg unless arg == :absent

  @demo = File.read(demo_file)
end

.demo_file(arg = :absent) ⇒ Object

Specify or get the path name containing a small demo for this lexer (can be overriden by demo).


74
75
76
77
78
# File 'lib/rouge/lexer.rb', line 74

def demo_file(arg=:absent)
  return @demo_file = Pathname.new(arg) unless arg == :absent

  @demo_file = Pathname.new(__FILE__).dirname.join('demos', tag)
end

.desc(arg = :absent) ⇒ Object

Specify or get this lexer's description.


64
65
66
67
68
69
70
# File 'lib/rouge/lexer.rb', line 64

def desc(arg=:absent)
  if arg == :absent
    @desc
  else
    @desc = arg
  end
end

.filenames(*fnames) ⇒ Object

Specify a list of filename globs associated with this lexer.

Examples:

class Ruby < Lexer
  filenames '*.rb', '*.ruby', 'Gemfile', 'Rakefile'
end

196
197
198
# File 'lib/rouge/lexer.rb', line 196

def filenames(*fnames)
  (@filenames ||= []).concat(fnames)
end

.find(name) ⇒ Object

Given a string, return the correct lexer class.


25
26
27
# File 'lib/rouge/lexer.rb', line 25

def find(name)
  registry[name.to_s]
end

.find_fancy(str, code = nil) ⇒ Object

Find a lexer, with fancy shiny features.

  • The string you pass can include CGI-style options

    Lexer.find_fancy('erb?parent=tex')
    
  • You can pass the special name 'guess' so we guess for you, and you can pass a second argument of the code to guess by

    Lexer.find_fancy('guess', "#!/bin/bash\necho Hello, world")
    

This is used in the Redcarpet plugin as well as Rouge's own markdown lexer for highlighting internal code blocks.


43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
# File 'lib/rouge/lexer.rb', line 43

def find_fancy(str, code=nil)
  name, opts = str ? str.split('?', 2) : [nil, '']

  # parse the options hash from a cgi-style string
  opts = CGI.parse(opts || '').map do |k, vals|
    [ k.to_sym, vals.empty? ? true : vals[0] ]
  end

  opts = Hash[opts]

  lexer_class = case name
  when 'guess', nil
    self.guess(:source => code, :mimetype => opts[:mimetype])
  when String
    self.find(name)
  end

  lexer_class && lexer_class.new(opts)
end

.guess(info = {}) ⇒ Object

Guess which lexer to use based on a hash of info.

Parameters:

  • info (Hash) (defaults to: {})

    a customizable set of options

Options Hash (info):

  • :mimetype (Object)

    A mimetype to guess by

  • :filename (Object)

    A filename to guess by

  • :source (Object)

    The source itself, which, if guessing by mimetype or filename fails, will be searched for shebangs, <!DOCTYPE …> tags, and other hints.

See Also:


104
105
106
107
108
109
110
111
112
113
114
115
116
# File 'lib/rouge/lexer.rb', line 104

def guess(info={})
  by_mimetype = guess_by_mimetype(info[:mimetype]) if info[:mimetype]
  return by_mimetype if by_mimetype

  by_filename = guess_by_filename(info[:filename]) if info[:filename]
  return by_filename if by_filename

  by_source = guess_by_source(info[:source]) if info[:source]
  return by_source if by_source

  # guessing failed, just parse it as text
  return Lexers::Text
end

.guess_by_filename(fname) ⇒ Object


124
125
126
127
128
129
130
131
# File 'lib/rouge/lexer.rb', line 124

def guess_by_filename(fname)
  fname = File.basename(fname)
  registry.values.detect do |lexer|
    lexer.filenames.any? do |pattern|
      File.fnmatch?(pattern, fname, File::FNM_DOTMATCH)
    end
  end
end

.guess_by_mimetype(mt) ⇒ Object


118
119
120
121
122
# File 'lib/rouge/lexer.rb', line 118

def guess_by_mimetype(mt)
  registry.values.detect do |lexer|
    lexer.mimetypes.include? mt
  end
end

.guess_by_source(source) ⇒ Object


133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
# File 'lib/rouge/lexer.rb', line 133

def guess_by_source(source)
  assert_utf8!(source)

  source = TextAnalyzer.new(source)

  best_result = 0
  best_match = nil
  registry.values.each do |lexer|
    result = lexer.analyze_text(source) || 0
    return lexer if result == 1

    if result > best_result
      best_match = lexer
      best_result = result
    end
  end

  best_match
end

.lex(stream, opts = {}, &b) ⇒ Object

Lexes `stream` with the given options. The lex is delegated to a new instance.

See Also:


14
15
16
# File 'lib/rouge/lexer.rb', line 14

def lex(stream, opts={}, &b)
  new(opts).lex(stream, &b)
end

.mimetypes(*mts) ⇒ Object

Specify a list of mimetypes associated with this lexer.

Examples:

class Html < Lexer
  mimetypes 'text/html', 'application/xhtml+xml'
end

206
207
208
# File 'lib/rouge/lexer.rb', line 206

def mimetypes(*mts)
  (@mimetypes ||= []).concat(mts)
end

.register(name, lexer) ⇒ Object


154
155
156
# File 'lib/rouge/lexer.rb', line 154

def register(name, lexer)
  registry[name.to_s] = lexer
end

.tag(t = nil) ⇒ Object

Used to specify or get the canonical name of this lexer class.

Examples:

class MyLexer < Lexer
  tag 'foo'
end

MyLexer.tag # => 'foo'

Lexer.find('foo') # => MyLexer

168
169
170
171
172
173
# File 'lib/rouge/lexer.rb', line 168

def tag(t=nil)
  return @tag if t.nil?

  @tag = t.to_s
  Lexer.register(@tag, self)
end

Instance Method Details

#debug(&b) ⇒ Object

Leave a debug message if the `:debug` option is set. The message is given as a block because some debug messages contain calculated information that is unnecessary for lexing in the real world.

Examples:

debug { "hello, world!" }

262
263
264
265
266
267
268
269
270
271
272
273
274
275
# File 'lib/rouge/lexer.rb', line 262

def debug(&b)
  # This method is a hotspot, unfortunately.
  #
  # For performance reasons, the "debug" option of a lexer cannot
  # be changed once it has begun lexing.  This method will redefine
  # itself on the first call to a noop if "debug" is not set.
  if option(:debug)
    def self.debug; puts yield; end
  else
    def self.debug; end
  end

  debug(&b)
end

#lex(string, opts = {}, &b) ⇒ Object

Given a string, yield [token, chunk] pairs. If no block is given, an enumerator is returned.

Parameters:

  • opts (Hash) (defaults to: {})

    a customizable set of options

Options Hash (opts):

  • :continue (Object)

    Continue the lex from the previous state (i.e. don't call #reset!)


289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
# File 'lib/rouge/lexer.rb', line 289

def lex(string, opts={}, &b)
  return enum_for(:lex, string) unless block_given?

  Lexer.assert_utf8!(string)

  reset! unless opts[:continue]

  # consolidate consecutive tokens of the same type
  last_token = nil
  last_val = nil
  stream_tokens(string) do |tok, val|
    next if val.empty?

    if tok == last_token
      last_val << val
      next
    end

    b.call(last_token, last_val) if last_token
    last_token = tok
    last_val = val
  end

  b.call(last_token, last_val) if last_token
end

#option(k, v = :absent) ⇒ Object

get or specify one option for this lexer


248
249
250
251
252
253
254
# File 'lib/rouge/lexer.rb', line 248

def option(k, v=:absent)
  if v == :absent
    options[k]
  else
    options({ k => v })
  end
end

#options(o = {}) ⇒ Object

get and/or specify the options for this lexer.


241
242
243
244
245
# File 'lib/rouge/lexer.rb', line 241

def options(o={})
  (@options ||= {}).merge!(o)

  self.class.default_options.merge(@options)
end

#reset!Object

This method is abstract.

Called after each lex is finished. The default implementation is a noop.


281
282
# File 'lib/rouge/lexer.rb', line 281

def reset!
end

#stream_tokens(stream, &b) ⇒ Object

This method is abstract.

Yield `[token, chunk]` pairs, given a prepared input stream. This must be implemented.

Parameters:

  • stream (StringScanner)

    the stream


327
328
329
# File 'lib/rouge/lexer.rb', line 327

def stream_tokens(stream, &b)
  raise 'abstract'
end

#tagObject

delegated to tag


316
317
318
# File 'lib/rouge/lexer.rb', line 316

def tag
  self.class.tag
end