Class: YARD::Parser::SourceParser

Inherits:
Object
  • Object
show all
Defined in:
lib/yard/parser/source_parser.rb

Overview

Responsible for parsing a source file into the namespace. Parsing also invokes handlers to process the parsed statements and generate any code objects that may be recognized.

Custom Parsers

SourceParser allows custom parsers to be registered and called when a certain filetype is recognized. To register a parser and hook it up to a set of file extensions, call SourceParser.register_parser_type

Constant Summary collapse

SHEBANG_LINE =
/\A\s*#!\S+/
ENCODING_LINE =
/\A(?:\s*#*!.*\r?\n)?\s*(?:#+|\/\*+|\/\/+).*coding\s*[:=]{1,2}\s*([a-z\d_\-]+)/i
FROZEN_STRING_LINE =
/frozen(-|_)string(-|_)literal: true/i
DEFAULT_PATH_GLOB =

The default glob of files to be parsed.

Since:

  • 0.9.0

["{lib,app}/**/*.rb", "ext/**/*.{c,cc,cxx,cpp}"]
ENCODING_BYTE_ORDER_MARKS =

Byte order marks for various encodings

Since:

  • 0.7.0

{
  'utf-8' => "\xEF\xBB\xBF",
  # Not yet supported
  #'utf-16be' => "\xFE\xFF",
  #'utf-16le' => "\xFF\xFE",
  #'utf-32be' => "\x00\x00\xFF\xFE",
  #'utf-32le' => "\xFF\xFE",
}

Parser Callbacks collapse

Class Attribute Summary collapse

Parser Callbacks collapse

Class Method Summary collapse

Constructor Details

#initialize(parser_type = SourceParser.parser_type, globals = nil) ⇒ SourceParser

Returns a new instance of SourceParser

Creates a new parser object for code parsing with a specific parser type.


404
405
406
407
408
409
# File 'lib/yard/parser/source_parser.rb', line 404

def initialize(parser_type = SourceParser.parser_type, globals1 = nil, globals2 = nil)
  globals = [true, false].include?(globals1) ? globals2 : globals1
  @file = '(stdin)'
  @globals = globals || OpenStruct.new
  self.parser_type = parser_type
end

Class Attribute Details

.parser_typeSymbol


84
85
86
# File 'lib/yard/parser/source_parser.rb', line 84

def parser_type
  @parser_type
end

Instance Attribute Details

#contentsString (readonly)

Returns the contents of the file to be parsed

Since:

  • 0.7.0


397
398
399
# File 'lib/yard/parser/source_parser.rb', line 397

def contents
  @contents
end

#fileString


384
385
386
# File 'lib/yard/parser/source_parser.rb', line 384

def file
  @file
end

#globalsOpenStruct (readonly)

Returns an open struct containing arbitrary global state shared between files and handlers.

Since:

  • 0.7.0


393
394
395
# File 'lib/yard/parser/source_parser.rb', line 393

def globals
  @globals
end

#parser_typeSymbol


388
389
390
# File 'lib/yard/parser/source_parser.rb', line 388

def parser_type
  @parser_type
end

Class Method Details

.after_parse_file {|parser| ... } ⇒ Proc

Registers a callback to be called after an individual file is parsed. The block passed to this method will be called on subsequent parse calls.

To register a callback that is called after the entire list of files is processed, see after_parse_list.

Examples:

Printing the length of each file after it is parsed

SourceParser.after_parse_file do |parser|
  puts "#{parser.file} is #{parser.contents.size} characters"
end
YARD.parse('lib/**/*.rb')
# prints:
"lib/foo.rb is 1240 characters"
"lib/foo_bar.rb is 248 characters"

Yields:

  • (parser)

    the yielded block is called once after each file that is parsed. This might happen many times for a single codebase.

Yield Parameters:

  • parser (SourceParser)

    the parser object that parsed the file.

Yield Returns:

  • (void)

    the return value for the block is ignored.

See Also:

Since:

  • 0.7.0


322
323
324
# File 'lib/yard/parser/source_parser.rb', line 322

def after_parse_file(&block)
  after_parse_file_callbacks << block
end

.after_parse_file_callbacksArray<Proc>

Returns the list of callbacks to be called after parsing a file. Should only be used for testing.

Since:

  • 0.7.0


350
351
352
# File 'lib/yard/parser/source_parser.rb', line 350

def after_parse_file_callbacks
  @after_parse_file_callbacks ||= []
end

.after_parse_list {|files, globals| ... } ⇒ Proc

Registers a callback to be called after a list of files is parsed via parse. The block passed to this method will be called on subsequent parse calls.

Examples:

Printing results after parsing occurs

SourceParser.after_parse_list do
  puts "Finished parsing!"
end
YARD.parse
# Prints "Finished parsing!" after parsing files

Yields:

  • (files, globals)

    the yielded block is called once before parsing all files

Yield Parameters:

Yield Returns:

  • (void)

    the return value for the block is ignored.

See Also:

Since:

  • 0.7.0


256
257
258
# File 'lib/yard/parser/source_parser.rb', line 256

def after_parse_list(&block)
  after_parse_list_callbacks << block
end

.after_parse_list_callbacksArray<Proc>

Returns the list of callbacks to be called after parsing a list of files. Should only be used for testing.

Since:

  • 0.7.0


336
337
338
# File 'lib/yard/parser/source_parser.rb', line 336

def after_parse_list_callbacks
  @after_parse_list_callbacks ||= []
end

.before_parse_file {|parser| ... } ⇒ Proc

Registers a callback to be called before an individual file is parsed. The block passed to this method will be called on subsequent parse calls.

To register a callback that is called before the entire list of files is processed, see before_parse_list.

Examples:

Installing a simple callback

SourceParser.before_parse_file do |parser|
  puts "I'm parsing #{parser.file}"
end
YARD.parse('lib/**/*.rb')
# prints:
"I'm parsing lib/foo.rb"
"I'm parsing lib/foo_bar.rb"
"I'm parsing lib/last_file.rb"

Cancel parsing of any test_*.rb files

SourceParser.before_parse_file do |parser|
  return false if parser.file =~ /^test_.+\.rb$/
end

Yields:

  • (parser)

    the yielded block is called once before each file that is parsed. This might happen many times for a single codebase.

Yield Parameters:

Yield Returns:

  • (Boolean)

    if the block returns false, parsing for the file is cancelled.

See Also:

Since:

  • 0.7.0


293
294
295
# File 'lib/yard/parser/source_parser.rb', line 293

def before_parse_file(&block)
  before_parse_file_callbacks << block
end

.before_parse_file_callbacksArray<Proc>

Returns the list of callbacks to be called before parsing a file. Should only be used for testing.

Since:

  • 0.7.0


343
344
345
# File 'lib/yard/parser/source_parser.rb', line 343

def before_parse_file_callbacks
  @before_parse_file_callbacks ||= []
end

.before_parse_list {|files, globals| ... } ⇒ Proc

Registers a callback to be called before a list of files is parsed via parse. The block passed to this method will be called on subsequent parse calls.

Examples:

Installing a simple callback

SourceParser.before_parse_list do |files, globals|
  puts "Starting to parse..."
end
YARD.parse('lib/**/*.rb')
# prints "Starting to parse..."

Setting global state

SourceParser.before_parse_list do |files, globals|
  globals.method_count = 0
end
SourceParser.after_parse_list do |files, globals|
  puts "Found #{globals.method_count} methods"
end
class MyCountHandler < Handlers::Ruby::Base
  handles :def, :defs
  process { globals.method_count += 1 }
end
YARD.parse
# Prints: "Found 37 methods"

Using a global callback to cancel parsing

SourceParser.before_parse_list do |files, globals|
  return false if files.include?('foo.rb')
end

YARD.parse(['foo.rb', 'bar.rb']) # callback cancels this method
YARD.parse('bar.rb') # parses normally

Yields:

  • (files, globals)

    the yielded block is called once before parsing all files

Yield Parameters:

Yield Returns:

  • (Boolean)

    if the block returns false, parsing is cancelled.

See Also:

Since:

  • 0.7.0


232
233
234
# File 'lib/yard/parser/source_parser.rb', line 232

def before_parse_list(&block)
  before_parse_list_callbacks << block
end

.before_parse_list_callbacksArray<Proc>

Returns the list of callbacks to be called before parsing a list of files. Should only be used for testing.

Since:

  • 0.7.0


329
330
331
# File 'lib/yard/parser/source_parser.rb', line 329

def before_parse_list_callbacks
  @before_parse_list_callbacks ||= []
end

.parse(paths = DEFAULT_PATH_GLOB, excluded = [], level = log.level) ⇒ void

This method returns an undefined value.

Parses a path or set of paths


98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
# File 'lib/yard/parser/source_parser.rb', line 98

def parse(paths = DEFAULT_PATH_GLOB, excluded = [], level = log.level)
  log.debug("Parsing #{paths.inspect} with `#{parser_type}` parser")
  excluded = excluded.map do |path|
    case path
    when Regexp; path
    else Regexp.new(path.to_s, Regexp::IGNORECASE)
    end
  end
  files = [paths].flatten.
    map {|p| File.directory?(p) ? "#{p}/**/*.{rb,c,cc,cxx,cpp}" : p }.
    map {|p| p.include?("*") ? Dir[p].sort_by {|f| f.length } : p }.flatten.
    reject {|p| !File.file?(p) || excluded.any? {|re| p =~ re } }

  log.enter_level(level) do
    parse_in_order(*files.uniq)
  end
end

.parse_string(content, ptype = parser_type) ⇒ Object

Parses a string content


121
122
123
# File 'lib/yard/parser/source_parser.rb', line 121

def parse_string(content, ptype = parser_type)
  new(ptype).parse(StringIO.new(content))
end

.parser_type_extensions=(value) ⇒ Object


162
# File 'lib/yard/parser/source_parser.rb', line 162

def parser_type_extensions=(value) @@parser_type_extensions = value end

.parser_type_for_extension(extension) ⇒ Symbol

Finds a parser type that is registered for the extension. If no type is found, the default Ruby type is returned.

Since:

  • 0.5.6


169
170
171
172
173
174
# File 'lib/yard/parser/source_parser.rb', line 169

def parser_type_for_extension(extension)
  type = parser_type_extensions.find do |t, exts|
    [exts].flatten.any? {|ext| ext === extension }
  end
  validated_parser_type(type ? type.first : :ruby)
end

.parser_types=(value) ⇒ Object


156
# File 'lib/yard/parser/source_parser.rb', line 156

def parser_types=(value) @@parser_types = value end

.register_parser_type(type, parser_klass, extensions = nil) ⇒ void

This method returns an undefined value.

Registers a new parser type.

Examples:

Registering a parser for “java” files

SourceParser.register_parser_type :java, JavaParser, 'java'

See Also:


144
145
146
147
148
149
150
# File 'lib/yard/parser/source_parser.rb', line 144

def register_parser_type(type, parser_klass, extensions = nil)
  unless Base > parser_klass
    raise ArgumentError, "expecting parser_klass to be a subclass of YARD::Parser::Base"
  end
  parser_type_extensions[type.to_sym] = extensions if extensions
  parser_types[type.to_sym] = parser_klass
end

.tokenize(content, ptype = parser_type) ⇒ Array

Tokenizes but does not parse the block of code


130
131
132
# File 'lib/yard/parser/source_parser.rb', line 130

def tokenize(content, ptype = parser_type)
  new(ptype).tokenize(content)
end

Instance Method Details

#parse(content = __FILE__) ⇒ Object?

The main parser method. This should not be called directly. Instead, use the class methods parse and parse_string.


416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
# File 'lib/yard/parser/source_parser.rb', line 416

def parse(content = __FILE__)
  case content
  when String
    @file = File.cleanpath(content)
    content = convert_encoding(File.read_binary(file))
    checksum = Registry.checksum_for(content)
    return if Registry.checksums[file] == checksum

    if Registry.checksums.has_key?(file)
      log.info "File '#{file}' was modified, re-processing..."
    end
    Registry.checksums[@file] = checksum
    self.parser_type = parser_type_for_filename(file)
  else
    content = content.read if content.respond_to? :read
  end

  @contents = content
  @parser = parser_class.new(content, file)

  self.class.before_parse_file_callbacks.each do |cb|
    return @parser if cb.call(self) == false
  end

  @parser.parse
  post_process

  self.class.after_parse_file_callbacks.each do |cb|
    cb.call(self)
  end

  @parser
rescue ArgumentError, NotImplementedError => e
  log.warn("Cannot parse `#{file}': #{e.message}")
  log.backtrace(e, :warn)
rescue ParserSyntaxError => e
  log.warn(e.message.capitalize)
  log.backtrace(e, :warn)
end

#tokenize(content) ⇒ Array

Tokenizes but does not parse the block of code using the current #parser_type


460
461
462
463
# File 'lib/yard/parser/source_parser.rb', line 460

def tokenize(content)
  @parser = parser_class.new(content, file)
  @parser.tokenize
end