Class: CodeRay::Scanners::Scanner
- Inherits:
-
StringScanner
- Object
- StringScanner
- CodeRay::Scanners::Scanner
- Extended by:
- Plugin
- Includes:
- Enumerable
- Defined in:
- lib/coderay/scanner.rb
Overview
Scanner
The base class for all Scanners.
It is a subclass of Ruby’s great StringScanner
, which makes it easy to access the scanning methods inside.
It is also Enumerable
, so you can use it like an Array of Tokens:
require 'coderay'
c_scanner = CodeRay::Scanners[:c].new "if (*p == '{') nest++;"
for text, kind in c_scanner
puts text if kind == :operator
end
# prints: (*==)++;
OK, this is a very simple example :) You can also use map
, any?
, find
and even sort_by
, if you want.
Direct Known Subclasses
C, CPlusPlus, CSS, Debug, Delphi, Diff, HTML, JSON, Java, JavaScript, NitroXHTML, Plaintext, Python, RHTML, Ruby, SQL, Scheme, YAML
Constant Summary collapse
- ScanError =
Raised if a Scanner fails while scanning
Class.new(Exception)
- DEFAULT_OPTIONS =
The default options for all scanner classes.
Define @default_options for subclasses.
{ :stream => false }
- KINDS_NOT_LOC =
[:comment, :doctype]
Class Method Summary collapse
- .file_extension(extension = nil) ⇒ Object
- .normify(code) ⇒ Object
-
.streamable? ⇒ Boolean
Returns if the Scanner can be used in streaming mode.
Instance Method Summary collapse
- #column(pos = self.pos) ⇒ Object
-
#each(&block) ⇒ Object
Traverses the tokens.
-
#initialize(code = '', options = {}, &block) ⇒ Scanner
constructor
Create a new Scanner.
-
#lang ⇒ Object
Returns the Plugin ID for this scanner.
-
#line ⇒ Object
The current line position of the scanner.
- #marshal_dump ⇒ Object
- #marshal_load(options) ⇒ Object
- #reset ⇒ Object
-
#streaming? ⇒ Boolean
Whether the scanner is in streaming mode.
- #string=(code) ⇒ Object (also: #code=)
-
#tokenize(new_string = nil, options = {}) ⇒ Object
Scans the code and returns all tokens in a Tokens object.
- #tokens ⇒ Object
Methods included from Plugin
helper, included, plugin_host, plugin_id, register_for, title
Constructor Details
#initialize(code = '', options = {}, &block) ⇒ Scanner
Create a new Scanner.
-
code
is the input String and is handled by the superclass StringScanner. -
options
is a Hash with Symbols as keys. It is merged with the default options of the class (you can overwrite default options here.) -
block
is the callback for streamed highlighting.
If you set :stream to true
in the options, the Scanner uses a TokenStream with the block
as callback to handle the tokens.
Else, a Tokens object is used.
120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 |
# File 'lib/coderay/scanner.rb', line 120 def initialize code='', = {}, &block raise "I am only the basic Scanner class. I can't scan "\ "anything. :( Use my subclasses." if self.class == Scanner @options = self.class::DEFAULT_OPTIONS.merge super Scanner.normify(code) @tokens = [:tokens] if @options[:stream] warn "warning in CodeRay::Scanner.new: :stream is set, "\ "but no block was given" unless block_given? raise NotStreamableError, self unless kind_of? Streamable @tokens ||= TokenStream.new(&block) else warn "warning in CodeRay::Scanner.new: Block given, "\ "but :stream is #{@options[:stream]}" if block_given? @tokens ||= Tokens.new end @tokens.scanner = self setup end |
Class Method Details
.file_extension(extension = nil) ⇒ Object
86 87 88 89 90 91 92 |
# File 'lib/coderay/scanner.rb', line 86 def file_extension extension = nil if extension @file_extension = extension.to_s else @file_extension ||= plugin_id.to_s end end |
.normify(code) ⇒ Object
69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 |
# File 'lib/coderay/scanner.rb', line 69 def normify code code = code.to_s if code.respond_to?(:encoding) && (code.encoding.name != 'UTF-8' || !code.valid_encoding?) code = code.dup original_encoding = code.encoding code.force_encoding 'Windows-1252' unless code.valid_encoding? code.force_encoding original_encoding if code.encoding.name == 'UTF-8' code.encode! 'UTF-16BE', :invalid => :replace, :undef => :replace, :replace => '?' end code.encode! 'UTF-8', :invalid => :replace, :undef => :replace, :replace => '?' end end code.to_unix end |
.streamable? ⇒ Boolean
Returns if the Scanner can be used in streaming mode.
65 66 67 |
# File 'lib/coderay/scanner.rb', line 65 def streamable? is_a? Streamable end |
Instance Method Details
#column(pos = self.pos) ⇒ Object
208 209 210 211 212 213 214 215 216 |
# File 'lib/coderay/scanner.rb', line 208 def column pos = self.pos return 0 if pos <= 0 string = string() if string.respond_to?(:bytesize) && (defined?(@bin_string) || string.bytesize != string.size) @bin_string ||= string.dup.force_encoding('binary') string = @bin_string end pos - (string.rindex(?\n, pos) || 0) end |
#each(&block) ⇒ Object
Traverses the tokens.
193 194 195 196 197 |
# File 'lib/coderay/scanner.rb', line 193 def each &block raise ArgumentError, 'Cannot traverse TokenStream.' if @options[:stream] tokens.each(&block) end |
#lang ⇒ Object
Returns the Plugin ID for this scanner.
165 166 167 |
# File 'lib/coderay/scanner.rb', line 165 def lang self.class.plugin_id end |
#line ⇒ Object
The current line position of the scanner.
Beware, this is implemented inefficiently. It should be used for debugging only.
204 205 206 |
# File 'lib/coderay/scanner.rb', line 204 def line string[0..pos].count("\n") + 1 end |
#marshal_dump ⇒ Object
218 219 220 |
# File 'lib/coderay/scanner.rb', line 218 def marshal_dump @options end |
#marshal_load(options) ⇒ Object
222 223 224 |
# File 'lib/coderay/scanner.rb', line 222 def marshal_load @options = end |
#reset ⇒ Object
144 145 146 147 |
# File 'lib/coderay/scanner.rb', line 144 def reset super reset_instance end |
#streaming? ⇒ Boolean
Whether the scanner is in streaming mode.
188 189 190 |
# File 'lib/coderay/scanner.rb', line 188 def streaming? !!@options[:stream] end |
#string=(code) ⇒ Object Also known as: code=
149 150 151 152 153 154 155 156 157 158 |
# File 'lib/coderay/scanner.rb', line 149 def string= code code = Scanner.normify(code) if defined?(RUBY_DESCRIPTION) && RUBY_DESCRIPTION['rubinius 1.0.1'] reset_state @string = code else super code end reset_instance end |
#tokenize(new_string = nil, options = {}) ⇒ Object
Scans the code and returns all tokens in a Tokens object.
170 171 172 173 174 175 176 177 178 179 180 181 |
# File 'lib/coderay/scanner.rb', line 170 def tokenize new_string=nil, = {} = @options.merge() self.string = new_string if new_string @cached_tokens = if @options[:stream] # :stream must have been set already reset unless new_string scan_tokens @tokens, @tokens else scan_tokens @tokens, end end |
#tokens ⇒ Object
183 184 185 |
# File 'lib/coderay/scanner.rb', line 183 def tokens @cached_tokens ||= tokenize end |