CLexer

A C port of the whitequark/parser's lexer.

Initially based on whitequark/parser#248 (thanks @alexdowad).

Can be use on Linux/Mac/Windows, requires MRI.

Installation

$ gem install c_lexer

Usage

c_lexer doesn't change any parser classes.

It provides 3 classes:

Parser::CLexer (C version of the lexer)
Parser::Ruby25WithCLexer (a subclass of parser/ruby25 that uses Parser::CLexer for lexing)
Parser::Ruby26WithCLexer (a subclass of parser/ruby26 that uses Parser::CLexer for lexing)

If you want CLexer to be a default lexer you can use the following patch:

require 'parser'
require 'c_lexer'

module Parser
  dedenter = Lexer::Dedenter

  remove_const(:Lexer)
  Lexer = CLexer
  Lexer::Dedenter = dedenter

  remove_const(:Ruby26)
  Ruby26 = Ruby26WithCLexer
end

Versioning

c_lexer follows versioning of the parser gem to avoid confusions like "which version of c_lexer should be used with the parser vX.Y.Z". For parser 2.5.1.0 you should use c_lexer 2.5.1.0.X.

c_lexer supports all versions of parser starting from 2.5.1.0

Development

Make sure that you have ragel installed.
Clone the repo and fetch submodules with git submodule update --init.
The parser gem is located under vendor/parser.
rake ruby_parser:generate generates lexer.rb and parsers for the parser gem.
rake c_lexer:generate generates lexer.c.
rake compile compiles lexer.c to lexer.so (or lexer.bundle depending on your platform).
rake test runs parser tests using c_lexer as a default lexer.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/opal/c_lexer.

Before submitting a bug report, please, make sure that parser is not affected by the same issue. If it's a parser bug please report it to the parser repo and we will backport it afterwards.