Class: ToknInternal::TokenDefParser

Inherits:
Object
  • Object
show all
Defined in:
lib/tokn/token_defn_parser.rb

Overview

Parses a token definition script, and generates an NFA that is capable of recognizing and distinguishing between the various tokens.

Each line in the script is one of

# ...comment... (the # must appear as the first character in the line)

<tokenname> ':' <regex>

A <tokenname> must be an ‘identifier’ (alphanumeric, with first character a letter (or ‘_’)). If the first character is ‘_’, the token is treated as an ‘anonymous’ token; these can appear in the curly brace portions of previous reg. expr. entries, but do not appear as tokens in the generated NFA.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(script, createPDF = false) ⇒ TokenDefParser

Compile a token definition script into a DFA



28
29
30
31
32
33
34
 
# File 'lib/tokn/token_defn_parser.rb', line 28

def initialize(script, createPDF = false)  
  @script = script 
  parseScript
  if createPDF
    dfa.startState.generatePDF("tokenizer_dfa")
  end
end

Instance Attribute Details

#dfa ⇒ Object (readonly)

Returns the value of attribute dfa.



24
25
26
 
# File 'lib/tokn/token_defn_parser.rb', line 24

def dfa
  @dfa
end