Module: ToknInternal

Included in:
Tokn::DFA
Defined in:
lib/tokn/tokn_const.rb,
lib/tokn/state.rb,
lib/tokn/code_set.rb,
lib/tokn/reg_parse.rb,
lib/tokn/dfa_builder.rb,
lib/tokn/range_partition.rb,
lib/tokn/token_defn_parser.rb

Overview

Namespace to encompass the portions of the Tokn gem used only internally

Defined Under Namespace

Classes: CodeSet, DFABuilder, ParseException, RPNode, RangePartition, RegParse, State, TokenDefParser

Constant Summary collapse

UNKNOWN_TOKEN =

Token id if text didn’t match any tokens in the DFA

-1
EPSILON =

Code for epsilon transitions

-1
CODEMAX =

One plus the maximum code represented

0x110000
CODEMIN =

Minimum code possible (e.g., indicating a token id)

-10000

Class Method Summary collapse

Class Method Details

.edgeLabelToTokenId(edgeLabel) ⇒ Object

Convert an edge label value ( < 0) to a token id (>=0)



31
32
33
 
# File 'lib/tokn/tokn_const.rb', line 31

def self.edgeLabelToTokenId(edgeLabel)
  EPSILON-1-edgeLabel
end

.tokenIdToEdgeLabel(tokenId) ⇒ Object

Convert a token id (>=0) to an edge label value ( < 0)



25
26
27
 
# File 'lib/tokn/tokn_const.rb', line 25

def self.tokenIdToEdgeLabel(tokenId)
  EPSILON-1-tokenId  
end