Module: Drudge::Parsers::Tokenizer

Extended by:
Tokenizer
Included in:
ArgumentParser, Tokenizer
Defined in:
lib/drudge/parsers/tokenizer.rb

Overview

tokenization of commandline arguments.

Instance Method Summary collapse

Instance Method Details

#tokenize(argv) ⇒ Object

tokenizes the arg-v list into an array of sexps the sexps are then suitable for the Drudge::parsers parser combinators



11
12
13
14
15
 
# File 'lib/drudge/parsers/tokenizer.rb', line 11

def tokenize(argv)
  argv.map.with_index do |arg, index|
    [:val, arg, loc(index, arg.length)]
  end
end

#underline_token(input, token, underline_char: '~') ⇒ Object

produces a string that underlines a specific token if no token is provided, the end of string is underlined



30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
 
# File 'lib/drudge/parsers/tokenizer.rb', line 30

def underline_token(input, token, underline_char: '~')
  line                 = untokenize(input)

  if token
    _, _, meta         = token
    location           = meta[:loc]
    _, _, token_length = location
    white_space        = index_of_sexp_in_untokenized(input, location)
  else
    white_space        = line.length + 1
    token_length       = 1
    underline_char     = '^'
  end

  " " * white_space + underline_char * token_length
end

#untokenize(sexps) ⇒ Object

given an array of sexps (as returned by tokenize) produce a string representatio of that



19
20
21
22
23
24
25
26
 
# File 'lib/drudge/parsers/tokenizer.rb', line 19

def untokenize(sexps)
  sexps.map do |type, arg, *_|
    case type
    when :val
      arg
    end
  end.join(" ")
end