Class: Iriq::CLI

Inherits:
Object
  • Object
show all
Defined in:
lib/iriq/cli.rb

Overview

Flag-driven CLI. The default action for an input is a combined parse + normalize + explain summary; the -p/-n/-e flags select individual sections. The only subcommand is cluster, which is structurally different (many inputs, not one). Construct with explicit IO so specs can run it without shelling out.

Defined Under Namespace

Classes: InputError

Constant Summary collapse

SECTION_FLAGS = 
i[parse normalize].freeze
TOP_N_STATS = 
10
LARGE_BATCH_THRESHOLD =

When extraction yields this many or more IRIs, the default pipe output switches from a URL list to clusters — a longer list is easier to read as route-shape groups.

10
USAGE = 
"iriq \u2014 find a URL's shape: the route template behind it (e.g. /users/{id}).\n\nUsage: iriq [options] <input>\n       iriq [options] < text\n       iriq cluster [options] [file]\n\n<input> may be an IRI, a file to extract IRIs from (an existing file wins\nunless the argument contains ://), or piped text via stdin.\n\nSections (combine freely):\n  -n, --normalize       Shape \u2014 variable parts become placeholders\n  -c, --canonical       Clean form \u2014 tidy scheme/host, keep the values\n  -p, --parse           Parsed fields\n  -e, --explain         Annotated trace \u2014 per-segment notes about why\n                        each placeholder / canonical value was chosen\n                        (mechanical rules only, even with a corpus)\n\nCorpus + stats:\n      --corpus PATH     Use a specific corpus file (overrides the default).\n                        Extension picks the backend: .db/.sqlite/.sqlite3\n                        are SQLite; anything else is JSON.\n  -C, --no-corpus       Disable corpus persistence for this invocation.\n                        Same as IRIQ_NO_CORPUS=1 in the environment.\n      --reset           Delete the corpus (default path or the one\n                        resolved via --corpus / IRIQ_CORPUS), its SQLite\n                        sidecars and JSON temp files, and exit.\n      --host MODE       Host-keying strategy for clustering:\n                        full (default), registrable (or reg) strips\n                        subdomains, none ignores host entirely. Keys\n                        IRIs as they're observed (-C included) and on\n                        --reinfer; existing clusters keep their keys.\n      --stats           Print rolling aggregates\n      --reinfer         Replay the source-IRI log through the current\n                        classifier + reducers; rebuilds materialized\n                        views from scratch.\n      --propose-recognizers\n                        Scan observed values for shape patterns that\n                        recur enough to suggest a new Recognizer.\n                        Combine with --json for structured output.\n      --cross-host-shapes\n                        List route shapes that recur across\n                        multiple hosts. Combine with --min-hosts.\n      --activate-above F  With --propose-recognizers, promote every\n                        proposal at or above CONFIDENCE F into a\n                        live Recognizer on the corpus, then\n                        reinfer. Confidence integrates coverage\n                        and cross-host corroboration.\n\nEnvironment:\n      IRIQ_CORPUS=PATH    Set the corpus path (overrides the default).\n      IRIQ_NO_CORPUS=1    Disable the default corpus (equivalent to -C).\n\nThresholds (apply to --propose-recognizers / --cross-host-shapes):\n      --min-observations N  proposal noise floor (default 20)\n      --min-coverage F      proposal coverage floor (default 0.7)\n      --min-hosts N         proposal: minimum hosts (default 1);\n                            cross-host-shapes: minimum hosts to\n                            list (default 2)\n\nOther:\n  -h, --help            Show this message\n  -j, --json            Emit JSON instead of human-readable output\n  -J, --ndjson          Newline-delimited JSON (one object per line). Implies --json.\n                        Streams per IRI only with -n/-p/-c/-e; alone, it\n                        prints the URL list or clusters at end of input.\n  -N, --no-hints        Use {integer} placeholders instead of {user_id}\n      --no-scheme-less  Skip foo.com/path extraction (explicit-scheme only)\n  -V, --version         Print version\n\nSubcommands:\n  cluster [file]        Observe file (or stdin), then show every cluster\n                        in the corpus (default view for \u226510 IRIs anyway)\n  completion <shell>    Print shell completion script (bash | zsh)\n\nExamples:\n  iriq foo.com/users/456\n  iriq -n https://foo.com/users/123\n  iriq access.log                       # extract URLs (request paths have no host)\n  cat README.md | iriq -n               # one normalized URL per line\n  tail -f app.log | iriq -nJ            # live stream \u2192 NDJSON per IRI\n  cat README.md | iriq --corpus c.json\n"

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(stdin: $stdin, stdout: $stdout, stderr: $stderr) ⇒ CLI

Returns a new instance of CLI.



107
108
109
110
111
 
# File 'lib/iriq/cli.rb', line 107

def initialize(stdin: $stdin, stdout: $stdout, stderr: $stderr)
  @stdin  = stdin
  @stdout = stdout
  @stderr = stderr
end

Instance Attribute Details

#stderr ⇒ Object (readonly)

Returns the value of attribute stderr.



105
106
107
 
# File 'lib/iriq/cli.rb', line 105

def stderr
  @stderr
end

#stdin ⇒ Object (readonly)

Returns the value of attribute stdin.



105
106
107
 
# File 'lib/iriq/cli.rb', line 105

def stdin
  @stdin
end

#stdout ⇒ Object (readonly)

Returns the value of attribute stdout.



105
106
107
 
# File 'lib/iriq/cli.rb', line 105

def stdout
  @stdout
end

Instance Method Details

#parseable_iri?(input) ⇒ Boolean

Returns:

  • (Boolean) 


195
196
197
198
199
200
 
# File 'lib/iriq/cli.rb', line 195

def parseable_iri?(input)
  Iriq.parse(input)
  true
rescue Iriq::ParseError
  false
end

#run(argv) ⇒ Object

Returns an integer exit code.



114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
 
# File 'lib/iriq/cli.rb', line 114

def run(argv)
  # Pre-scan so an error during option parsing can still honor --json.
  # Re-set authoritatively from opts once parsing succeeds.
  @json = json_requested?(argv)
  args, opts = parse_options(argv)
  @json = opts[:json]

  return print_usage(stdout, 0) if opts[:help]
  return print_version          if opts[:version]

  # `iriq completion <shell>` short-circuits — no corpus, no IRI input,
  # just emit the script bundled with the gem.
  if args.first == "completion"
    args.shift
    return cmd_completion(args)
  end

  explicit_cluster = (args.first == "cluster")
  args.shift if explicit_cluster

  # A positional that names an existing file is read as a file, so
  # `iriq access.log` works without ./ (it also parses as a host). An
  # argument containing "://" is always an IRI.
  positional_is_file = args.first && !args.first.include?("://") && File.file?(args.first)

  batch_mode = explicit_cluster || positional_is_file ||
               (args.empty? && piped_stdin?)

  # --reset short-circuits: delete the resolved corpus file (+ sidecars)
  # and exit. Resolves through the same precedence chain as the normal
  # path so `--reset --corpus other.db` and `IRIQ_CORPUS=… --reset` Just Work.
  if opts[:reset]
    return cmd_reset(opts)
  end

  # Before any corpus is opened, so a typo doesn't create one.
  if (missing = missing_input_file(args.first, explicit_cluster))
    raise InputError.new("file_not_found", "no such file: #{missing}")
  end

  return print_usage(stdout, 0) if args.empty? && !batch_mode && !opts[:reinfer] && !opts[:propose] && !opts[:cross_host_shapes]

  corpus_path = resolve_corpus_path(opts)
  corpus = corpus_path ? load_corpus(corpus_path, host_strategy: opts[:host_strategy], announce_create: true) : nil

  code = if opts[:reinfer]
    cmd_reinfer(corpus, opts)
  elsif opts[:propose]
    cmd_propose(corpus, opts)
  elsif opts[:cross_host_shapes]
    cmd_cross_host_shapes(corpus, opts)
  elsif batch_mode
    cmd_batch(args, opts, corpus, explicit_cluster: explicit_cluster)
  elsif opts[:stats]
    cmd_stats(corpus, opts)
  else
    cmd_summary(args, opts, corpus)
  end

  corpus.save(corpus_path) if corpus && corpus_path
  code
rescue Iriq::ParseError => e
  emit_error("parse_error", e.message, 2, human: "iriq: parse error: #{e.message}")
rescue OptionParser::ParseError => e
  emit_error("option_error", e.message, 1)
rescue InputError => e
  emit_error(e.code, e.message, 1)
rescue Iriq::CorpusError => e
  emit_error("corpus_error", e.message, 1)
end