Iriq — IRI Query

Iriq finds the shape of a URL — the structural template you get when you erase the parts that vary and keep the parts that don't. …/users/123 and …/users/999 are the same shape: /users/{user_id}. Feed iriq a pile of messy URLs — a log file, a column of links, free-text prose — and it collapses them into a small set of stable, deterministic route templates. Fifty thousand distinct URLs become twelve shapes.

What's an IRI? Internationalized Resource Identifiers cover everyday URLs https://…, plus URNs like urn:isbn:0451450523, other schemes like mailto:, and internationalized addresses with non-ASCII characters like https://例え.jp/パス. Formally it's the Unicode superset of URI/URL. The name is IRI Query: iriq queries an IRI for its structure.

Everything iriq does — parsing, normalizing, classifying path and query components, clustering, learning new patterns — exists to derive, render, or group by that shape.

And it gets sharper the more you feed it. A corpus — on by default — records what it sees and improves classifications as data flows in: high-churn slots get promoted to placeholders, and whole types emerge that no single URL can reveal (a position that's always 100–599 is an HTTP status; one bounded to a dozen values is an enum).

$ iriq -n https://foo.com/users/123
https://foo.com/users/{user_id}

It answers questions like:

"What routes does this service actually expose?" (cluster a log file)
"Which params are stable identifiers vs. churning IDs vs. enums?" (--stats)
"Are these 50,000 distinct URLs really just 12 templates?" (clustering)
"What does /api/v1/users/abc-123-def become as a route shape?" (/api/{version}/users/{user_id})

Iriq ships as a command-line tool (iriq) and a Rust library.

Quick start

$ iriq https://foo.com/users/123
# parse
original:      https://foo.com/users/123
kind:          url
scheme:        https
host:          foo.com
path_segments: ["users", "123"]
canonical:     https://foo.com/users/123

# normalize
https://foo.com/users/{user_id}

$ iriq -n https://foo.com/users/123
https://foo.com/users/{user_id}

$ iriq -n https://shop.com/pricing/usd?currency=eur
https://shop.com/pricing/USD?currency=EUR     # currency upcased

Pipe in text, or name a file, and iriq extracts every URL in it:

$ cat urls.log | iriq                         # ≥ 10 IRIs → cluster view
[6] api.example.com  /api/{version}/users/{user_id}
    https://api.example.com/api/v1/users/123
    https://api.example.com/api/v1/users/456
    https://api.example.com/api/v1/users/789
    + 3 more

[3] api.example.com  /orders/{order_uuid}
    https://api.example.com/orders/5f0c6a52-8b2e-4c1a-9f3d-2e7b1c9a0d11?status=open
    https://api.example.com/orders/0a1b2c3d-4e5f-4a6b-8c7d-9e0f1a2b3c4d?status=closed
    https://api.example.com/orders/7c9e6679-7425-40de-944b-e07fc1f90ae7?status=open
    status  string  conf 0.17  (2 distinct, 100%)

[3] api.example.com  /products/{product_id}
    https://api.example.com/products/blue-widget
    https://api.example.com/products/red-gadget
    https://api.example.com/products/green-gizmo

$ cat urls.log | iriq --stats                 # rolling aggregates
$ iriq urls.log -n                            # a file argument → normalize each URL
$ iriq -nJ < urls.log                         # one JSON line per URL
$ iriq --corpus team.db < urls.log            # use a specific corpus file

Reading a web server's access log? Its request lines have no host, so see Access logs first.

Per-IRI sections (-n, -c, -p, -e) stream: each line is read, observed, rendered from the corpus as it stands, and flushed, so iriq works on an unbounded live feed:

$ tail -f app.log | iriq -n                   # one shape per line, as logs land
$ tail -f app.log | iriq -nJ                  # same, as newline-delimited JSON

-J on its own doesn't stream. Like the default view, it waits for the end of input, then prints the URL list (fewer than 10 IRIs) or one object per cluster.

Every invocation observes into a persistent corpus by default, so iriq gets smarter the more you run it. The corpus-only types (e.g. enum / http_status) emerge from the distribution of values observed.

# Feed a stream where ?status only ever holds a couple of words:
$ for n in $(seq 1 20); do
    iriq --corpus demo.db "https://api.foo.com/orders/$n?status=open"   >/dev/null
    iriq --corpus demo.db "https://api.foo.com/orders/$n?status=closed" >/dev/null
  done

# Ask what it learned. ?status is now an enum — a verdict no single URL
# could support, since one URL shows only one value:
$ iriq --corpus demo.db cluster
[40] api.foo.com  /orders/{order_id}
    https://api.foo.com/orders/1?status=open
    https://api.foo.com/orders/1?status=closed
    https://api.foo.com/orders/2?status=open
    + 37 more
    status  enum  conf 0.73  (2 distinct, 100%)

conf is how much evidence backs the type, from 0 to 1. These learned types also flow into normalized output:

$ iriq --corpus demo.db -n 'https://api.foo.com/orders/99?status=open'
https://api.foo.com/orders/{order_id}?status={enum}

The corpus only acts on evidence: it changes a shape only at a position or param it has seen at least 5 times. Until then, -n prints exactly what -C would. Dates and currencies always print canonically (2024-01-15, USD).

The default corpus lives at $XDG_DATA_HOME/iriq/default.db, or ~/.local/share/iriq/default.db when XDG_DATA_HOME is unset (macOS and Linux alike; %LOCALAPPDATA%/iriq/default.db on Windows). First-run creation prints a one-line stderr notice. Three knobs control it:

$ iriq --no-corpus -n https://foo.com/users/123    # one-shot ephemeral; or -C
$ IRIQ_NO_CORPUS=1 iriq -n https://foo.com/users/123  # globally disable
$ IRIQ_CORPUS=/path/to/work.db iriq -n https://foo.com/users/123  # override path
$ iriq --corpus team.db https://foo.com/users/123  # explicit override (wins over env)
$ iriq --reset                                     # delete the corpus and exit

Two ways to normalize

Pick by the question you're asking:

--canonical — clean up this URL, keeping the specifics. HTTP://Foo.com:80/pull/42 → http://foo.com/pull/42 (scheme/host lowercased, default port dropped; path and query left alone). Handy, but table stakes — plenty of libraries do it.
--normalize (the default) — find the URL's shape, erasing the specifics into placeholders. …/pull/42 → …/pull/{pull_id}. A shape ignores the #fragment, so -n drops it. This is the part you came to iriq for.

Same input, two questions: "what's the clean form of this URL?" vs "what kind of URL is this?" The second is iriq's reason to exist.

Install

# Homebrew (recommended)
brew install dpep/tools/iriq

# Cargo, from crates.io (Rust 1.85 or newer)
cargo install iriq

One crate ships both the library and the iriq binary. Corpora persist to SQLite (bundled, WAL) out of the box — nothing to flag, install, or rebuild.

Segment classification

Iriq classifies each path/query segment into one of ~25 types — the first matching rule wins, and heuristics are deterministic:

literal — plain word (users, orders, Profile, こんにちは)
integer — pure digits below the timestamp range
float — decimal with digits on both sides (3.14, -2.5, 1.0)
boolean — true / false (any case)
version — semver-ish with v prefix (v1, v2.0.1, v1.2.3-beta)
locale — BCP 47-ish (en-US, fr_CA, zh-Hant, bare en/fr/ja)
currency — ISO 4217 codes (USD, EUR, JPY)
uuid — f47ac10b-58cc-4372-a567-0e02b2c3d479
date — 2024-05-23, 2024/05/23, 20240523, 05/23/2024. Canonicalized to ISO in --normalize output.
timestamp — ISO 8601, or 10/13-digit UNIX epoch
hash — 32+ hex chars (md5 / sha)
slug — my-cool-post, my_cool_post
ipv4 / ipv6 — collapsed to {ip} in normalized output
url — https://..., ftp://..., also scheme-less foo.com/path
email — [email protected]
phone — E.164 (+15551234567) or NANP (555-666-7777, (555) 666-7777)
jwt — three base64url segments separated by dots
mime — image/png, application/vnd.api+json
file — name.ext for known extensions; per-kind grouping (image/document/data/...)
color — hex form (#fff, #ffffff, #ffffff80)
coordinate — lat,lng pair with plausible-range validation
country — ISO 3166-1 alpha-2 codes (US, JP, GB)
base64 — standard base64 blobs with disambiguating +///=
opaque_id — short alphanumeric mix that doesn't fit elsewhere

RESTful hints

When a variable segment follows a literal one, iriq derives a hint by singularizing the literal and suffixing _id (or _uuid for UUIDs). That's what produces {user_id} from /users/123 and {order_id} from /orders/456. Semantic types (version, locale, currency, date, boolean) skip the hint and surface as {type} — /api/v1/status renders as /api/{version}/status, not the misleading /api/{api_id}/status. Pass -N / --no-hints for mechanical placeholders ({integer} instead of {user_id}); a slot only the corpus knows is variable renders {value}.

Types only the corpus can see

Four types emerge from the distribution of values across many observations:

Type	Emerges when a position…
`number`	holds both integers and floats
`year`	holds integers that all land in 1900–2100
`http_status`	holds integers that all land in 100–599
`enum`	holds a small, bounded set of distinct values

Mechanically, 200 is just an integer. Across ten thousand URLs where that slot is always 100–599, it's likely an HTTP status.

Corpus (streaming + learning)

The corpus maintains rolling aggregates and per-(host, prefix) frequency stats, so classification improves as more data comes in — handy for an unbounded stream of identifiers. The default corpus already persists; --corpus PATH points iriq at a specific file instead, to keep separate corpora or share one across runs.

The extension picks the backend, and the two behave differently:

.db / .sqlite / .sqlite3 (SQLite) — the default, and the one to share. Many iriq processes can write at once by taking turns: a writer waits up to 10 seconds for its turn. A big cluster commits about a second at a time, and --reinfer rebuilds on the side and holds the corpus only to swap the result in, so a tail -f stream keeps flowing beside either. Use SQLite for streams and concurrent writers.
Anything else (JSON) — read when iriq starts and written once, when it exits cleanly. It's single-writer: when two processes use one file, the last to exit wins. A streaming run that's killed, Ctrl-C included, saves nothing.

A few things to know:

The cluster view (iriq cluster, or 10+ piped IRIs) shows the whole corpus, not just this input. Add -C to cluster one input on its own.
iriq keeps every IRI it observes, repeats included, so --reinfer can replay them. That log grows without bound.
On SQLite, cluster and --stats commit their input about a second at a time. If one is killed part-way, what it committed stays: feed it the same input again and those IRIs count twice.
--reinfer (and --activate-above) rebuilds in temporary tables before swapping the result in, so it needs free space in TMPDIR: plan on more than the corpus file's own size.
--reset deletes the corpus file, its SQLite -wal / -shm sidecars, and any temp files a JSON save left behind. Don't reset a corpus another process is writing: that process carries on, exits 0, and its writes are lost with the deleted file.
iriq refuses a corpus file it can't safely use rather than overwrite it — a JSON file that isn't an iriq corpus, or a SQLite corpus written by a newer iriq (upgrade to open it).

Host keying

By default every hostname gets its own clusters. --host reg keys by registrable domain, so api.foo.com and www.foo.com both cluster under foo.com; --host none ignores the host. The mode applies when observations are recorded, when you --reinfer, and to a -C run's throwaway corpus. It doesn't re-key a report of an existing corpus: iriq --host reg cluster shows the clusters as they were recorded. To re-key a corpus, reinfer it:

$ iriq --corpus c.db --host reg --reinfer
reinferred 3 observations: 3 → 1 cluster

Re-runnable inference

A corpus persists the source-IRI log alongside the materialized views. --reinfer drops every view and replays the log through the current classifier and reducers. Tune a threshold, swap in a different classifier, or activate new recognizers (below) — then reinfer to see the new results without re-feeding URLs.

$ iriq --corpus c.db --reinfer

Learning new types

Iriq doesn't just classify against a fixed list — it watches the stream and proposes new recognizers for patterns it keeps seeing. Notice ghp_… or cus_… recurring at a slug position and iriq will suggest a recognizer for it, with evidence: coverage, host count, confidence.

# Print proposals (human-readable, or --json)
$ iriq --corpus c.db --propose-recognizers

# Auto-activate every proposal with confidence ≥ 0.9, then reinfer
$ iriq --corpus c.db --propose-recognizers --activate-above 0.9

Cross-host shape learning

A route shape that recurs across multiple hosts is independent evidence of a semantic pattern — two unrelated hosts inventing the same /users/{integer} structure by accident is unlikely.

$ iriq --corpus c.db --cross-host-shapes [--min-hosts N]

The same signal feeds back into proposal confidence: each additional host beyond the first adds 0.05 to the score (capped at 1.0), so a prefix proposed on 5 hosts is meaningfully stronger than the same coverage seen on 1 host.

Extracting IRIs from text

Pipe-mode extraction picks up explicit-scheme URLs (http, https, ftp, ws, wss, urn) and foo.com/path-style scheme-less URLs (small TLD allow-list, required path). It trims trailing sentence punctuation and preserves balanced parens (https://en.wikipedia.org/wiki/Ruby_(programming_language) stays intact; (see https://foo.com) drops the outer paren).

Known limitations (intentional):

Comma is a URL boundary, so query strings like ?q=37.7,-122.4 truncate. Trade-off picked to keep CSV-shaped text working.
No HTML entity decoding (& stays as-is).
Scheme-less mode skips bare hostnames without a path (too noisy in prose).

Disable scheme-less extraction with --no-scheme-less.

Access logs

Extraction needs URLs with a host. A web server's request line ("GET /api/v1/users/123 HTTP/1.1") has none, so on a raw access log iriq finds only the full URLs on each line, usually the Referer:

$ cat access.log | iriq
[12] example.com  /referrer
    https://example.com/referrer
    + 11 more

Pull out the path and give it a host first. In the common and combined log formats, the path is the seventh field:

$ awk '{print "https://api.example.com" $7}' access.log | iriq
[6] api.example.com  /api/{version}/users/{user_id}
    https://api.example.com/api/v1/users/123
    https://api.example.com/api/v1/users/456
    https://api.example.com/api/v1/users/789
    + 3 more

[3] api.example.com  /orders/{order_uuid}
    https://api.example.com/orders/5f0c6a52-8b2e-4c1a-9f3d-2e7b1c9a0d11?status=open
    https://api.example.com/orders/0a1b2c3d-4e5f-4a6b-8c7d-9e0f1a2b3c4d?status=closed
    https://api.example.com/orders/7c9e6679-7425-40de-944b-e07fc1f90ae7?status=open
    status  string  conf 0.17  (2 distinct, 100%)

[3] api.example.com  /products/{product_id}
    https://api.example.com/products/blue-widget
    https://api.example.com/products/red-gadget
    https://api.example.com/products/green-gizmo

How it works

Under the shape sits one idea: Position + Evidence. A Position is a slot in a host's structure — a typed path prefix, or a query-param name. Evidence is everything the corpus has observed about that slot: which values, how often, across how many hosts. Strings are observations; types are inferences drawn from the pile. Shape is the surface you see; Position + Evidence is the engine underneath. See docs/ARCHITECTURE.md for the full model.

CLI reference

Single input — combined parse + normalize summary; trim with section flags (-p, -n).

Piped stdin, or a file argument — extraction runs by default. With no section flag, iriq reads all of the input, then prints a deduplicated URL list (fewer than 10 IRIs) or the cluster view of the corpus (10 or more). With a section flag, it prints each IRI's result as the line arrives. -n is corpus-informed; -e is mechanical even with a corpus.

Flag	Effect
`-p, --parse`	Show parsed fields
`-n, --normalize`	Show the shape-normalized form
`-c, --canonical`	Show the canonical form (no shape normalization)
`-e, --explain`	Annotated trace — per-segment notes about why each placeholder / canonical value was chosen. Mechanical, even with a corpus
`-j, --json`	Emit JSON
`-J, --ndjson`	Newline-delimited JSON; implies `--json`. With a section flag (`-nJ`), one line per IRI as it arrives; alone, the URL list or clusters at end of input
`-N, --no-hints`	Use `{integer}` etc. instead of `{user_id}`
`--no-scheme-less`	Skip `foo.com/path`-style extraction (explicit-scheme only)
`--corpus PATH`	Use a specific corpus file (`.json` or `.db`/`.sqlite`/`.sqlite3`). Overrides the default
`-C, --no-corpus`	Disable corpus persistence for this invocation (same as `IRIQ_NO_CORPUS=1`)
`--reset`	Delete the corpus file, its SQLite sidecars and JSON temp files, and exit
`--host MODE`	Host keying: `full` (default), `reg` strips subdomains, `none` ignores host. Applies when observing, to `--reinfer`, and with `-C`
`--stats`	Print rolling aggregates
`--reinfer`	Drop the materialized views and replay the source-IRI log through the current classifier + reducers
`--propose-recognizers`	Scan observed values for shape patterns that recur enough to suggest a new recognizer. Combine with `--json` for structured output
`--cross-host-shapes`	List route shapes that recur across multiple hosts
`--min-observations N`	Proposal threshold; default 20
`--min-coverage F`	Proposal threshold; default 0.7
`--min-hosts N`	Threshold for both proposals and cross-host shapes; default 1 / 2 respectively
`--activate-above F`	With `--propose-recognizers`, auto-activate every proposal whose confidence is ≥ F
`cluster [file]`	Force the cluster view
`completion bash\	zsh`	Print shell completion script (Homebrew installs this automatically)
`-V, --version`	Print version

Environment variables:

Variable	Effect
`IRIQ_CORPUS=PATH`	Set the corpus path (overrides the default)
`IRIQ_NO_CORPUS=1`	Disable the default corpus (equivalent to `-C`)

A positional argument that names an existing file is read as a file, unless it contains :// — iriq access.log and iriq /var/log/foo.log both work. A path-like argument (/x, ./x, ../x) that doesn't exist is an error; a bare name that isn't a file, like foo.log, parses as a host (https://foo.log/).

Errors go to stderr as iriq: MESSAGE, or, with --json / -J, as {"error":{"code":"…","message":"…"}}. A corpus error names the file: iriq: corpus team.db: attempt to write a readonly database.

Exit	Meaning	JSON codes
`0`	Success
`1`	Bad option or argument, missing or unreadable input, unusable corpus, or stdout failed	`option_error`, `unknown_shell`, `file_not_found`, `read_error`, `invalid_utf8`, `corpus_error`, `stdout_error`
`2`	The input isn't a parseable IRI	`parse_error`
`141`	The reader went away (`iriq … \	head`); iriq stops quietly

Rust library

cargo add iriq

use iriq::{normalize, parse, Corpus};

fn main() -> iriq::Result<()> {
    let iri = parse("https://foo.com/users/123")?;
    println!("{} {:?}", iri.host, iri.path_segments); // foo.com ["users", "123"]
    println!("{}", normalize("https://foo.com/users/123")?); // https://foo.com/users/{user_id}

    // A persistent corpus: SQLite for .db, JSON otherwise.
    let mut corpus = Corpus::open("c.db")?;
    for n in 1..=3 {
        corpus.observe(&format!("https://foo.com/users/{n}"))?;
    }
    for cluster in corpus.clusters()? {
        println!("[{}] {} {}", cluster.count, cluster.host, cluster.shape); // [3] foo.com /users/{user_id}
    }
    corpus.save("c.db")?; // flushes in place; a .json corpus is written only here
    Ok(())
}

Every Corpus operation returns iriq::Result, whose iriq::Error names the corpus that failed. SQLite comes from the default-on sqlite feature; cargo add iriq --no-default-features drops it and keeps in-memory and JSON corpora. Requires Rust 1.85 or newer. A long-lived Corpus sees recognizers another process activated at its next batch or observe, not in reads outside one.

The crate README is the library tour: reading clusters and params, batches, sharing a corpus, and errors. Full API on docs.rs/iriq.

Limitations (intentional)

Iriq does not:

Implement RFC 3986, RFC 3987, or the WHATWG URL standard fully.
Convert between Unicode (IRI) and punycode (URI) — the display form is preserved as-is.
Percent-encode or decode path/query bytes. Bytes are kept as written.
Validate scheme-specific structure beyond URL vs. URN.
Resolve relative references against a base URL.
Round-trip canonical back to the exact original byte-for-byte (whitespace is stripped, default ports are dropped, dot segments are collapsed).

Iriq's focus is the analysis side: classification, normalization, and clustering — not a complete URL implementation.