🪙 Token::Resolver
if ci_badges.map(&:color).detect { it != "green"} ☝️ let me know, as I may have missed the discord notification.
if ci_badges.map(&:color).all? { it == "green"} 👇️ send money so I can do more of this. FLOSS maintenance is now my full-time job.
👣 How will this project approach the September 2025 hostile takeover of RubyGems? 🚑️
I've summarized my thoughts in [this blog post](https://dev.to/galtzo/hostile-takeover-of-rubygems-my-thoughts-5hlo).🌻 Synopsis
Token::Resolver is a configurable PEG-based token parser and resolver for structured token detection and replacement in arbitrary text.
Detects structured tokens like {KJ|GEM_NAME} in any file format and resolves them against a replacement map. The token structure (delimiters, separators, segment count) is fully configurable.
# One-liner: parse and resolve
result = Token::Resolver.resolve(
"Hello {KJ|NAME}, welcome to {KJ|PROJECT}!",
{"KJ|NAME" => "World", "KJ|PROJECT" => "my-app"},
)
# => "Hello World, welcome to my-app!"
💡 Info you can shake a stick at
| Tokens to Remember |   |
|---|---|
| Works with JRuby |   |
| Works with Truffle Ruby | -AABBCC?style=for-the-badge&logo=ruby&logoColor=pink){kind=icon} -AABBCC?style=for-the-badge&logo=ruby&logoColor=pink){kind=icon}   |
| Works with MRI Ruby 4 |   |
| Works with MRI Ruby 3 |    |
| Support & Community |     |
| Source |     |
| Documentation |      |
| Compliance |      |
| Style |     |
| Maintainer 🎖️ |      |
... 💖 |
    🧊 🐙 🛖 🧪 |
Compatibility
Compatible with MRI Ruby 3.2.0+, and concordant releases of JRuby, and TruffleRuby.
| 🚚 Amazing test matrix was brought to you by | 🔎 appraisal2 🔎 and the color 💚 green 💚 |
|---|---|
| 👟 Check it out! | ✨ github.com/appraisal-rb/appraisal2 ✨ |
Federated DVCS
Find this repo on federated forges (Coming soon!)
| Federated [DVCS][💎d-in-dvcs] Repository | Status | Issues | PRs | Wiki | CI | Discussions | |-------------------------------------------------|-----------------------------------------------------------------------|---------------------------|--------------------------|---------------------------|--------------------------|------------------------------| | 🧪 [kettle-rb/token-resolver on GitLab][📜src-gl] | The Truth | [💚][🤝gl-issues] | [💚][🤝gl-pulls] | [💚][📜gl-wiki] | 🐭 Tiny Matrix | ➖ | | 🧊 [kettle-rb/token-resolver on CodeBerg][📜src-cb] | An Ethical Mirror ([Donate][🤝cb-donate]) | [💚][🤝cb-issues] | [💚][🤝cb-pulls] | ➖ | ⭕️ No Matrix | ➖ | | 🐙 [kettle-rb/token-resolver on GitHub][📜src-gh] | Another Mirror | [💚][🤝gh-issues] | [💚][🤝gh-pulls] | [💚][📜gh-wiki] | 💯 Full Matrix | [💚][gh-discussions] | | 🎮️ [Discord Server][✉️discord-invite] | [![Live Chat on Discord][✉️discord-invite-img-ftb]][✉️discord-invite] | [Let's][✉️discord-invite] | [talk][✉️discord-invite] | [about][✉️discord-invite] | [this][✉️discord-invite] | [library!][✉️discord-invite] |Enterprise Support 
Available as part of the Tidelift Subscription.
Need enterprise-level guarantees?
The maintainers of this and thousands of other packages are working with Tidelift to deliver commercial support and maintenance for the open source packages you use to build your applications. Save time, reduce risk, and improve code health, while paying the maintainers of the exact packages you use. [![Get help from me on Tidelift][🏙️entsup-tidelift-img]][🏙️entsup-tidelift] - 💡Subscribe for support guarantees covering _all_ your FLOSS dependencies - 💡Tidelift is part of [Sonar][🏙️entsup-tidelift-sonar] - 💡Tidelift pays maintainers to maintain the software you depend on!📊`@`Pointy Haired Boss: An [enterprise support][🏙️entsup-tidelift] subscription is "[never gonna let you down][🧮kloc]", and *supports* open source maintainers Alternatively: - [![Live Chat on Discord][✉️discord-invite-img-ftb]][✉️discord-invite] - [![Get help from me on Upwork][👨🏼🏫expsup-upwork-img]][👨🏼🏫expsup-upwork] - [![Get help from me on Codementor][👨🏼🏫expsup-codementor-img]][👨🏼🏫expsup-codementor]
✨ Installation
Install the gem and add to the application's Gemfile by executing:
bundle add token-resolver
If bundler is not being used to manage dependencies, install the gem by executing:
gem install token-resolver
🔒 Secure Installation
For Medium or High Security Installations
This gem is cryptographically signed, and has verifiable [SHA-256 and SHA-512][💎SHA_checksums] checksums by [stone_checksums][💎stone_checksums]. Be sure the gem you install hasn’t been tampered with by following the instructions below. Add my public key (if you haven’t already, expires 2045-04-29) as a trusted certificate: ```console gem cert --add <(curl -Ls https://raw.github.com/galtzo-floss/certs/main/pboling.pem) ``` You only need to do that once. Then proceed to install with: ```console gem install token-resolver -P HighSecurity ``` The `HighSecurity` trust profile will verify signed gems, and not allow the installation of unsigned dependencies. If you want to up your security game full-time: ```console bundle config set --global trust-policy MediumSecurity ``` `MediumSecurity` instead of `HighSecurity` is necessary if not all the gems you use are signed. NOTE: Be prepared to track down certs for signed gems and add them the same way you added mine.⚙️ Configuration
Token Config Options
| Option | Default | Description |
|---|---|---|
pre |
"{" |
Opening delimiter |
post |
"}" |
Closing delimiter |
separators |
["|"] (pipe) |
Segment separators (sequential; last repeats) |
min_segments |
2 |
Minimum segments for a valid token |
max_segments |
nil |
Maximum segments (nil = unlimited) |
segment_pattern |
"[A-Za-z0-9_]" |
Parslet character class for valid segment content |
Segment Character Constraints
Token segments (the parts between delimiters and separators) only match characters that
conform to the segment_pattern. By default, this is word characters: uppercase and
lowercase letters, digits, and underscores.
This prevents false positives with syntax that structurally resembles tokens but isn't:
# These are NOT parsed as tokens (spaces, punctuation disqualify them):
"items.map { |x| x.to_s }" # Ruby block parameters
"${CLASSPATH:+:$CLASSPATH}" # Shell variable expansion
"cert_chain.select! { |fp| File.exist? }" # Ruby block with expressions
If you need different characters in your token segments, provide a custom pattern:
# Allow hyphens in segments: {NS|my-key}
config = Token::Resolver::Config.new(segment_pattern: "[A-Za-z0-9_-]")
🔧 Basic Usage
Basic Token Resolution
require "token/resolver"
# Parse a document to inspect tokens
doc = Token::Resolver.parse("Deploy {KJ|GEM_NAME} to {KJ|GH_ORG}")
doc.token_keys # => ["KJ|GEM_NAME", "KJ|GH_ORG"]
doc.text_only? # => false
# Resolve tokens
result = Token::Resolver.resolve(
"Deploy {KJ|GEM_NAME} to {KJ|GH_ORG}",
{"KJ|GEM_NAME" => "my-gem", "KJ|GH_ORG" => "my-org"},
)
# => "Deploy my-gem to my-org"
Handling Missing Tokens
# Default: raise on unresolved tokens
Token::Resolver.resolve("{KJ|MISSING}", {})
# => raises Token::Resolver::UnresolvedTokenError
# Keep unresolved tokens as-is
Token::Resolver.resolve("{KJ|MISSING}", {}, on_missing: :keep)
# => "{KJ|MISSING}"
# Remove unresolved tokens
Token::Resolver.resolve("{KJ|MISSING}", {}, on_missing: :remove)
# => ""
Custom Token Structure
# Tokens like <<SECTION:NAME>>
config = Token::Resolver::Config.new(
pre: "<<",
post: ">>",
separators: [":"],
)
Token::Resolver.resolve("Hello <<NS:NAME>>!", {"NS:NAME" => "World"}, config: config)
# => "Hello World!"
Multi-Segment Tokens with Sequential Separators
# Tokens like {KJ|SECTION:SUBSECTION}
config = Token::Resolver::Config.new(
separators: ["|", ":"], # First boundary uses |, second uses :, rest repeat :
)
doc = Token::Resolver.parse("{KJ|META:AUTHOR}", config: config)
doc.tokens.first.key # => "KJ|META:AUTHOR"
doc.tokens.first.prefix # => "KJ"
doc.tokens.first.segments # => ["KJ", "META", "AUTHOR"]
Step-by-Step API
# Parse
doc = Token::Resolver::Document.new("Hello {KJ|NAME}!")
# Inspect
doc.nodes # => [Text("Hello "), Token(["KJ", "NAME"]), Text("!")]
doc.tokens # => [Token(["KJ", "NAME"])]
doc.token_keys # => ["KJ|NAME"]
doc.to_s # => "Hello {KJ|NAME}!" (roundtrip fidelity)
# Resolve
resolver = Token::Resolver::Resolve.new(on_missing: :raise)
result = resolver.resolve(doc, {"KJ|NAME" => "World"})
# => "Hello World!"
Design
Grammar Never Fails
The parslet grammar is designed so that any input is valid. When the parser encounters
{ but it doesn't start a valid token, the { is consumed as plain text. No input can
cause a parse failure.
Single-Pass Resolution
Replacement values are emitted as-is and are not re-scanned for tokens. This prevents infinite loops and ensures predictable behavior when replacement values contain token-like strings.
Performance
If the input doesn't contain the pre delimiter at all, the parser fast-paths and returns
a single Text node without invoking parslet.
📊 Benchmarks
Token-resolver prioritizes flexibility, configurability, and maintainability over raw speed.
⚠️ Important: Token-resolver is 100-3000x slower than simple alternatives like String#gsub and Kernel#sprintf.
because it does significantly more work:
- Full PEG parsing of the input string
- Token validation and structure enforcement
- Building an AST for introspection
- Flexible error handling for missing tokens
This performance difference is expected and acceptable because these are fundamentally different approaches solving different problems.
See BENCHMARK.md for detailed performance comparisons and guidance on:
- When to use token-resolver: Configurable token structures, validation, introspection, flexible error handling
- When to use String#gsub: Fixed token patterns, maximum performance, simple one-shot replacements
- When to use Kernel#sprintf: Positional formatting, fixed templates, printf-style output
The choice should be based on your actual requirements, not just raw performance metrics.
To run benchmarks on your system:
bundle exec rake bench:comparison
False Positive Prevention
The grammar constrains segment content to the configured segment_pattern (default: word
characters). This ensures that syntax using the same delimiter characters — such as Ruby
block parameters ({ |x| expr }) or shell variable expansion (${VAR:+val}) — is never
mistakenly parsed as a token. Replacement keys that contain characters outside the
segment_pattern are rejected with an ArgumentError at resolve time.
🦷 FLOSS Funding
While kettle-rb tools are free software and will always be, the project would benefit immensely from some funding. Raising a monthly budget of... "dollars" would make the project more sustainable.
We welcome both individual and corporate sponsors! We also offer a wide array of funding channels to account for your preferences (although currently Open Collective is our preferred funding platform).
If you're working in a company that's making significant use of kettle-rb tools we'd appreciate it if you suggest to your company to become a kettle-rb sponsor.
You can support the development of kettle-rb tools via GitHub Sponsors, Liberapay, PayPal, Open Collective and Tidelift.
| 📍 NOTE |
|---|
| If doing a sponsorship in the form of donation is problematic for your company from an accounting standpoint, we'd recommend the use of Tidelift, where you can get a support-like subscription instead. |
Open Collective for Individuals
Support us with a monthly donation and help us continue our activities. [Become a backer]
NOTE: kettle-readme-backers updates this list every day, automatically.
No backers yet. Be the first!
Open Collective for Organizations
Become a sponsor and get your logo on our README on GitHub with a link to your site. [Become a sponsor]
NOTE: kettle-readme-backers updates this list every day, automatically.
No sponsors yet. Be the first!
Another way to support open-source
I’m driven by a passion to foster a thriving open-source community – a space where people can tackle complex problems, no matter how small. Revitalizing libraries that have fallen into disrepair, and building new libraries focused on solving real-world challenges, are my passions. I was recently affected by layoffs, and the tech jobs market is unwelcoming. I’m reaching out here because your support would significantly aid my efforts to provide for my family, and my farm (11 🐔 chickens, 2 🐶 dogs, 3 🐰 rabbits, 8 🐈 cats).
If you work at a company that uses my work, please encourage them to support me as a corporate sponsor. My work on gems you use might show up in bundle fund.
I’m developing a new library, floss_funding, designed to empower open-source developers like myself to get paid for the work we do, in a sustainable way. Please give it a look.
Floss-Funding.dev: 👉️ No network calls. 👉️ No tracking. 👉️ No oversight. 👉️ Minimal crypto hashing. 💡 Easily disabled nags
🔐 Security
See SECURITY.md.
🤝 Contributing
If you need some ideas of where to help, you could work on adding more code coverage, or if it is already 💯 (see below) check reek, issues, or PRs, or use the gem and think about how it could be better.
We so if you make changes, remember to update it.
See CONTRIBUTING.md for more detailed instructions.
🚀 Release Instructions
See CONTRIBUTING.md.
Code Coverage
🪇 Code of Conduct
Everyone interacting with this project's codebases, issue trackers,
chat rooms and mailing lists agrees to follow the .
🌈 Contributors
Made with contributors-img.
Also see GitLab Contributors: https://gitlab.com/kettle-rb/token-resolver/-/graphs/main
⭐️ Star History
📌 Versioning
This Library adheres to .
Violations of this scheme should be reported as bugs.
Specifically, if a minor or patch version is released that breaks backward compatibility,
a new version should be immediately released that restores compatibility.
Breaking changes to the public API will only be introduced with new major versions.
dropping support for a platform is both obviously and objectively a breaking change
—Jordan Harband (@ljharb, maintainer of SemVer) in SemVer issue 716
I understand that policy doesn't work universally ("exceptions to every rule!"), but it is the policy here. As such, in many cases it is good to specify a dependency on this library using the Pessimistic Version Constraint with two digits of precision.
For example:
spec.add_dependency("token-resolver", "~> 1.0")
📌 Is "Platform Support" part of the public API? More details inside.
SemVer should, IMO, but doesn't explicitly, say that dropping support for specific Platforms is a *breaking change* to an API, and for that reason the bike shedding is endless. To get a better understanding of how SemVer is intended to work over a project's lifetime, read this article from the creator of SemVer: - ["Major Version Numbers are Not Sacred"][📌major-versions-not-sacred]See CHANGELOG.md for a list of releases.
📄 License
The gem is available as open source under the terms of
the MIT License .
See LICENSE.txt for the official Copyright Notice.
© Copyright
-
Copyright (c) 2026 Peter H. Boling, of
Galtzo.com
🤑 A request for help
Maintainers have teeth and need to pay their dentists. After getting laid off in an RIF in March, and encountering difficulty finding a new one, I began spending most of my time building open source tools. I'm hoping to be able to pay for my kids' health insurance this month, so if you value the work I am doing, I need your support. Please consider sponsoring me or the project.
To join the community or get help 👇️ Join the Discord.
To say "thanks!" ☝️ Join the Discord or 👇️ send money.
💌 
Please give the project a star ⭐ ♥.
Thanks for RTFM. ☺️