Class: W3cApi::Hal

Inherits:
Object
  • Object
show all
Includes:
Singleton
Defined in:
lib/w3c_api/hal.rb

Constant Summary collapse

API_URL = 
"https://api.w3.org/"
RETRY_EXCEPTIONS =

Exceptions the Faraday retry middleware treats as retriable.

faraday-retry implements retry_statuses by raising Faraday::RetriableResponse internally and rescuing it with a matcher built from exceptions:. Passing exceptions: REPLACES faraday-retry's DEFAULT_EXCEPTIONS, it does not merge with them -- so omitting Faraday::RetriableResponse silently disables retry_statuses and lets the internal raise escape to the caller on the very first matching response. Always build this list on top of the defaults.

(
  Faraday::Retry::Middleware::DEFAULT_EXCEPTIONS + [Faraday::ConnectionFailed]
).freeze
DEFAULT_RETRY_OPTIONS =

Retry policy for the W3C-specific transient failures: HTTP 403 (how the W3C API signals rate-limiting) plus connection/timeout errors. The computed backoff grows 1, 2, 4, 8, 16s, matching rate_limiting_options.

max_interval sits far above the largest computed backoff (16s) because it also caps Retry-After: faraday-retry's calculate_sleep_amount returns nil -- aborting the retry loop entirely, with zero retries -- when a response's Retry-After exceeds max_interval. Too small a cap turns a polite server hint into an immediate give-up.

{
  max: 5,
  interval: 1.0,
  backoff_factor: 2,
  max_interval: 60.0,
  # Both nested values are frozen too: Hash#freeze is shallow, and
  # retry_options hands out a shallow dup, so an unfrozen array here would
  # let a caller corrupt this baseline for the whole process.
  retry_statuses: [403].freeze,
  exceptions: RETRY_EXCEPTIONS,
}.freeze
DEFAULT_USER_AGENT =

api.w3.org is fronted by Cloudflare, and a bare HTTP-library User-Agent such as "Faraday v2.12.2" is a well-known trigger for its bot heuristics — which is how a polite crawl turns into an all-or-nothing 429 storm. W3C also asks API consumers to identify themselves, so name the gem, its version, and a contact URL.

"w3c_api/#{VERSION} (+https://github.com/relaton/w3c_api)".freeze
USER_AGENT_ENV_VAR =

Lets an operator put their own contact details on the wire without a code change. An explicit configure_user_agent call still wins.

"W3C_API_USER_AGENT"

Instance Method Summary collapse

Constructor Details

#initialize ⇒ Hal

Returns a new instance of Hal.



94
95
96
 
# File 'lib/w3c_api/hal.rb', line 94

def initialize
  # Don't call setup here - it will be called when register is first accessed
end

Instance Method Details

#cache_options ⇒ Object

Cache options for the model register

lutaml-hal caches realized objects keyed by their (canonical) URL, so a document linked from many places is fetched once. In-memory by default; pass an adapter config such as { adapter: { type: :filesystem, options: { path: "..." } } } for cross-run persistence. Returns nil when caching is disabled.



211
212
213
214
215
 
# File 'lib/w3c_api/hal.rb', line 211

def cache_options
  return @cache_options if defined?(@cache_options)

  @cache_options = { adapter: :memory }
end

#client ⇒ Object 



98
99
100
101
102
103
104
 
# File 'lib/w3c_api/hal.rb', line 98

def client
  @client ||= Lutaml::Hal::Client.new(
    api_url: API_URL,
    connection: connection,
    rate_limiting: rate_limiting_options,
  )
end

#configure_cache(options = {}) ⇒ Object

Set cache options (merged into the current ones)



218
219
220
221
 
# File 'lib/w3c_api/hal.rb', line 218

def configure_cache(options = {})
  @cache_options = (cache_options || {}).merge(options)
  rebuild_register
end

#configure_rate_limiting(options = {}) ⇒ Object

Set rate limiting options



186
187
188
189
190
191
192
 
# File 'lib/w3c_api/hal.rb', line 186

def configure_rate_limiting(options = {})
  @rate_limiting_options = rate_limiting_options.merge(options)
  # Reset the client *and* the register to pick up the new options: the
  # register keeps the client it was built with, so dropping @client alone
  # left an already-built register on the old rate-limiting settings.
  reset_client
end

#configure_retry(options = {}) ⇒ Object

Set retry options (merged into the current ones)

The retry middleware is baked into the Faraday connection when the connection is built, so this goes through the usual reset_connection -> reset_client -> rebuild_register cascade. Like the other configure_* setters it starts a fresh object cache, so it is a start-up knob rather than a mid-crawl one.

There is no separate disable switch -- configure_retry(max: 0) turns retrying off, returning the response as-is on the first attempt.



164
165
166
167
 
# File 'lib/w3c_api/hal.rb', line 164

def configure_retry(options = {})
  @retry_options = retry_options.merge(options)
  reset_connection
end

#configure_user_agent(user_agent) ⇒ Object

Set the User-Agent sent with every request.

Applications embedding this gem should identify themselves and keep the gem's own token, e.g. configure_user_agent("my-crawler/1.4 (+https://example.com) #DEFAULT_USER_AGENT")

Pass nil or a blank string to fall back to W3C_API_USER_AGENT / the gem default. Rebuilds the connection, so call it before issuing requests.



122
123
124
125
126
 
# File 'lib/w3c_api/hal.rb', line 122

def configure_user_agent(user_agent)
  @user_agent = normalize_user_agent(user_agent)
  reset_connection
  self.user_agent
end

#connection ⇒ Object

Faraday connection mirroring lutaml-hal's default middleware stack, with a retry layer for the failures lutaml-hal's RateLimiter does not cover: the W3C API signals rate-limiting with HTTP 403, plus transient connection and timeout errors. (lutaml-hal still retries 429 and 5xx.) Owning retries here means every consumer of the client is resilient without its own wrapper.



133
134
135
136
137
138
139
140
141
142
143
144
 
# File 'lib/w3c_api/hal.rb', line 133

def connection
  @connection ||= Faraday.new(
    url: API_URL.delete_suffix("/"),
    headers: { "User-Agent" => user_agent },
  ) do |conn|
    conn.request :retry, retry_options
    conn.use Faraday::FollowRedirects::Middleware
    conn.request :json
    conn.response :json, content_type: /\bjson$/
    conn.adapter Faraday.default_adapter
  end
end

#disable_cache ⇒ Object

Disable caching of realized objects



224
225
226
227
 
# File 'lib/w3c_api/hal.rb', line 224

def disable_cache
  @cache_options = nil
  rebuild_register
end

#disable_rate_limiting ⇒ Object

Disable rate limiting



195
196
197
 
# File 'lib/w3c_api/hal.rb', line 195

def disable_rate_limiting
  configure_rate_limiting(enabled: false)
end

#enable_cache(options = nil) ⇒ Object

Enable caching of realized objects



230
231
232
233
 
# File 'lib/w3c_api/hal.rb', line 230

def enable_cache(options = nil)
  @cache_options = options || { adapter: :memory }
  rebuild_register
end

#enable_rate_limiting ⇒ Object

Enable rate limiting



200
201
202
 
# File 'lib/w3c_api/hal.rb', line 200

def enable_rate_limiting
  configure_rate_limiting(enabled: true)
end

#rate_limiting_options ⇒ Object

Configure rate limiting options

lutaml-hal's RateLimiter retries 429 and 5xx responses with exponential backoff (base_delay * backoff_factor**(attempt - 1), capped at max_delay). These defaults grow 1, 2, 4, 8, 16s so a rate-limited or briefly overloaded W3C API is given real room to recover during a bulk crawl.



175
176
177
178
179
180
181
182
183
 
# File 'lib/w3c_api/hal.rb', line 175

def rate_limiting_options
  @rate_limiting_options ||= {
    enabled: true,
    max_retries: 5,
    base_delay: 1.0,
    max_delay: 30.0,
    backoff_factor: 2.0,
  }
end

#rebuild_register ⇒ Object

Rebuild the register immediately rather than lazily. Link#realize looks the register up in lutaml-hal's GlobalRegister, which raises when the name is absent — so leaving it torn down would break .realize on every model fetched before a configure_* call.



270
271
272
273
 
# File 'lib/w3c_api/hal.rb', line 270

def rebuild_register
  reset_register
  register
end

#register ⇒ Object 



235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
 
# File 'lib/w3c_api/hal.rb', line 235

def register
  return @register if @register

  @register = Lutaml::Hal::ModelRegister.new(
    name: :w3c_api,
    client: client,
    cache: cache_options,
  )
  Lutaml::Hal::GlobalRegister.instance.register(:w3c_api, @register)

  # Re-run setup to register all endpoints with the new register
  setup

  @register
end

#reset_client ⇒ Object

Drop the memoized lutaml-hal client. The register holds the client it was constructed with (and hands it to its cache manager), so it must go too.



261
262
263
264
 
# File 'lib/w3c_api/hal.rb', line 261

def reset_client
  @client = nil
  rebuild_register
end

#reset_connection ⇒ Object

Drop the memoized Faraday connection. Memoization runs register -> client -> connection, so a connection-level change (headers, middleware) only takes effect once all three are rebuilt.



254
255
256
257
 
# File 'lib/w3c_api/hal.rb', line 254

def reset_connection
  @connection = nil
  reset_client
end

#reset_register ⇒ Object 



275
276
277
278
279
280
 
# File 'lib/w3c_api/hal.rb', line 275

def reset_register
  # Drop the global registration too, otherwise rebuilding the register
  # raises "replacing another one" when it re-registers the same name.
  Lutaml::Hal::GlobalRegister.instance.unregister(:w3c_api)
  @register = nil
end

#retry_options ⇒ Object

Retry policy for the W3C-specific transient failures. See DEFAULT_RETRY_OPTIONS for why exceptions and max_interval are what they are -- both have non-obvious failure modes.



149
150
151
152
 
# File 'lib/w3c_api/hal.rb', line 149

def retry_options
  # dup: DEFAULT_RETRY_OPTIONS is frozen and must stay the pristine baseline.
  @retry_options ||= DEFAULT_RETRY_OPTIONS.dup
end

#user_agent ⇒ Object

User-Agent sent with every request. Precedence: an explicit configure_user_agent call, then W3C_API_USER_AGENT, then the gem default.



108
109
110
111
112
 
# File 'lib/w3c_api/hal.rb', line 108

def user_agent
  @user_agent ||=
    normalize_user_agent(ENV.fetch(USER_AGENT_ENV_VAR, nil)) ||
    DEFAULT_USER_AGENT
end