ChimeraHttpClient

When starting to split monolithic apps into smaller services, you need an easy way to access the remote data from the other apps. This chimera_http_client gem should serve as a comfortable and unifying way to access endpoints from other apps.

And what works for the internal communication between your own apps, will also work for external APIs that do not offer a client for simplified access.

It offers an easy to learn interface and nice error handling. And it enables you to queue HTTP requests to run them in parallel for better performance.

Dependencies

The chimera_http_client gem is using the libcurl wrapper Typhoeus. This allows for fast requests, for caching, and for queueing requests to run them in parallel (not all features are implemented in the gem yet).

It does not have any other runtime dependencies.

ENV variables

Setting the environment variable ENV['CHIMERA_HTTP_CLIENT_LOG_REQUESTS'] to true (or 'true') will provide more detailed error messages for logging and also add additional information to the Error JSON. It is recommended to use this only in development environments.

Table of Contents

• ChimeraHttpClient
  • Table of Contents
  • Dependencies
    • ENV variables
  • The Connection class
    • Initialization
      • Mandatory initialization parameter base_url
      • Optional initialization parameters
    • Request methods
      • Mandatory request parameter endpoint
      • Optional request parameters
      • Basic auth
      • Timeout duration
      • Custom logger
      • Caching responses
    • Example usage
  • The Request class
  • The Response class
  • Error classes
  • The Queue class
    • Queueing requests
    • Executing requests in parallel
  • Installation
  • Maintainers and Contributors
    • Roadmap
  • Chimera

The Connection class

The basic usage looks like this:

connection = ChimeraHttpClient::Connection.new(base_url: 'http://localhost/namespace')
response = connection.get!(endpoint, params: params)

Initialization

connection = ChimeraHttpClient::Connection.new(base_url: 'http://localhost:3000/v1', logger: logger, cache: cache)

Mandatory initialization parameter base_url

The mandatory parameter is base_url which should include the host, port and base path to the API endpoints you want to call, e.g. 'http://localhost:3000/v1'.

Setting the base_url is meant to be a comfort feature, as you can then pass short endpoints to each request like /users. You could set an empty string '' as base_url and then pass full qualified URLs as endpoint of the requests.

Optional initialization parameters

The optional parameters are:

• logger - an instance of a logger class that implements #info and #warn methods
• timeout - the timeout for all requests, can be overwritten in any request, the default are 3 seconds
• user_agent - if you would like your calls to identify with a specific user agent
• verbose - the default is false, set it to true while debugging issues
• cache - an instance of your cache solution, can be overwritten in any request

Request methods

The available methods are:

• get / get!
• post / post!
• put / put
• patch / patch!
• delete / delete!

where the methods ending on a bang! will raise an error (which you should handle in your application) while the others will return an error object.

Mandatory request parameter endpoint

The base_url set in the connection will together with the endpoint determine the URL to make a request to.

connection.get([:users, id])
connection.get(["users", id])
connection.get("users/#{id}")
connection.get("/users/#{id}")

All forms above ave valid and will make a request to the same URL.

• Please take note that the endpoint can be given as a String, a Symbol, or an Array.
  
• While they do no harm, there is no need to pass leading or trailing / in endpoints.
  
• When passing the endpoint as an Array, it's elements are converted to Strings and concatenated with /.
  

Optional request parameters

All request methods expect a mandatory endpoint and an optional hash as parameters. In the latter the following keywords are treated specially:

• body - the mandatory body of a post, put or patch request
• headers - a hash of HTTP headers
• params - parameters of a HTTP request
• username - used for a BasicAuth login
• password - used for a BasicAuth login
• timeout - set a custom timeout per request (the default is 3 seconds)
• cache - optionally overwrite the cache store set in Connection in any request

Example:

connection.post(
  :users,
  body: { name: "Andy" },
  params: { origin: `Twitter`},
  headers: { "Authorization" => "Bearer #{token}" },
  timeout: 10,
  cache: nil
)

Basic auth

In case you need to use an API that is protected by basic_auth just pass the credentials as optional parameters:
username: 'admin', password: 'secret'

Timeout duration

The default timeout duration is 3 seconds.

If you want to use a different timeout, you can pass the key timeout when initializing the Connection. You can also overwrite it on every call.

Custom logger

By default no logging is happening. If you need request logging, you can pass your custom Logger to the key logger when initializing the Connection. It will write to logger.info when starting and when completing a request.

Caching responses

To cache all the reponses of a connection, just pass the optional parameter cache to its initializer. You can also overwrite the connection's cache configuration by passing the parameter cache to any get call.

It could be an instance of an implementation as simple as this:

class Cache
  def initialize
    @memory = {}
  end

  def get(request)
    @memory[request]
  end

  def set(request, response)
    @memory[request] = response
  end
end

Or use an adapter for Dalli, Redis, or Rails cache that also support an optional time-to-live default_ttl parameter.

Read more about how to use it: https://github.com/typhoeus/typhoeus#caching

Example usage

To use the gem, it is recommended to write wrapper classes for the endpoints used. While it would be possible to use the get, get!, post, post!, put, put!, patch, patch!, delete, delete! or also the bare request.run methods directly, wrapper classes will unify the usage pattern and be very convenient to use by veterans and newcomers to the team. A wrapper class could look like this:

require 'chimera_http_client'

class Users
  def initialize(base_url: 'http://localhost:3000/v1')
    @base_url = base_url
  end

  # GET one user by id and instantiate a User
  #
  def find(id:)
    response = connection.get!(['users', id])

    user = response.parsed_body
    User.new(id: id, name: user['name'], email: user['email'])

  rescue ChimeraHttpClient::Error => error
    # handle / log / raise error
  end

  # GET a list of users and instantiate an Array of Users
  #
  def all(filter: nil, page: nil)
    params = {}
    params[:filter] = filter
    params[:page] = page

    response = connection.get!('users', params: params, timeout: 10) # set longer timeout

    all_users = response.parsed_body
    all_users.map { |user| User.new(id: user['id'], name: user['name'], email: user['email']) }

  rescue ChimeraHttpClient::Error => error
    # handle / log / raise error
  end

  # CREATE a new user by sending attributes in a JSON body and instantiate the new User
  #
  def create(body:)
    response = connection.post!('users', body: body.to_json) # body.to_json (!!)

    user = response.parsed_body
    User.new(id: user['id'], name: user['name'], email: user['email'])

  rescue ChimeraHttpClient::Error => error
    # handle / log / raise error
  end

  private

  def connection
    # base_url is mandatory
    # logger and timeout are optional
    @connection ||= ChimeraHttpClient::Connection.new(base_url: @base_url, logger: Logger.new(STDOUT), timeout: 2)
  end
end

To create and fetch a user from a remote service with the Users wrapper listed above, calls could be made like this:

  users = Users.new

  new_user = users.create(body: { name: "Andy", email: "andy@example.com" })
  id = new_user.id

  user = users.find(id: id)
  user.name # == "Andy"

The Request class

Usually it does not have to be used directly. It is the class that executes the Typhoeus::Requests, raises Errors on failing and returns Response objects on successful calls.

The body which it receives from the Connection class has to be in the in the (serialized) form in which the endpoint expects it. Usually this means you have to pass a JSON string to the body (it will not be serialized automatically).

Upcoming feature:

It will be expanded by a .queue method, that will queue (sic) calls and run them in parallel and not run every call directly, like the .run method does.

The Response class

The ChimeraHttpClient::Response objects have the following interface:

* body             (content the call returns)
* code             (http code, should be 200 or 2xx)
* time             (for monitoring)
* response         (the full response object, including the request)
* error?           (returns false)
* parsed_body      (returns the result of JSON.parse(body))

If your API does not use JSON, but a different format e.g. XML, you can either monkey patch a parsed_xml method to the Response class, or let your wrapper handle the parsing of body.

Error classes

All errors inherit from ChimeraHttpClient::Error and therefore offer the same attributes:

* code             (http error code)
* body             (alias => message)
* time             (for monitoring)
* response         (the full response object, including the request)
* error?           (returns true)
* error_class      (e.g. ChimeraHttpClient::NotFoundError)
* to_s             (information for logging / respects ENV['CHIMERA_HTTP_CLIENT_LOG_REQUESTS'])
* to_json          (information to return to the API consumer / respects ENV['CHIMERA_HTTP_CLIENT_LOG_REQUESTS'])

The error classes and their corresponding http error codes:

ConnectionError           # 0
RedirectionError          # 301, 302, 303, 307
BadRequestError           # 400
UnauthorizedError         # 401
ForbiddenError            # 403
NotFoundError             # 404
MethodNotAllowedError     # 405
ResourceConflictError     # 409
UnprocessableEntityError  # 422
ClientError               # 400..499
ServerError               # 500..599
TimeoutError              # timeout

The Queue class

Instead of making single requests immediately, the ChimeraHttpClient allows to queue requests and run them in parallel.

The number of parallel requests is limited by your system. There is a hard limit for 200 concurrent requests. You will have to measure yourself where the sweet spot for optimal performance is - and when things start to get flaky. I recommend to queue not much more than 20 requests before running them.

Queueing requests

The initializer of the Queue class expects and handles the same parameters as the Connection class.

queue = ChimeraHttpClient::Queue.new(base_url: 'http://localhost:3000/v1')

queue.add expects and handles the same parameters as the requests methods of a connection.

queue.add(method, endpoint, options = {})

The only difference is that a parameter to set the HTTP method has to prepended. Valid options for method are:

• :get / 'get' / 'GET'
• :post / 'post' / 'POST'
• :put / 'put' / 'PUT'
• :patch / 'patch' / 'PATCH'
• :delete / 'delete' / 'DELETE'

Executing requests in parallel

Once the queue is filled, run all the requests concurrently with:

responses = queue.execute

responses will contain an Array of ChimeraHttpClient::Response objects when all calls succeed. If any calls fail, the Array will also contain ChimeraHttpClient::Error objects. It is in your responsibility to handle the errors.

Tip: every Response and every Error make the underlying Typheous::Request available over object.response.request, which could help with debugging, or with building your own retry mechanism.

Empty the queue

The queue is emptied after execution. You could also empty it at any other point before by calling queue.empty.

To inspect the requests waiting for execution, call queue.queued_requests.

Installation

Add this line to your application's Gemfile:

gem 'chimera_http_client', '~> 0.3'

And then execute:

$ bundle

When updating the version, do not forget to run

$ bundle update chimera_http_client

Maintainers and Contributors

After checking out the repo, run bundle install and then bundle execute rake to run the tests and rubocop.

You can also run rake console to open an irb session with the ChimeraHttpClient pre-loaded that will allow you to experiment.

To build and install this gem onto your local machine, run bundle exec rake install.

Maintainers only:

To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Bug reports and pull requests are welcome on GitHub at https://github.com/mediafinger/chimera_http_client

Roadmap

https://github.com/mediafinger/chimera_http_client/blob/master/TODO.markdown

Chimera

Why this name? First of all, I needed a unique namespace. HttpClient is already used too often. And as this gem is based on Typhoeus I picked the name of one of his (mythological) children.

https://en.wikipedia.org/wiki/Chimera_(mythology)