Vayacondios
Vayacondios is a server-client program designed to make it simple to collect and centralize information and metrics from a large number of disparate sources from multiple application domains.
Vayacondios has the following design goals:
The client is simple enough to use in a shell script and the server is performant enough to support ubiquitous use across a large installation with many clients.
- Decentralized -- Any client can dispatch stashes or events from anywhere
- Dynamic -- No data types or schemas need to be created in advance
- Ubiquitous -- Clients require minimal dependencies because the API is simple to use and access
- Simple -- Clients can write data in whatever way is natural for them
- Scalable -- Server and storage can be scaled horizontally to allow for ever-increasing loads
- Fast -- No client should have to worry that sending data to Vayacondios will affect its performance
The basic objects of Vayacondios are stash and the event:
- a stash is an "object", a "configuration", or "setting" designed to be shared among many services
- an event is a "fact", "measurement", or "metric" announced by an arbitrary service, possibly related to some stash
Stashes and events are each documents which can contain arbitrary JSON-serializable data: hashes/maps/dictionarys, arrays/lists, strings, numbers, floats, null, &c.
The client and server communicate over a RESTful, HTTP-based API which speaks JSON.
See also Coda Hale's metrics.
Architecture
Database
Vayacondios stores all its data in a database. Access to the database
within Vayacondios is strictly contained within model classes within
lib/vayacondios/server/models
. This is so that the backend database
can one day be changed easily without affecting the rest of the
application.
MongoDB is currently the only supported database. MongoDB is a natural choice because it exposes atomic query primitives which map very closely to the operations exposed by the Vayacondios API.
Server
The Vayacondios server process is a Goliath web server which implements the Vayacondios API over HTTP using JSON.
A single server process can easily handle hundreds of client requests per second. Multiple Vayacondios servers can easily be deployed behind a load-balancer.
Each running server process reads and writes all its data in a single MongoDB database specified at runtime.
Client
Clients communicate with the Vayacondios server via the HTTP API it exposes. This makes it extremely simply for applications in any language to communicate with the server.
Vayacondios comes with several clients:
- a Ruby-language client (
Vayacondios::HttpClient
) - a Java-language client (
com.infochimps.vayacondios.HTTPClient
) - a command-line client (the
vcd
program)
The Ruby-language client and the command-line client are bundled with
the vayacondios-client
Ruby gem. The Java-language client is part
of the com.infochimps.vayacondios
package.
Data Model
Vayacondios uses a two-level hierarchical data model to organize events and stashes.
The top-level is the organization. Data from multiple organizations is stored together but accessed separately by a running Vayacondios server. An organization could be the name of a user, workgroup, application, or service using Vayacondios.
The next level is the topic. Each topic within Vayacondios has a single stash and can have multiple events. An "object" like a server, a database, an application, a service, or a user maps to the concept of "topic".
Topics and organizations are strings which can only contain letters,
digits, underscores, periods, and hypens, though periods cannot be the
first or last character. Organizations cannot begin with the string
system.
.
Events
Events belong to a topic within an organization. Each event additionally has
- an ID which is automatically set by the server to a random, unique value if none is provided when the event is announced. Provided IDs cannot contain periods or dollar signs.
- a timestamp which is automatically set by the server to the current UTC time if none is provided when the event is announced. Provided timestamps will attempt to be parsed either from a string or from an integer UNIX timestamp.
- arbitrary key/value data. Keys cannot contain periods or dollar signs.
Events are used for storing facts, measurements, metrics, errors, occurrences, &c. If you anticipate wanting to see a time series or a histogram of a certain kind of data then you should consider writing that data into Vayacondios as events on some topic.
Events are stored in MongoDB in a collection named after their
organization and topic: an event on the ci
topic for the example
organization would be stored in the MongoDB collection
example.ci.events
. The ID of the event, whether auto-generated by
the server or specified by the client, will be used as the _id
field
of the resulting document within this collection.
Here are some examples of data that it would make sense to store as events (in JSON format):
- the output of a build from a CI system might be written to topic
ci
{ "environment": "Jenkins CI v. 1.519", "project": { "name": "website", "version": "0b4d99ded50a19e495d2472477bbb0784d8a18d8", "url": "https://github.com/mycompany/website.git", }, "build": { "time": 182, "status": "success" }, "test": { "time": 97, "ran": 102, "passed": 102, "failed": 0 } }
- an intrusion event picked up by the firewall might be written to topic
firewall.intrusions
{ "ip": "74.210.29.117", "port": 22, "type": "ssh", "reason": "blacklisted" }
- some performance statistics for a running server might be written topic
phoenix.servers.webserver-16
{ "data_center": "Phoenix", "rack": "14", "server": "webserver-16", "cpu": { user: 3.17, nice: 0.01, system: 0.27, iowait: 0.18, steal: 0.00, idle: 96.38 }, "mem": { "total": 12304632, "used": 10335900, "free": 1968732 }, "net": { "out": 2.25, "in": 10.28, }, "disk": { "write": 16.182, "read": 0.11 }
Stashes
Stashes belong to a topic within an organization. Each stash additionally has arbitrary key/value data that it can store. Keys cannot contain dollar signs or periods.
Stashes are used for storing objects, configuration, settings, &c. If you anticipate wanting to lookup a value by name then you should consider writing that data into Vayacondios as (or within) a stash on some topic.
The names of top-level keys within a stash can be used as the "ID" when retrieving/setting/deleting values via the API.
Stashes are stored in MongoDB in a collection named after their
organization: a stash for the example
organization would be stored
in the MongoDB collection example.stash
. The topic of the stash
will be used as the _id
field of the resulting document within this
collection.
Here are some examples of data that it would make sense to store as stashes (in JSON format):
- a collection of projects to run through a CI system might be stored on topic
ci
{ "projects": { { "name": "website", "url": "https://github.com/mycompany/website.git", }, { "name": "client_tool", "url": "https://github.com/mycompany/client_tool.git", }, ... } }
- firewall settings might be stored on topic
firewall
{ "firewall": { "rules": [ { "range": "0.0.0.0", "port": 80, "protocol": "tcp" }, { "range": "10.0.0.0", "port"; 22, "protocol": "ssh" } ] } }
- a mapping of servers within some data center might be stored on topic
data_centers.phoenix
{ "name": "PHX", "location": "Phoenix, AZ", "servers": [ "webserver-0", "webserver-1", "webserver-2", ... ] }
Installation & Configuration
Database
Vayacondios server depends on a database to store all its data. Currently, only MongoDB is supported: here are some installation instructions.
Server
Vayacondios server is distributed via Rubygems:
$ sudo gem install vayacondios-server
Once installed, you can launch a copy of the server from the command-line running locally on port 9000:
$ vcd-server --verbose --stdout
Ports, logging, the location of MongoDB, and much more can be
configured via command-line options. Try vcd-server --help
for more
details.
Client
The server exposes its API via HTTP so all sorts of clients can talk to Vayacondios server. Most simply, a command like
$ curl -X POST http://localhost:9000/v2/my_organization/event/some_topic -d '{"event": "data"}'
will work "right out of the box".
You can also install some pre-written clients that are aware of the Vayacondios API.
Command-Line
The vcd
command-line client is installed via Rubygems:
$ sudo gem install vayacondios-client
You can now run the vcd
program. The equivalent to the above curl
command would be
$ vcd announce 'some_topic' '{"event": "data"}'
The vcd
program looks for its configuration (where is the
Vayacondios server? what organization am I in?) in the files
/etc/vayacondios/vayacondios.yml
and ~/.vayacondios.yml
. The
following can be put in either location to customize the behavior of
vcd
for a given server or user.
---
host: vcd.example.com
port: 9000
organization: my_company
Ruby Client
A Ruby client is also avialable via Rubygems:
$ sudo gem install vayacondios-client
You can now use the Vayacondios::HttpClient
class in your code:
require 'vayacondios-client'
client = Vayacondios::HttpClient.new(organization: 'my_company')
client.announce('some_topic', foo: 'bar')
The Ruby client exposes several API requests as named methods (like
announce
above, which maps to a announce event API endpoint).
Java Client
A Java client is also available. Put the following into your
pom.xml
:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
...
<repositories>
...
<repository>
<id>infochimps.releases</id>
<name>Infochimps Internal Repository</name>
<url>https://s3.amazonaws.com/artifacts.chimpy.us/maven-s3p/releases</url>
</repository>
...
</repositories>
...
<dependencies>
...
<dependency>
<groupId>com.infochimps</groupId>
<artifactId>vayacondios</artifactId>
<version>2.0.0</version>
</dependency>
...
</dependencies>
...
</project>
You can now use the com.infochimps.vayacondios.HTTPClient
class in
your code:
import com.infochimps.vayacondios.VayacondiosClient;
import com.infochimps.vayacondios.HTTPClient;
class public HelloVayacondios {
public static void main(String[] args) throws Exception {
VayacondiosClient client = new HTTPClient("my_organization");
Map event = new HashMap();
event.put("foo", "bar");
client.announce("my_topic", event);
Thread.sleep(50) // ensures async HTTP request finishes
client.close();
}
}
The Java client exposes several API requests as named methods (like
announce
above, which maps to a announce event API endpoint).
API (v2)
All HTTP endpoints defined by the Vayacondios server API share a
common structure: /:version/:organization/:type/[:topic]/[:id]/...
Parameter | Required | Definition | Examples |
---|---|---|---|
version | required | Vayacondios API version | v1, v2 (current) |
organization | required | Name of organization, service, or app | security , accounting , customerX |
type | required | Request type | event , stash , events , stashes |
topic | varies | Topic for event or stash | firewall , servers.webserver-3 |
id | varies | ID of event or field within stash | cpu , liua38923u2389f |
The version
, organization
, and type
parameters are always
required. Other parameters are required depending on the endpoint.
Vayacondios server only listens for a single value of the top-level
version
parameter (curently: v2
). A frontend webserver (Apache,
nginx, &c.) can be used to split traffic to backend Vayacondios
servers running different versions of the Vayacondios API by routing
based on this parameter.
The type
parameter is fixed and defines the type of a Vayacondios
request: event
, stash
, events
, or stashes
.
All other parameters are completely free for clients to specify under the following constraints:
the
organization
parameter can only contain letters, digits, hyphens, and underscores and it must begin with a letterthe
topic
parameter can only contain letters, digits, hyphens, underscores, and periods and it cannot start or end with a periodthe
id
parameter cannot contain the dollar sign or period
The type
, organization
, topic
, and id
parameter together
constitue the vayacondios route.
The document is the request body sent to the server with a given
request. Requests to Vayacondios should have JSON-encoded bodies but
the body can be any JSON datatype: Hash, Array, String, Integer,
Float, Boolean, or null
.
The response is the JSON-encoded response body sent back to the
client from the server. If an error occurred, in addition to the
appropriate HTTP response code, the response will be a Hash containing
the key error
with a message detailing the error.
In the case of a record which is not found, the response may be empty but the HTTP response code will be 404.
In the case of a successful request, the response code will be 200 and the response body will the requested/written object.
Events
A topic within an organization can have many events.
Events are Hash-like data structures which have an associated timestamp and ID.
Events can be announced, retrieved, and searched. Events cannot be updated or deleted, though announcing an event with the same ID as an existing event overwrites the existing event.
Announce a new event
An event can be created without an ID. The server will generate a random, unique ID and include it with the event in the response. This is the most common way to write an event. If you don't intend to ever retrieve this specific event (as opposed to searching across events) then this is the right choice.
Events can also be created with an explicit ID. This is less common but can be useful if your events naturally contain a unique identifier.
Method | Path | Request | Response | Action |
---|---|---|---|---|
POST | /v2/:organization/event/:topic | Hash | Hash | Stores a new event with an auto-generated ID |
POST | /v2/:organization/event/:topic/:id | Hash | Hash | Stores/overwrites a new event with the given ID |
All requests to announce a new event accept a Hash-like request body. Key/value pairs in this request body constitute the body of the event. The following parameters have special meaning:
Parameter | Description | Default | Example Values |
---|---|---|---|
time | Set the timestamp of the event | current time | 2013-06-20 16:20:48 -0500 , 1371763237 |
The response body will contain a Hash that is the original request Hash but with the (possibly auto-generated) ID and timestamp included.
Get an existing event
Events can be retrieved if their ID is known.
Method | Path | Request | Response | Action |
---|---|---|---|---|
GET | /v2/:organization/event/:topic/:id | N/A | Hash | Retrieve an existing event given its ID |
The response will contain the event Hash if found or will be empty if not.
Search for events
You can search for events matching a query.
Method | Path | Request | Response | Action |
---|---|---|---|---|
GET | /v2/:organization/events/:topic | Hash | Array |
Search for events on the given topic. |
The default behavior (which will occur with an empty request body) is
to return the most recent 50 events on the given topic
sorted in
descending order by their timestamps.
Each key in the query body will be interpeted as a condition that the data of each event must match in order to be returned. Keys with periods are interpreted as nested fields. The following parameters have special meaning and can be used to adjust the time window, number of returned events, the sort behavior, and the fields within each event to return:
Parameter | Description | Default | Example Values |
---|---|---|---|
from | Occurred after this time | 1 hour ago | 2013-06-20 Thu 00:00:00 -0500 , 1371704400 |
upto | Occurred before this time | current time | 2013-06-20 Thu 23:59:59 -0500 , 1371790799 |
limit | Return up to this many events | 50 | 100, 200 |
fields | Return only these fields from the event body | all fields | ["account_id", "ip_address"] |
sort | Sort returned events by this field | descending by time | ["time", "ascending"] , ["ip_address", "ascending"] |
id | Regular expression search on event ID | N/A | sensor-data-.* , 2013-06-20-.* |
The response will be an Array of the matching events, possibly an empty Array if no events were found.
Stashes
A topic within an organization can have a stash.
Stashes are Hash-like data structures. Each key/value pair with the stash can be accessed directly by using the name of its key as the ID in requests.
Stashes can be set, merged, retrieved, searched, and destroyed.
Set a value
You can set a value for a stash or one of the fields within a stash. Your value will override whatever value is currently stored for that stash or for that ID within the stash.
Method | Path | Request | Response | Action |
---|---|---|---|---|
POST | /v2/:organization/stash/:topic | Hash | Hash | Overwrites the stash with the given topic. |
POST | /v2/:organization/stash/:topic/:id | varies | varies | Overwrites the ID field of the stash with the given topic. |
When setting the stash itself, your value must be Hash-like. When setting an ID within a stash, your value can have any datatype.
The response for setting a stash will be the (Hash-like) stash you just set. When setting an ID within a stash, the response will be of the same datatype as the request.
Merge a value
You can merge a value for a stash or one of the fields within a stash.
Method | Path | Request | Response | Action |
---|---|---|---|---|
PUT | /v2/:organization/stash/:topic | Hash | Hash | Merges into the stash with the given topic. |
PUT | /v2/:organization/stash/:topic/:id | varies | varies | Merges into the ID field of the stash with the given topic. |
When merging the stash itself, your value must be Hash-like and will be merged on top of the existing (Hash-like) stash's value.
When merging one of the ID fields within the stash, your value can have any datatype and it will be intelligently merged:
- if your value is Hash-like and the existing value is Hash-like , your new value will be merged on top of the existing value
- if your value is Array-like and the existing value is Array-like , your new value will be concatenated to the end of the existing value
- if your value is String-like and the existing value is String-like , your new value will be concatenated to the end of the existing value
- if your value is Numeric-like and the existing value is Numeric-like , your new value will be added to the existing value
The response for merging a stash will be the Hash-like combination of your old and new value. The response for merging an ID within a stash will be of the same type as the request.
Get a value
You can get the value of an existing stash or one of the fields within that stash.
Method | Path | Request | Response | Action |
---|---|---|---|---|
GET | /v2/:organization/stash/:topic | N/A | Hash | Return the stash with the given topic. |
GET | /v2/:organization/stash/:topic/:id | N/A | varies | Return the ID field of the stash with the given topic. |
The response for retreiving a stash will be the Hash-like stash while the response for retreving an ID field within a stash will vary based on the datatype of that value.
Delete a value
You can get delete a stash or one of the fields within a stash.
Method | Path | Request | Response | Action |
---|---|---|---|---|
DELETE | /v2/:organization/stash/:topic | N/A | Hash | Deletes the stash with the given topic. |
DELETE | /v2/:organization/stash/:topic/:id | N/A | Hash | Deletes the ID field of the stash with the given topic. |
The response for deleting a stash or an ID within a stash will be a Hash naming the topic (and ID if given in the request) deleted.
Search for stashes
You can search for stashes.
Method | Path | Request | Response | Action |
---|---|---|---|---|
GET | /v2/:organization/stashes | Hash | Array |
Search for stashes matching the given query. |
The default behavior (which will occur with an empty request body) is to return 50 stashes in sorted in ascending order by their topic.
Each key in the query body will be interpreted as a condition that the data of each stash must match in order to be returned. Keys with periods are interpreted as nested fields. The following parameters have special meaning and can be used to adjust the number of returned stashes, the fields within each stash, and the the sort behavior.
Parameter | Description | Default | Example Values |
---|---|---|---|
limit | Return up to this many stashes | 50 | 100, 200 |
sort | Sort returned stashes by this field | ascending by topic | ["ip_address", "ascending"] |
fields | Array of fields to include in the response | all fields | ["name", "address", "phone"] |
topic | Regular expression search on the stash's topic | N/A | servers-.* , firewall\..*\.rule |
topic_in | List of explicit topics to match | N/A | ["servers.bob", "servers.alan"] |
topic_not_in | List of explicit topics to not match | N/A | ["servers.bob", "servers.alan"] |
The response will be an Array of the matching stashes, possibly an empty Array if no events were found.
Update Multiple Stashes
There are two related methods that let you update multiple stashes in place according to the same rule.
Method | Path | Request | Response | Action |
---|---|---|---|---|
PUT | /v2/:organization/stashes | Hash | Hash | Apply an update to all stashes matching a query |
POST | /v2/:organization/stashes | Hash | Hash | Apply a replacement to all stashes matching a query |
Each of these methods accepts a Hash request body that is supposed to have the following parameters:
Parameter | Description | Default | Example Values |
---|---|---|---|
query | A Hash of matching criteria the stash must satisfy | N/A | {"region": "dakota"} |
update | An update to apply to each matched stash | N/A | {"status": "disabled", "service.ftp": "down"} |
The methods differ in the way they process the update
. The POST
method will always replace the existing values of the named fields
(top-level or nested) in the update Hash with their new, updated
values. This is useful when you want to explicitly set a property to
some fixed value for all stashes which match some criteria.
The PUT
method (like the PUT /v2/:organization/stash/:topic
method) tries to be more clever and attempts to merge the update
into the record according to the following rules
- if your value is Numeric-like, your new value will be added to the existing value
- if your value is Array-like, your new value will be concatenated to the end of the existing value
- if your value is Hash-like, it will replace the existing value (Note:: this is different than the behavior of
PUT /v2:organization/stash/:stopic
.) - if your value is String-like, it will replace the existing value (Note:: this is different than the behavior of
PUT /v2:organization/stash/:stopic
.)
Notice that the behavior for Hashes and Strings isn't as nice as it is
for PUT /v2:organization/stash/:stopic
. Still, this method is
useful for when you want to increment some value across all stashes
which match some criteria.
For both of these methods, the query Hash also contains parameters that are very similar to when searching for a stash:
Parameter | Description | Default | Example Values |
---|---|---|---|
topic | Regular expression search on the stash's topic | N/A | servers-.* , firewall\..*\.rule |
topic_in | List of explicit topics to match | N/A | ["servers.bob", "servers.alan"] |
topic_not_in | List of explicit topics to not match | N/A | ["servers.bob", "servers.alan"] |
Other fields in the query Hash are interpreted as requirements on the data in the stash itself.
Deleting Multiple Stashes
The following method can be used to delete multiple stashes which all match some criteria:
Method | Path | Request | Response | Action |
---|---|---|---|---|
DELETE | /v2/:organization/stashes | Hash | Hash | Delete all stashes matching a query |
The following request parameters define the query which is used to match stashes to be deleted:
Parameter | Description | Default | Example Values |
---|---|---|---|
topic | Regular expression search on the stash's topic | N/A | servers-.* , firewall\..*\.rule |
topic_in | List of explicit topics to match | N/A | ["servers.bob", "servers.alan"] |
topic_not_in | List of explicit topics to not match | N/A | ["servers.bob", "servers.alan"] |
Other fields in the query Hash are interpreted as requirements on the data in the stash itself in order for it to match and be deleted.
Copyright
Copyright (c) 2011 - 2013 Infochimps. See LICENSE.md for further details.