smoke
smoke is a Ruby based DSL that allows you to query web services such as YQL, RSS / Atom and JSON or XML in an elegant manner.
These "services" can then be re-represented, sorted and filtered. Data can be collected from multiple sources, sorted via a common property, chopped up and refried.
Then you can output as a plain ruby object or one of your other favourites (JSON, YAML, XML)
Examples of use
- The
examples
directory has something to get you running straight away - I powered my entire site using smoke, until further documentation exists, this is probably a good place to start.
- Read further details in the rdoc documentation or the wiki
Media
- Presentation from Melbourne #roro
- Early screencast to get developer / peer feedback
The concept
The concept comes from using Yahoo Pipes to make web based mash ups: Get a list of tv shows for my torrent client, compile a recipe book or make tools to give me a list of albums that artists in my music library are about to be released.
How or what to contribute
- Test everything you do
- Add a way to output (XML, anyone?)
- Examples of queries you'd like to be able to do (email / github message them to me)
API Examples
YQL
# This will use yahoo search to get an array of search results about Ruby
Smoke.yql(:ruby) do
select :all
from "search.web"
where :query, "ruby"
discard :title, /tuesday/i
end
Smoke.yql(:python) do
select :all
from "search.web"
where :query, "python"
end
Join sources and use them together
Smoke.join(:ruby, :python)
or even
Smoke.join(:python, :ruby) do
emit do
sort :title
rename :shit_name => :title
end
end
Define a source allowing for variables to be injected later
Source definition:
Smoke.feed :delicious do
prepare do
url "http://feeds.delicious.com/v2/rss/#{username}?count=15"
end
end
Execution:
Smoke[:delicious].username("bschwarz").output
TODO (working on, just mental notes)
Later / maybe
- YQL w/oAuth
- YQL Subqueries?
- Implement basic auth for sources
For wiki pages (docs, later)
- Document all sources with their irrespective differential methods
- How to use
path
- YQL Definitions
- Tranformations
- Insert
- Joining
- Variable injection
- Sort, Reverse
- Keep, Discard
- Truncate
- Manually setting the content type for a url
Copyright
Copyright (c) 2009 Ben Schwarz. See LICENSE for details.