An utility to archive webpages through time.

Takes snapshots and make incremental backups of webpages assets so you can follow the pages' evolutions through time.

  • Assets are stored in a git respository to simplify incremental storage and easy retrieval
  • Snapshots and thumbails are stored in a plain repository so they can easily be served by a webserver
  • List of webpages and archives instances are stored in an SQL database
  • Some caching data are stored in the same database

Required tools:

Installation

  • Install the required tools
  • Install the gem
  • All configuration items have default value, have a look bellow if you want to customize them (default database configuration require the sqlite3 gem)
  • Use it !: all the required files and database structure will be created at first call

API

The public API is provided by WebpageArchivist::WebpageArchivist, example:

require 'webpage-archivist'
archivist = WebpageArchivist::WebpageArchivist.new
webpage = archivist.add_webpage('http://www.nytimes.com/' , 'The New York Times')
archivist.fetch_webpages [webpage.id]

Models are available in the lib/webpag-archivist/models.rb file, have a look at the Sequel API if you want to querry them.

Configuration

Basic configuration is done through environment variables:

  • DATABASE_URL : database url, default to sqlite://#{Dir.pwd}/webpage-archivist.sqlite3 syntax is described here, remember to add the required database gem
  • ARCHIVIST_ASSETS_PATH : path to store the assets, default to ./archivist_assets
  • ARCHIVIST_SNAPSHOTS_PATH : path to store the thumbnail, default to ./archivist_snapshots
  • ARCHIVIST_MAX_RUNNING_REQUESTS : number of elements requests running in parallel (not so important since requests are run using EventMachine, default to 20
  • PHANTOMJS_PATH: path to PhantomJS executable if they aren't in the path
  • GRAPHICS_MAGICK_PATH : path to GraphicsMagick executable if it isn't in the path
  • BACKGROUND_THREAD_POOL_SIZE: EventMachine pool size for background tasks like taking the snapshots (default to 20) Configuration for snapshoting is done through the WebpageArchivist::Snapshoter class.

To enable debugging use

WebpageArchivist.log= true

Connect to the database / run migrations

The database connection is available as WebpageArchivist::DATABASE and if you want to run your own migrations use

require 'webpage-archivist/migrations'
WebpageArchivist::Migrations.migration 'create table foo' do
WebpageArchivist::DATABASE.create_table :foos do
  primary_key :id
  # ...
end
end

WebpageArchivist::Migrations.new.run

this way your migrations will be run when the corresponding class is loaded

Released under the MIT license