Kithe

An in-progress experiment in shareable tools/components for building a digital collections app in Rails.

Build Status Gem Version

What is kithe?

Kithe is a toolkit for building digital collections/repository applications in Rails. It comes out of experience in the samvera community of open source library-archives-museums digital collections/preservation work (but is not a samvera project).

Kithe does not use fedora or valkyrie, but stores all metadata using ActiveRecord. Kithe requires you use postgres 9.5+ as your db. While kithe provides some additional architecture and support on top of "ordinary" ActiveRecord, it tries to mostly let you develop as you would any Rails app, inclduing all choices and you'd normally have, and taking advantage of standard ActiveRecord-based functionality. Kithe bases it's file handling framework on shrine, supporting cloud or local file storage.

Kithe will not give you a working "turnkey" application. It is a collection of tools to help you write a Rails app. You may end up using some but not all of kithe's tools. The range of tools provided and areas of an app given some support in kithe will probably grow over time, in hopefully a careful and cautious way.

In that it provides tools and not a turnkey app, develping an app based on kythe in some ways similar to developing an app based on valkyrie. They both provide basic architecture for modelling/persistence, although in quite different ways. Kithe also provides tools in addition to modelling/persistence, but does not provide the data-mapper/repository pattern valkyrie does, or any built-in abstraction for persisting anywhere but a postgres DB.

If you are comparing it to a "solution bundle" digital collections platform, kithe may seem like more work. But experience has shown me that in our domain, historically "solution bundles" can be less of a "turnkey" approach than they seem, and can have greater development cost over total app lifecycle than anticipated. If you have similar experience that leads you to consider a more 'bespoke' app approach -- you may want to consider kithe. We hope to provide architecturally simple support and standardization for your custom app, taking care of some of the common "hard parts" and leaving you with flexibility to build out the app that meets your needs.

Kithe is at the beginning stages of development. It is pre-1.0 and can change in backwards incompat ways. It is probably not ready for using seriously unless you really know what you're getting into. But trying it out is very invited and encouraged.

You are encouraged to try out kithe, but also welcome to copy any code or just ideas from kithe.

Any questions or feedback of any kind are very welcome and encouraged, in the github project issues, samvera slack, or wherever is convenient.

Kithe is being developed in tandem with the Science History Institute's in-development replacement digital collections app, and you can look there for a model/canonical/demo kithe use.

Kithe parts

Some guide documentation is available to explain each of kithe's major functionality areas. Definitely start with the modelling guide.

  • Modelling and Persistence: It can be somewhwat challenging to figure out a good way to model our data in an rdbms. We give you a hopefully flexible and understandable architecture that is designed to support efficient performance. It's influenced by PCDM and traditional samvera modelling. It's based on attr_json to let you model arbitrary and complex object-oriented data that gets persisted as a serialized json hash. It uses rails Single Table Inheritance to support hetereogenous associations and collections. The modelling classes in some places use postgres-specific features for efficiency.

    • Work representatives. Built in associations to support "representative", using postgres recursive CTE's to compute the "leaf" representative, designed to support efficient use of the DB including pre-loading leaf representatives.
    • Kithe objects use UUIDv4's as internal primary keys, but also provide a "friendlier_id" with a shorter unique alphanumeric identifier for URLs and other UI. By default they are supplied by a postgres stored procedure, but your code can set them to whatever you like.
  • Form support: Dealing with complex and repeatable data, as our modelling layer allows, can be tricky in an HTML form. We supply javascript-backed Rails form input help for repeatable and compound/nested data.

  • File handling: Handling files is at the core of digital repository use cases. We need a file handling framework that is flexible, predictable and reliable, and architected for performance. We try to give you one based on the shrine file attachment toolkit for ruby.

    • Derivatives A flexible and reliable derivatives architecture, designed to ensure data consistency without race conditions, and support efficient DB usage patterns.
  • Solr Indexing: Uses traject for defining mappings from your model objects to what you want in a Solr index. Uses ActiveRecord callbacks to automatically sync saves to solr, with many opportunities for customization.

    • Not coupled to any other kithe components, could be used independently, hypothetically on any ActiveRecord model.
    • Written after review of "prior art" in sunspot and searchkick (which both used AR callback-based indexing), and others.

Also

Setting up your app to use kithe

So you want to start an app that uses kithe. We should later provide better 'getting started' guide. For now some sketchy notes:

  • Again re-iterate that kithe requires your Rails app use postgres, 9.5+.

  • kithe requires Rails 5.2 or 6.0.

  • To install migrations from kithe to setup your database for it's models: rake kithe_engine:install:migrations

  • Kithe view support generally assumes your app uses bootstrap 4, and uses simple form configured with bootstrap settings. See https://github.com/plataformatec/simple_form#bootstrap . So you should install simple_form and bootstrap 4.

  • Specific additional pre-requisites/requirements can sometimes be found in individual feature docs. And include the Javascript from cocoon, for form support for repeatable-field editing forms. We haven't quite figured out our preferred sane approach for sharing Javascript via kithe.

Note that at present kithe will end up forcing your app to use :sql style schema dumps. We may try to fix this.

To be done

Considering some blacklight integration support.

Other components/features may become more clear as we continue to develop. It's possible that kithe won't (at least for a long time) contain controllers themselves (it may contain some helper methods for controllers), or generalized permissions architecture. Both of these are some of the things most particular to specific apps, that are hard to generalize without creating monsters.

Development

This is a Rails 'engine' whose template was created with: rails plugin new kithe --full --skip-test-unit --dummy-path=spec/dummy --database=postgresql

  • Note we have chosen not to make it 'mountable' or 'isolated', I think that would be inappropriate for this kind of gem. It is an engine so it can hook into Rails load paths and config as needed.

  • Note we are currently using the standard rails-generated dummy app in spec/dummy for testing, rather than engine_cart or combustion.

    • Before you run the tests for the first time, create the database by running: rails db:setup. This will create two databases, kithe_development and kithe_test.
    • We do use appraisal to test under multiple rails versions,but still wtih the standard dummy app. It works for both Rails 5.2 and 6.0, because Rails structure changes have settled down.
    • Locally you can run bundle exec appraisal rspec to run tests multiple times for each rails we have configured, or eg bundle exec appraisal rails-60 rspec for a particular one.
    • If the project Gemfile or Appraisal file changes, you may need to re-run bundle exec appraisal install and commit changes.

You can use all rails generators (eg rails g model foo) and it will generate properly for engine, including module namespace. You can generally use rake tasks and other rails commands for dummy app, like rake db:create etc.

We use rspec for testing, [bundle exec] rake spec, rake, or rspec.

Release new gem versions with bundle exec rake release (after making sure ./lib/kithe/version.rb is appropriate)

License

The gem is available as open source under the terms of the MIT License.