keinaufwand-sync

Reusable Rails persistence for the Keinaufwand API. The gem supports full/delta API sync, schema-backed Active Record models, SQLite and PostgreSQL JSON queries, local asset mirroring, generated JSON indexes, and signed incremental webhooks.

Part of Keinaufwand.

Install

gem "keinaufwand"
gem "keinaufwand-sync"

bin/rails generate keinaufwand:sync:install
bin/rails keinaufwand:indices:generate
bin/rails db:migrate

The install generator creates keinaufwand_records, keinaufwand_sync_states, keinaufwand_assets, and keinaufwand_webhook_receipts. It also mounts /media/:id and the webhook Engine at /webhooks/keinaufwand.

Configuration

Keinaufwand::Sync.configure do |config|
  config.client = -> { MyKeinaufwandClient.build }
  config.webhook_secret = -> { Rails.application.credentials.dig(:keinaufwand, :webhook_secret) }
  config.storage_adapter = :sqlite3
end

The consumer selects resources and JSON indexes in config/keinaufwand.yml. The versioned API schema remains bundled in keinaufwand.

bin/rails keinaufwand:sync
bin/rails keinaufwand:resync
bin/rails keinaufwand:indices:generate

Application models inherit from Keinaufwand::Record. Their id is the upstream ID; id and STI type form the composite primary key. Schema fields and associations use normal Active Record-style find, exists?, where, and order calls.

Console progress

Version 0.1.1 adds optional console output without changing the default quiet sync:

Keinaufwand::Sync.sync!(output: $stdout)

Output includes the effective sync cursor, committed per-page New/Upd/Del/Skip counts, the API record total, percentage, records per second, estimated time remaining for the current resource, per-resource fetch/media/database-write timings, and a final completion summary. Lines are flushed immediately, including when stdout is redirected. Upd counts existing records written by the sync, even if their data was unchanged. Skip counts stale snapshots and deletion markers for records already absent. Counts describe this polling run; concurrent webhook writes are not included.

Progress updates after each committed page (up to 100 records). ETA uses the current resource’s average processing rate, including fetch and media time; it is approximate and shown as unknown when the API omits its total. Page fetch and media-processing messages identify work between updates.

Failed resources report the error and stop the run with the original exception. Rolled-back pages are not counted as processed, and the resource cursor only advances after the complete resource sync succeeds.

Webhooks

The mounted Engine accepts the exact JSON body signed by Keinaufwand in X-Keinaufwand-Signature. Processing is synchronous so a 200 response means the SQLite/PostgreSQL transaction committed. Event receipts make retries idempotent and preserve the latest source timestamp so an older delayed update cannot overwrite or resurrect newer data.

Regular keinaufwand:sync calls request each aggregate root with updated_since and apply _deleted records returned by widened soft-visibility scopes. Inline associations are stored inside the root JSON and exposed as schema-backed model objects without duplicate STI rows. Retried webhooks carry hard-delete tombstones; keinaufwand:resync repairs exceptional drift after a permanently failed delivery.

Polling compares source timestamps with both local records and webhook receipts after downloading media, then applies each page in a database transaction. Older polling snapshots cannot overwrite a newer webhook update or resurrect a record deleted by a newer webhook. Index generation skips unchanged indexes and uses distinct migration names for later configuration changes.