DiffTest

Configuration

enabled

  DiffTest.configure do |config|
    config.enabled = ENV['CI'].present?

    config.project_root = Rails.root

    config.vendor_bundle_path = File.join(Rails.root, 'vendor')

    # Default: All files generated by git
    config.project_files = [
      -> { `git -C "#{Rails.root}" ls-files`.split("\n") }
    ]

    # These paths don't factor into determining if tests should be run at all.
    # Its as if they aren't even there.
    config.ignored_paths = [
      'db/migrate/**/*',
      '**/*.md',
      # 'doc/**/*',
      # 'services/video_encoding/**/*
      # 'simplero-mobile-app/**/*
      # **/*.css
      # **/*.scss
    ]

    # Add patterns where you ruby/template files live.
    # If you change any file that does not match these patterns, we will run all the tests.
    config.tracked_paths = [
      'app/**/*.rb',
      'app/**/*.html.erb',
      'lib/**/*.rb',
      'lib/**/*.html.erb',
      'test/**/*.rb',
    ]

    # For this to work correctly,
    # You need to ensure that these files are annotated
    # You can automatically annotate your files with
    #   rake diff_test:annotate
    # This also happens automatically when you run RAILS_ENV=test rails assets:precompile
    config.tracked_js_paths = [
      'app/javascript/**/*.{js,jsx,ts,tsx,mjs,cjs,coffee}',
      'app/assets/javascripts/**/*.{js,jsx,ts,tsx,mjs,cjs,coffee}',
    ]

  end

TODO

  • Testing strategy

* Documentation

  • Fixture tracker
  • Factory tracker

  • CSS tracker

  • TracePointTracker

  • ConstantTracker (depends_on TracePointTracker)

  • FileTracker

  • FixtureTracker

- JavascriptTracker

  • ImpactedFileTracker
    • FileTracker
    • ConstantTracker
    • TracePointTracker

Number of commits (out of last 2000) with any changes NOT matching allowed patterns: base_only => 592 with_factories_fixtures => 583 with_js_ts => 422 with_js_ts_css => 325 with_js_css_db_except_schema => 298 with_factories_fixtures_js_css => 316 with_factories_fixtures_js_css_db_except_schema => 288 with_factories_fixtures_except_schema => 558 with_factories_fixtures_css_except_schema => 487 with_css => 521 with_css_except_schema => 497 ruby cookbooks/track_non_matching.rb 1000 23.75s user 8.41s system 95% cpu 33.642 total

  • Write tests on diff_test_web

    • These are all system tests.
    • Creates test suite executions
    • Creates test executions
    • Runs tests when impacted files change
    • Runs tests when project files change
    • Runs system test when js version change (in unrelated file) but no previous js run
    • Skips system test when no js version change
    • Runs system test when related js file changes
    • Skip system test when related js file changes after a succesful run

  • ImpactedFileTracker

    • Ruby
    • HTML
    • Javascript

Test TODO

Test rails app [ ] UserTest model - class access [ ] AccountTest model - calls searchable - uses fixtures for users and accounts [ ] Controller/Integration test [ ] System test - AccountController#index - views, controller, js, css, fixtures - AccountController#partial - AccountController#stream - AccountController#components - AccountController#layouts - AccountController#helpers [ ] Mailer test [ ] Job test

Tasks

X Allow login X Allow signup xpassword confirmation controller X Allow account creation xhandle check controller xinherit from controller X Account#show xMake account#show view render a user_card partial X Allow customer creation (assoc to user & account) x Write model unit tests for User x Write model unit tests for Account x View component with html x View component with call x Write integration tests for AccountController around the handle_connect helper x Make a mailer that sends a email about the details of a newly created account x Make a job that runs every hour deleting accounts with invalid users x View comopnent test x Write system test for signup password mismatch test x Write system test for login x Write system test for account creation

  • handle check
  • currency (USD)
  • user x Write system test for customer creation -

DiffTestWeb [ ] Test each test written above assert impacted file. run it again assert skipped change some unrelated files run it again change some related files assert not skipped

[ ] Test various configurations Changing JS version hash Changing project version hash All JS files annotated tracking

Works locally

Won't auto-annotate files locally

Annotate files in CI through precompile or db:prepare

LAter

config.enabled = Rails.env.test?

config.enabled = Rails.env.test? && ENV['CI'].present?

Annotates JS files so they are be able to tracked as well.

By default, we only do this in CI env.

As an alternative, you can run bundle exec rake diff_test:annotate manually.

To unannotate, simply run bundle exec rake diff_test:unannoate

config.auto_annotate_js_files = :ci

TODO

x Perforamnce x Limit results x Tests pass weh nwe are down x Same key /workflow_id succeeds x Unskippable branches x Skip from branch/commit/author name

  • Initialier/install script
  • Set unskippable x Wrap run_one_method ?
    • skip setup hooks
    • tracks stuff in before_setup as well
    • path/skip setup_fixtures x batch send beacon

46s - plain 50s - with rubyfile enable/disable clear 68s - with tracepoint but immediate next 118s - with tracepoint but push 424s - with tracepoint but processing like it should 186s - with tracepoint but processing like it should (Except c_call) 73s - just reading event.path inside breakpoint

18s - all skipped

test/services/zoom_webhook_event_processor_test.rb 06.00s - No Diff Test

41.15s - Diff Testc 37.52 - Diff Testc(only file) 13.52 - Diff Testc(only constant) 10.00 - Diff Testc(only tracepoint)

21.62s - Diff Test 20.08 - Diff Test (only file) 11.50 - Diff Test (only constant) 10.50 - Diff Test (only tracepoint) 12.68 - Diff Test (constant + file_in_project + add to set)

/Users/owaiskhan/.rbenv/versions/3.3.1/lib/ruby/gems/3.3.0/gems/diff_test-0.8.0/lib/diff_test/test_suite_execution.rb:43: warning: mismatched indentations at 'end' with 'def' at 41 /Users/owaiskhan/.rbenv/versions/3.3.1/lib/ruby/gems/3.3.0/gems/diff_test-0.8.0/lib/diff_test/test_suite_execution.rb:69: warning: assigned but unused variable - response

overhead of const TracePoint and file path prefix native method_added/const_added

WRITE test for it setting time from previous successful run

Perf improvements

TestExecution

- New Test table (just for test name)

- Drop updated_at

- impacted_file_hash becomes integer

ImpactedFile

- add file_path and file_hash

TestExecutionImpactedFile

- Store file hash with impacted file
- TestExecutionImpactedFile then just belongs to test execution and impacted_file