pmdtester

home

pmd.github.io

code

github.com/pmd/pmd-regression-tester

bugs

github.com/pmd/pmd-regression-tester/issues

build-status

<img src=“github.com/pmd/pmd-regression-tester/workflows/build/badge.svg?branch=main” alt=“Build Status” />

gem-version

<img src=“badge.fury.io/rb/pmdtester.svg” alt=“Gem Version” />

DESCRIPTION:

A regression testing tool ensure that new problems and unexpected behaviors will not be introduced to PMD project after fixing an issue , and new rules can work as expected.

FEATURES/PROBLEMS:

The diff report can be generated according to the base and patch branch of PMD on a list of standard projects(Spring Framework, Hibernate, Solr, etc.)

SYNOPSIS:

Options:

-r, --local-git-repo        path to the local PMD repository
-b, --base-branch           name of the base branch in local PMD repository
-p, --patch-branch          name of the patch branch in local PMD repository
-bc, --base-config          path to the base PMD configuration file
-pc, --patch-config         path to the patch PMD configuration file
-c, --config                path to the base and patch PMD configuration file
-l, --list-of-project       path to the file which contains the list of standard projects
-m, --mode                          the mode of the tool: 'local', 'online' or 'single'
      single: Set this option to 'single' if your patch branch contains changes
        for any option that can't work on main/base branch
      online: Set this option to 'online' if you want to download
        the PMD report of main/base branch rather than generating it locally
      local: Default option is 'local', PMD reports for the base and patch branches are generated locally.

-t, --threads               Sets the number of threads used by PMD. Set threads to 0 to disable multi-threading processing.
-f, --html-flag             whether to not generate the html diff report in single mode
-a, --auto-gen-config       whether to generate configurations automatically based on branch differences,this option only works in online and local mode
--filter-with-patch-config  whether to use patch config to filter baseline result as if --auto-gen-config has been used. This option only works in online mode.
--keep-reports              whether to keep old reports and skip running PMD again if possible
-d, --debug                 whether change log level to DEBUG to see more information
--error-recovery            enable error recovery mode when executing PMD. Might help to analyze errors.
--baseline-download-url     download url prefix from where to download the baseline in online mode
-v, --version               
-h, --help

Quick start

Run local mode

pmdtester -b main -p YOUR_DEVELOPMENT_BRANCH -r PATH_TO_LOCAL_PMD_REPO -a

Run single mode

pmdtester -p YOUR_DEVELOPMENT_BRANCH -pc CONFIG_ONLY_CONTAINS_NEW_PMD_JAVA_RULE -m single

Run online mode

pmdtester -b main -p YOUR_DEVELOPMENT_BRANCH -r PATH_TO_LOCAL_PMD_REPO -m online -a

Output

The tool creates the following folders:

target
├── repositories         <- the analyzed projects are cloned here
│   ├── PROJECT_NAME_1
│   ├── ......
│   └── PROJECT_NAME_n
├── reports
│   ├── BASE_BRANCH_NAME        <- the base baseline is placed here
│   ├── PATCH_BRANCH_NAME       <- the patch baseline is placed here
│   └── diff
│       ├── index.html          <- the summary report of diff reports
│       ├── base_config.xml     <- pmd config from the base branch
│       ├── patch_config.xml    <- pmd config from the patch branch
│       ├── css  <- css resources are placed here
│       ├── js   <- js resources
│       ├── PROJECT_NAME_1
│       │   ├── project_data.js <- contains the violations as js/json
│       │   └── index.html      <- the diff report of PROJECT_1
│       ├── .......
│       └── PROJECT_NAME_n
│           ├── project_data.js <- contains the violations as js/json
│           └── index.xml       <- the diff report of PROJECT_N
├── pmd-bin-<version>-<branch_name>-<sha1>  <- cached pmd builds that are reused
└── pmd-bin-....

The baseline format

branch_name
├── branch_info.json
├── config.xml
├── STANDARD_PROJECT_NAME_1
│   ├── report_info.json
│   └── pmd_report.xml
├── ......................
│   ├── report_info.json
│   └── pmd_report.xml
└── STANDARD_PROJECT_NAME_n
    ├── report_info.info
    └── pmd_report.xml

REQUIREMENTS:

  • Ruby 3.3 or higher

Runtime dependency

nokogiri ~> 1.18
slop ~> 4.10
differ ~> 0.1
rufus-scheduler ~> 3.9
logger-colors ~> 1.0
liquid ~> 5.8
base64 ~> 0.2
bigdecimal ~> 3.1
logger ~> 1.6

Development dependency

hoe ~> 4.2
hoe-bundler ~> 1.5
hoe-git ~> 1.6
minitest ~> 5.25
mocha ~> 2.7
rubocop ~> 1.74
test-unit ~> 3.6
rdoc ~> 6.12
rake ~> 13.2

INSTALL:

gem install pmdtester --pre

DEVELOPERS:

git clone https://github.com/pmd/pmd-regression-tester.git
cd pmd-regression-tester
gem install bundler
bundle config set path "vendor/cache"
bundle install # once
bundle exec rake verify # run this command before commit your changes
bundle exec pmdtester ... # run this to directly execute pmdtester from source

Run all unit tests:
bundle exec rake clean test

Run all integration tests:
bundle exec rake clean integration-test

Run a single test class, e.g.:
bundle exec ruby -I test test/test_project_diff_report.rb
bundle exec ruby -I test test/integration_test_runner.rb

Run a single test, e.g.:
bundle exec ruby -I test test/test_project_diff_report.rb -n test_diff_report_builder

Releasing

  • Update History.md (version and date)

  • Update lib/pmdtester.rb (version)

  • Run “bundle exec rake verify” and add new pmdtester.gemspec (new version)

  • Commit (“Prepare release x.y.z”).

  • Tag this commit (“git tag vx.y.z”).

  • Update History.md and lib/pmdtester.rb for the next development version, run again “bundle exec rake verify”

  • Commit (“Prepare next development version x.y.z-SNAPSHOT”).

  • Push to main.

  • Push the tag. Github Actions will build and publish the new gem

  • A github release is automatically created, verify it on github.com/pmd/pmd-regression-tester/releases

  • To make pmd’s main CI use the new version (in [pmd/pmd](github.com/pmd/pmd/)), go to the directory “.ci/files” and run ‘bundle lock –update`. Commit these changes.

  • Rename milestone to version, close it and create a new “Next” milestone: github.com/pmd/pmd-regression-tester/milestones