+==============================+
| rpminspect and librpminspect |
+==============================+

The tests below are the categories handled by rpmdiff-checker, which
is the ancestor to rpminspect.  Red Hat associates may recognize these
as the grid categories on the rpmdiff webui.  Each test category may
perform a couple of tests.  As tests are documented and implemented,
the specifics will be noted during the work in progress.

* indicates the logic for the tests has moved to librpminspect
! means the work is in-progress


Unranked tests categories to migrate (might also decide to skip them):
----------------------------------------------------------------------
 TEST_SCRIPT
 TEST_TRIGGERS


Other:
    - Check for presence of required external program in abidiff,
      kmidiff, and other functions that fork and exec another program.

    - Add a 'rebase' inspection that just fails if packages are being
      rebased but not on the rebaseable list.  This should be off by
      default but then enabled in a maintenance profile.  Maybe???

    - Modify downloading builds and packages to go to a download cache
      directory.  Modify -k/--keep so it retains builds in the
      download cache and then -k with -d (debug) specified, keep the
      builds but also the workdir files.


Test categories that have been migrated:
----------------------------------------
*TEST_CHANGED (changedfiles)
*TEST_DESKTOP (desktop)
*TEST_EMPTYRPM (emptyrpm)
*TEST_FILELIST (addedfiles and removedfiles)
*TEST_JAVABYTECODE (javabytecode)
*TEST_MANPAGE (manpage)
*TEST_OWNERSHIP (ownership)
*TEST_SHELLSYNTAX (shellsyntax)
*TEST_XML (xml)
*TEST_IPV6 (part of elf inspection in rpminspect)
*TEST_EXECSHIELD (elf)
*TEST_NEEDED (DT_NEEDED)
*TEST_UPSTREAM (upstream; the BZ 'Rebase' keyword check against the
                Errata Tool bug list was not migrated, see below)
*TEST_PERMS (permissions and capabilities)
*TEST_MODPARM (kmod)
*TEST_MODPCIID (kmod)
*TEST_RPMCHANGE (arch and subpackages)
*TEST_CHANGELOG (changelog)
*TEST_PATHNAMES (pathmigration; the forbidden init paths are added
                 to forbidden_path_prefixes in the config file and
                 those are caught by 'addedfiles')
*TEST_SIZE (filesize)
*TEST_SYMLINK (symlinks)
*TEST_SPECFILE (disttag, license, and files)
*TEST_TYPES (types)
*TEST_FILEMOVE (movedfiles)
*TEST_ABI (abidiff and kmidiff)
*TEST_CONFIG (config and doc)
*TEST_PATCHES (patches)
*TEST_VIRUS (virus)
*TEST_POLITICS (politics)
*TEST_RPATH (runpath)
*TEST_METADATA (only checks for bad words, valid license, valid vendor,
                and valid buildhost)
*TEST_REQUIRES (rpmdeps)
*TEST_STRIP    (debuginfo)


Test categories excluded (see MISSING):
---------------------------------------
 TEST_BUILDLOG
 TEST_RPMLINT
 TEST_UPSTREAM (only the rebase keyword check)
 TEST_ELFLINT
 TEST_MULTILIB
 TEST_SPECFILE (only the 32-bit x86 $RPM_OPT_FLAGS check)
 TEST_METADATA (the 'fedora without red hat' string check)


+============+
| test suite |
+============+

Integration tests to write for rpminspect
-----------------------------------------

The test suite needs more integration tests written.  rpminspect is a
program you run at the command line and it reads RPM files for input.
The test suite is not simply a unit test suite, but rather a detailed
integration test suite that creates simulated RPM packages to trigger
pass and fail cases in each of the different inspections.

There are six input types to rpminspect:

    * Single source RPM
    * Two source RPMs to compare
    * Single binary RPM
    * Two binary RPMs to compare
    * A single Koji build
    * Two Koji builds to compare

The test suite carries baseclass.py that defines some base classes
that you can build on for the different integration tests.  They are:

    * Single source RPM              --> TestSRPM
    * Two source RPMs to compare     --> TestCompareSRPM
    * Single binary RPMs             --> TestRPMs
    * Two binary RPMs to compare     --> TestCompareRPMs
    * A single Koji build            --> TestKoji
    * Two Koji builds to compare     --> TestCompareKoji
    * A single module build          --> TestModule
    * Two module build to compare    --> TestCompareModules

The TestRPMs and TestCompareRPMs in practice are used to test single
binary RPMs and not multiple RPMs, but I chose the name a long time
ago for the class.

These classes set up the test environment with rpminspect from the
source tree, simulated configuration data, and then test RPM inputs
built using the 'rpmfluff' module (https://pagure.io/rpmfluff).  In
the case of base classes that deal with a single SRPM, RPM, or Koji
build, the class contains "self.rpm" that you can modify.  In the
compare bare classes, you have "self.before_rpm" and "self.after_rpm".

The base classes will invoke rpminspect and capture the output as
JSON.  You set "self.inspection", "self.result", and "self.waiverauth"
to the values you expect in the output from that test input.  The idea
is that the actual integration tests are mostly fill-in-the-blanks
type classes.  Some get more complicated due to limitations in
rpmfluff or something that just needs to be handled in a specific way.

The integration test suite needs to test each inspection with each
input type above and check the result of each test, both in the
passing and failing case.  All of the inspections are written as
lib/inspect_*.c files.  Read through those source files and look for
the lines with "add_result()" calls.  When add_result() is called,
that's where a specific test was performed.  Read the code here to
determine what it's looking for and then construct the test case to
test for the passing case and then another set of classes for the
failing case.

For example, if there's an inspection called "uppercase" and its job
is to check and make sure no files are installed with uppercase
letters, the integration test suite needs 12 different tests.  The
first 6 are for the inputs noted above and to test for the passing
case.  That is a file is installed with lowercase letters.  The next 6
are for the inputs again, but for the failing case.  That is,
construct one of the test RPMs to have a file with an uppercase name.

Keep in mind that some test cases only apply to SRPM files and some
only apply to binary RPMS, so that will impact the number of test
cases you will need.

The following inspections lack integration tests:

    arch
    dsodeps
    javabytecode
    kmidiff
    movedfiles
    subpackages

There are plenty of functions that can use unit tests too.  Look
through the source in lib/ for functions that this would apply to and
feel free to add more unit tests under test/lib/.


+=======+
| Other |
+=======+

General functionality
---------------------

* In the [inspections] section in config files, all for "ALL = on"
  and "ALL = off" to toggle everything.  Similar to the command
  line option.

* Add verbose runtime reporting.  This is currently only available
  for downloading packages.  Need to figure out how to separate it
  from the test results.

* Text mode reporting formatting.  Allow a user-specified column width
  and align output on columns.

  * Add a new output format that looks like rpmdiff from rpmlint.
    Specifically something like this:

    $ rpmdiff ./auditwheel-4.0.0-1.fc36.noarch.rpm ./auditwheel-5.0.0-1.fc36.noarch.rpm
    removed     REQUIRES python3.10dist(setuptools)
    removed     PROVIDES auditwheel = 4.0.0-1.fc36
    removed     PROVIDES python3.10dist(auditwheel) = 4
    removed     PROVIDES python3dist(auditwheel) = 4
    added       PROVIDES auditwheel = 5.0.0-1.fc36
    added       PROVIDES python3.10dist(auditwheel) = 5
    added       PROVIDES python3dist(auditwheel) = 5
    ..........T /usr/bin/auditwheel
    ..........T /usr/lib/python3.10/site-packages/auditwheel
    removed     /usr/lib/python3.10/site-packages/auditwheel-4.0.0.dist-info
    removed     /usr/lib/python3.10/site-packages/auditwheel-4.0.0.dist-info/INSTALLER
    removed     /usr/lib/python3.10/site-packages/auditwheel-4.0.0.dist-info/LICENSE
    removed     /usr/lib/python3.10/site-packages/auditwheel-4.0.0.dist-info/METADATA
    removed     /usr/lib/python3.10/site-packages/auditwheel-4.0.0.dist-info/WHEEL
    removed     /usr/lib/python3.10/site-packages/auditwheel-4.0.0.dist-info/entry_points.txt
    removed     /usr/lib/python3.10/site-packages/auditwheel-4.0.0.dist-info/top_level.txt
    added       /usr/lib/python3.10/site-packages/auditwheel-5.0.0.dist-info
    added       /usr/lib/python3.10/site-packages/auditwheel-5.0.0.dist-info/INSTALLER
    added       /usr/lib/python3.10/site-packages/auditwheel-5.0.0.dist-info/LICENSE
    added       /usr/lib/python3.10/site-packages/auditwheel-5.0.0.dist-info/METADATA
    added       /usr/lib/python3.10/site-packages/auditwheel-5.0.0.dist-info/WHEEL
    added       /usr/lib/python3.10/site-packages/auditwheel-5.0.0.dist-info/entry_points.txt
    added       /usr/lib/python3.10/site-packages/auditwheel-5.0.0.dist-info/pbr.json
    added       /usr/lib/python3.10/site-packages/auditwheel-5.0.0.dist-info/top_level.txt
    ..........T /usr/lib/python3.10/site-packages/auditwheel/__init__.py
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/__main__.py
    ..........T /usr/lib/python3.10/site-packages/auditwheel/__pycache__
    ..5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/__init__.cpython-310.opt-1.pyc
    ..5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/__init__.cpython-310.pyc
    ..5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/__main__.cpython-310.opt-1.pyc
    ..5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/__main__.cpython-310.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/condatools.cpython-310.opt-1.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/condatools.cpython-310.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/elfutils.cpython-310.opt-1.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/elfutils.cpython-310.pyc
    added       /usr/lib/python3.10/site-packages/auditwheel/__pycache__/error.cpython-310.opt-1.pyc
    added       /usr/lib/python3.10/site-packages/auditwheel/__pycache__/error.cpython-310.pyc
    ..5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/genericpkgctx.cpython-310.opt-1.pyc
    ..5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/genericpkgctx.cpython-310.pyc
    ..5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/hashfile.cpython-310.opt-1.pyc
    ..5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/hashfile.cpython-310.pyc
    S.5.N.....T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/lddtree.cpython-310.opt-1.pyc
    S.5.N.....T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/lddtree.cpython-310.pyc
    added       /usr/lib/python3.10/site-packages/auditwheel/__pycache__/libc.cpython-310.opt-1.pyc
    added       /usr/lib/python3.10/site-packages/auditwheel/__pycache__/libc.cpython-310.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/main.cpython-310.opt-1.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/main.cpython-310.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/main_addtag.cpython-310.opt-1.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/main_addtag.cpython-310.pyc
    ..5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/main_lddtree.cpython-310.opt-1.pyc
    ..5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/main_lddtree.cpython-310.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/main_repair.cpython-310.opt-1.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/main_repair.cpython-310.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/main_show.cpython-310.opt-1.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/main_show.cpython-310.pyc
    added       /usr/lib/python3.10/site-packages/auditwheel/__pycache__/musllinux.cpython-310.opt-1.pyc
    added       /usr/lib/python3.10/site-packages/auditwheel/__pycache__/musllinux.cpython-310.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/patcher.cpython-310.opt-1.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/patcher.cpython-310.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/repair.cpython-310.opt-1.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/repair.cpython-310.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/tmpdirs.cpython-310.opt-1.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/tmpdirs.cpython-310.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/tools.cpython-310.opt-1.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/tools.cpython-310.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/wheel_abi.cpython-310.opt-1.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/wheel_abi.cpython-310.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/wheeltools.cpython-310.opt-1.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/__pycache__/wheeltools.cpython-310.pyc
    ..........T /usr/lib/python3.10/site-packages/auditwheel/_vendor
    ..........T /usr/lib/python3.10/site-packages/auditwheel/_vendor/__init__.py
    ..........T /usr/lib/python3.10/site-packages/auditwheel/_vendor/__pycache__
    ..5.......T /usr/lib/python3.10/site-packages/auditwheel/_vendor/__pycache__/__init__.cpython-310.opt-1.pyc
    ..5.......T /usr/lib/python3.10/site-packages/auditwheel/_vendor/__pycache__/__init__.cpython-310.pyc
    ..........T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel
    added       /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/LICENSE.txt
    ..........T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/__init__.py
    ..........T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/__pycache__
    ..5.......T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/__pycache__/__init__.cpython-310.opt-1.pyc
    ..5.......T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/__pycache__/__init__.cpython-310.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/__pycache__/pkginfo.cpython-310.opt-1.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/__pycache__/pkginfo.cpython-310.pyc
    ..5.......T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/__pycache__/util.cpython-310.opt-1.pyc
    ..5.......T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/__pycache__/util.cpython-310.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/__pycache__/wheelfile.cpython-310.opt-1.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/__pycache__/wheelfile.cpython-310.pyc
    ..........T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/cli
    ..........T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/cli/__init__.py
    ..........T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/cli/__pycache__
    ..5.......T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/cli/__pycache__/__init__.cpython-310.opt-1.pyc
    ..5.......T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/cli/__pycache__/__init__.cpython-310.pyc
    ..5.......T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/cli/__pycache__/convert.cpython-310.opt-1.pyc
    ..5.......T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/cli/__pycache__/convert.cpython-310.pyc
    ..5.......T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/cli/__pycache__/pack.cpython-310.opt-1.pyc
    ..5.......T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/cli/__pycache__/pack.cpython-310.pyc
    ..5.......T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/cli/__pycache__/unpack.cpython-310.opt-1.pyc
    ..5.......T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/cli/__pycache__/unpack.cpython-310.pyc
    ..........T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/cli/convert.py
    ..........T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/cli/pack.py
    ..........T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/cli/unpack.py
    ..........T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/pkginfo.py
    ..........T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/util.py
    ..........T /usr/lib/python3.10/site-packages/auditwheel/_vendor/wheel/wheelfile.py
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/condatools.py
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/elfutils.py
    added       /usr/lib/python3.10/site-packages/auditwheel/error.py
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/genericpkgctx.py
    ..........T /usr/lib/python3.10/site-packages/auditwheel/hashfile.py
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/lddtree.py
    added       /usr/lib/python3.10/site-packages/auditwheel/libc.py
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/main.py
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/main_addtag.py
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/main_lddtree.py
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/main_repair.py
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/main_show.py
    added       /usr/lib/python3.10/site-packages/auditwheel/musllinux.py
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/patcher.py
    ..........T /usr/lib/python3.10/site-packages/auditwheel/policy
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/policy/__init__.py
    ..........T /usr/lib/python3.10/site-packages/auditwheel/policy/__pycache__
    S.5.N.....T /usr/lib/python3.10/site-packages/auditwheel/policy/__pycache__/__init__.cpython-310.opt-1.pyc
    S.5.N.....T /usr/lib/python3.10/site-packages/auditwheel/policy/__pycache__/__init__.cpython-310.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/policy/__pycache__/external_references.cpython-310.opt-1.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/policy/__pycache__/external_references.cpython-310.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/policy/__pycache__/versioned_symbols.cpython-310.opt-1.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/policy/__pycache__/versioned_symbols.cpython-310.pyc
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/policy/external_references.py
    added       /usr/lib/python3.10/site-packages/auditwheel/policy/manylinux-policy.json
    added       /usr/lib/python3.10/site-packages/auditwheel/policy/musllinux-policy.json
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/policy/policy-schema.json
    removed     /usr/lib/python3.10/site-packages/auditwheel/policy/policy.json
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/policy/versioned_symbols.py
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/repair.py
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/tmpdirs.py
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/tools.py
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/wheel_abi.py
    S.5.......T /usr/lib/python3.10/site-packages/auditwheel/wheeltools.py
    ..........T /usr/share/doc/auditwheel
    S.5.......T /usr/share/doc/auditwheel/README.rst

    $ rpmdiff ./auditwheel-4.0.0-1.fc36.src.rpm ./auditwheel-5.0.0-1.fc36.src.rpm
    removed     REQUIRES python3dist(setuptools)
    removed     REQUIRES python3dist(tox) >= 1.6
    removed     REQUIRES python3dist(tox-current-env) >= 0.0.6
    added       REQUIRES python3dist(setuptools) >= 42
    removed     PROVIDES auditwheel = 4.0.0-1.fc36
    added       PROVIDES auditwheel = 5.0.0-1.fc36
    removed     319.patch
    removed     auditwheel-4.0.0.tar.gz
    added       auditwheel-5.0.0.tar.gz
    S.5.......T auditwheel.spec

* Drop the need to fork/exec msgunfmt and just read in the po files
  directly.  This should be possible and will give us the information
  necessary for comparison purposes.

* Drop the need to fork/exec desktop-file-validate if a library
  mechanism can be found instead.

* Translations.  Use po/rpminspect.pot as a template and add more
  languages.


Documentation
-------------

Developer guide
Testing guide
User guide
    Use Doxygen for the C source but tie that in with Sphinx so
    publishing can be done on readthedocs.io

Also rpminspect.yaml.5 documenting the configuration file and the
profile configuration files.
