Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-71369

Rewrite testlib's selftest to generate its own expected data files

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Reported
    • Priority: P4: Low
    • Resolution: Unresolved
    • Affects Version/s: 5.12
    • Fix Version/s: None
    • Component/s: Testing: qtestlib
    • Labels:
      None
    • Platform/s:
      All
    • Technical Risk:
      Normal

      Description

      At present, qtbase/tests/auto/testlib/selftests/ contains a generate_expected_output.py script which runs sub-tests to generate the output that tst_selftests.cpp should expect.
      This involves keeping the script in sync with the C++ code so that what it generates does match what the subtests produce when the test runs them. Both the test and the script do some massaging of the test output so as to ignore irrelevant changes (e.g. paths to files, which vary depending on who's running the test and how they organise their source tree; times taken to run tests; and line numbers within files mentioned in output), in somewhat different ways, that have to be kept compatible.

      It would be better to adapt the C++ selftest to have some way of generating the expected output itself. It could even just generate the output as actual_$sub.$fmt to match each expected_$sub.$fmt (see below) file, making diff-ing easier. (A .gitignore would need to mention /actual_* of course.) A developer satisfied that the generated output is actually correct would just rename actual_$sub.$fmt to expected_$sub.$fmt and save the changed version as part of whatever commit changes the output that should be expected.

      Another thing worth considering in the process of any such re-write: the expected output files are all kept in qtbase/tests/auto/testlib/selftests/ with file-names of form expected_$sub.$fmt where $sub is the sub-test name and $fmt is the output format file-suffix. There would be some sense in moving the files to the sub-directories, renaming each to $sub/expected.$fmt (and the directory tree would be less cluttered if we did this). This could simplify some of the code for the selftest, too (it could just have a QDir for the sub-test and ask it for files to run, without further mention of the sub-directory name when asking for the expected output files).

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

            Assignee:
            macadder Jason McDonald
            Reporter:
            Eddy Edward Welbourne
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:

                Time Tracking

                Estimated:
                Original Estimate - 2 weeks
                2w
                Remaining:
                Remaining Estimate - 2 weeks
                2w
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Gerrit Reviews

                  There are no open Gerrit changes