qt_internal_add_test's non builtin_testdata mode can't be relied upon for QFINDTESTDATA

qt_internal_add_test has 2 code paths for handling test data, either embedding it into a resource, or trying to install the test data somewhere and hoping it gets picked up by test binaries.

The latter doesn't work in the current state, for multiple reasons.

But first here's the order that QFINDTESTDATA looks for data with Qt 6.5, with examples.

Assuming the test

/Volumes/T3/Dev/qt/worktrees/dev/qtbase/tests/auto/widgets/widgets/qframe/tst_qframe.cpp

built in-tree with the qtbase build at

/Volumes/T3/Dev/qt/builds/dev-mac-prefix/qtbase/tests/auto/widgets/widgets/qframe/tst_qframe.app/Contents/MacOS/tst_qframe

which was configured with

configure -developer-build -make tests -prefix /Volumes/T3/Dev/qt/builds/dev-mac-prefix/installed

with the following defines

-DQT_TESTCASE_BUILDDIR=\"/Volumes/T3/Dev/qt/builds/dev-mac-prefix/qtbase/tests/auto/widgets/widgets/qframe\" 
-DQT_TESTCASE_SOURCEDIR=\"/Volumes/T3/Dev/qt/worktrees/dev/qtbase/tests/auto/widgets/widgets/qframe\"

$ pwd 
/Volumes/T3/Dev/qt/builds/dev-mac-prefix/qtbase/tests/auto/widgets/widgets/qframe
$ ./tst_qframe.app/Contents/MacOS/tst_qframe

QFINDTESTDATA("images/box_noshadow_0_0.png") resolves to the following call

QTest::qFindTestData(
base="images/box_noshadow_0_0.png", file="/Volumes/T3/Dev/qt/worktrees/dev/qtbase/tests/auto/widgets/widgets/qframe/tst_qframe.cpp", 
line=165, 
builddir="/Volumes/T3/Dev/qt/builds/dev-mac-prefix/qtbase/tests/auto/widgets/widgets/qframe", 
sourcedir="/Volumes/T3/Dev/qt/worktrees/dev/qtbase/tests/auto/widgets/widgets/qframe")

and looks for the file in the following order:

1. relative to test binary
uses QCoreApplication::applicationDirPath()

/Volumes/T3/Dev/qt/builds/dev-mac-prefix/qtbase/tests/auto/widgets/widgets/qframe/tst_qframe.app/Contents/MacOS/images/box_noshadow_0_0.png

2. installed tests path
uses QLibraryInfo::path(QLibraryInfo::TestsPath);

/Volumes/T3/Dev/qt/builds/dev-mac-prefix/qtbase/tests/tst_qframe/images/box_noshadow_0_0.png

3. relative to test source

        // srcdir is the directory containing the calling source file.
        QFileInfo srcdir(QFileInfo(QFile::decodeName(file)).path());

/Volumes/T3/Dev/qt/worktrees/dev/qtbase/tests/auto/widgets/widgets/qframe/images/box_noshadow_0_0.png

4. resources

5. current working directory

const QString candidate = QDir::currentPath() + u'/' + base;

/Volumes/T3/Dev/qt/builds/dev-mac-prefix/qtbase/tests/auto/widgets/widgets/qframe/images/box_noshadow_0_0.png

6. main source directory

const QString candidate = QTest::mainSourcePath % u'/' % base;

/Volumes/T3/Dev/qt/worktrees/dev/qtbase/tests/auto/widgets/widgets/qframe/images/box_noshadow_0_0.png

7. supplied source directory

const QString candidate = QFile::decodeName(sourcedir) % u'/' % base;

/Volumes/T3/Dev/qt/worktrees/dev/qtbase/tests/auto/widgets/widgets/qframe/images/box_noshadow_0_0.png

Now, looking at the (2) installed tests path

/Volumes/T3/Dev/qt/builds/dev-mac-prefix/qtbase/tests/tst_qframe/images/box_noshadow_0_0.png

That's actually a path in the build dir!
And it's a lower-cased executable name, not the name of the subdir of the test.

And our qt_install() call in qt_internal_add_tests installs into

/Volumes/T3/Dev/qt/builds/dev-mac-prefix/installed/tests/auto/widgets/widgets/qframe/winpanel_noshadow_0_0.png

which won't be picked up by any of QFINDTESTDATA candidates.

What's worse, in a standalone tests builds done using qt-internal-configure-tests script, we install into a fake prefix which definitely won't be picked up. This was done to ensure we don't accidentally ship test artifacts with the qt libraries in an official release.

/Volumes/T3/Dev/qt/builds/dev-mac-prefix/st_qtbase/fake_prefix/tests/auto/widgets/widgets/qframe/winpanel_noshadow_0_0.png/winpanel_noshadow_0_0.png

Also, the qt_install() call doesn't handle subdirectories properly, they are not preserved upon installation, so the images subdir above is not preserved.

Our CI currently zips up the the standalone tests build dir, unpacks it on a different VM, and runs ctest -V.
If the binaries were to depend on the installed test data, we'd have to also run ninja install, and zip up the installed data and then unpack it in the new VM, which would be weird, why would build-time binaries depend on installed data.
Especially considering that we don't install the test binaries themselves.

The only reason tests pass in the CI is because we always unpack the qt sources into the new VMs at the exact locations where they were when built.
And for some of the cross-compiling platforms, we depend on the builtin testdata that uses resources.

The testdata installation mechanism was ported over from qmake times, but it's not entirely clear to me why qmake had it either.
I would personally rely either on builtin test data using resources, or implement copying of the testdata to the build dir next to the binaries, and hook up QFINDTESTDATA to find those. Although that seemed to cause some issues for certain tests.
See https://codereview.qt-project.org/c/qt/qtbase/+/290319