Qt uses Macros and config files from install tree when building qtdeclarative "again"

If I build and install Qt with qtdeclarative module and try to build qtdeclarative as standalone module against the installed Qt the Qt6Qml package is used from install directory instead of one I'm trying to build.
Steps:

mkdir build && cd build
<path/to/>qt5/configure -submodule qtdeclarative -nomake tests -nomake examples -prefix ../install
ninja install
cd ..
mkdir build_qtdeclarative_only && cd build_qtdeclarative_only
../install/bin/qt-configure-module ../qt5/qtdeclarative -- -DQT_BUILD_TESTS=ON

Note that issue is not reproducible if I don't enable QT_BUILD_TESTS flag.