Uploaded image for project: 'Qt for Python'
  1. Qt for Python
  2. PYSIDE-1053

[Shiboken] "Shiboken2Targets-*.cmake" Erroneously Links to Single Python Version

    XMLWordPrintable

Details

    • Bug
    • Resolution: Won't Do
    • Not Evaluated
    • None
    • 5.12.4, 5.13.0
    • Shiboken
    • None
    • Gentoo Linux
    • Linux/Wayland, Linux/X11, Linux/Other display system

    Description

      Firstly, a hearty congratulations to the hard-working "Qt for Python" team. PySide2 has made staggering inroads over the past several years – a testament to the active dedication of Alex Blasche and friends. You guys collectively rock!

      Secondly, I've co-maintained the official Shiboken2, PySide2, and pyside2-tools ebuilds for Gentoo Linux AKA, "Fighting Man's Linux" since their inception. All's gone mostly well... until the most recent minor version bump from Shiboken2 5.12.3 to 5.12.4.

      Shiboken2 ≥ 5.12.4 contains a significant cmake bug that Shiboken2 ≤ 5.12.3 assuredly did not. Although we've since resolved this bug on our end with a sed-based patch against the problematic cmake file in question, an upstream resolution in the Shiboken2 codebase would be dramatically preferable to our fragile distribution-specific kludge.

      This bug has yet to be reported by other package maintainers on other Linux distributions. Nonetheless, the generality of this bug suggests that Linux distributions other than Gentoo Linux will probably suffer a similar fate.

      I lost two languorous summer days to this bug. For humanity, I write this bug report that nobody else should suffer my fate.

      Yes, Yes. What's This All About?

      Business as usual then, is it? Very well.

      The top-level
      pyside-setup-everywhere-src-5.12.4/sources/shiboken2/CMakeLists.txt makefile auto-generates the /usr/lib64/cmake/Shiboken2-5.12.4/Shiboken2Targets-${BUILD_TYPE}.cmake file at installation time, where ${BUILD_TYPE} is either Release or Debug (e.g. Shiboken2Targets-release.cmake for release builds).

      In the case of Gentoo Linux, we globally override ${BUILD_TYPE} for all cmake-based projects to Gentoo... for "reasons." Shiboken2 thus auto-generates the /usr/lib64/cmake/Shiboken2-5.12.4/Shiboken2Targets-gentoo.cmake file for us.

      Regardless of filename, this file erroneously targets only the last libshiboken2*.so library to have been installed rather than the current libshiboken2*.so library being linked against. That's bad. That's so bad, in fact, that I almost gave up supporting Shiboken2, PySide2, and pyside2-tools under Gentoo. I persevered despite the perpetual anguish – and so can you!

      To help explain this subtle insanity, let's examine the contents of an example Shiboken2Targets-gentoo.cmake file:

      #----------------------------------------------------------------
# Generated CMake target import file for configuration "Gentoo".
#----------------------------------------------------------------

# Commands may need to know the format version.
set(CMAKE_IMPORT_FILE_VERSION 1)

# Import target "Shiboken2::libshiboken" for configuration "Gentoo"
set_property(TARGET Shiboken2::libshiboken APPEND PROPERTY IMPORTED_CONFIGURATIONS GENTOO)
set_target_properties(Shiboken2::libshiboken PROPERTIES
  IMPORTED_LOCATION_GENTOO "/usr/lib64/libshiboken2-python3.6.so.5.12.4"
  IMPORTED_SONAME_GENTOO "libshiboken2-python3.6.so.5.12"
  )

list(APPEND _IMPORT_CHECK_TARGETS Shiboken2::libshiboken )
list(APPEND _IMPORT_CHECK_FILES_FOR_Shiboken2::libshiboken "/usr/lib64/libshiboken2-python3.6.so.5.12.4" )

# Import target "Shiboken2::shiboken2" for configuration "Gentoo"
set_property(TARGET Shiboken2::shiboken2 APPEND PROPERTY IMPORTED_CONFIGURATIONS GENTOO)
set_target_properties(Shiboken2::shiboken2 PROPERTIES
  IMPORTED_LOCATION_GENTOO "${_IMPORT_PREFIX}/bin/shiboken2"
  )

list(APPEND _IMPORT_CHECK_TARGETS Shiboken2::shiboken2 )
list(APPEND _IMPORT_CHECK_FILES_FOR_Shiboken2::shiboken2 "${_IMPORT_PREFIX}/bin/shiboken2" )

# Commands beyond this point should not need to know the version.
set(CMAKE_IMPORT_FILE_VERSION)

      Note the repeated references to /usr/lib64/libshiboken2-python3.6.so.5.12.4. That's bad. Why? Because that forces downstream cmake projects (namely, PySide2) to unconditionally link against this single libshiboken2*.so library. As the -python3.6 in this filename suggests, this library is in fact specific to Python 3.6. Is this a problem?

      This is a problem. Attempting to link against this library when building a Python 2.7-specific version of PySide2 produces the following fatal error at PySide2 compilation time:

      ...
(test) [3767ms] Running Header generator...                                 [OK]
Done, (test) 3770ms (567 known issues)
[197/1326] cd /var/tmp/portage/dev-python/pyside-5.12.4/work/pyside-setup-everywhere-src-5.12.4/sources/pyside2-python2_7/PySide2/QtCore && /usr/bin/cmake -E env LD_LIBRARY_PATH=/var/tmp/portage/dev-python/pyside-5.12.4/work/pyside-setup-everywhere-src-5.12.4/sources/pyside2-python2_7/libpyside:/usr/lib64 /usr/bin/python2.7 /var/tmp/portage/dev-python/pyside-5.12.4/work/pyside-setup-everywhere-src-5.12.4/sources/pyside2/PySide2/QtCore/../support/generate_pyi.py QtCore --sys-path /var/tmp/portage/dev-python/pyside-5.12.4/work/pyside-setup-everywhere-src-5.12.4/sources/pyside2-python2_7 /usr/lib64/python2.7/site-packages/shiboken2
FAILED: PySide2/QtCore/CMakeFiles/QtCore_pyi 
cd /var/tmp/portage/dev-python/pyside-5.12.4/work/pyside-setup-everywhere-src-5.12.4/sources/pyside2-python2_7/PySide2/QtCore && /usr/bin/cmake -E env LD_LIBRARY_PATH=/var/tmp/portage/dev-python/pyside-5.12.4/work/pyside-setup-everywhere-src-5.12.4/sources/pyside2-python2_7/libpyside:/usr/lib64 /usr/bin/python2.7 /var/tmp/portage/dev-python/pyside-5.12.4/work/pyside-setup-everywhere-src-5.12.4/sources/pyside2/PySide2/QtCore/../support/generate_pyi.py QtCore --sys-path /var/tmp/portage/dev-python/pyside-5.12.4/work/pyside-setup-everywhere-src-5.12.4/sources/pyside2-python2_7 /usr/lib64/python2.7/site-packages/shiboken2
Traceback (most recent call last):
  File "/var/tmp/portage/dev-python/pyside-5.12.4/work/pyside-setup-everywhere-src-5.12.4/sources/pyside2/PySide2/QtCore/../support/generate_pyi.py", line 290, in <module>
    generate_all_pyi(outpath, options=options)
  File "/var/tmp/portage/dev-python/pyside-5.12.4/work/pyside-setup-everywhere-src-5.12.4/sources/pyside2/PySide2/QtCore/../support/generate_pyi.py", line 269, in generate_all_pyi
    generate_pyi(import_name, outpath, options)
  File "/var/tmp/portage/dev-python/pyside-5.12.4/work/pyside-setup-everywhere-src-5.12.4/sources/pyside2/PySide2/QtCore/../support/generate_pyi.py", line 190, in generate_pyi
    top = __import__(import_name)
ImportError: /usr/lib64/libshiboken2-python3.6.so.5.12: undefined symbol: _Py_FalseStruct
[198/1326] /usr/bin/x86_64-pc-linux-gnu-g++ -DQT_CORE_LIB -DQT_GUI_LIB -DQT_NO_DEBUG -DQtGui_EXPORTS -I/var/tmp/portage/dev-python/pyside-5.12.4/work/pyside-setup-everywhere-src-5.12.4/sources/pyside2/PySide2/QtGui/QtGui -I/var/tmp/portage/dev-python/pyside-5.12.4/work/pyside-setup-everywhere-src-5.12.4/sources/pyside2/PySide2/QtGui -IPySide2/QtGui -I/var/tmp/portage/dev-python/pyside-5.12.4/work/pyside-setup-everywhere-src-5.12.4/sources/pyside2/PySide2 -I/var/tmp/portage/dev-python/pyside-5.12.4/work/pyside-setup-everywhere-src-5.12.4/sources/pyside2/libpyside -IPySide2/QtCore/PySide2/QtCore -isystem /usr/include/qt5 -isystem /usr/include/qt5/QtCore -isystem /usr/lib64/qt5/mkspecs/linux-g++ -isystem /usr/include/qt5/QtGui -isystem /usr/include/shiboken2 -isystem /usr/include/python2.7  -DNDEBUG -march=native -O2 -pipe -Wall -fvisibility=hidden -Wno-strict-aliasing -fPIC   -fPIC -std=gnu++11 -MD -MT PySide2/QtGui/CMakeFiles/QtGui.dir/PySide2/QtGui/qhoverevent_wrapper.cpp.o -MF PySide2/QtGui/CMakeFiles/QtGui.dir/PySide2/QtGui/qhoverevent_wrapper.cpp.o.d -o PySide2/QtGui/CMakeFiles/QtGui.dir/PySide2/QtGui/qhoverevent_wrapper.cpp.o -c PySide2/QtGui/PySide2/QtGui/qhoverevent_wrapper.cpp
[199/1326] /usr/bin/x86_64-pc-linux-gnu-g++ -DQT_CORE_LIB -DQT_GUI_LIB -DQT_NO_DEBUG -DQtGui_EXPORTS -I/var/tmp/portage/dev-python/pyside-5.12.4/work/pyside-setup-everywhere-src-5.12.4/sources/pyside2/PySide2/QtGui/QtGui -I/var/tmp/portage/dev-python/pyside-5.12.4/work/pyside-setup-everywhere-src-5.12.4/sources/pyside2/PySide2/QtGui -IPySide2/QtGui -I/var/tmp/portage/dev-python/pyside-5.12.4/work/pyside-setup-everywhere-src-5.12.4/sources/pyside2/PySide2 -I/var/tmp/portage/dev-python/pyside-5.12.4/work/pyside-setup-everywhere-src-5.12.4/sources/pyside2/libpyside -IPySide2/QtCore/PySide2/QtCore -isystem /usr/include/qt5 -isystem /usr/include/qt5/QtCore -isystem /usr/lib64/qt5/mkspecs/linux-g++ -isystem /usr/include/qt5/QtGui -isystem /usr/include/shiboken2 -isystem /usr/include/python2.7  -DNDEBUG -march=native -O2 -pipe -Wall -fvisibility=hidden -Wno-strict-aliasing -fPIC   -fPIC -std=gnu++11 -MD -MT PySide2/QtGui/CMakeFiles/QtGui.dir/PySide2/QtGui/qaccessibletablemodelchangeevent_wrapper.cpp.o -MF PySide2/QtGui/CMakeFiles/QtGui.dir/PySide2/QtGui/qaccessibletablemodelchangeevent_wrapper.cpp.o.d -o PySide2/QtGui/CMakeFiles/QtGui.dir/PySide2/QtGui/qaccessibletablemodelchangeevent_wrapper.cpp.o -c PySide2/QtGui/PySide2/QtGui/qaccessibletablemodelchangeevent_wrapper.cpp
ninja: build stopped: subcommand failed.

      Everything behaves as expected until the header generation phase, at which point all compilation hell breaks loose. Note the ImportError: /usr/lib64/libshiboken2-python3.6.so.5.12: undefined symbol: _Py_FalseStruct exception message, which Python 2.7 raises on attempting to import a C extension erroneously linked against a Python 3 interpreter.

      Woops.

      We're Lost. Can You Explain This in Layman's Terms?

      Probably... not. Yet I shall valiantly try.

      Let's start with the -python3.6 suffix in the library basename above. Of course, this suffix is how Gentoo uniquifies the same version of Shiboken2 (e.g., Shiboken 5.12.4) to concurrently target multiple versions of Python (e.g., Python 2.7 and Python 3.6).

      Python 2 and Python 3 are distinct languages. While similar, these languages are dissimilar enough that Gentoo allows (and encourages) Python packages to be concurrently installed under multiple versions of Python – typically, Python 2.7 and the newest version of Python 3 (e.g., Python 3.7).

      The top-level pyside-setup-everywhere-src-5.12.4/sources/shiboken2/CMakeLists.txt makefile enables us to effectively create different Shiboken2 "profiles," each targeting a different version of Python. How is this ludicrous magic accomplished? Via the mostly undocumented ${PYTHON_CONFIG_SUFFIX} cmake variable, which defaults to:

      • -python2.7 when building Shiboken2 against Python 2.7. This is deterministic and thus useful.
      • .cpython-36m-x86_64-linux-gnu when building Shiboken2 against Python 3.6. This is less deterministic and thus less useful.

      To enforce parity between these two common cases, we explicitly override this variable on the command line by passing DPYTHON_CONFIG_SUFFIX="${EPYTHON}". Of course, ${EPYTHON} is a Gentoo-specific variable expanding to -python${PYTHON_VERSION_MAJOR}.${PYTHON_VERSION_MINOR}. Passing this option thus sets this variable to:

      • -python2.7 when building Shiboken2 against Python 2.7. (Same as above.)
      • -python3.6 when building Shiboken2 against Python 3.6. (Different from above, but significantly more orthogonal with the Python 2.7 case.)

      So far so good, right?

      Forget We Even Asked.

      No, wait! This is going somewhere meaningful. (Scout's honour.)

      By overriding the ${BUILD_TYPE} and ${PYTHON_CONFIG_SUFFIX} cmake variables as detailed above, we guarantee that Shiboken2 5.12.4 generates the following files in the /usr/lib64/cmake/Shiboken2-5.12.4/ directory:

      • Shiboken2Targets-gentoo.cmake, detailed above.
      • Shiboken2Targets.cmake, which unconditionally includes Shiboken2Targets-gentoo.cmake via lines 61—66:
      # Load information for each installed configuration.
get_filename_component(_DIR "${CMAKE_CURRENT_LIST_FILE}" PATH)
file(GLOB CONFIG_FILES "${_DIR}/Shiboken2Targets-*.cmake")
foreach(f ${CONFIG_FILES})
  include(${f})
endforeach()
      • Shiboken2Config-python2.7.cmake, which unconditionally includes Shiboken2Targets.cmake via line 50:
        include("${CMAKE_CURRENT_LIST_DIR}/Shiboken2Targets.cmake")
      • Shiboken2Config-python3.6.cmake, which also unconditionally includes Shiboken2Targets.cmake via the exact same line.

      You see the conundrum, I trust. The Python 2.7- and 3.6-specific Shiboken2 cmake configurations both transitively include the exact same Shiboken2Targets-gentoo.cmake file, which unconditionally links to the /usr/lib64/libshiboken2-python3.6.so.5.12.4 library, which is only safely importable under Python 3.6.

      Chaos rapidly ensues. When the Python 2.7-specific PySide2 build includes the Python 2.7-specific Shiboken2 cmake configuration, it erroneously imports the Python 3.6-specific version of libshiboken2. Import exceptions are raised; the Super Blood Wolf Moon howls for vengeance; the ceremony of innocence is drowned; the meek inherit the Earth; and other terrifying things happen, too.

      Whadda You Want Us to Do About Your Problems?

      I think we can all agree that the above description of this issue is nightmarish and convoluted. (It took 200 lines of 80 column-wrapped plaintext in Vim just to get us this far.)

      Is the solution as nightmarish? Nope! The solution is actually trivial. That's nice, right?

      Specifically, the contents of the example Shiboken2Targets-gentoo.cmake file auto-generated by Shiboken2 above should instead resemble:

      #----------------------------------------------------------------
# Generated CMake target import file for configuration "Gentoo".
#----------------------------------------------------------------

# Commands may need to know the format version.
set(CMAKE_IMPORT_FILE_VERSION 1)

# Import target "Shiboken2::libshiboken" for configuration "Gentoo"
set_property(TARGET Shiboken2::libshiboken APPEND PROPERTY IMPORTED_CONFIGURATIONS GENTOO)
set_target_properties(Shiboken2::libshiboken PROPERTIES
  IMPORTED_LOCATION_GENTOO "/usr/lib64/libshiboken2${SHIBOKEN_PYTHON_SHARED_LIBRARY_SUFFIX}.so.5.12.4"
  IMPORTED_SONAME_GENTOO "libshiboken2${SHIBOKEN_PYTHON_SHARED_LIBRARY_SUFFIX}.so.5.12"
  )

list(APPEND _IMPORT_CHECK_TARGETS Shiboken2::libshiboken )
list(APPEND _IMPORT_CHECK_FILES_FOR_Shiboken2::libshiboken "/usr/lib64/libshiboken2${SHIBOKEN_PYTHON_SHARED_LIBRARY_SUFFIX}.so.5.12.4" )

# Import target "Shiboken2::shiboken2" for configuration "Gentoo"
set_property(TARGET Shiboken2::shiboken2 APPEND PROPERTY IMPORTED_CONFIGURATIONS GENTOO)
set_target_properties(Shiboken2::shiboken2 PROPERTIES
  IMPORTED_LOCATION_GENTOO "${_IMPORT_PREFIX}/bin/shiboken2"
  )

list(APPEND _IMPORT_CHECK_TARGETS Shiboken2::shiboken2 )
list(APPEND _IMPORT_CHECK_FILES_FOR_Shiboken2::shiboken2 "${_IMPORT_PREFIX}/bin/shiboken2" )

# Commands beyond this point should not need to know the version.
set(CMAKE_IMPORT_FILE_VERSION)

      Critically, note that all static references to a specific libshiboken2*.so library (e.g., /usr/lib64/libshiboken2-python3.6.so.5.12.4) have been globally replaced by dynamic references to the current libshiboken2*.so library being linked against (e.g., /usr/lib64/libshiboken2${SHIBOKEN_PYTHON_SHARED_LIBRARY_SUFFIX}.so.5.12.4).

      As expected, the ${SHIBOKEN_PYTHON_SHARED_LIBRARY_SUFFIX} cmake variable dynamically expands to the library suffix unique to the current libshiboken2*.so library being linked against. This variable is, in turn, defined by the Python 2.7- and 3.6-specific cmake files for Shiboken2:

      • Shiboken2Config-python2.7.cmake, via line 67:
      set(SHIBOKEN_PYTHON_SHARED_LIBRARY_SUFFIX "-python2.7")
      • Shiboken2Config-python3.6.cmake, also via line 67:
      set(SHIBOKEN_PYTHON_SHARED_LIBRARY_SUFFIX "-python3.6")

      Voilà! Thus were the wrong things made right again.

      What? Is That It?

      Well... not quite. There's always the rub. This is that rub.

      Shiboken2Config-python2.7.cmake and Shiboken2Config-python3.6.cmake both define the necessary ${SHIBOKEN_PYTHON_SHARED_LIBRARY_SUFFIX} variable after including the Shiboken2Targets-gentoo.cmake file. Since cmake variables default to the empty string when undefined, this fails to behave as expected.

      The solution is, again, trivial. That's nice, right?

      Lines 47—67 of the example Shiboken2Config-python2.7.cmake file auto-generated by Shiboken2 currently resemble:

      # Import targets and call variable set up functions  only when using an installed shiboken config
# file (so not during a regular shiboken build, or during a super project build).
if (NOT TARGET Shiboken2::shiboken2)
    include("${CMAKE_CURRENT_LIST_DIR}/Shiboken2Targets.cmake")
    include("${CMAKE_CURRENT_LIST_DIR}/shiboken_helpers.cmake")

    # Compute the python include and libraries path if needed (aka not part of super project build).
    shiboken_find_required_python(2)
    shiboken_check_if_built_and_target_python_are_compatible()
    shiboken_check_if_limited_api()
    shiboken_compute_python_includes(IS_CALLED_FROM_EXPORT)
    shiboken_compute_python_libraries(IS_CALLED_FROM_EXPORT)
endif()

# Get the "python interpreter" dynamic global property as a variable instead. It brings it into
# scope for super project builds.
get_property(SHIBOKEN_PYTHON_INTERPRETER GLOBAL PROPERTY SHIBOKEN_PYTHON_INTERPRETER)

# Set static variables.
set(SHIBOKEN_PYTHON_EXTENSION_SUFFIX "")
set(SHIBOKEN_PYTHON_SHARED_LIBRARY_SUFFIX "-python2.7")

      That's bad. Those lines should instead resemble:

      # Set static variables.
set(SHIBOKEN_PYTHON_EXTENSION_SUFFIX "")
set(SHIBOKEN_PYTHON_SHARED_LIBRARY_SUFFIX "-python2.7")

# Import targets and call variable set up functions  only when using an installed shiboken config
# file (so not during a regular shiboken build, or during a super project build).
if (NOT TARGET Shiboken2::shiboken2)
    include("${CMAKE_CURRENT_LIST_DIR}/Shiboken2Targets.cmake")
    include("${CMAKE_CURRENT_LIST_DIR}/shiboken_helpers.cmake")

    # Compute the python include and libraries path if needed (aka not part of super project build).
    shiboken_find_required_python(2)
    shiboken_check_if_built_and_target_python_are_compatible()
    shiboken_check_if_limited_api()
    shiboken_compute_python_includes(IS_CALLED_FROM_EXPORT)
    shiboken_compute_python_libraries(IS_CALLED_FROM_EXPORT)
endif()

# Get the "python interpreter" dynamic global property as a variable instead. It brings it into
# scope for super project builds.
get_property(SHIBOKEN_PYTHON_INTERPRETER GLOBAL PROPERTY SHIBOKEN_PYTHON_INTERPRETER)

      That is, static variables should be defined before including the Shiboken2Targets.cmake file and thus the Shiboken2Targets-gentoo.cmake file expanding those variables.

      On Gentoo Linux, we trivially resolve this latter issue by simply overriding the ${SHIBOKEN_PYTHON_SHARED_LIBRARY_SUFFIX} cmake variable to its desired value from the command line when building PySide2 (e.g., by passing -DSHIBOKEN_PYTHON_SHARED_LIBRARY_SUFFIX="-python2.7" when building PySide2 for Python 2.7).

      Again, none of this is ideal. It works... technically. But it's fragile, scary, and non-trivial to document. We'd much rather the Qt Company get this to officially work for the mutual good of Linux-kind everywhere.

      Please Tell Us You're Done Now

      I'm done now. Freya be praised! If anyone actually read this far, may the mercurial heavens bless your every earthly move – for you are a veritable saint on this Earth.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            kleint Friedemann Kleint
            leycec Cecil Curry
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes