Details
P1: Critical Description
When PySide and shiboken are tested, they are built as wheels. After the normal unit tests, the script wheel_tester.py performs additional integration tests. These tests show some problems on certain platforms.
Instead of removing the small problems of the script, completely unrelated problems had to be circumvented in absence of the COIN staff.
These errors may be hiding since a while, already. They became a problem, when I tried to build an extension for wheel_tester.py, that should use PyInstaller to ensure that a built demo script works on all platforms. Automatically Test Small Example With PyInstaller
Linux openSUSE_42_3 Problem
Error on Linux openSUSE_42_3: At the end of the log, you can see certain lines which report a problem with the sphinx package. The intended installer test could therefore not run.
This was circumvented by COIN: Skip doc build for Python 2, but erroneously classified as a sphinx bug.
If you take a closer look, you see:
build.go:192: Traceback (most recent call last): build.go:192: File "/home/qt/work/pyside/pyside-setup/env/bin/sphinx-build", line 7, in <module> build.go:192: from sphinx.cmd.build import main build.go:192: File "/home/qt/work/pyside/pyside-setup/env/lib/python2.7/site-packages/sphinx/cmd/build.py", line 39 build.go:192: file=stderr) build.go:192: ^ build.go:192: SyntaxError: invalid syntax
When inspecting the source code of build.py, you realize that this code cannot belong to Python 2.7, which has some comment instead in line 39. The file=stderr belongs to the Python 3 version of sphinx/build.py line 39.
That means: The Python 2 library contains Python 3 code, which of course is not suitable for Python 2. It is hard to imagine how such an error is possible in an automated environment.
Linux RHEL_7_4 Problem
Error on Linux RHEL_7_4: The end of the log shows PyInstaller trying to build a test script. It fails on Python 3 with the following traceback:
build.go:192: Traceback (most recent call last): build.go:192: File "/opt/rh/rh-python36/root/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main build.go:192: "__main__", mod_spec) build.go:192: File "/opt/rh/rh-python36/root/usr/lib64/python3.6/runpy.py", line 85, in _run_code build.go:192: exec(code, run_globals) build.go:192: File "/home/qt/work/pyside/pyside-setup/env3/lib/python3.6/site-packages/PyInstaller/__main__.py", line 118, in <module> build.go:192: run() build.go:192: File "/home/qt/work/pyside/pyside-setup/env3/lib/python3.6/site-packages/PyInstaller/__main__.py", line 111, in run build.go:192: run_build(pyi_config, spec_file, **vars(args)) build.go:192: File "/home/qt/work/pyside/pyside-setup/env3/lib/python3.6/site-packages/PyInstaller/__main__.py", line 63, in run_build build.go:192: PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs) build.go:192: File "/home/qt/work/pyside/pyside-setup/env3/lib/python3.6/site-packages/PyInstaller/building/build_main.py", line 838, in main build.go:192: build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build')) build.go:192: File "/home/qt/work/pyside/pyside-setup/env3/lib/python3.6/site-packages/PyInstaller/building/build_main.py", line 784, in build build.go:192: exec(text, spec_namespace) build.go:192: File "<string>", line 17, in <module> build.go:192: File "/home/qt/work/pyside/pyside-setup/env3/lib/python3.6/site-packages/PyInstaller/building/build_main.py", line 241, in __init__ build.go:192: self.__postinit__() build.go:192: File "/home/qt/work/pyside/pyside-setup/env3/lib/python3.6/site-packages/PyInstaller/building/datastruct.py", line 158, in __postinit__ build.go:192: self.assemble() build.go:192: File "/home/qt/work/pyside/pyside-setup/env3/lib/python3.6/site-packages/PyInstaller/building/build_main.py", line 573, in assemble build.go:192: self._check_python_library(self.binaries) build.go:192: File "/home/qt/work/pyside/pyside-setup/env3/lib/python3.6/site-packages/PyInstaller/building/build_main.py", line 676, in _check_python_library build.go:192: raise IOError(msg) build.go:192: OSError: Python library not found: libpython3.6mu.so.1.0, libpython3.6m.so.1.0, libpython3.6.so.1.0 build.go:192: This would mean your Python installation doesn't come with proper library files. build.go:192: This usually happens by missing development package, or unsuitable build parameters of Python installation. build.go:192: build.go:192: * On Debian/Ubuntu, you would need to install Python development packages build.go:192: * apt-get install python3-dev build.go:192: * apt-get install python-dev build.go:192: * If you're building Python by yourself, please rebuild your Python with `--enable-shared` (or, `--enable-framework` on Darwin)
The error message seems to make not much sense. For some reason, the correct library cannot be found.
As an intermediate circumvention, the build process was simply skipped on any failure, because we wanted to test running the compiled script, and not the compilation itself.
How To Resolve The Problem
These errors became visible while developing the above mentioned PyInstaller test. Although PyInstaller has some issues as well, these problems are purely COIN misconfigurations and out of the scope of the original scripting task.
The following steps are necessary to fix them completely:
- make sure that Error on Linux openSUSE_42_3: gets solved
- make sure that Error on Linux RHEL_7_4: gets solved
- revert patch COIN: Skip doc build for Python 2
- change patch Automatically Test Small Example With PyInstaller to react again on build errors
Amendment 2019-04-28: Follow-Up PyInstaller Problem
While these two COIN problems were fixed, another issue came up:
macOS with Python 2.7.16 crashes the generated PyInstaller test program.
Reason:
Python 2.7.16 includes the Tcl/Tk library in the Python distribution in a way that does not work with the usual framework layout that PyInstaller expects on macOS.
More info: https://www.python.org/download/mac/tcltk/
The error message is:
Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyInstaller/loader/rthooks/pyi_rth__tkinter.py", line 28, in <module> raise FileNotFoundError('Tcl data directory "%s" not found.' % (tcldir))
The hard-to-find reason for this error lies in the numpy module that was installed for other tests.
It turns out that Python 3.7 has the same problem when Tcl/Tk is used, and this has been reported since a while in https://github.com/pyinstaller/pyinstaller/pull/3830 .
That problem is not very visible because PyInstaller is not used often with Tcl/Tk. But on Python 2.7.16 the problem is more tricky, because the bug is triggered by the installation of numpy, which is a very frequently used module.
The proposed patch is the same for both Python versions and works fine. Until this is fixed, we excluded Tcl/Tk in the spec file.