Uploaded image for project: 'Qt for Python'
  1. Qt for Python
  2. PYSIDE-1587

WIP: Change the structure of the build process and sources

    XMLWordPrintable

    Details

    • Type: Epic
    • Status: Open
    • Priority: P2: Important
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: PySide, Shiboken
    • Labels:
      None
    • Epic Name:
      New build structure

      Description

      Overview

      When the work of porting the old PySide (Qt4) to Qt5 started in 2015, the repositories
      were still split into multiple parts, also when the new team took over in 2016.
      After about a year, the team decided to move all the different repositories
      into a mono-repository structure, which helped to forget about
      version problems, and a quicker development, branching, etc.

      Before the initial official release for 5.12, the repository was
      configured to build one big wheel containing everything: PySide, Shiboken,
      and Shiboken Generator.

      Even if this was convenient, again, for having everything on the same
      place, there were many user stories, where people were relying
      on the Shiboken generator, or only on PySide as user, so generating
      different wheels was the next step.

      Building three wheels from the same setup.py file was a challenging
      task, that required a large set of changes related the build scripts
      structure we currently have, and successfully ended into the 3 main
      wheels that we currently have:

      • PySide (on PyPi)
      • Shiboken, the python helper module (on PyPi)
      • Shiboken, the generator (not on PyPi due to the libclang dependency)

      The Problem

      Having a process that generates 3 wheels from the same setup.py file
      is problematic if people are relying on editable build/install https://pip.pypa.io/en/latest/cli/pip_install/#editable-installs
      because that approach is not compatible with more than one .egg-info
      directory:

      See errors here: https://gist.github.com/cmaureir/82fcb52bcbea5ce70362cb4270fc3f8c

      Additionally,
      when a user on a non-supported platform runs a pip install pyside6 command
      if there is no pre-built wheel, pip will start building the source package
      with a simple pip install, which generates the same errors.

      Even if this approach would mean that users would need to build the whole
      project in case they want to use it on an unsupported platform,
      a process that will be really slow depending on the host, it's a solution
      to at least be available on those platforms.

      Complementary to that, the current source package we include
      contains everything, and we cannot securely split the source only
      for pyside (or shiboken), so people can build it directory with
      a setup.py call.

      Last but not least, going back to a many-repo approach for the project
      will introduce a lot of extra work regarding the synchronization between
      repositories, branches, etc.

      Current scenario

      At the moment the structure of the repository looks like this:

      ./build_scripts/
      ./setup.py
      ./sources/pyside6/
      ./sources/shiboken6/shibokenmodule
      ./sources/shiboken6/generator
      ./...
      

      when calling globally python setup.py build ... that will trigger
      three calls, with an option --internal-build-type for each project:

      python setup.py build ... --internal-build-type=shiboken6
      python setup.py build ... --internal-build-type=shiboken_module
      python setup.py build ... --internal-build-type=pyside6
      

      and thus, having independent processes, allows us to generate 3 different
      wheels.

      Solution proposal

      Since having many repositories is not a solution,
      the only solution will mean to have source directories for each project
      so we could be able to build them independently:

      ./setup.py
      ./sources/pyside6/build_scripts/
      ./sources/pyside6/setup.py
      ./sources/shiboken6/shibokenmodule/build_scripts/
      ./sources/shiboken6/shibokenmodule/setup.py
      ./sources/shiboken6/generator/build_scripts/
      ./sources/shiboken6/generator/setup.py
      ./...
      

      Now when the global setup.py is called, that will explicitly
      go into the 3 different projects, and call internally
      the same command:

      python setup.py build ...
      -> go on sources/shiboken6/shibokenmodule/
      -> python setup.py build ...
      -> go on sources/shiboken6/generator
      -> python setup.py build ...
      -> go on sources/pyside6
      -> python setup.py build ...
      

      This will ensure, that creating a zip file on the sources/pyside6, for example,
      will contain the necessary scripts to build the project, as long
      as the dependencies are fulfilled,
      enabling us to provide the source packages, so people can build it on any
      infrastructure.

      Additionally, this new structure will help the efforts of making the whole project more cross-compatible friendly, in the sense that due to the independence of each project-source-structure, the options to cross compile tools for host and target will be simple than taking care of the general build process, and tackle all the possible conditions.

      Lastly, this new structure might help the generation of new 'addons' wheels that people can install on top of a PySide6 one, like for example M2M Protocols, creating a new special directory:

      ...
      ./sources/pyside6m2m/build_scripts
      ./sources/pyside6m2m/setup.py
      ...
      

      Complications

      We rely on CMake for our build process, and distributions usually rely
      on calling cmake directly, instead of using the setup.py file, so we need
      to ensure, that also that process is properly supported.

      PEP517 will probably become the standard to build python projects
      https://www.python.org/dev/peps/pep-0517
      and with the help of a pyproject.toml file https://www.python.org/dev/peps/pep-0518/
      projects will change a bit the way of using setup.py,
      however, that will not be the solution for C-extensions all the time,
      and projects still would need to rely on them.
      See https://stackoverflow.com/questions/66157987/how-to-build-a-c-extension-in-keeping-with-pep-517-i-e-with-pyproject-toml-ins

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

            Assignee:
            crmaurei Cristian Maureira-Fredes
            Reporter:
            crmaurei Cristian Maureira-Fredes
            PM Owner:
            Vladimir Minenko Vladimir Minenko
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:

                Gerrit Reviews

                There are no open Gerrit changes