Qt Creator: Support build environments in Docker containers

Qt Creator should support building applications with build environments inside a Docker container. Primary use cases are Linux toolchains for embedded targets.

Business value

Being able to ship build environments for embedded targets in containers would help R&D to streamline the effort of providing native toolchains for Linux and Windows separately. Furthermore, the isolation that Docker provides would avoid issues that some customers might face due to different local environments. Finally, it would make it easy to provide the same build environments also for macOS. For more details see mother Epic QTBUG-81947

Required functionality

Registration of containers

A user or the SDK needs to be able to register a container to Qt Creator. Qt Creator needs to have enough metadata about the container, or needs a standard way to query a toolchain for the information.

Startup and shutdown of containers

Qt Creator needs to be able to start and stop containers, either upon user request or transparently (if a Kit is configured).

Code model

Qt Creator needs to be able to parse projects that are supposed to be built in the toolchain.

Build process

Qt Creator needs to be able to configure and build an application inside the one container.

Debugger

Qt Creator needs to be able to start & connect to a debugger in the container.

Definition of Done

The Qt SDK can register containers containing a Linux with embedded toolchain and Qt libraries in Qt Creator. A user can load, edit, and compile projects with this container.

Technical Aspects

Background

To build Qt applications using Qt Creator with container-based toolchain, all relevant build steps (instance of ProjectExplorer::BuildStep-derived classes) and supporting functionality needs to have the ability to work with containers.

Build steps typically access files and run processes. They need to be able to access certain parameters like environment variables.

Traditionally, Qt Creator assumed that the "build device" is the same but as the "host machine", and used ordinary Qt classes like QDir, QFileInfo, QProcess, QProcessEnvironment, QFileDialog which also operate on the host machine only.

Moving towards containerized builds means that all relevant code using these classes has to be updated to use some abstraction layer.

Some of the code that needs to be converted is in a "don't touch" state: It is complex, it works, but nobody alive has ever seen it shift currently .

Conversion will have to cover a lot (thousands of pieces) of code. Some changes will be mechanical, but a significant part will require human inspection including adaptation of supporting functionality. And testing.

There might be further challenges related to synchronicity of operations. Currently almost all file accesses and a lot of short process execution assume they can be done in the main user thread without negative impact on UX. This is not the case for real "remote" execution. Docker is somewhere in between , as long as container file system accesses can be done directly in the host file system. 

Current situation

We have already classes Utils::FilePath and Utils::QtcProcess as convenience layers on top of QString based file names and QProcessm respectively.

We are currently extending these to become the core of the needed abstraction layer for containerized builds.

Utils::FilePath is has been changed to be a structure of three strings, 'scheme', 'host' and 'path', somewhat aligned to the url-terminology. QUrl can unfortunately not be used as we need to pass host and path information that are not valid RFC 8089.

Utils::QtcProcess is currently being unter re-construction to accept also tasks of other process-like classes in Creator to make it the primary interface.

Time schedule

The following /might/ be feasible if no unforeseen additional problems arise:

For Creator 5.0 (Feature freeze wk26 (~ Jun 22 2021))

• Proof of concept (with severe limitations):
  • Simple autodetection of Qt and toolchain installations on docker
  • Building simple Qt (core, widgets, declarative) applications with
    dockerized toolchain and qmake and cmake build systems
• Limitations:
  • Only Linux host systems will be supported
  • Possible UX degradation due to the mentioned synchronicity issues
  • Possibly a requirement to map certain directories 1:1 between host and build device

For Creator 6.0 (Feature freeze wk40 (~ Sep 28 2021)

• "Experimental"

• • Simple autodetection of Qt and toolchain installations on docker
  • Automatic setup of kits from user-select docker images
  • Coordination with installer-provided kits
  • Building Qt applications with
    dockerized toolchain and qmake and cmake build systems
  • No obvious UX degradation due to the synchronocity issues
  • Support for Linux and Windows as build host (Windows is a stretch target)

• Limitations:
  • Possibly a requirement to map certain directories 1:1 between host