Customizable rendering pipelines User story

As a user, I want to be able to have the ability to customize the rendering pipeline. Qt Quick 3D provides many ways to customize the rendering pipeline, but this is mostly suggestions to a black box.  Instead there should be an alternative way to customize how and what is rendered, and to what.  And while even this exists already in Qt Quick 3D, those API’s are C++ only and require internal engine knowledge to use.  Instead it should be possible to define custom rendering pipelines in Qt Quick via QML or QDS, which should be in line with the goals of Qt Quick 3D.

Much like it is possible to define multiple passes in post processing effects, it should also be possible to define additional or alternative passes during the “main” rendering of a scene.  That is the parts of the rendering that involve the actual scene renderable.  It is possible todo this today in C++ with the RendererExtensions API, but it should also be possible to do so more generically in QML.  Conceptually this is similar to the Frame Graph in Qt 3D, but a bit more high level.  On the backend however we do indeed built a frame graph to determine the render strategy.

Roughly speaking, the API would define an alternative way to renderer a scene.  So for example, I may want to collect all opaque items in a scene, sort them by distance from furthest to nearest, and then render each of them with an alternative material (say rendering each with a solid random color without shading).  This would render to a texture, which I could then use in a post processing effect on the “main” pass of the View3D.  In this post processing effect, I can mix the two textures such that each opaque item is “highlighted” with an alternative color.  I could alternatively use this texture in a custom picking algorithm, which could be useful for providing accurate picking for models that are only animated at runtime (rigged animations).

We should provide ready-made building blocks for defining a custom render pass in QML/QDS, such that building a tree of these primitives leads to a frame graph that results in passes.

This should be considered an alternative render pipeline, not a customization of the built in pipeline.  It’s an alternative, not a replacement.  The ease of use in Qt Quick 3D comes from our control of the default forward+ renderer, and us being able to determine what the “default” passes look like by how the API is used.  Defining your own pipeline means that it will be used instead of the default, and render to it’s own render target, or be used in conjunction with the default pipeline (with the resulting textures/buffers generated being accessible from custom materials or post processing effects via the usage of Texture).