Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-114769

Multiview and related in the Qt graphics stack and Quick 3D XR

    XMLWordPrintable

Details

    • User Story
    • Resolution: Unresolved
    • P2: Important
    • None
    • 6.7, 6.8
    • Qt RHI, Quick: 3D
    • None

    Description

      Enable multiview in the graphics stack.

      Use it in Qt Quick 3D XR, and by extension, in every layer below in stack; unfortunately multiview affects everything down to the QRhi backend (with implications for the shaders and qsb/shadertools.

      This is not merely a performance optimization for VR/AR: the naive separate left/right eye rendering of Quick3D XR breaks API contracts in Qt Quick for instance (because one "frame" suddenly consists of two (left/right eye) frames; and so QQuickWindow signals suddenly and unexpectedly get emitted twice per frame etc.). For proper compatibility with the non-XR world multiview must be used.

      On the QRhi level it is known that this can be implemented for OpenGL (ES), Vulkan, and D3D12. Perhaps Metal as well.

      To prove the low-level integration works (QRhi, QRhi backends), have a manual test that renders a triangle or other geometry with two different transforms into two layers of a texture array using a single draw call, and then draw two textured quads to show what got rendered in that single draw call.

      To enable this for D3D12, it involves a bunch of related work for pipeline and shader management as well since we need HLSL 6.1 and fxc/D3DCompile will not support this. To do HLSL->DXIL compilation with dxc/IDxcCompiler needs some work both in qsb (simple) and in the runtime HLSL compilation support in the D3D12 backend (not so simple). Interesting build and deployment problems as well (ancient headers in MinGW, missing DLLs on regular PCs, etc.) Perhaps will be limited to MSVC? Also update windeployqt.
      On top, view instancing in D3D12 relies on the extensible PSO creation method (CreatePipelineState()), not the old CreateGraphicsPipelineState(). Have to switch.

      The multiview feature is not going to be supported on D3D11.

      Test MSAA as well. (must work with a texture array, so QRhiRenderBuffer is out; we never even exercised MSAA and auto-resolve with things other than 2D textures)

      Layers:

      • qtbase, QRhi, QRhi backends
      • qtshadertools Infrastructure for specifying view count, storing metadata, new HLSL versions, etc.
      • qtdeclarative, Qt Quick scenegraph (internal), built-in materials, public API additions and implications for QSGMaterial and ShaderEffect (plus build system consequences for qt_add_shaders, thinking of suitable HLSL versions for example, etc.)
      • qtquick3d Internal support, not sure if there are public API implications.
      • qtquick3dxr Changes from the XrSwapChain up to how cameras and the rendering logic work. May need to deal with various implications.

      Related work and features to investigate:

      • Multisampling. Even non-multiview MSAA is impossible to achieve for XrView without further enablers in Quick/Gui, whereas multiview support needs extra care when using in combination with MSAA. (investigate both the XrSwapchains MSAA support (which is usually not available) and also doing MSAA ourselves via the 3D API)
      • Issues due to VR compositor / OpenXR implementations insistence of handing out textures with a _SRGB format. Quick 3D XR does not want extra linear->sRGB conversions on already sRGB fragment colors. Needs further enablers in qtbase.
      • If we want to "submit the depth buffer", to help improving reprojections (XR_KHR_composition_layer_depth), that presents further interesting questions and complications, both with and without multiview. (one does not submit the depth buffer, but has to create a "depth swapchain", i.e. an XrSwapchain with a format like D32 or D24S8 and get the subimages from that, and render into that, instead of Qt creating its own texture/renderbuffer to depth-stencil)

      Attachments

        Issue Links

          1.
          Support GL_OVR_multiview(2) for OpenGL and OpenGL ES Sub-task Closed Laszlo Agocs
          2.
          Support VK_KHR_multiview for Vulkan Sub-task Closed Laszlo Agocs
          3.
          Support D3D12 view instancing Sub-task Closed Laszlo Agocs
          4.
          Use dxc/IDxcCompiler in qsb and the D3D12 backend when shader model >= 6.0 is specified Sub-task Closed Laszlo Agocs
          5.
          Multiview on Metal Sub-task Closed Laszlo Agocs
          6.
          Multisampling with multiview rendering Sub-task Closed Laszlo Agocs
          7.
          Extend the multiview manual test to exercise instancing Sub-task Closed Laszlo Agocs
          8.
          Depth buffer in a multiview render pass Sub-task Closed Laszlo Agocs
          9.
          Stencil buffer in a multiview render pass Sub-task Closed Laszlo Agocs
          10.
          Add autotest for QRhi-level multiview support Sub-task Closed Laszlo Agocs
          11.
          Add qt_add_shaders convenience option to auto-generate 2-view versions of the shaders Sub-task Closed Laszlo Agocs
          12.
          Enable multiview render targets with the Qt Quick scenegraph (covers render target plumbing, not actual full multiview rendering support) Sub-task Closed Laszlo Agocs
          13.
          Add QQuickRenderTarget overloads to adopt an existing texture array for multiview rendering Sub-task Closed Laszlo Agocs
          14.
          Make Qt Quick materials multiview capable Sub-task Closed Laszlo Agocs
          15.
          Quick 3D multiview plumbing (umbrella) Sub-task Open Laszlo Agocs
          16.
          Multiview: Depth textures in custom materials (and effects) Sub-task Closed Laszlo Agocs
          17.
          Multiview: Screen textures in custom materials (and effects) Sub-task Closed Laszlo Agocs
          18.
          Enable SSAA with multiview Sub-task Closed Laszlo Agocs
          19.
          Postprocessing effects with multiview Sub-task Closed Laszlo Agocs
          20.
          Verify/fix (Extended)SceneEnvironment effects with multiview Sub-task Closed Laszlo Agocs
          21.
          Temporal and progressive AA in multiview mode Sub-task Open Laszlo Agocs
          22.
          Property to control multiview rendering Sub-task Closed Laszlo Agocs
          23.
          Add a doc page about multiview Sub-task Closed Laszlo Agocs
          24.
          Qt Quick 3D XR Multiview Sub-task Closed Laszlo Agocs
          25.
          Avoid creating a new depth-stencil (and msaa color) buffer every frame Sub-task Closed Laszlo Agocs
          26.
          MSAA in XrView Sub-task Closed Laszlo Agocs
          27.
          Rendering into multisample texture (not renderbuffer) and resolving does not work with OpenGL ES (3.1+ and only ES) Sub-task Closed Laszlo Agocs
          28.
          Rendering is too bright in some combinations of MSAA, multiview, and 3D API Sub-task Closed Laszlo Agocs
          29.
          Investigate the case of missing models with multiview on OpenGL ES on Android Sub-task Closed Laszlo Agocs
          30.
          Revisit multiview on Metal, why does the autotest fail on Intel Mac Mini Sub-task Closed Laszlo Agocs
          31.
          Fixed foveated rendering Sub-task Closed Laszlo Agocs
          32.
          Option to submit depth buffer to compositor (XR_KHR_composition_layer_depth) Sub-task Closed Laszlo Agocs
          33.
          Demonstrate in examples/tests that custom multiview-capable QSGMaterials and ShaderEffects can be created by the end users as planned Sub-task Reported Laszlo Agocs
          34.
          Figure out what to do with the lack of dxc DLLs Sub-task Closed Laszlo Agocs
          35.
          Figure out what to do with MingGW that ships outdated/unofficial Windows headers Sub-task Reported Laszlo Agocs
          36.
          Multiview rendering on VisionOS Sub-task Closed Christian
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              lagocs Laszlo Agocs
              lagocs Laszlo Agocs
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are no open Gerrit changes