Variable Rate Shading support

VRS investigation.

1. Set shading rate (coarse pixel size) on command list, per-draw (per-pipeline):
1x1, 1x2, 2x1, 2x2, and optionally, if supported, 2x4, 4x2, 4x4.

Sample count - coarse pixel size support map; as per D3D12 docs, optional ones in ( ), with Vulkan check all this at run time with vkGetPhysicalDeviceFragmentShadingRatesKHR instead

1 - 1x2, 2x1, 2x2, (2x4, 4x2, 4x4)
2 - 1x2, 2x1, 2x2, (2x4)
4 - 1x2, 2x1, 2x2
8 - none
16 - none

D3D12:

• Supported if D3D12_FEATURE_DATA_D3D12_OPTIONS6::VariableShadingRateTier != D3D12_VARIABLE_SHADING_RATE_TIER_NOT_SUPPORTED
• Support for the additional rates are indicated by D3D12_FEATURE_DATA_D3D12_OPTIONS6::AdditionalShadingRatesSupported
• Usage: commandList->RSSetShadingRate(D3D12_SHADING_RATE_2X2, nullptr);
  • Needs ID3D12GraphicsCommandList5. Perhaps ifdef in case someone tries to build with old Windows SDK that does not have it.

Vulkan:

• When creating the device, pipelineFragmentShadingRate must be enabled in VkPhysicalDeviceFragmentShadingRateFeaturesKHR (if VkGetPhysicalDeviceFeatures2 says it is supported)
• Query VkPhysicalDeviceFragmentShadingRatePropertiesKHR via vgGetPhysicalDeviceProperties2 for stuff, but not sure which of them are relevant here.
• Call vkGetPhysicalDeviceFragmentShadingRatesKHR to check which coarse pixel sizes are supported for which sample counts.
• Similarly to viewport or scissor, it can be either baked in to the graphics pipeline, or declared a dynamic state in the pipeline and then set via vkCmdSetFragmentShadingRateKHR.

Metal:

• Not supported.

2. Per-primitive, in the vertex shader by writing to SV_ShadingRate (HLSL) or whatever the GLSL/SPIR-V equivalent is.

• Supported by D3D12 and Vulkan.
• Not supported with Metal.
• Won't be supported by Qt.

3. Image-based

D3D12:

• Supported if D3D12_FEATURE_DATA_D3D12_OPTIONS6::VariableShadingRateTier == D3D12_VARIABLE_SHADING_RATE_TIER_2
• Create a 2D texture with R8 (DXGI_FORMAT_R8_UINT), layout D3D12_TEXTURE_LAYOUT_UNKNOWN.
• Get tileSize = D3D12_FEATURE_DATA_D3D12_OPTIONS6::ShadingRateImageTileSize
• Texture pixel size is ceil(rt_pixel_width / tileSize), ceil(rt_pixel_height / tile_size)
• Each byte is a value from D3D12_SHADING_RATE. (indicating the coarse pixel size for one tile)
• Then later just: commandList->RSSetShadingRateImage(texture); (must be in D3D12_RESOURCE_STATE_SHADING_RATE_SOURCE state)
• Must be preceded by calling RSSetShadingRate at least once, to set the combiners, the image is ignored otherwise.
• Probably RSSetShadingRate(1x1, D3D12_SHADING_RATE_COMBINER_MAX for both combiners) is sufficient so that it picks the coarsest, i.e. what's in the image. (the combiner says which/how of the three setting (per-draw, per-primitive, image-based) should be taken into account)
• NB with multiview, the texture is a 2D texture array, with the number of layers matching the render target.

Vulkan:

• When creating the device, attachmentFragmentShadingRate must be enabled in VkPhysicalDeviceFragmentShadingRateFeaturesKHR, if VkGetPhysicalDeviceFeatures2 says it is supported.
• Take maxFragmentShadingRateAttachmentTexelSize from VkFragmentShadingRateAttachmentInfoKHR. This is the tileSize as in the D3D12 example above. (typical values are 32x32, 16x16, and 8x8, based on the vulkan.gpuinfo.org)
• Create a 2D image with format VK_FORMAT_R8_UINT and usage VK_IMAGE_USAGE_FRAGMENT_SHADING_RATE_ATTACHMENT_BIT_KHR. (optimal tiling, device local)
• Call vkGetPhysicalDeviceFragmentShadingRatesKHR to check which coarse pixel sizes are supported for which sample counts.
• Write to the texture. (staging buffer, one coarse pixel size per byte, vkCmdCopyBufferToImage)
• So that if texel is a byte in the image, then coarse_width = 2^(texel/4)&3) and coarse_height = 2^(texel&3) [NB the possible values are likely identical to the values in D3D12_SHADING_RATE]
• The VkFramebuffer must be created with this image as an attachment.
• Then chain in a VkFragmentShadingRateAttachmentInfoKHR to the VkSubpassDescription2KHR, and then must use vkCreateRenderPass2KHR.
• Then in the render pass, set the combiners like with D3D12. The shading rate image then used in that pass.
• NB with multiview, the texture is a 2D texture array, with the number of layers matching the render target.

Metal:

• Check for support with supportsRasterizationRateMapWithLayerCount: 1 (or 2 if multiview) on the device.
• A dedicated MTLRasterizationRateMap type that is not a texture.
• Set up a MTLRasterizationRateMapDescriptor with the output pixel size as screenSize. Multiple layers if multiview.
  • Note how this is different from anyone else: there is no query for a tile size and then calculating the size of the map based on that. Instead, the output size is passed in and then the user specifices the zone counts (number of 0.0-1.0 rate values specified horizontally and vertically).
• MTLRasterizationRateLayerDescriptor: Horizontal and vertical rates, in range 0.0-1.0, are specified separately.
• Then it wants us to set up intermediate textures with a smaller size calculated based on the number of zones presumably, render into those, while setting rasterizationRateMap on the MTLRenderPassDescriptor to the rate map object created above.
• Then scale it up. (by doing a whole separate render pass to draw a textured quad?)
• For non-uniform coarse pixel sizes this seems even more complicated since it requires asking the rate map to copy data to a MTLBuffer, expose that to the shader, and then do stuff based on that.
• However, I assume that this interemediate texture and scaling business is not relevant on VisionOS, where we get a MTLRasterizationRateMap when foveation is enabled. And all we need to do is to associate that rate map with the render pass. The rest (texture sizes and such) are taken care of by the compositor, I guess? But cannot confirm due to insufficient Apple documentation.

Plan for now:

As we are anyway exploring this because of the Vision Pro's (alleged) insufficient performance without dynamic foveation, we should try adding proper, cross-platform VRS support to the extent it makes sense. But we cannot aim for full portable support, due to Apple doing things differently. Also some of the options (#2, per-primitive via shaders) has shader tooling (compiler/transpiler) implications, which we will not touch now.

• Expose the per-draw setting of the coarse pixel size as an optional feature when on D3D12 and Vulkan. For OpenGL ES there is a Qualcomm-specific extension that may be relevant for the Quest 3, that should be explored too.
  • With the appropriate QRhi::Feature flag, and some getter for the supported coarse pixel sizes (for a given sample count, probably).
  • Perhaps will bake it into the QRhiGraphicsPipeline for now, not in QRhiCommandBuffer. (so with Vulkan it's baked, not dynamic state)? Might be too limiting, perhaps do the opposite. (declare UsesShadingRate on the pipeline, call setShadingRate on the command buffer)
  • Create a manual test to demo this.

• Image-based approach:
  • Create a new type, QRhiShadingRateMap, or something, that internally is backed by D3D12 texture, Vulkan image, or MTLRasterizationRateMap.
  • At minimum, we must support creating a QRhiShadingRateMap with an existing native object, whatever that is with a given backend (ID3D12Resource*, VkImage, MTLRasterizationRateMap), similarly to how a QRhiTexture can wrap (not own) a native texture object.
  • Then add the option to associate a QRhiShadingRateMap with the ??? (what exactly? for Vulkan it needs to be in the render pass descriptor so might need to add setShadingRateMap to both QRhiTextureRenderTarget and QRhiSwapChain?). If these are in place, that might be sufficient for VisionOS / Quick 3D XR purposes, unless there's a misunderstanding of what exactly is required to support dynamic foveation there (i.e. this assumes there is no "scaling" step performed by Qt).
  • The next, nice to have, level is the ability to create a QRhiShadingRateMap with D3D12 and Vulkan at least without manually creating a native texture first, and then providing its content in a convenient manner (say, a QByteArray that gets uploaded automatically into the texture/image). This needs to be fleshed out. This will likely not be supported with Metal for now.

References:

https://microsoft.github.io/DirectX-Specs/d3d/VariableRateShading.html
https://wickedengine.net/2020/09/variable-rate-shading-first-impressions/
https://developer.qualcomm.com/sites/default/files/docs/adreno-gpu/snapdragon-game-toolkit/gdg/gpu/best_practices_other.html#use-variable-rate-shading
https://registry.khronos.org/vulkan/specs/1.3-extensions/man/html/VK_KHR_fragment_shading_rate.html
https://developer.apple.com/documentation/metal/render_passes/rendering_at_different_rasterization_rates?language=objc
https://developer.qualcomm.com/sites/default/files/docs/adreno-gpu/snapdragon-game-toolkit/gdg/gpu/best_practices_other.html#use-variable-rate-shading
https://github.com/gpuweb/gpuweb/issues/450 is a useful comparison, but note that this is 2019 and partly out of date, e.g. when it comes to Vulkan, VK_KHR_fragment_shading_rate is different from the older NVIDIA extensions.