Vulkan QSGBatchRenderer::Renderer crash on pipelineCache access (with fix)

The crash is easily reproducible once you hit the right circumstances. You need to be running Qt 5.15.1 (or 5.15.0) QML with VulkanRhi backend. The issue is not reproducible with OpenGL or DirectX backends.

Symptoms:

The general situation is this: construct multiple (30+) objects in QML, these objects must use some kind of effect, like OpacityMask. Then destroy the objects, and create them again. Repeat a few times, and the application will crash.

The crash always produces the same callstack:

Qt5Guid.dll!QVarLengthArray<VkAttachmentDescription,8>::count() Line 120
Qt5Guid.dll!QVkRenderPassDescriptor::isCompatible(const QRhiRenderPassDescriptor * other) Line 5742
Qt5Quickd.dll!QSGBatchRenderer::operator==(const QSGBatchRenderer::GraphicsPipelineStateKey & a, const QSGBatchRenderer::GraphicsPipelineStateKey & b) Line 4693
Qt5Quickd.dll!QHashNode<QSGBatchRenderer::GraphicsPipelineStateKey,QRhiGraphicsPipeline *>::same_key(unsigned int h0, const QSGBatchRenderer::GraphicsPipelineStateKey & key0) Line 156
Qt5Quickd.dll!QHash<QSGBatchRenderer::GraphicsPipelineStateKey,QRhiGraphicsPipeline *>::findNode(const QSGBatchRenderer::GraphicsPipelineStateKey & akey, unsigned int h) Line 931
Qt5Quickd.dll!QHash<QSGBatchRenderer::GraphicsPipelineStateKey,QRhiGraphicsPipeline *>::findNode(const QSGBatchRenderer::GraphicsPipelineStateKey & akey, unsigned int * ahp) Line 951
Qt5Quickd.dll!QHash<QSGBatchRenderer::GraphicsPipelineStateKey,QRhiGraphicsPipeline *>::constFind(const QSGBatchRenderer::GraphicsPipelineStateKey & akey) Line 907
Qt5Quickd.dll!QSGBatchRenderer::Renderer::ensurePipelineState(QSGBatchRenderer::Element * e, const QSGBatchRenderer::ShaderManagerShader * sms) Line 3291
Qt5Quickd.dll!QSGBatchRenderer::Renderer::prepareRenderMergedBatch(QSGBatchRenderer::Batch * batch, QSGBatchRenderer::Renderer::PreparedRenderBatch * renderBatch) Line 3696
Qt5Quickd.dll!QSGBatchRenderer::Renderer::renderBatches() Line 4139
Qt5Quickd.dll!QSGBatchRenderer::Renderer::render() Line 4365
Qt5Quickd.dll!QSGRenderer::renderScene(const QSGBindable & bindable) Line 265
Qt5Quickd.dll!QSGRenderer::renderScene(unsigned int fboId) Line 201
Qt5Quickd.dll!QSGDefaultRenderContext::renderNextRhiFrame(QSGRenderer * renderer) Line 257
Qt5Quickd.dll!QSGRhiLayer::grab() Line 416
Qt5Quickd.dll!QSGRhiLayer::updateTexture() Line 122
Qt5Quickd.dll!QSGRhiShaderEffectNode::preprocess() Line 772
Qt5Quickd.dll!QSGRenderer::preprocess() Line 339
Qt5Quickd.dll!QSGRenderer::renderScene(const QSGBindable & bindable) Line 241
Qt5Quickd.dll!QSGRenderer::renderScene(unsigned int fboId) Line 201
Qt5Quickd.dll!QSGDefaultRenderContext::renderNextRhiFrame(QSGRenderer * renderer) Line 257
Qt5Quickd.dll!QQuickWindowPrivate::renderSceneGraph(const QSize & size, const QSize & surfaceSize) Line 615
Qt5Quickd.dll!QSGRenderThread::syncAndRender(QImage * grabImage) Line 839
Qt5Quickd.dll!QSGRenderThread::run() Line 1028
Qt5Cored.dll!QThreadPrivate::start(void * arg) Line 405
kernel32.dll!00007ffd41ba7bd4()
ntdll.dll!00007ffd42dcce51()

At frame 2, `const QRhiRenderPassDescriptor * other` pointer is dangling - the address is fine, but the content is invalid.

Analysis:

The problem is in \qtdeclarative\src\quick\scenegraph\qsgrhilayer.cpp, and \qtdeclarative\src\quick\scenegraph\coreapi\qsgbatchrenderer.cpp. The following sequence of events causes the crash:

1. QSGRhiLayer::grab() method is called
   1. Creates newCompatibleRenderPassDescriptor m_rtRp
   2. Creates QSGBatchRenderer::Renderer
   3. Calls renderer setRenderPassDescriptor with m_rtRp
2. In QSGBatchRenderer::Renderer constructor
   1. Creates ShaderManager object m_shaderManager
   2. Assigns m_shaderManager as a child to QSGRhiLayer::m_context with name "__qt_ShaderManager"
3. QSGBatchRenderer::Renderer::ensurePipelineState() method is called
   1. Constructs a key using the current render pass descriptor (set with m_rtRp)
   2. Looks up in m_shaderManager->pipelineCache by this key
   3. If not found, creates QRhiGraphicsPipeline object and saves in the hash using m_rtRp pointer as a part of the key
4. All good and well, until we start removing objects
5. QSGRhiLayer::releaseResources() method is called
   1. Deletes m_rtRp
6. QSGRhiLayer::grab() method is called
   1. Creates new or reuses objects
   2. Calls renderer setRenderPassDescriptor with a new m_rtRp
7. QSGBatchRenderer::Renderer::ensurePipelineState() method is called
   1. Constructs a key using the current render pass descriptor (set with m_rtRp)
   2. Looks up in m_shaderManager->pipelineCache by this key
   3. Old entry with old m_rtRp is still in the cache, at some point tries to call a.compatibleRenderPassDescriptor->isCompatible(b.compatibleRenderPassDescriptor)
8. Crash

Solution:

Perhaps it is more of a workaround rather than proper solution, but nevertheless.
Modify qtdeclarative\src\quick\scenegraph\qsgrhilayer.cpp:
Add include:

#include "coreapi/qsgbatchrenderer_p.h"

Modify QSGRhiLayer::releaseResources:

void QSGRhiLayer::releaseResources()
{
    if (auto shaderManager = m_context->findChild<QSGBatchRenderer::ShaderManager *>(QStringLiteral("__qt_ShaderManager"), Qt::FindDirectChildrenOnly)) {
        for (auto it = shaderManager->pipelineCache.begin(); it != shaderManager->pipelineCache.end();) {
            if (it.key().compatibleRenderPassDescriptor == m_rtRp) {
                delete it.value();
                it = shaderManager->pipelineCache.erase(it);
            } else {
                ++it;
            }
        }
    }
...

releaseResources() deletes m_rtRp used previously in shaderManager->pipelineCache, resulting in a dangling pointer. So here before we delete m_rtRp, clean up the cache.