Regression: Qt Wayland Compositor allocates massive amounts of shared memory, never frees it

Problem originally observed with custom QtWayland compositor and application. However it is reproducible using slightly modified versions of Qt exampleprojects:

qtwayland/examples/wayland/multi-screen - unmodified

qtdeclarative/examples/quick/demos/samegame - change Settings.qml so that the application spans the two screens (not sure if this is essential to trigger the bug, but it seems to exaggerate it):

    property int screenHeight: 1080
    property int screenWidth: 2 * 1920

Reproduce:

1. Start monitoring memory usage: e.g. `sar -r 1` in one terminal, `watch 'cat /proc/meminfo | grep Shmem\:' in another.
2. Run the multi-screen compositor, with the platform set to eglfs. Note the total memory usage once it has loaded.
3. Run samegame, using wayland as the platform and touch/click "Single Player Game", but do not interact any further with the application.
4. Watch the memory usage for a few minutes. It will gradually step up. Looking at htop and /proc/meminfo shows that most of the memory is "Shmem" shared memory.
5. After a while, stop samegame. You will see a small drop in memory usage, but the shared memory isn't freed (compare memory usage with what it was after starting the compositor, but before starting samegame).
6. Stop the multi-screen compositor. The shared memory is freed
7. Alternatively, leave both programs running. Eventually the system runs out of memory and one of them is killed.

Observations

Tested with Qt5.11.2, Qt5.12.2 and Qt5.12.3

Qt5.11.2 does not exhibit the bug, Qt5.12.2 and Qt5.12.3 do.

If the compositor is built against 5.11 and samegame against 5.12, the issue doesn't happen.

Stopping and re-running samegame exacerbates the problem: it never seems to reclaim or re-use the shared memory as long as the compositor keeps running.

Using a more complex real-world application in place of samegame makes the problem much worse (With Qt5.11.2, our compositor + application stabilise at ~35% RAM usage. With Qt5.12, it quickly shoots up to ~97% and the application gets OOM-killed).

See attached file, 'sar-output.txt' for an example of memory usage while running the programs as described above:

multi-screen is started at 14:08:25

samegame is started at 14:08:36

samegame is stopped at 14:19:37

multi-screen is stopped at 14:20:44