If there's no space available on the screen to the right from the parent menu, even if there's available space to the left, a nested menu gets placed on top of the parent menu in wayland, but not in X11 (xwayland).

Move the window to the right edge of the screen so that the parent menu touches the right screen edge, open its submenu, observe it being shown on top of its parent, even though there's space to the left.

Edit:

The cause is a known QtWayland issue - https://wiki.qt.io/QtWayland_Remaining_Issues. It seems qtwayland currently has no way of getting information required for configuring xdg_positioner, anchor rect always has size 1x1, anchor top_right, gravity bottom_right, constraint adjustment slide_x | slide_y.

I've made a patch that somewhat "fixes" this issue - just a hack to enable flip_x adjustment for popups that have xdg_popup parent.