If dockOptions are set to GroupedDragging in mainWindow, and dockWindows are undocked and grouped outside mainWindow, tab position is hardcoded to QTabBar::RoundedSouth in
qmainwindowlayout.cpp line no. 2576

                QDockAreaLayoutInfo *info = floatingTabs->layoutInfo();     
                *info = QDockAreaLayoutInfo(&layoutState.dockAreaLayout.sep, QInternal::LeftDock,
                                            Qt::Horizontal, QTabBar::RoundedSouth,
                                            static_cast<QMainWindow *>(parentWidget()));

Ideally it should pick this information from parent window i.e mainWindow.
I have attached a sample, when you dockwindows inside Main window, tab position is set to north, but when they are grouped after undocking, it is always south.