The model is not efficient for large amount of data, especially when you want to sort all but a few items. The sort itself is not the problem. The problem comes from the fucntion indexesFromRange that goes through all the indexes:
static void indexesFromRange(const QItemSelectionRange &range, QModelIndexList &result) in qitemselectionmodel.cpp

Test case - to reproduce:

• select the entire table (click in top left)
• then, deselect a couple of rows using ctrl-click
• then, click the column 2 header to trigger a sort

Also while it would help, reimplementing QItemSelectionModel is not a solution here as the virtual functions exposed are limited.