Stack overflow in nativeSelect()

QNativeSocketEnginePrivate code is vulnerable to fd_set overflow. As a result, an attacker could set at least one arbitrary bit on program's stack. Due to the nature of attack, stack smashing protection doesn't help much.

Details: the QNativeSocketEnginePrivate::nativeSelect() calls do use select() system call. The latter relies upon setting desired file descriptor being waited on in fd_set (semi-opaque) structure. "Semi-opaque" means that while its actual implementation is system-dependent, maximum number of file descriptors it could address is hardcoded at the compile time. On systems like *BSD and GNU/Linux fd_set structures contain 1024 bits by default. The FD_SET() routine (usually implemented as macro), which is used for setting exact bits, doesn't check actual boundaries, allowing to access bits outside default fd_set size.

Let's see what will happen if some malicious application opens >1021 files and then exec()'s the Qt-based binary. If such binary will call anything in Qt that results in QNativeSocketEnginePrivate::nativeSelect() being run, then out-of-bounds access will happen. In the worst case, this will case application to misbehave, for example, due to some flag variable like "allow current user to do something" being set instead of being zero.

The problematic code existed since Qt 4.x at least.

The supposed solution is to change select() to poll(), which is as well standardized as select(), but doesn't suffer from the same diseases.

Attached are:
1. Proposed patch (tested in the environment below);
2. PoC code.

Sample PoC output:

$ ./qtsmashtest
qtsmashtest: trying socket descriptor 1029
qtsmashtest: trying socket descriptor 1037
qtsmashtest: trying socket descriptor 1045
qtsmashtest: trying socket descriptor 1053
qtsmashtest: trying socket descriptor 1061
qtsmashtest: trying socket descriptor 1069
qtsmashtest: trying socket descriptor 1077
qtsmashtest: trying socket descriptor 1085
qtsmashtest: trying socket descriptor 1093
Abort trap (core dumped)