Details
-
Bug
-
Resolution: Done
-
P1: Critical
-
5.7, 5.9.6, 5.10.1
-
Linux 3.14 (TI AM355x with wl-18xx drivers)
-
c89d0f9d532a2719118614b9fa9b8efffbe12f2f
Description
We are running Qt on an embedded ARM am355x device (qt 5.7.0 with a 3.14 kernel and TI wl18xx drivers). We found that all our daemons using HTTP crashed intermittently on certain machines, with a segfault in malloc. Looking at the coredump, we were able to trace that back to a recursion in QNetworkAccessManager.cpp (note the frame depth), with the stack at some point starting to overwrite heap data structures.
#185619 0xb6daa020 in QNetworkAccessManagerPrivate::_q_networkSessionStateChanged (this=this@entry=0x1d0af0, state=<optimized out>) at /usr/src/debug/qtbase/5.7.0+gitAUTOINC+69b43e74d7-r0/git/src/network/access/qnetworkaccessmanager.cpp:1654 #185620 0xb6daa430 in QNetworkAccessManagerPrivate::createSession (this=this@entry=0x1d0af0, config=...) at /usr/src/debug/qtbase/5.7.0+gitAUTOINC+69b43e74d7-r0/git/src/network/access/qnetworkaccessmanager.cpp:1593 #185621 0xb6daa020 in QNetworkAccessManagerPrivate::_q_networkSessionStateChanged (this=this@entry=0x1d0af0, state=<optimized out>) at /usr/src/debug/qtbase/5.7.0+gitAUTOINC+69b43e74d7-r0/git/src/network/access/qnetworkaccessmanager.cpp:1654 #185622 0xb6daa430 in QNetworkAccessManagerPrivate::createSession (this=this@entry=0x1d0af0, config=...) at /usr/src/debug/qtbase/5.7.0+gitAUTOINC+69b43e74d7-r0/git/src/network/access/qnetworkaccessmanager.cpp:1593 #185623 0xb6daa020 in QNetworkAccessManagerPrivate::_q_networkSessionStateChanged (this=this@entry=0x1d0af0, state=<optimized out>) at /usr/src/debug/qtbase/5.7.0+gitAUTOINC+69b43e74d7-r0/git/src/network/access/qnetworkaccessmanager.cpp:1654 #185624 0xb6daa430 in QNetworkAccessManagerPrivate::createSession (this=this@entry=0x1d0af0, config=...) at /usr/src/debug/qtbase/5.7.0+gitAUTOINC+69b43e74d7-r0/git/src/network/access/qnetworkaccessmanager.cpp:1593 #185625 0xb6daa020 in QNetworkAccessManagerPrivate::_q_networkSessionStateChanged (this=this@entry=0x1d0af0, state=<optimized out>) at /usr/src/debug/qtbase/5.7.0+gitAUTOINC+69b43e74d7-r0/git/src/network/access/qnetworkaccessmanager.cpp:1654 #185626 0xb6daa430 in QNetworkAccessManagerPrivate::createSession (this=this@entry=0x1d0af0, config=...) at /usr/src/debug/qtbase/5.7.0+gitAUTOINC+69b43e74d7-r0/git/src/network/access/qnetworkaccessmanager.cpp:1593 #185627 0xb6daa020 in QNetworkAccessManagerPrivate::_q_networkSessionStateChanged (this=this@entry=0x1d0af0, state=<optimized out>) at /usr/src/debug/qtbase/5.7.0+gitAUTOINC+69b43e74d7-r0/git/src/network/access/qnetworkaccessmanager.cpp:1654 #185628 0xb6daa430 in QNetworkAccessManagerPrivate::createSession (this=this@entry=0x1d0af0, config=...) at /usr/src/debug/qtbase/5.7.0+gitAUTOINC+69b43e74d7-r0/git/src/network/access/qnetworkaccessmanager.cpp:1593 #185629 0xb6daa020 in QNetworkAccessManagerPrivate::_q_networkSessionStateChanged (this=this@entry=0x1d0af0, state=<optimized out>) at /usr/src/debug/qtbase/5.7.0+gitAUTOINC+69b43e74d7-r0/git/src/network/access/qnetworkaccessmanager.cpp:1654 #185630 0xb6daa430 in QNetworkAccessManagerPrivate::createSession (this=this@entry=0x1d0af0, config=...) at /usr/src/debug/qtbase/5.7.0+gitAUTOINC+69b43e74d7-r0/git/src/network/access/qnetworkaccessmanager.cpp:1593 #185631 0xb6daa020 in QNetworkAccessManagerPrivate::_q_networkSessionStateChanged (this=this@entry=0x1d0af0, state=<optimized out>) at /usr/src/debug/qtbase/5.7.0+gitAUTOINC+69b43e74d7-r0/git/src/network/access/qnetworkaccessmanager.cpp:1654 #185632 0xb6daa430 in QNetworkAccessManagerPrivate::createSession (this=this@entry=0x1d0af0, config=...) at /usr/src/debug/qtbase/5.7.0+gitAUTOINC+69b43e74d7-r0/git/src/network/access/qnetworkaccessmanager.cpp:1593 #185633 0xb6daa020 in QNetworkAccessManagerPrivate::_q_networkSessionStateChanged (this=this@entry=0x1d0af0, state=<optimized out>) at /usr/src/debug/qtbase/5.7.0+gitAUTOINC+69b43e74d7-r0/git/src/network/access/qnetworkaccessmanager.cpp:1654 #185634 0xb6daa430 in QNetworkAccessManagerPrivate::createSession (this=this@entry=0x1d0af0, config=...) at /usr/src/debug/qtbase/5.7.0+gitAUTOINC+69b43e74d7-r0/git/src/network/access/qnetworkaccessmanager.cpp:1593 #185635 0xb6daa020 in QNetworkAccessManagerPrivate::_q_networkSessionStateChanged (this=this@entry=0x1d0af0, state=<optimized out>) at /usr/src/debug/qtbase/5.7.0+gitAUTOINC+69b43e74d7-r0/git/src/network/access/qnetworkaccessmanager.cpp:1654 #185636 0xb6daa430 in QNetworkAccessManagerPrivate::createSession (this=this@entry=0x1d0af0, config=...) at /usr/src/debug/qtbase/5.7.0+gitAUTOINC+69b43e74d7-r0/git/src/network/access/qnetworkaccessmanager.cpp:1593 #185637 0xb6daa020 in QNetworkAccessManagerPrivate::_q_networkSessionStateChanged (this=this@entry=0x1d0af0, state=<optimized out>) at /usr/src/debug/qtbase/5.7.0+gitAUTOINC+69b43e74d7-r0/git/src/network/access/qnetworkaccessmanager.cpp:1654 #185638 0xb6daa430 in QNetworkAccessManagerPrivate::createSession (this=this@entry=0x1d0af0, config=...) at /usr/src/debug/qtbase/5.7.0+gitAUTOINC+69b43e74d7-r0/git/src/network/access/qnetworkaccessmanager.cpp:1593 #185639 0xb6daa020 in QNetworkAccessManagerPrivate::_q_networkSessionStateChanged (this=this@entry=0x1d0af0, state=<optimized out>) at /usr/src/debug/qtbase/5.7.0+gitAUTOINC+69b43e74d7-r0/git/src/network/access/qnetworkaccessmanager.cpp:1654 #185640 0xb6daa430 in QNetworkAccessManagerPrivate::createSession (this=this@entry=0x1d0af0, config=...) at /usr/src/debug/qtbase/5.7.0+gitAUTOINC+69b43e74d7-r0/git/src/network/access/qnetworkaccessmanager.cpp:1593 #185641 0xb6daa020 in QNetworkAccessManagerPrivate::_q_networkSessionStateChanged (this=this@entry=0x1d0af0, state=<optimized out>) at /usr/src/debug/qtbase/5.7.0+gitAUTOINC+69b43e74d7-r0/git/src/network/access/qnetworkaccessmanager.cpp:1654
This chain is triggered by a state = Disconnected transition. I think the following happens in QNetworkAccessManager, and is due to a race condition introduced by https://bugreports.qt.io/browse/QTBUG-49760 .
void QNetworkAccessManagerPrivate::_q_networkSessionStateChanged(QNetworkSession::State state)
{
Q_Q(QNetworkAccessManager);
bool reallyOnline = false;
//Do not emit the networkSessionConnected signal here, except for roaming -> connected
//transition, otherwise it is emitted twice in a row when opening a connection.
if (state == QNetworkSession::Connected && lastSessionState != QNetworkSession::Roaming)
emit q->networkSessionConnected();
lastSessionState = state;
if (online && state == QNetworkSession::Disconnected) {
const auto cfgs = networkConfigurationManager.allConfigurations();
for (const QNetworkConfiguration &cfg : cfgs) {
// XXX the configuration being active does not mean that the session is actually connected
if (cfg.state().testFlag(QNetworkConfiguration::Active)) {
reallyOnline = true;
}
}
} else if (state == QNetworkSession::Connected || state == QNetworkSession::Roaming) {
reallyOnline = true;
}
if (!reallyOnline) {
if (state != QNetworkSession::Connected && state != QNetworkSession::Roaming) {
if (networkAccessible != QNetworkAccessManager::NotAccessible) {
networkAccessible = QNetworkAccessManager::NotAccessible;
emit q->networkAccessibleChanged(networkAccessible);
}
}
} else {
if (defaultAccessControl)
if (networkAccessible != QNetworkAccessManager::Accessible) {
networkAccessible = QNetworkAccessManager::Accessible;
emit q->networkAccessibleChanged(networkAccessible);
}
}
online = reallyOnline;
if (online && (state != QNetworkSession::Connected && state != QNetworkSession::Roaming)) {
_q_networkSessionClosed();
createSession(q->configuration());
}
}
The state is Disconnected, yet there is an active QNetworkConfiguration. QNetworkAccessManager decides to close the current session, and create a new session from the active configuration.
Little does it know that this will again create a disconnected session, which causes the infinite recursion.
Instead of assuming that an active configuration necessarily means that the device is online, QNetworkAccessManager should also check the bearer engine's state. I don't have enough knowledge of the network access management layer to provide a high quality patch, I would be happy to write one with some guidance.
It is worth pointing out that the wifi network is hazardous at best, and the drivers misbehave fairly often. I wasn't able to reproduce the case easily, which leads me to believe there is only a short window of time before the configuration itself gets marked inactive.
This is a regression from 5.4 which we have been using for the last 1.5 years on thousands of devices. This was spotted on our initial test run on about 20 machines running for a couple of days (it happened on at least 2 machines).
An additional detail is that the SSID of the connman service that triggered this crash seems to point to a verizon mobile hotspot, which indeed would be in and out if that person was moving around.