Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-67161

QNetworkAccessManager can silently resubmit non idempotent HTTP requests

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • P3: Somewhat important
    • None
    • 5.9.4
    • Network: HTTP
    • None
    • Windows 1O Pro (1709)

      QT 5.9.4

    Description

      We have an application that uses the QNetworkAccessManager to submit requests to a SOAP webservice, ie does an HTTP post of an XML request.  We have occasionally seen that multiple submissions of these requests from the QT HTTP networking stack, even though only one submission was made from the application.  These repeated submissions cause application problems for us, if the initial error was propagated to the client, we could decide whether or not to resubmit the request ( for example querying the backend before deciding to resubmit ).

      This can be fairy easily reproduced, I will attach a test case that does 1000 POST's to a simple CGI script, which returns a fixed payload.

      The setup to reproduce this is as follows:

      QT Client Prog(HTTP POST * 1000) --> HAProxy Load Balancer --> Apache with CGI Page

      By simply restarting the HAProxy LB every few seconds during the test, this causes the re-submissions to occur.  Another method is to simply introduce a network with a small degree of packet loss between the client and server.

      The requests are setup to POST urls of the form:

      POST /cgi-bin/qt-test.cgi?a=FIXEDTIMESTAMP-REQUESTINDEX

      This allows easy identification of duplicate requests.

      The apache log looks like this:

      127.0.0.1 - - [19/Mar/2018:13:20:19 +0000] "POST /cgi-bin/qt-test.cgi?a=1521465603-996 HTTP/1.1" 200 24768 "-" "Mozilla/5.0"
      127.0.0.1 - - [19/Mar/2018:13:20:19 +0000] "POST /cgi-bin/qt-test.cgi?a=1521465603-997 HTTP/1.1" 200 24768 "-" "Mozilla/5.0"
      127.0.0.1 - - [19/Mar/2018:13:20:19 +0000] "POST /cgi-bin/qt-test.cgi?a=1521465603-998 HTTP/1.1" 200 24768 "-" "Mozilla/5.0"
      127.0.0.1 - - [19/Mar/2018:13:20:19 +0000] "POST /cgi-bin/qt-test.cgi?a=1521465603-999 HTTP/1.1" 200 24768 "-" "Mozilla/5.0"
      127.0.0.1 - - [19/Mar/2018:13:20:19 +0000] "POST /cgi-bin/qt-test.cgi?a=1521465603-1000 HTTP/1.1" 200 24768 "-" "Mozilla/5.0"

      Duplicates can be found like this:

      [root@qt-http-test httpd]# grep 1521465603 access_log | awk -F= '{print $2}' | sort | wc -l
      1005

      [root@qt-http-test httpd]# grep 1521465603 access_log | awk -F= '{print $2}' | sort | uniq -c | grep -v " 1"
      2 1521465603-250 HTTP/1.1" 200 24768 "-" "Mozilla/5.0"
      2 1521465603-438 HTTP/1.1" 200 24768 "-" "Mozilla/5.0"
      2 1521465603-44 HTTP/1.1" 200 24768 "-" "Mozilla/5.0"
      2 1521465603-639 HTTP/1.1" 200 24768 "-" "Mozilla/5.0"
      2 1521465603-849 HTTP/1.1" 200 24768 "-" "Mozilla/5.0"

      [root@qt-http-test httpd]# grep 1521465603-849 access_log
      127.0.0.1 - - [19/Mar/2018:13:20:16 +0000] "POST /cgi-bin/qt-test.cgi?a=1521465603-849 HTTP/1.1" 200 24768 "-" "Mozilla/5.0"
      127.0.0.1 - - [19/Mar/2018:13:20:17 +0000] "POST /cgi-bin/qt-test.cgi?a=1521465603-849 HTTP/1.1" 200 24768 "-" "Mozilla/5.0" 

      I believe these re-submissions are coming from logic in qhttpnetworkconnectionchannel.cpp

      https://github.com/qt/qtbase/blob/5.9/src/network/access/qhttpnetworkconnectionchannel.cpp#LC72

      It looks to me as if this behavior has been added by design, however I can't find it documented in the Public APIs anywhere.

      From my reading of the code, no attempt is made to distinguish between "safe" or "idempotent" methods as defined in the HTTP RFC : https://tools.ietf.org/html/rfc7231#section-4.2 or other methods, so this re-submission could have un-intended consequences, as in our case ( think of double booking or double payment scenarios )

      1) Is my understanding correct?

      2) Could this be documented somewhere - we spent a long time identifying this behaviour?

      3) Could this behavior be made configurable in QNAM, with the current behavior as the default, allowing clients to programatically decide if they wanted silent re-submissions?

      Our current workaround is patch the value of reconnectAttemptsDefault to 0 in qhttpnetworkconnectionchannel.cpp, but we would prefer not to have to patch QT.

      Attachments

        1. qt-67161.zip
          6 kB
          Dave Riseley
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            cnn Qt Core & Network
            daver_oa Dave Riseley
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes