Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-80674

Large memory consumption (>35G!!!) while downloading a http request

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Not Evaluated
    • Resolution: Duplicate
    • Affects Version/s: 5.12.2
    • Fix Version/s: None
    • Component/s: GUI: Drag and Drop
    • Labels:
      None
    • Platform/s:
      Linux/X11

      Description

      Minimal example

      Here you can find minimal reproducible example. It tries to download just one file - http://storage.googleapis.com/books/ngrams/books/googlebooks-eng-all-3gram-20120701-an.gz.

      I measure memory consumption via two tools: standard KDE's System Monitor, and google's tcmalloc heapprofile. More precisely, I link against tcmalloc, and then run the binary from the example as follows: $ HEAPPROFILE=mybin.hprof ./google_ngram_qt_bug.

      I also used strace to figure out what is going on.

      The problem

      So, heapprofile reports two things:

      • Lines in logs like "Dumping heap profile to mybin.hprof.0361.heap (36215 MB currently in use)".
      • and, more importantly, mybin.hprof.0361.heap files that can later be visualised (see the attached screenshot). In particular, on the screenshot you can see how some mysterious (non-main) thread (QTcpServer -> QAbstractSocket -> <signal-slot> -> QHttpPart) in the background downloaded 36GB.

      Apart from the tool, I have a very useful cout in my app itself:

            std::cout << Reply->url().toString().toStdString()
                      << " Reply->bytesAvailable(): " << Reply->bytesAvailable()
                      << ", Reply->readBufferSize(): " << Reply->readBufferSize()
                      << ", bytesAvailable Before Polling: " << BeforePolling
                      << " => diff = " << Reply->bytesAvailable() - BeforePolling << ", Was Requested: " << MaxSize
                      << std::endl;
      

      which gives the following output (slightly redacted):

      ...an.gz bytesAvailable: 988118, bytesAvailable Before Polling: 988118, Was Requested: 16384
      an.gz Reply->bytesAvailable(): 971734 <--- NOTE bytesAvailable is decreasing
      an.gz Reply->bytesAvailable(): 955350
      an.gz Reply->bytesAvailable(): 938966
      an.gz Reply->bytesAvailable(): 21462  <--- Finally it becomes 21K
      
      !!! All of a sudden, from thin air, bytesAvailable() become 500MB(!):
      
      an.gz Reply->bytesAvailable(): 540618996, Reply->readBufferSize(): 1048576, bytesAvailable Before Polling: 5078 => diff = 540613918, Was Requested: 16384
      ...
      

      So, code in the main

      • request 16K,
      • but in response to get 16K, Qt got 500MB.

      strace shows huge amount of read traffic from socket 8 in thread id 4418:

      [pid  4418] ioctl(8, FIONREAD, [7090])  = 0
      [pid  4418] ioctl(8, FIONREAD, [12762]) = 0
      [pid  4418] ioctl(8, FIONREAD, [25524]) = 0
      [pid  4418] read(8, "%i\200}\31\370\210\1\233\\\361o`\251/\325\235\6\300\254Va\33`\331\26\34\17\274\261\252\346"..., 25524) = 25524
      [pid  4418] write(3, "\1\0\0\0\0\0\0\0", 8) = 8
      [pid  4418] ioctl(8, FIONREAD, [26942]) = 0
      [pid  4418] read(8, "\363\206\233\316\300\r/A\371*\361\246\21\336\\\v\322\343o\311\373>\260\220=\366\346\226\277\6\355W"..., 26942) = 26942
      

      Now if we grep for the thread id we will get:
      $ ps -e -T | grep 4418
      4415 4418 pts/1 00:00:20 QNetworkAccessM
       
      strace reported that file descriptor #8 was used for reading. let's have a look at what it is associated with:

      lsof -i -a -p 4443
      COMMAND    PID    USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
      google_ng 4443 dimanne    8u  IPv4 5585565      0t0  TCP fb4c8169a8dd:38282->lhr35s07-in-f16.1e100.net:http (ESTABLISHED)
      

      8u IPv4 fb4c8169a8dd:38282->lhr35s07-in-f16.1e100.net:http

       

      My surmise

      I presume that someone decided that it would be perfectly fine to implicitly download a page in the background in a different thread. It also means that:

      • QIODevice (from the point of view of end-user) does not behave like a stream - it eagerly reads everything in memory. (while I understand that it is not QIODevice that does it, but rather a different component in a different thread).
      • setReadBufferSize() is completely broken.

      So, since QIOStream/setReadBufferSize are not working, is there a way to disable eager and implicit downloading in the background?

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              dimanne DimanNe
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Gerrit Reviews

                  There are no open Gerrit changes