Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-19169

Qt 5: readAny() API for zero-copy in QIODevice

    XMLWordPrintable

Details

    Description

      The API is designed to provide zero-copy access to the data of a QIODevice.
      The current read() and readAll() calls all do a memory copy instead.

      For implementing:

      virtual QByteArray QIODevice::readAnyData();
      

      For using:

      QByteArray QIODevice::readAny();
      

      A readAny() call would avoid a memory copy: It could return a QByteArray of an arbitrary size
      from the data it has stored anyway "inside".

      Depending on the specific QIODevice implementation different things could be done:

      QTcpSocket:
      This class is getting notifications from the OS for new data. The new data is roughly in ~MTU
      sized chunks. Instead of the user getting a copy of the data, he would get the
      data itself, so only one copy from OS to user.

      QSslSocket:
      Similar to QTcpSocket, although the decryption process is obviously a copy.

      QNetworkReply:
      There are two cases here. For the case where the server sends a content-length,
      the zero-copy QNetworkReply would kick in. This allocates a big buffer where
      all the network reply data is stored in. A readAny() there returns a QByteArray
      created with QByteArray::fromRawData() which represents a part of the network reply
      that is already downloaded.
      For servers not sending a content-length (e.g. that stream data of unknown length)
      the QNetworkReply behaves very similar to the QTcpSocket/QSslSocket.
      With clever implementation only one copy from OS to user happens.

      QIODevice returned by a QAbstractNetworkCache:
      Depending on the network cache implementation various things are imaginable,
      e.g. the usage of memory mapped files or for an in-memory cache.

      QIODevice given to a QNetworkAccessManager for POST/PUT:
      Let's take QtWebKit with the POSTing of an HTML form as an example.
      QNetworkAccessManager reads from QtWebKit's QIODevice (=copy) and then
      at some point writes that data to the OS. This copy could be avoided by
      QtWebKit's QIODevice returning QByteArrays for its form elements.

      QFile:
      Who knows, on some OSes or with some file systems it might make sense to
      have QFile always memory-map stuff and then readAny provides the access to that
      buffer similar to the QNetworkReply zero-copy.
      If memory-mapping of the file doesn't make sense, we could read an "optimal" buffer amount to read from the OS.

      QBuffer:
      If at the start position, readAny() is basically readAll().
      If in the middle of the buffer, a slice of the QByteArray is returned via QByteArray::fromRawData().

      Any buffered QIODevice:
      If there is data in the QIODevice's internal buffer, return that.

      Default implementation:
      We should find something nice for that.

      Attachments

        Issue Links

          For Gerrit Dashboard: QTBUG-19169
          # Subject Branch Project Status CR V

          Activity

            People

              Unassigned Unassigned
              mgoetz Markus Goetz (Inactive)
              Votes:
              5 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are no open Gerrit changes