Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-19169

Qt 5: readAny() API for zero-copy in QIODevice




      The API is designed to provide zero-copy access to the data of a QIODevice.
      The current read() and readAll() calls all do a memory copy instead.

      For implementing:

      virtual QByteArray QIODevice::readAnyData();

      For using:

      QByteArray QIODevice::readAny();

      A readAny() call would avoid a memory copy: It could return a QByteArray of an arbitrary size
      from the data it has stored anyway "inside".

      Depending on the specific QIODevice implementation different things could be done:

      This class is getting notifications from the OS for new data. The new data is roughly in ~MTU
      sized chunks. Instead of the user getting a copy of the data, he would get the
      data itself, so only one copy from OS to user.

      Similar to QTcpSocket, although the decryption process is obviously a copy.

      There are two cases here. For the case where the server sends a content-length,
      the zero-copy QNetworkReply would kick in. This allocates a big buffer where
      all the network reply data is stored in. A readAny() there returns a QByteArray
      created with QByteArray::fromRawData() which represents a part of the network reply
      that is already downloaded.
      For servers not sending a content-length (e.g. that stream data of unknown length)
      the QNetworkReply behaves very similar to the QTcpSocket/QSslSocket.
      With clever implementation only one copy from OS to user happens.

      QIODevice returned by a QAbstractNetworkCache:
      Depending on the network cache implementation various things are imaginable,
      e.g. the usage of memory mapped files or for an in-memory cache.

      QIODevice given to a QNetworkAccessManager for POST/PUT:
      Let's take QtWebKit with the POSTing of an HTML form as an example.
      QNetworkAccessManager reads from QtWebKit's QIODevice (=copy) and then
      at some point writes that data to the OS. This copy could be avoided by
      QtWebKit's QIODevice returning QByteArrays for its form elements.

      Who knows, on some OSes or with some file systems it might make sense to
      have QFile always memory-map stuff and then readAny provides the access to that
      buffer similar to the QNetworkReply zero-copy.
      If memory-mapping of the file doesn't make sense, we could read an "optimal" buffer amount to read from the OS.

      If at the start position, readAny() is basically readAll().
      If in the middle of the buffer, a slice of the QByteArray is returned via QByteArray::fromRawData().

      Any buffered QIODevice:
      If there is data in the QIODevice's internal buffer, return that.

      Default implementation:
      We should find something nice for that.


        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.



              Unassigned Unassigned
              mgoetz Markus Goetz (Inactive)
              5 Vote for this issue
              6 Start watching this issue



                Gerrit Reviews

                  There are no open Gerrit changes