Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-119740

QDoc: Parallelize source file parsing in ClangCodeParser

    XMLWordPrintable

Details

    • Task
    • Resolution: Unresolved
    • P2: Important
    • None
    • None
    • Build tools: qdoc
    • None

    Description

      In QTBUG-78197 profiling QDoc showed that most of the running time (generally up to a good 50-70% of the total time) is spent by Clang parsing files.

      QDoc uses Clang C and C++ APIs to extract both the user provided documentation and information about the code-base itself when parsing a project that has is written, at least partially, in C++.

       

      Generally, QDoc delegates to Clang in two different stages.

       

      QDoc first parses a "module header" to build a PCH of all the relevant headers in the project. The PCH is later used when parsing implementation files.

       

      This step, due to how Clang is built, is to be considered as a single atomic operation from QDoc's side. That is, to build a PCH, Clang requires that we use a single translation unit.

       

      In this sense, there is little that we can do to improve the performances at this step.

      QDoc will analyze the AST that clang produces, so that some optmizations could be done in that part, but the profiling that was performed shows that we cannot yet consider this a bottleneck, as the parsing itself takes a far bigger part of the running time.

       

      After the PCH is built, QDoc will actually parse the implementation files of the project, using the PCH that was previously built.

       

      The main interest during this step is to retrieve and analyze the user-provided documentation blocks.

       

      Currently, QDoc parses each file sequentially, analyzes the documentation that is found in the file, perform certain sanity checking operation and performs some cleanups.

      While, generally, a single file does not seem to require a huge amount of time to be processed, the sequential aspect of it creates a huge waiting time, especially for projects like Qt where many files are present.

       

      To reduce the time taken by this step, which seems to take most of the time of an actual run of QDoc, source files should be read in parallel.

       

      While the parallelization itself should be relatively trivial to implement after certain architectural clean-ups, this might de-stabilize the output that QDoc produces for a project, even when no change is performed on the project itself.

       

      Stabilizing the output might require certain bigger architectural changes which are a prerequisite to completing this task.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              docinfrastructure Documentation Infrastructure Team
              diseraluca Luca Di Sera
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are no open Gerrit changes