Uploaded image for project: 'Qt Visual Studio Tools'
  1. Qt Visual Studio Tools
  2. QTVSADDINBUG-916

QtMsbuild provokes concurrency issues on .tlog files access

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: P1: Critical
    • Resolution: Done
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.7.2
    • Component/s: Build-time integration
    • Labels:
      None
    • Commits:
      ac101910a367b3c34b3fb12ac758dd3d583fa62a (qt-labs/vstools/dev)

      Description

      Time to time our build on CI fails with different, but similar and common mysterous errors:

      17:26:59.743     9>C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Microsoft\VC\v160\Microsoft.CppBuild.targets(396,5): error MSB3374: The last access/last write time on file "x64\Release\Notifications\Notifications.tlog\Notifications.lastbuildstate" cannot be set. The process cannot access the file 'C:\app\appCore\Notifications\x64\Release\Notifications\Notifications.tlog\Notifications.lastbuildstate' because it is being used by another process. [C:\app\appCore\Notifications\Notifications.vcxproj]
      17:26:59.748     9>Done Building Project "C:\app\appCore\Notifications\Notifications.vcxproj" (default targets) -- FAILED.
      17:26:59.755    37>Done Building Project "C:\app\appCore\Core\Core.vcxproj" (default targets) -- FAILED.
      
      17:17:50.341    12>C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Microsoft\VC\v160\Microsoft.CppBuild.targets(346,5): error MSB3501: Could not read lines from file "x64\Release\Utils\Utils.tlog\Utils.lastbuildstate". The process cannot access the file 'C:\app\Shared\Utils\x64\Release\Utils\Utils.tlog\Utils.lastbuildstate' because it is being used by another process. [C:\app\Shared\Utils\Utils_vs2019.vcxproj]
      17:17:50.133  93:2>Project "C:\app\setsManager\setsManager.vcxproj" (93:2) is building "C:\app\appCore\sets\sets.vcxproj" (10:109) on node 10 (default targets).
      17:17:50.196 10:109>Project "C:\app\appCore\sets\sets.vcxproj" (10:109) is building "C:\app\3rdParty\Blowfish\vs2019\blowfish.vcxproj" (22:92) on node 4 (default targets).
      
      16:41:33.504    19>C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(1522,5): error MSB6003: The specified task executable "Lib.exe" could not be run. System.IO.IOException: The process cannot access the file 'C:\app\3rdParty\Cryptopp\x64\Release\cryptlib_static\cryptlib_static.tlog\Lib.command.1.tlog' because it is being used by another process. [C:\app\3rdParty\Cryptopp\cryptlib_static.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(1522,5): error MSB6003:    at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) [C:\app\3rdParty\Cryptopp\cryptlib_static.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(1522,5): error MSB6003:    at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) [C:\app\3rdParty\Cryptopp\cryptlib_static.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(1522,5): error MSB6003:    at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) [C:\app\3rdParty\Cryptopp\cryptlib_static.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(1522,5): error MSB6003:    at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost) [C:\app\3rdParty\Cryptopp\cryptlib_static.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(1522,5): error MSB6003:    at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost) [C:\app\3rdParty\Cryptopp\cryptlib_static.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(1522,5): error MSB6003:    at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding) [C:\app\3rdParty\Cryptopp\cryptlib_static.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(1522,5): error MSB6003:    at Microsoft.Build.CPPTasks.TrackedVCToolTask.WriteSourcesToCommandLinesTable(IDictionary`2 sourcesToCommandLines) [C:\app\3rdParty\Cryptopp\cryptlib_static.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(1522,5): error MSB6003:    at Microsoft.Build.CPPTasks.TrackedVCToolTask.PostExecuteTool(Int32 exitCode) [C:\app\3rdParty\Cryptopp\cryptlib_static.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(1522,5): error MSB6003:    at Microsoft.Build.CPPTasks.TrackedVCToolTask.ExecuteTool(String pathToTool, String responseFileCommands, String commandLineCommands) [C:\app\3rdParty\Cryptopp\cryptlib_static.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(1522,5): error MSB6003:    at Microsoft.Build.Utilities.ToolTask.Execute() [C:\app\3rdParty\Cryptopp\cryptlib_static.vcxproj]
      16:41:33.510    19>Done Building Project "C:\app\3rdParty\Cryptopp\cryptlib_static.vcxproj" (default targets) -- FAILED.
      

      We use multithreaded command-line msbuild build with a command line:

      msbuild app.sln /p:configuration=release /p:platform=x64 /m /nr:false

      or

      msbuild app.sln /t:mainApp /p:configuration=release /p:platform=x64 /m /nr:false

      Our solution contains of both Windows C++ applications and project with QtMsBuild actions.

      All errors have in common that they occur on files from *.tlog folder, so I started to dig into logs.

      During my research I've found out that there is nothing to do with msbuild so it does pretty well evaluate the dependency tree and build each project only once. This can be pretty noticed when build simple windows project with many dependencies

      What I was looking for are repeated lines of actions with files in tlog folder. E.g.
      Deleting file "x64\Release\winConso.483b606c.tlog\unsuccessfulbuild".
      Our builds contains number of repeated lines per each project, however building just Windows app with same dependecies yields just to only one action per project.

      Prepared simple solution I've noticed that project with QtMsbuild actions being built produces repetition of this line. While simple Windows project produces only one line.
      Also I see that previous version of qt projects with CustomActions for moc and other tools produce also single line per each dependent project.

      This means that the problem of repeated touching of files in *.tlog directory is common only for projects with QtMsBuild actions inside.

      You can notice this in build_qt.log that StaticLib1 was build first then

      QtVars:
                 WHERE /Q cl.exe
      

      block was called, then StaticLib1 was triggered to build again on different node (3:3).

      Scaling this up we'll get our case where we have 80 projects in a solution with ~16 of them using QtMsBuild and others being dependencies.

      I've attached screens from simple solution's log, also simple solution itself, and msbuild logs from it.

        Attachments

        1. build_qt.log
          10 kB
        2. build_win.log
          6 kB
        3. loop_msbuild.bat
          1 kB
        4. qt_bin_msbuild.binlog
          459 kB
        5. qt_log_screen.png
          qt_log_screen.png
          70 kB
        6. simple_src.zip
          562 kB
        7. win_bin_msbuild.binlog
          338 kB
        8. win_log_screen.png
          win_log_screen.png
          63 kB

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

              People

              Assignee:
              mecfc Miguel Costa
              Reporter:
              oleg.poloinikov oleg.poloinikov
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Gerrit Reviews

                  There are no open Gerrit changes