Uploaded image for project: 'Coin'
  1. Coin
  2. COIN-497

Tests fails due to git hanging up because of too many concurrent connections

    XMLWordPrintable

Details

    • User Story
    • Resolution: Done
    • P3: Somewhat important
    • None
    • None
    • Repository Manager, Tests
    • None

    Description

      On fresh install of coin, 'make citest' is likely to fail as a result of gerrit slamming the connection because of too many concurrent connections:

       

      INFO:repositorymanager.py:112(87540): Cloning missing repository qt/qtcanvas3d from ssh://codereview.qt-project.org:29418/qt/qtcanvas3d to /tmp/coin-testpit6040s/git-repos/qt-project/qt/qtcanvas3d
      INFO:repositorymanager.py:112(87540): Cloning missing repository qt/qtwebview from ssh://codereview.qt-project.org:29418/qt/qtwebview to /tmp/coin-testpit6040s/git-repos/qt-project/qt/qtwebview
      INFO:repositorymanager.py:112(87540): Cloning missing repository qt/qtquickcontrols2 from ssh://codereview.qt-project.org:29418/qt/qtquickcontrols2 to /tmp/coin-testpit6040s/git-repos/qt-project/qt/qtquickcontrols2
      INFO:repositorymanager.py:112(87540): Cloning missing repository qt/qtpurchasing from ssh://codereview.qt-project.org:29418/qt/qtpurchasing to /tmp/coin-testpit6040s/git-repos/qt-project/qt/qtpurchasing
      INFO:repositorymanager.py:112(87540): Cloning missing repository qt/qtcharts from ssh://codereview.qt-project.org:29418/qt/qtcharts to /tmp/coin-testpit6040s/git-repos/qt-project/qt/qtcharts
      INFO:repositorymanager.py:112(87540): Cloning missing repository qt/qtdatavis3d from ssh://codereview.qt-project.org:29418/qt/qtdatavis3d to /tmp/coin-testpit6040s/git-repos/qt-project/qt/qtdatavis3d
      INFO:repositorymanager.py:112(87540): Cloning missing repository qt/qtvirtualkeyboard from ssh://codereview.qt-project.org:29418/qt/qtvirtualkeyboard to /tmp/coin-testpit6040s/git-repos/qt-project/qt/qtvirtualkeyboard
      INFO:repositorymanager.py:112(87540): Cloning missing repository qt/qtgamepad from ssh://codereview.qt-project.org:29418/qt/qtgamepad to /tmp/coin-testpit6040s/git-repos/qt-project/qt/qtgamepad
      INFO:repositorymanager.py:112(87540): Cloning missing repository qt/qtscxml from ssh://codereview.qt-project.org:29418/qt/qtscxml to /tmp/coin-testpit6040s/git-repos/qt-project/qt/qtscxml
      INFO:repositorymanager.py:112(87540): Cloning missing repository qt/qtspeech from ssh://codereview.qt-project.org:29418/qt/qtspeech to /tmp/coin-testpit6040s/git-repos/qt-project/qt/qtspeech
      INFO:repositorymanager.py:112(87540): Cloning missing repository qt/qtnetworkauth from ssh://codereview.qt-project.org:29418/qt/qtnetworkauth to /tmp/coin-testpit6040s/git-repos/qt-project/qt/qtnetworkauth
      INFO:repositorymanager.py:112(87540): Cloning missing repository qt/qtremoteobjects from ssh://codereview.qt-project.org:29418/qt/qtremoteobjects to /tmp/coin-testpit6040s/git-repos/qt-project/qt/qtremoteobjects
      INFO:repositorymanager.py:112(87540): Cloning missing repository qt/qtwebglplugin from ssh://codereview.qt-project.org:29418/qt/qtwebglplugin to /tmp/coin-testpit6040s/git-repos/qt-project/qt/qtwebglplugin
      INFO:repositorymanager.py:112(87540): Cloning missing repository qt/qtlottie from ssh://codereview.qt-project.org:29418/qt/qtlottie to /tmp/coin-testpit6040s/git-repos/qt-project/qt/qtlottie
      INFO:repositorymanager.py:112(88172): Cloning missing repository qt/qt5 from ssh://codereview.qt-project.org:29418/qt/qt5 to /tmp/coin-testbub5ref4/git-repos/qt-project/qt/qt5
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtsvg from ssh://codereview.qt-project.org:29418/qt/qtsvg to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtsvg
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtdeclarative from ssh://codereview.qt-project.org:29418/qt/qtdeclarative to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtdeclarative
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtactiveqt from ssh://codereview.qt-project.org:29418/qt/qtactiveqt to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtactiveqt
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtscript from ssh://codereview.qt-project.org:29418/qt/qtscript to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtscript
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtmultimedia from ssh://codereview.qt-project.org:29418/qt/qtmultimedia to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtmultimedia
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qttools from ssh://codereview.qt-project.org:29418/qt/qttools to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qttools
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtxmlpatterns from ssh://codereview.qt-project.org:29418/qt/qtxmlpatterns to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtxmlpatterns
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qttranslations from ssh://codereview.qt-project.org:29418/qt/qttranslations to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qttranslations
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtdoc from ssh://codereview.qt-project.org:29418/qt/qtdoc to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtdoc
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtrepotools from ssh://codereview.qt-project.org:29418/qt/qtrepotools to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtrepotools
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtqa from ssh://codereview.qt-project.org:29418/qt/qtqa to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtqa
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtlocation from ssh://codereview.qt-project.org:29418/qt/qtlocation to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtlocation
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtsensors from ssh://codereview.qt-project.org:29418/qt/qtsensors to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtsensors
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtsystems from ssh://codereview.qt-project.org:29418/qt/qtsystems to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtsystems
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtfeedback from ssh://codereview.qt-project.org:29418/qt/qtfeedback to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtfeedback
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtdocgallery from ssh://codereview.qt-project.org:29418/qt/qtdocgallery to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtdocgallery
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtpim from ssh://codereview.qt-project.org:29418/qt/qtpim to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtpim
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtconnectivity from ssh://codereview.qt-project.org:29418/qt/qtconnectivity to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtconnectivity
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtwayland from ssh://codereview.qt-project.org:29418/qt/qtwayland to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtwayland
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qt3d from ssh://codereview.qt-project.org:29418/qt/qt3d to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qt3d
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtimageformats from ssh://codereview.qt-project.org:29418/qt/qtimageformats to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtimageformats
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtquick1 from ssh://codereview.qt-project.org:29418/qt/qtquick1 to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtquick1
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtgraphicaleffects from ssh://codereview.qt-project.org:29418/qt/qtgraphicaleffects to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtgraphicaleffects
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtquickcontrols from ssh://codereview.qt-project.org:29418/qt/qtquickcontrols to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtquickcontrols
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtserialbus from ssh://codereview.qt-project.org:29418/qt/qtserialbus to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtserialbus
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtserialport from ssh://codereview.qt-project.org:29418/qt/qtserialport to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtserialport
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtx11extras from ssh://codereview.qt-project.org:29418/qt/qtx11extras to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtx11extras
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtmacextras from ssh://codereview.qt-project.org:29418/qt/qtmacextras to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtmacextras
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtwinextras from ssh://codereview.qt-project.org:29418/qt/qtwinextras to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtwinextras
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtandroidextras from ssh://codereview.qt-project.org:29418/qt/qtandroidextras to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtandroidextras
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtenginio from ssh://codereview.qt-project.org:29418/qt/qtenginio to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtenginio
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtwebsockets from ssh://codereview.qt-project.org:29418/qt/qtwebsockets to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtwebsockets
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtwebchannel from ssh://codereview.qt-project.org:29418/qt/qtwebchannel to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtwebchannel
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtwebengine from ssh://codereview.qt-project.org:29418/qt/qtwebengine to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtwebengine
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtcanvas3d from ssh://codereview.qt-project.org:29418/qt/qtcanvas3d to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtcanvas3d
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtwebview from ssh://codereview.qt-project.org:29418/qt/qtwebview to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtwebview
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtquickcontrols2 from ssh://codereview.qt-project.org:29418/qt/qtquickcontrols2 to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtquickcontrols2
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtpurchasing from ssh://codereview.qt-project.org:29418/qt/qtpurchasing to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtpurchasing
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtcharts from ssh://codereview.qt-project.org:29418/qt/qtcharts to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtcharts
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtdatavis3d from ssh://codereview.qt-project.org:29418/qt/qtdatavis3d to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtdatavis3d
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtvirtualkeyboard from ssh://codereview.qt-project.org:29418/qt/qtvirtualkeyboard to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtvirtualkeyboard
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtgamepad from ssh://codereview.qt-project.org:29418/qt/qtgamepad to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtgamepad
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtscxml from ssh://codereview.qt-project.org:29418/qt/qtscxml to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtscxml
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtspeech from ssh://codereview.qt-project.org:29418/qt/qtspeech to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtspeech
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtnetworkauth from ssh://codereview.qt-project.org:29418/qt/qtnetworkauth to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtnetworkauth
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtremoteobjects from ssh://codereview.qt-project.org:29418/qt/qtremoteobjects to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtremoteobjects
      INFO:repositorymanager.py:112(87532): Cloning missing repository qt/qtwebglplugin from ssh://codereview.qt-project.org:29418/qt/qtwebglplugin to /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtwebglplugin
      ERROR:dependency.py:131(87532): Failed to resolve dependencies
      Traceback (most recent call last):
        File "/home/citest/tqtc-coin-ci/src/repositorymanager.py", line 581, in resolve_submodule
          repo = await self.resolvedRepository(repo)
        File "/home/citest/tqtc-coin-ci/src/repositorymanager.py", line 207, in resolvedRepository
          await self.ensureRepoExists(repositoryState.gerritInstance, repositoryState.project)
        File "/home/citest/tqtc-coin-ci/src/utils.py", line 268, in async_lru_cached_wrapper
          result = await future_result
        File "/home/citest/tqtc-coin-ci/src/repositorymanager.py", line 116, in ensureRepoExists
          await clone("--bare", repoUrl, diskPath)
        File "/home/citest/tqtc-coin-ci/src/ash.py", line 145, in __call__
          raise sh.ErrorReturnCode_128(" ".join(processed_args), stdout, stderr)
      sh.ErrorReturnCode_128:   RAN: git --no-pager clone --bare ssh://codereview.qt-project.org:29418/qt/qtsensors /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtsensors  STDOUT:
        STDERR:
      Cloning into bare repository '/tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtsensors'...
      Received disconnect from 54.194.93.196 port 29418:12: Too many concurrent connections (64) - max. allowed: 64
      Disconnected from 54.194.93.196 port 29418
      fatal: Could not read from remote repository.Please make sure you have the correct access rights
      and the repository exists.
      During handling of the above exception, another exception occurred:Traceback (most recent call last):
        File "/home/citest/tqtc-coin-ci/src/dependency.py", line 124, in get_submodule_dependencies
          resolvedRepositories = await self._resolve_repositories_against_product(modules, superRepo)
        File "/home/citest/tqtc-coin-ci/src/dependency.py", line 136, in _resolve_repositories_against_product
          submodules = await self.repoManager.readSubModules(superRepo.gerritInstance, superRepo.project, superRepo.sha1)
        File "/home/citest/tqtc-coin-ci/src/repositorymanager.py", line 597, in readSubModules
          await asyncio.gather(*(resolve_submodule(submodule, results[submodule["path"]]) for submodule in modules.values() if not submodule.get('sha1')))
        File "/home/citest/tqtc-coin-ci/src/repositorymanager.py", line 584, in resolve_submodule
          raise Exception(f"Resolving submodule {submodule} failed:\n{traceback.format_exc()}")
      Exception: Resolving submodule <Section: submodule "qtsensors"> failed:
      Traceback (most recent call last):
        File "/home/citest/tqtc-coin-ci/src/repositorymanager.py", line 581, in resolve_submodule
          repo = await self.resolvedRepository(repo)
        File "/home/citest/tqtc-coin-ci/src/repositorymanager.py", line 207, in resolvedRepository
          await self.ensureRepoExists(repositoryState.gerritInstance, repositoryState.project)
        File "/home/citest/tqtc-coin-ci/src/utils.py", line 268, in async_lru_cached_wrapper
          result = await future_result
        File "/home/citest/tqtc-coin-ci/src/repositorymanager.py", line 116, in ensureRepoExists
          await clone("--bare", repoUrl, diskPath)
        File "/home/citest/tqtc-coin-ci/src/ash.py", line 145, in __call__
          raise sh.ErrorReturnCode_128(" ".join(processed_args), stdout, stderr)
      sh.ErrorReturnCode_128:   RAN: git --no-pager clone --bare ssh://codereview.qt-project.org:29418/qt/qtsensors /tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtsensors  STDOUT:
        STDERR:
      Cloning into bare repository '/tmp/coin-test1cp7ts48/git-repos/qt-project/qt/qtsensors'...
      Received disconnect from 54.194.93.196 port 29418:12: Too many concurrent connections (64) - max. allowed: 64
      Disconnected from 54.194.93.196 port 29418
      fatal: Could not read from remote repository.Please make sure you have the correct access rights
      and the repository exists.
      

      The connection should be either limited to max_connections=64 or alternatively or protect the git operations by a retry after sleep.

       

      Attachments

        For Gerrit Dashboard: COIN-497
        # Subject Branch Project Status CR V

        Activity

          People

            aakeskimo Aapo Keskimölö
            aakeskimo Aapo Keskimölö
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes