Uploaded image for project: 'Qt Quality Assurance Infrastructure'
  1. Qt Quality Assurance Infrastructure
  2. QTQAINFRA-1171

Race conditions between WebServer and Scheduler while cancelling integration

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • P3: Somewhat important
    • production
    • unversioned
    • Coin (obsolete)
    • None

    Description

      When cancelling an integration, the running integration task file is removed by the Scheduler. However, this creates a race condition between the Webserver and Scheduler, because the Webserver tries to read a symlink that might already been deleted by the Scheduler.

      To reproduce:
      1. In webui, add a new integration:
      2. In webui, cancel the integration.
      3. In tmux, compare scheduler and webserver pane outputs:

      2017-02-25 17:31:14,926 DEBUG:utils: Removed running symlink: /home/akeskimo/ci-working-dir/storage/running-integrations/run_builds_1488036666.thrift_bin

      webserver:2017/02/25 17:31:14.927219 websocket.go:183: Error reading symlink: readlink /home/akeskimo/ci-working-dir/storage/running-integrations/run_builds_1488036666.thrift_bin: no such file or directory /home/akeskimo/ci-working-dir/storage/running-integrations/run_builds_1488036666.thrift_bin

      As we compare the timestamps, we can see that the Webserver tries to read the symlink that has already been destroyed by the Scheduler.

      Attachments

        For Gerrit Dashboard: QTQAINFRA-1171
        # Subject Branch Project Status CR V

        Activity

          People

            aakeskimo Aapo Keskimölö
            aakeskimo Aapo Keskimölö
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes