Details
-
Bug
-
Resolution: Done
-
P3: Somewhat important
-
unversioned
-
None
Description
When cancelling an integration, the running integration task file is removed by the Scheduler. However, this creates a race condition between the Webserver and Scheduler, because the Webserver tries to read a symlink that might already been deleted by the Scheduler.
To reproduce:
1. In webui, add a new integration:
2. In webui, cancel the integration.
3. In tmux, compare scheduler and webserver pane outputs:
2017-02-25 17:31:14,926 DEBUG:utils: Removed running symlink: /home/akeskimo/ci-working-dir/storage/running-integrations/run_builds_1488036666.thrift_bin
webserver:2017/02/25 17:31:14.927219 websocket.go:183: Error reading symlink: readlink /home/akeskimo/ci-working-dir/storage/running-integrations/run_builds_1488036666.thrift_bin: no such file or directory /home/akeskimo/ci-working-dir/storage/running-integrations/run_builds_1488036666.thrift_bin
As we compare the timestamps, we can see that the Webserver tries to read the symlink that has already been destroyed by the Scheduler.
Attachments
For Gerrit Dashboard: QTQAINFRA-1171 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
186721,2 | WIP: Implement notifications between Webserver and Scheduler | master | qtqa/tqtc-coin-ci | Status: ABANDONED | -2 | 0 |
186741,7 | Fix WebUI trying to read removed links during integration cancellation | master | qtqa/tqtc-coin-ci | Status: ABANDONED | -1 | 0 |
205709,10 | Fix symlink error during cancellation of integration | master | qtqa/tqtc-coin-ci | Status: MERGED | +2 | 0 |