Uploaded image for project: 'Qt Quality Assurance Infrastructure'
  1. Qt Quality Assurance Infrastructure
  2. QTQAINFRA-1949

Remove insignificant tests in Qt

    XMLWordPrintable

Details

    • User Story
    • Resolution: Out of scope
    • P2: Important
    • None
    • 5.11, 5.12, 5.14
    • Metrics / Test Results
    • None

    Description

      We have a lot of old "insignificant_test" flags in different .pro -files. These tests are totally skipped while testing and they should be replaced with blacklisting when possible or just fix it.

      Situation as of 20th April 2018:

      ./qtpim/tests/auto/versit/qvcard21writer/qvcard21writer.pro:CONFIG+=insignificant_test # QTBUG-25382
./qtquick1/tests/auto/declarative/qmlvisual/qmlvisual.pro:CONFIG+=insignificant_test # QTBUG-26705
./qtquick1/tests/auto/declarative/qdeclarativetext/qdeclarativetext.pro:CONFIG += insignificant_test # QTBUG-21015
./qtquick1/tests/auto/declarative/qdeclarativedebug/qdeclarativedebug.pro:CONFIG+=insignificant_test # QTBUG-26702
./qtquick1/tests/auto/declarative/qdeclarativedebugjs/tst_qdeclarativedebugjs.pro:CONFIG+=insignificant_test # QTQAINFRA-428
./qtquick1/tests/auto/declarative/qdeclarativedebugobservermode/tst_qdeclarativedebugobservermode.pro:CONFIG+=insignificant_test # QTBUG-28298
./qtdeclarative/tests/auto/quick/qquickcanvasitem/qquickcanvasitem.pro:CONFIG += insignificant_test # QTBUG-41043
./qtlocation/tests/auto/qnmeapositioninfosource/qnmeapositioninfosource_simulation/qnmeapositioninfosource_simulation.pro:win32:CONFIG+=insignificant_test
./qtlocation/tests/auto/qnmeapositioninfosource/qnmeapositioninfosource_simulation_generic/qnmeapositioninfosource_simulation_generic.pro:win32:CONFIG+=insignificant_test
./qtlocation/tests/auto/qnmeapositioninfosource/qnmeapositioninfosource_realtime/qnmeapositioninfosource_realtime.pro:win32:CONFIG+=insignificant_test
./qtlocation/tests/auto/qnmeapositioninfosource/qnmeapositioninfosource_realtime_generic/qnmeapositioninfosource_realtime_generic.pro:win32:CONFIG+=insignificant_test
./qtlocation/tests/auto/qnmeapositioninfosource/dummynmeapositioninfosource/dummynmeapositioninfosource.pro:win32:CONFIG+=insignificant_test
./qtscript/tests/auto/qscriptjstestsuite/qscriptjstestsuite.pro:win32:CONFIG += insignificant_test # QTBUG-36632
./qtqa/tests/postbuild/symbols/symbols.pro:CONFIG += insignificant_test # QTQAINFRA-325
./qtqa/tests/postbuild/guiapplauncher/guiapplauncher.pro:CONFIG += insignificant_test # QTQAINFRA-323
./qtqa/scripts/generic/t/data/test-projects/tests/tests.pro: passing_insignificant_test\
./qtqa/scripts/generic/t/data/test-projects/tests/tests.pro: failing_insignificant_test\
./qtqa/scripts/generic/t/data/test-projects/tests/passing_insignificant_test/passing_insignificant_test.pro:CONFIG+=insignificant_test
./qtqa/scripts/generic/t/data/test-projects/tests/failing_insignificant_test/failing_insignificant_test.pro:CONFIG+=insignificant_test
./qtqa/scripts/scripts.pro:linux-*:system(". /etc/lsb-release && [ $DISTRIB_CODENAME = precise ]"):CONFIG+=insignificant_test # QTQAINFRA-708
./qtbase/tests/auto/network/access/spdy/spdy.pro:win32:CONFIG += insignificant_test # QTBUG-47128
./qtbase/tests/auto/widgets/dialogs/qfilesystemmodel/qfilesystemmodel.pro:CONFIG += insignificant_test # QTBUG-29403
./qtbase/tests/auto/widgets/dialogs/qcolordialog/qcolordialog.pro:linux*: CONFIG += insignificant_test # Crashes on different Linux distros
./qtbase/tests/auto/widgets/itemviews/qlistview/qlistview.pro:linux*: CONFIG += insignificant_test # Crashes
./qtbase/tests/auto/gui/kernel/qopenglwindow/qopenglwindow.pro:win32:CONFIG+=insignificant_test # QTBUG-46452, QTBUG-49630
./qtbase/tests/auto/gui/kernel/qtouchevent/qtouchevent.pro:osx: CONFIG += insignificant_test # QTBUG-46266, crashes
./qtbase/tests/auto/corelib/io/qprocess/test/test.pro:mac:CONFIG += insignificant_test # QTBUG-25895 - sometimes hangs
./qtfeedback/tests/auto/qfeedbackmmk/qfeedbackmmk.pro:linux-*:system(". /etc/lsb-release && [ $DISTRIB_CODENAME = oneiric ]"):CONFIG += insignificant_test # QTBUG-25448
./qtfeedback/tests/auto/qfeedbackmmk/qfeedbackmmk.pro:win32: CONFIG += insignificant_test # QTBUG-25448

      Attachments

        Issue Links

          depends on

          Sub-task - The sub-task of the issue QTBUG-29403 tst_qfilesystemmodel (widgets/dialogs) hangs.

          • P1: Critical - Urgent and Important, will STOP the release if matched with set FixVersion field. This includes regressions from the last version that are not edge cases; Data loss; Build issues; All but the most unlikely crashes/lockups/hanging; Serious usability issues and embarrassing bugs & Bugs critical to a deliverable. This is the highest possible priority for requirements.
          • Closed

          Sub-task - The sub-task of the issue QTQAINFRA-428 Temporarily disable failing tests in qt#4.8

          • P1: Critical - Urgent and Important, will STOP the release if matched with set FixVersion field. This includes regressions from the last version that are not edge cases; Data loss; Build issues; All but the most unlikely crashes/lockups/hanging; Serious usability issues and embarrassing bugs & Bugs critical to a deliverable. This is the highest possible priority for requirements.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. QTBUG-28298 tst_qdeclarativedebugobservermode fails randomly in CI

          • P3: Somewhat important - Should be fixed, but doesn't affect the release in any way.
          • Reported

          Bug - A problem which impairs or prevents the functions of the product. QTBUG-50842 tst_qcolordialog crashes under heavy load in linux

          • P3: Somewhat important - Should be fixed, but doesn't affect the release in any way.
          • Reported

          Bug - A problem which impairs or prevents the functions of the product. QTBUG-25382 Some qtpim autotests fail due to assumptions of QHash being ordered

          • P4: Low - Would be nice to fix, but it's not very important.
          • Reported

          Bug - A problem which impairs or prevents the functions of the product. QTBUG-26705 qmlvisual qutotest is unstable

          • P2: Important - Urgent, should be fixed, but will not stop the release.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. QTBUG-36632 Failures in tst_QScriptJsTestSuite

          • P2: Important - Urgent, should be fixed, but will not stop the release.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. QTBUG-90441 Update to 20H2 broke auto test for winrt

          • P2: Important - Urgent, should be fixed, but will not stop the release.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. QTBUG-21015 tst_qdeclarativetext declarative autotest fails

          • P3: Somewhat important - Should be fixed, but doesn't affect the release in any way.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. QTBUG-41043 tst_qquickcanvasitem tends to fail on CI

          • P4: Low - Would be nice to fix, but it's not very important.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. QTBUG-26702 qdeclarativedebug autotest is unstable

          • P2: Important - Urgent, should be fixed, but will not stop the release.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. QTBUG-46266 tst_QTouchEvent fails

          • P2: Important - Urgent, should be fixed, but will not stop the release.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. QTBUG-46452 tst_qopenglwindows fails on windows 7

          • P2: Important - Urgent, should be fixed, but will not stop the release.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. QTBUG-47128 spdy test crashes on Windows

          • P2: Important - Urgent, should be fixed, but will not stop the release.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. QTBUG-50747 tst_qlistview crashes

          • P2: Important - Urgent, should be fixed, but will not stop the release.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. QTQAINFRA-708 qtqa tests are unstable on Ubuntu 12.04 x64

          • P2: Important - Urgent, should be fixed, but will not stop the release.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. QTBUG-25895 tst_qprocess::lockupsInStartDetached occasionally locks up in startDetached on mac

          • P3: Somewhat important - Should be fixed, but doesn't affect the release in any way.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. QTBUG-49630 tst_qopenglwindows crashes

          • P3: Somewhat important - Should be fixed, but doesn't affect the release in any way.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. QTBUG-67855 qnmeapositioninfosource tests are marked as insignificant

          • P4: Low - Would be nice to fix, but it's not very important.
          • Closed

          Task - A task that needs to be done. QTBUG-25448 qtfeedbackmmk tests fails on Windows and Ubtuntu 11.10

          • P2: Important - Urgent, should be fixed, but will not stop the release.
          • Closed

          User Story - Created by Jira Software - do not edit or delete. Issue type for a user story. QTQAINFRA-325 Validate and fix tst_symbols

          • P5: Not important - Not relevant and not urgent.
          • Open

          User Story - Created by Jira Software - do not edit or delete. Issue type for a user story. QTQAINFRA-323 Make guiapplauncher test stable

          • P3: Somewhat important - Should be fixed, but doesn't affect the release in any way.
          • Withdrawn
          mentioned in

          Page Loading...

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              rampe Rami Potinkara
              tosaraja Tony Sarajärvi
              Veli-Pekka Heinonen Veli-Pekka Heinonen
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes